Celery messages TTL. Make sure your queue is never overflown

Posted: 2013-03-20 15:50 | More posts about python celery

Sometimes we have quite huge boosts of messages rushing to our Sentry error logger. Providing that Sentry generates a couple of Celery tasks for every incoming message, it periodicaly overflows our RabbitMQ queue.

We don't mind to lose some messages as long as we have responsive logger, handling the rest of our tasks.

The solution is quite easy, but it took some time for me to come to a decision. I added the argument "x-message-ttl" to the queue to ensure it doesn't stuck. My settings.py file comes to look like this:

CELERY_QUEUES = (
    Queue('default', routing_key='default', queue_arguments={'x-message-ttl': 600}),
    Queue('celery', routing_key='celery', queue_arguments={'x-message-ttl': 600}),
    Queue('alerts', routing_key='alerts', queue_arguments={'x-message-ttl': 600}),
    Queue('cleanup', routing_key='cleanup', queue_arguments={'x-message-ttl': 600}),
    Queue('sourcemaps', routing_key='sourcemaps', queue_arguments={'x-message-ttl': 600}),
    Queue('search', routing_key='search', queue_arguments={'x-message-ttl': 600}),
    Queue('counters', routing_key='counters', queue_arguments={'x-message-ttl': 600}),
    Queue('events', routing_key='events', queue_arguments={'x-message-ttl': 600}),
    Queue('triggers', routing_key='triggers', queue_arguments={'x-message-ttl': 600}),
)

In short, this means that no messages will last longer than 10 minutes in a queue. Stale messages will be quietly removed.

Some notes on top of that:

Adding arguments alone isn't enough if your queues have been already created. With every change of queue_arguments you should re-create your queues. Actually, all you have to do is to remove them, and Celery creates new ones upon the next start up. I use RabbitMQ Management Plugin for this. By the way, it works with RabbitMQ 3.x only, and don't forget to remove the default "guest" RabbitMQ user, if you install it!
I've got the list of Celery queues, used by Sentry, from the sentry.conf.server file (this is its latest version). Ensure you don't forget any queues.
Try undocumented settings parameter SENTRY_USE_SEARCH = False to reduce the number of tasks in your queue. Sentry does nasty things when this option is turned on (proof)

Happy logging!