Celery, RabbitMQ and sending messages directly

2010-10-17 Computing

Been using RabbitMQ quite a lot at Cantemo, where we use it for long standing jobs, working with other languages and for notifications.

But I needed to decouple some of the code from my application, to be able to distribute it and talk from other systems – exactly what we needed to do, run tasks that are in the application from another application via posting messages to RabbitMQ. Tasks are held in Celery, and started usually within the code base.

Staying with Python, though not always using CPython, I have a task “notification_of_upload_handler” which is registered in Celery (it actually has a full path to namespace it correctly, but there is no need to show that for my example).

Obviously to do this you will need, Celery (which includes carrot), and RabbitMQ with a user and virtual host.

Using the Carrot library, I can start this task by sending a message to RabbitMQ like so:

[cc lang=“python”]

from carrot.connection import BrokerConnection

from carrot.messaging import Publisher

from celery.utils import gen_unique_id

conn = BrokerConnection(hostname=”rabbithost”, port=5672, userid=”myrabbituser”, password=”mypassword”, virtual_host=”rabbitvhost”)

publisher = Publisher(connection=conn, exchange=”celery”, routing_key=”celery”)

publisher.send({“task”:”notification_of_upload_handler”, “args”: (“more”,), “kwargs”: {}, “id”: gen_unique_id()})

publisher.close()

[/cc]

Of course you could use any language and any AQMP library to do this. Check out http://celeryq.org/docs/internals/protocol.html for more on the protocol.

My next task is to broadcast this so that messages don’t stay around.