Celery is an asynchronous task queue/job queue based on distributed message passing. This post is not detailed introduction but rather a short how-to start using Celery.
Using Celery supposes having of several components. It’s a:
- broker. Think it as a transport. You can choose among RabbitMQ, Redis or SQL servers;
- worker application which executes task;
- client application which should add tasks to the queue.
Let’s get started. At the very beginning there’s a need to install Celery. I run Fedora server. If you use Debian use apt-get.
yum install python-celery.noarch
For the sake of simplicity we’ll use Redis as a broker. It’s fast, simple to setup and doesn’t consume a lot of resources.
yum install redis
Now we can tune some options. Here’s redis.conf example:
save 900 1
save 300 10
save 60 10000
We will also need celery-with-redis package which Celery requires to work with Redis:
python-pip install -U celery-with-redis
Keep in mind that this command would also update your current Celery installation with its dependencies. It’s not big deal, but you might need to know.
Now let’s create our worker application called tasks.py:
from celery import Celery
celery = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/1')
def add(x, y):
return x + y
Now we can launch it:
celery -A tasks worker --loglevel=info
You should get similar output:
-------------- celery@turtle v3.0.15 (Chiastic Slide)
---- **** -----
--- * *** * -- [Configuration]
-- * - **** --- . broker: redis://localhost:6379/0
- ** ---------- . app: tasks:0x1d1b690
- ** ---------- . concurrency: 1 (processes)
- ** ---------- . events: OFF (enable -E to monitor this worker)
- ** ----------
- *** --- * --- [Queues]
-- ******* ---- . celery: exchange:celery(direct) binding:celery
--- ***** -----
[2013-02-19 23:52:42,339: WARNING/MainProcess] celery@turtle ready.
[2013-02-19 23:52:42,361: INFO/MainProcess] consumer: Connected to redis://localhost:6379/0.
Here’s our client application:
from tasks import add
result = add.delay(4, 4)
Note that here we use Celeray in synchronous mode. It means that we wait till the result is ready. I believe in most cases one would use Celery in asynchronous mode. Here we use it just to get a result to make sure everything works.
[dande@turtle ~]# python client.py
Now as everything is ready we can start to think about what we can do with described solution.
By the way, if you are interested in how Celery uses Redis run: