Celery is an asynchronous task queue/job queue based on distributed message passing. This post is not detailed introduction but rather a short how-to start using Celery.
Using Celery supposes having of several components. It’s a:
- broker. Think it as a transport. You can choose among RabbitMQ, Redis or SQL servers;
- worker application which executes task;
- client application which should add tasks to the queue.
Let’s get started. At the very beginning there’s a need to install Celery. I run Fedora server. If you use Debian use apt-get.
yum install python-celery.noarch
For the sake of simplicity we’ll use Redis as a broker. It’s fast, simple to setup and doesn’t consume a lot of resources.
yum install redis
Now we can tune some options. Here’s redis.conf example:
daemonize no pidfile /var/run/redis/redis.pid port 6379 bind 127.0.0.1 timeout 0 loglevel notice logfile /var/log/redis/redis.log databases 16 save 900 1 save 300 10 save 60 10000 rdbcompression yes dbfilename dump.rdb dir /var/lib/redis/ slave-serve-stale-data yes appendonly no appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb slowlog-log-slower-than 10000 slowlog-max-len 128 vm-enabled no vm-swap-file /tmp/redis.swap vm-max-memory 0 vm-page-size 32 vm-pages 134217728 vm-max-threads 4 hash-max-zipmap-entries 512 hash-max-zipmap-value 64 list-max-ziplist-entries 512 list-max-ziplist-value 64 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 activerehashing yes
We will also need celery-with-redis package which Celery requires to work with Redis:
python-pip install -U celery-with-redis
Keep in mind that this command would also update your current Celery installation with its dependencies. It’s not big deal, but you might need to know.
Now let’s create our worker application called tasks.py:
from celery import Celery celery = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/1') @celery.task def add(x, y): return x + y
Now we can launch it:
celery -A tasks worker --loglevel=info
You should get similar output:
-------------- celery@turtle v3.0.15 (Chiastic Slide) ---- **** ----- --- * *** * -- [Configuration] -- * - **** --- . broker: redis://localhost:6379/0 - ** ---------- . app: tasks:0x1d1b690 - ** ---------- . concurrency: 1 (processes) - ** ---------- . events: OFF (enable -E to monitor this worker) - ** ---------- - *** --- * --- [Queues] -- ******* ---- . celery: exchange:celery(direct) binding:celery --- ***** ----- [Tasks] . tasks.add [2013-02-19 23:52:42,339: WARNING/MainProcess] celery@turtle ready. [2013-02-19 23:52:42,361: INFO/MainProcess] consumer: Connected to redis://localhost:6379/0.
Here’s our client application:
from tasks import add result = add.delay(4, 4) print result.get(timeout=1)
Note that here we use Celeray in synchronous mode. It means that we wait till the result is ready. I believe in most cases one would use Celery in asynchronous mode. Here we use it just to get a result to make sure everything works.
[dande@turtle ~]# python client.py 8 [dande@turtle ~]#
Now as everything is ready we can start to think about what we can do with described solution.
By the way, if you are interested in how Celery uses Redis run: