Celery is the technology this blog entry will look at. Celery uses the following components:
Once Python, Erlang, and Rabbitmq are installed, Celery can be installed.
BROKER_URL = "amqp://guest:email@example.com:5672//"
CELERY_RESULT_BACKEND = "amqp"
CELERY_TASK_RESULT_EXPIRES = 300
CELERY_IMPORTS = ("tasks", )
The "CELERY_IMPORTS" line lists the files that contain tasks that this Celery daemon can process. "tasks" above points to a file called "tasks.py" which contains the code for the actual task to perform. In this example, the tasks is a simple brute-force search for a prime number:
from celery.task import task
# Determine if a number is prime. This is slow -- the point of this
# is a computation that is slow enough it should be handed off to another
# computer or thread
s = x ** .5 # Square root of x
if (x % 2) == 0:
return 0 # Not prime; even number
q = 3
while q < s + 1:
if (x % q) == 0:
return 0 # Not prime
q += 2
return x # Prime
The above code is run on a celery server, and determines whether or not a given number is prime.
Now that the above code is in place, we can have a client that determines the first number that is prime after a given number:
from tasks import is_prime
p = 3 # We can run up to three tasks at once
results = 
Below, we fill up the task queue by testing the first p (maximum simultaneous tasks run at one time) numbers.
a = 0
while a < p:
x += 1
a += 1
Now that the task queue is full, we wait for tasks to finish. If a task is finished and the candidate number, in fact, is prime, we output the prime number and stop spawning new tasks. If a given task is finished and the candidate number is not prime, we use the now-empty task slot to spawn a new task testing the next-higher to see if it is prime
a = 0
while a < 2000: # Infinite loop protection
b = 0
while b < p:
if results[b].result != 0:
results[b] = is_prime.delay(x)
x += 1
b += 1
a += 1
return 0 # No prime found :(
That finishes the code looking for a prime number. We now run this code to find the first prime number for a series of power of 10s.
This simple example shows how Celery can be used with Python programs to make programs run across multiple computers, which increases the scalability and speed of Python applications.
While this blog post is copyright 2011 Sam Trenholme, all code contained here is public domain and can be used for any purpose whatsoever. To post a comment about an entry, send me an email and I may or may not post your comment (with or without editing)