Posted on July 27, 2019

How to Develop a Django + Celery app in Kubernetes

Django + Celery is probably the most popular solution to develop websites that require running tasks in the background. Developing a Django + Celery app locally is complex, as you need to run different services: Django, Celery worker, Celery beat, Redis, databases… docker-compose is a very convenient tool in this case. You can spin up your local environment with docker-compose in just one single command. And thanks to the use of volume mounts, you are able to hot reload your application in seconds.

In this blog post, we want to go a step forward and explain why you should develop your Django + Celery app directly in Kubernetes.

The benefits are:

Reduce integration issues by developing in a more production-like environment, consuming Kubernetes manifests, secrets, volumes or config maps from development.
Overcome local development limitations. You will be able to develop against any Kubernetes cluster, local or remote. And having a lot of microservices makes it harder and harder to run the entire development environment locally.

But it is well-known that developing in Kubernetes is tedious. Let's explore together how to develop in Kubernetes the Cloud Native way 💥💥💥💥.

Deploy the Django + Celery Sample App

Get a local version of the Django + Celery Sample App by executing the following commands in your local terminal:

$ git clone https://github.com/okteto/math
$ cd math

The Django + Celery Sample App is a multi-service application that calculates math operations in the background. It consists of a web view, a worker, a queue, a cache, and a database.

Execute the command below to deploy the application in your Kubernetes cluster:

You can deploy to your own Kubernetes cluster or give Okteto Cloud a try. Okteto Cloud is a development platform for Kubernetes applications. Sign up today to get a free developer account with 4 CPUs and 8GB of RAM.

$ kubectl apply -f manifests

statefulset.apps "cache" created
service "cache" created
statefulset.apps "db" created
service "db" created
statefulset.apps "queue" created
service "queue" created
deployment.apps "web" created
service "web" created
deployment.apps "worker" created

Wait for a few seconds for the app to be ready. Check that all pods are ready by executing the command below:

$ kubectl get pod

NAME READY STATUS RESTARTS AGE
cache-0 1/1 Running 0 2m
db-0 1/1 Running 0 2m
queue-0 1/1 Running 0 2m
web-7bccc4bc99-2nwtc 1/1 Running 0 2m
worker-654d7b8bd5-42rq2 1/1 Running 0 2m

Efficient Kubernetes Development with Okteto

Now that we have the Django + Celery Sample App running in a Kubernetes cluster, we can use it as our development environment. When working in Kubernetes, you would have to rebuild your docker images, push them to a registry and redeploy your application every time you want to test a code change. This cycle is complex and time-consuming. Don't kill your productivity this way.

If you are familiar with docker-compose, you know how useful is to mount local folders into your containers to avoid the docker build/push/redeploy cycle. Volume mounts are a game changer for developing Docker applications and they are the missing piece to speed up your Kubernetes development cycle.

We developed Okteto to solve this problem. Okteto is open source, and the code is available in github. Feel free to check it out, contribute, and star it 🤗!

To install the Okteto CLI in your computer, follow the installation instructions and check that it is properly installed by running:

$ okteto version
okteto version 1.12.3

Okteto CLI >= 1.8.18 is required to follow this tutorial.

In order to start developing the Django + Celery Sample App, execute:

$ okteto up

 ✓  Development container activated
 ✓  Files synchronized
    Namespace: pchico83
    Name:      web
    Forward:   8080 -> 8080

curl: (52) Empty reply from server
Database is ready
No changes detected in app ‘myproject’
Created migrations
Operations to perform:
Apply all migrations: auth, contenttypes, myproject, sites
Running migrations:
No migrations to apply.
Migrated DB to latest version
Performing system checks…
System check identified no issues (0 silenced).
June 26, 2019–11:00:02
Django version 1.11.21, using settings ‘myproject.settings’
Starting development server at http://0.0.0.0:8080/
Quit the server with CONTROL-C.

Let's have a look at the okteto.yml file to understand what the okteto up command does:

name: web
command: ./run_web.sh
sync:
  - .:/app
services:
  - name: worker
    command: ./run_celery.sh
    sync:
      - .:/app
forward:
  - 8080:8080

okteto up synchronizes your local source code to the containers of the deployment web and worker. Your source code is synchronized to the container path /app. Also, the port 8080 is automatically forwarded between your development container and your computer. If you want to know more about how Okteto works, follow this link.

Let's Write some Code and Fix a Bug

Verify that the application is up and running by opening your browser and navigating to http://localhost:8080/jobs/. Go ahead and calculate the fibonacci for the number 5:

Fibonacci calculator

Press the POST button to submit the operation. The response payload will include the url of the job. Go to http://localhost:8080/jobs/1/ and you will notice that the result is wrong (hint: the fibonacci number of 5 is not 32). This is because our worker has a bug 🙀!

Typically, fixing this would involve you running the app locally, fixing the bug, building a new container, pushing it and redeploying your app. Instead, we’re going to do it the Cloud Native way .

Open myproject/myproject/models.py in your favorite local IDE. Take a look at the value of the task variable in line 29. It looks like someone hard-coded the name of the operation instead of reading it from the job. Let's fix it by changing it to self.type , as shown below:

task = TASK_MAPPING[‘power’]

task = TASK_MAPPING[self.type]

Save your file and go back to http://localhost:8080/jobs/. Submit a new fibonacci calculation for the same values as before. Go to http://localhost:8080/jobs/2/ and verify the result. The result looks correct this time, success!

Fibonacci calculator working correctly

How did this happen? With Okteto your changes were automatically applied to the remote containers as soon as you saved them. No commit, build, push or redeploy required 💪!

Final recommendations:

Use the .stignore file to skip files from file synchronization.
Execute okteto restart to restart the worker containers when needed. Celery does not work with hot-reloads mode.
Create a virtual environment to install your pip dependencies directly in your development container. This is a convenient way to install your pip dependencies incrementally, and don't rebuild containers when you update your dependencies.

Conclusions

We have shown the advantages of developing directly in Kubernetes while keeping the same developer experience than working on a local machine. Working on the Cloud is always better. You don’t work on your spreadsheets and listen to media files locally, do you? Stop dealing with local environments and become a Cloud Native Developer today!