Back to articles

Jul 2, 2019 | by Maksim Larkin

Deployment Django+Celery on Heroku using Bitbucket

Introduction

In this article we will analyze how to deploy a Django project from Celery to Heroku from Bitbucket.

In most cases, such articles on the Internet are already outdated, so now we will go through all the steps and find out how to do it quickly and easily from scratch.

Creating a project

First of all, we create a virtual environment using virtualenv.
If you already have a django project, go directly to the project configuration step.

virtualenv myproject-env

We activate the virtual environment

virtualenv myproject-env/bin/activate

Install Django to create a project

pip install Django

Create a project where myproject is the name of your project.

django-admin startproject myproject

Create some new app where we will create models later

python manage.py startapp myapp

Let’s create a simple model that we will do “experiments” on

class MyModel(models.Model):
    counter = models.IntegerField(verbose_name='counter', default=0)

Add our new app “myapp” to INSTALLED_APPS and apply all the manipulations to the database.

python manage.py makemigrations
python manage.py migrate

To work with Data Base (DB) we will use PostgreSQL.

To do this we need to install the following library. psycopg2.

pip install psycopg2

Now we install celery and all the necessary components for its work.

pip install "celery[redis]"

To run the server on Heroku we need gunicorn.

pip install gunicorn

We also need django-heroku this will help us to ease the deployment to heroku.

pip install django-heroku

We save all installed libraries and their versions in requirements.txt

pip freeze > requirements.txt

Project configuration

Now let’s deal with the settings. For the configuration, we recommend using Twelve-Factor App methodology, which means we will not break the project configuration into development и production, but will depend on the curing variables.
Lets proceed directly to the settings themselves. In settings.py we write the following config at the end of the file

# REDIS CONFIG
REDIS_URL = os.environ.get('REDIS_URL', 'redis://127.0.0.1:6379')
BROKER_CONNECTION_MAX_RETRIES = os.environ.get('BROKER_CONNECTION_MAX_RETRIES', None)
BROKER_POOL_LIMIT = os.environ.get('BROKER_POOL_LIMIT', None)

# CELERY CONFIG
CELERY_BROKER_URL = os.environ.get('CELERY_BROKER_URL', REDIS_URL)
CELERY_RESULT_BACKEND = os.environ.get('CELERY_RESULT_BACKEND', REDIS_URL)
CELERY_REDIS_MAX_CONNECTIONS = os.environ.get('CELERY_REDIS_MAX_CONNECTIONS', 5)
CELERYD_CONCURRENCY = os.environ.get('CELERYD_CONCURRENCY', 1)

# Configure Django App for Heroku.
django_heroku.settings(locals())

In development, you can use any settings for convenient work on the local machine.
Now lets analyze all the settings in more detail.

REDIS_URL – link to redis (Heroku will also automatically set the variable REDIS_URL in Config Vars)
BROKER_CONNECTION_MAX_RETRIES – maximum number of retries before heroku server give up re-establishing a connection to the Redis. We set None for connecting to Redis an infinite number of times.
BROKER_POOL_LIMIT – the maximum number of connections that can be open in the connection pool. If set to None the connection pool will be disabled and connections will be established and closed for every use.
CELERY_BROKER_URL and CELERY_RESULT_BACKEND – link to redis which will act as a broker.
CELERY_REDIS_MAX_CONNECTIONS – the maximum number of connections available in the Redis connection pool used for sending and retrieving results.
CELERYD_CONCURRENCY – The number of concurrent worker processes/threads/green threads executing tasks.
django_heroku.settings(locals()) – set the variabes configuration for Heroku (database, allowed hosts, etc).

We set for CELERY_REDIS_MAX_CONNECTIONS и CELERYD_CONCURRENCY small values due to the use of free dyno on Heroku, otherwise sometimes you may have some problems with the maximum number of connections to Redis and memory overflow. If you do not have “heavy” tasks and a small number of connections to redis, you can ignore these celery settings.

Let’s take a configuration of the celery itself. Go to the root folder “myproject /” and create the file celery.py. Write to it:

from __future__ import absolute_import, unicode_literals
from celery import Celery
import os

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('screenmatter')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

Lets create a small celery task and view to start this task from displaying the result of the task.

myapp/tasks.py

from myapp.models import MyModel
from myproject.celery import app


@app.task
def counter():
    instance, created = MyModel.objects.get_or_create(id=1)
    instance.counter += 1
    instance.save()

myapp/views.py

from django.views.generic import TemplateView
from myapp.models import MyModel
from myapp.tasks import counter


class HomeView(TemplateView):

    template_name = 'home.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        instance_counter = 0
        if MyModel.objects.exists():
            instance_counter = MyModel.objects.get(id=1).counter
        context['counter'] = instance_counter
        counter.delay()
        return context

myproject/urls.py

from django.contrib import admin
from django.urls import path
from myapp.views import HomeView

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', HomeView.as_view(), name='home'),
]

myproject/templates/home.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>MyProject</title>
</head>
<body>
  Counter: {{ counter }}
</body>
</html>

Now we move to the configuration of Django + Heroku. To do this, you should create the following files in the root.

In the file runtime.txt write the used python version

python-3.7.3

In the Procfile file, we describe what needs to be done after successful deployment through release, specify the main dyno via the web using gunicorn and celery itself through the worker.

release: python manage.py migrate
web: gunicorn myproject.wsgi --log-file -
worker: celery worker -A myproject -l info

In the bitbucket-pipelines.yml file we indicate the version of python we use and tell Bitbucket that after the merge we need to send the code to Heroku.

image: python:3.7.3

clone:
  depth: full

pipelines:
  default:
    - step:
        caches:
          - pip
        script:
          - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_NAME.git HEAD

In this file the variables HEROKU_API_KEY and HEROKU_APP_NAME are used. We will discuss them hereunder.

Here you can read about deploy to Heroku more.

Heroku Configuration

We turn to the Heroku itself. Let’s create a new application on heroku and add all the add-ons we need.

Select “Create new app” in Heroku account

Enter application name and select region

After creating the application on heroku, we need to register the variables on bitbucket, which are used in the bitbucket-pipelines.yml file (HEROKU_API_KEY and HEROKU_APP_NAME). You must first turn on pipelines in the Pipelines tab -> Settings

Now copy the API-Key from your personal account on Heroku and the name of the application

As a result it should turn out as at the screenshot below.

Now we need to add the following add-ons:

Heroku Postgres – as I said above we need it as a DB.
Heroku Redis – we will use it to work with Celery.

All add-ons are set equally with the following steps.

And now we are entering the home stretch. Now you can send your code to the master branch of your repository and the code will be automatically deployed to heroku and run successfully.

Now you may open the application and make sure everything works.

You may find our result on Heroku here.
If you are confused at some step or you have failed then try to look sourcecode on our github.

Now you can expand your application, use more complex tasks, expand functionality, etc. But do not forget that heroku’s free dyno has server load restrictions (RAM, number of connections, size of DB, etc)

Bonus Feature

Also, if you use deferred tasks, do not forget that the Heroku server goes to bed if no one is accessing it for a long time. You can solve this problem with add-on – Heroku Scheduler.

To do this, we will create a small command in which we will send a request to our site every 10 minutes in order to prevent the server from falling asleep.
To do this, first install the library requests (do not forget to update the file requirements.txt).

pip install requests

And create the command that will be run every 10 minutes

myproject/management/commands/wake_up_server.py

from django.core.management.base import BaseCommand
import requests


class Command(BaseCommand):
    """ class Command """

    def handle(self, *args, **options):
        print('Wake up server')
        requests.get('https://django-celery-heroku.herokuapp.com/')

It remains only to tell Scheduler to run this command every 10 minutes. To do this, go to the settings Scheduler (<heroku app> -> resources -> Heroku Scheduler -> Create job), specify the frequency of the command and the name of the command

Now your server will not fall asleep, but do not practise on it, because Heroku’s free dyno server has a wake up time limit.

But there is a very cool life hack! To increase the wakefulness limit of the heroku free dyno server, add your card in your personal account, then the number of hours will increase from 550 to 1000 and in most cases this is enough for a month.