Deployment Django+Celery on Heroku using Bitbucket

Introduction

In this article we will analyze how to deploy a Django project from Celery to Heroku from Bitbucket.

In most cases, such articles on the Internet are already outdated, so now we will go through all the steps and find out how to do it quickly and easily from scratch.

Creating a project

First of all, we create a virtual environment using virtualenv. If you already have a django project, go directly to the project configuration step_._

virtualenv myproject-env

We activate the virtual environment

virtualenv myproject-env/bin/activate

Install Django to create a project

pip install Django

Create a project where _myproject_ is the name of your project.

django-admin startproject myproject

Create some new _app_ where we will create models later

python manage.py startapp myapp

Let's create a simple model that we will do "experiments" on

class MyModel(models.Model):
    counter = models.IntegerField(verbose_name='counter', default=0)

Add our new app "myapp" to INSTALLED\_APPS and apply all the  manipulations to the database.

python manage.py makemigrations
python manage.py migrate

To work with Data Base (DB) we will use PostgreSQL.

To do this we need to install the following library. psycopg2.

pip install psycopg2

Now we install celery and all the necessary components for its work.

pip install "celery[redis]"

To run the server on Heroku we need gunicorn.

pip install gunicorn

We also need django-heroku this will help us to ease the deployment to heroku.

pip install django-heroku

We save all installed libraries and their versions in _requirements.txt_

pip freeze > requirements.txt

Project configuration

Now let's deal with the settings. For the configuration, we recommend using  Twelve-Factor App methodology, which means we will not break the project configuration into _development_ и _production_, but will depend on the curing variables. Lets proceed directly to the settings themselves. In se_ttings.py_ we write the following config at the end of the file

# REDIS CONFIG
REDIS_URL = os.environ.get('REDIS_URL', 'redis://127.0.0.1:6379')
BROKER_CONNECTION_MAX_RETRIES = os.environ.get('BROKER_CONNECTION_MAX_RETRIES', None)
BROKER_POOL_LIMIT = os.environ.get('BROKER_POOL_LIMIT', None)

# CELERY CONFIG
CELERY_BROKER_URL = os.environ.get('CELERY_BROKER_URL', REDIS_URL)
CELERY_RESULT_BACKEND = os.environ.get('CELERY_RESULT_BACKEND', REDIS_URL)
CELERY_REDIS_MAX_CONNECTIONS = os.environ.get('CELERY_REDIS_MAX_CONNECTIONS', 5)
CELERYD_CONCURRENCY = os.environ.get('CELERYD_CONCURRENCY', 1)

# Configure Django App for Heroku.
django_heroku.settings(locals())

In development, you can use any settings for convenient work on the local machine. Now lets analyze all the settings in more detail.

We set for _CELERY\_REDIS\_MAX\_CONNECTIONS_ и _CELERYD\_CONCURRENCY_ small values due to the use of free dyno on Heroku, otherwise sometimes you may have some problems with the maximum number of connections to Redis and memory overflow. If you do not have "heavy" tasks and a small number of connections to redis, you can ignore these celery settings.

Let's take a configuration of the celery itself. Go to the root folder "myproject /" and create the file celery.py. Write to it:

from __future__ import absolute_import, unicode_literals
from celery import Celery
import os

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('screenmatter')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

Lets create a small celery task and view to start this task from displaying the result of the task.

_myapp/tasks.py_

from myapp.models import MyModel
from myproject.celery import app


@app.task
def counter():
    instance, created = MyModel.objects.get_or_create(id=1)
    instance.counter += 1
    instance.save()

_myapp/views.py_

from django.views.generic import TemplateView
from myapp.models import MyModel
from myapp.tasks import counter


class HomeView(TemplateView):

    template_name = 'home.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        instance_counter = 0
        if MyModel.objects.exists():
            instance_counter = MyModel.objects.get(id=1).counter
        context['counter'] = instance_counter
        counter.delay()
        return context

_myproject/urls.py_

from django.contrib import admin
from django.urls import path
from myapp.views import HomeView

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', HomeView.as_view(), name='home'),
]

_myproject/templates/home.html_

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>MyProject</title>
</head>
<body>
  Counter: {{ counter }}
</body>
</html>

Now we move to the configuration of Django + Heroku. To do this, you should create the following files in the root.

In the file runtime.txt write the used python version

python-3.7.3

In the _Procfile_ file, we describe what needs to be done after successful deployment through _release_, specify the main dyno via the _web_ using _gunicorn_ and celery itself through the _worker_.

release: python manage.py migrate
web: gunicorn myproject.wsgi --log-file -
worker: celery worker -A myproject -l info

In the _bitbucket-pipelines.yml_ file we indicate the version of python we use and tell Bitbucket that after the merge we need to send the code to Heroku.

image: python:3.7.3

clone:
  depth: full

pipelines:
  default:
    - step:
        caches:
          - pip
        script:
          - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_NAME.git HEAD

In this file the variables HEROKU\_API\_KEY and HEROKU\_APP\_NAME are used. We will discuss them hereunder.

Here you can read about deploy to Heroku more.

Heroku Configuration

We turn to the Heroku itself. Let's create a new application on heroku and add all the add-ons we need.

 Select "Create new app" in Heroku account

 Enter application name and select region

After creating the application on heroku, we need to register the variables on bitbucket, which are used in the _bitbucket-pipelines.yml_ file (HEROKU\_API\_KEY and HEROKU\_APP\_NAME). You must first turn on pipelines in the _Pipelines tab -> Settings_


Now copy the API-Key from your personal account on Heroku and the name of the application


As a result it should turn out as at the screenshot below.


Now we need to add the following add-ons:

1. Heroku Postgres - as I said above we need it as a DB. 2. Heroku Redis - we will use it to work with Celery.

All add-ons are set equally with the following steps.

 Tap the button "Install "

 Select our application

And now we are entering the home stretch. Now you can send your code to the master branch of your repository and the code will be automatically deployed to heroku and run successfully.

 Результат auto deployment on Heroku

Now you may open the application and make sure everything works.


You may find our result on Heroku here. If you are confused at some step or you have failed then try to look sourcecode on our github.

Now you can expand your application, use more complex tasks, expand functionality, etc. But do not forget that heroku's free dyno has server load restrictions (RAM, number of connections, size of DB, etc)

Bonus Feature

Also, if you use deferred tasks, do not forget that the Heroku server goes to bed if no one is accessing it for a long time. You can solve this problem with add-on - Heroku Scheduler.

To do this, we will create a small command in which we will send a request to our site every 10 minutes in order to prevent the server from falling asleep. To do this, first install the library requests (do not forget to update the file _requirements.txt_).

pip install requests

And create the command that will be run every 10 minutes

_myproject/management/commands/wake\_up\_server.py_

from django.core.management.base import BaseCommand
import requests


class Command(BaseCommand):
    """ class Command """

    def handle(self, *args, **options):
        print('Wake up server')
        requests.get('https://django-celery-heroku.herokuapp.com/')

It remains only to tell Scheduler to run this command every 10 minutes. To do this, go to the settings Scheduler (_ -> resources -> Heroku Scheduler -> Create job_), specify the frequency of the command and the name of the command


Now your server will not fall asleep, but do not practise on it, because Heroku's free dyno server has a wake up time limit.

But there is a very cool life hack! To increase the wakefulness limit of the heroku free dyno server, add your card in your personal account, then the number of hours will increase from 550 to 1000 and in most cases this is enough for a month.


That's all! Good luck in your endeavors.

Larkin Maksim, Start Matter 2019