Automate frequent deployment tasks in Django with Fabric

If you're working in heist, you may not even have a staging server, mirroring you production instance. In this case a mistake may cause you headache and a digraceful e-mail to your admin inbox with stacktrace. While bigger projects use continous integration tools, how should we manage smaller ones, to avoid the problems stated above? The answer is automating these mundane jobs with Fabric.

The steps I use to forget when being eager to push a new changset:

1. Not running tests.
2. Pulling from the wrong branch.
3. Not running migrations.
4. Not installing requirements.
5. Forgetting to restart app to see results.

As we love developing in Python, Fabric was a natural choice. If you're not familiar with this library, it's a command-line utility that allows for running commands remotely over SSH for streamlining deployment or system administration tasks.

Lets say we have a new project. All requirements are already provisioned (database, virtualenv, web server etc.). Now after we commit some changes, it'd be great to get it into our server quickly for preview.

The desired result of our script is like that:

• (12:59:13) Running tests locally...
• (12:59:24) ...finished running tests locally.
• (12:59:25) Pulling mercurial repository...
• (12:59:29) ...finished pulling mercurial repository.
• (12:59:29) Collecting static files...
• (12:59:32) ...finished collecting static files.
• (12:59:32) Migrating models...
• (12:59:35) ...finished migrating models.
• (12:59:35) Installing requirements...
• (12:59:36) ...finished installing requirements.
• (12:59:36) Restarting application...
• (12:59:37) ...finished restarting application.

The script consists of 5 main jobs, where each one has to succeed for the next to begin.

First we need to enter data specific for our installation. We're typically developing with staging (development) and production (stable) servers. We use a standarized Django project template, following best practices from the authors of Two Scoops of Django.

{
"project_name": ,
"vcs_type": ,
"git_repository": ,
"stages": {
"stable": {
"name": "stable",
"host": "",
"user": "",
"vcs_branch": "",
"venv_directory": "",
"requirements_file": "",
"code_src_directory": "",
"restart_command": ""
},
"development": {
"name": "stage",
"host": "",
"user": "",
"vcs_branch": "",
"venv_directory": "",
"requirements_file": "",
"code_src_directory": "",
"restart_command": ""
}
},
"local": {
"code_src_directory": "",
"venv_python_executable": ""
}
}


We keep a settings module with versioned settings files for each stage and an untracked json document serving as source of sensitive data (passwords, keys, ports). Let's split the file above into parts:

{
"project_name": ,
"vcs_type": ,
"git_repository": ,
}
project name - as simple as it sounds. We keep repository with fabric script inside Django project template repository and put here '{{ project_name }}', so that it can be automatically filled in when running startproject.

vcs_type - 'mercurial' or 'git'

git_repository - if you're using git, put name of your repo (usually 'origin' goes here, but you can adjust it to your needs); otherwise just type "None"

Then come two stages. As the structure is exactly the same, lets analyze its content:

{
"stable": {
"name": "stable",
"host": "",
"user": "",
"vcs_branch": "",
"venv_directory": "",
"requirements_file": "",
"code_src_directory": "",
"restart_command": ""
}
}


name - name of stage

host - hostname or IP address of your server

user - user to run your tasks

vcs_branch - branch to use for this installation; set according to your naming conventions, we stick to 'stable' and 'development'

venv_directory - path to your virtualenv; needed to run tasks in installation context

requirement_file - path to requirements file for this installation

code_src_directory - path to directory containing source code, in particular your manage.py file

restart_command - we use supervisord for keeping track of processes; in this case the command could be 'supervisorctl restart project_name'

The last section is specifically for local environment to provide paths for running tests:

{
"local": {
"code_src_directory": "",
"venv_python_executable": ""
}
}


code_src_directory - path to directory containing source code, in particular your manage.py file

venv_python_executable - path to your Python executable; in case you work locally on a Windows machine

Having filled configuration details, let's analyze the script part by part. We're using two external libraries:

1. fabric-virtualenv - simplifies running tasks in virtualenv context
2. unipath - replacement for os.path

import json
from datetime import datetime

from fabric.api import *
from fabric.operations import require
from fabric.context_managers import settings
from fabric.utils import fastprint

from fabvenv import virtualenv
from unipath import Path

SETTINGS_FILE_PATH = Path(__file__).ancestor(1).child('project_settings.json')

with open(SETTINGS_FILE_PATH, 'r') as f:
# Load settings.
project_settings = json.loads(f.read())

env.prompts = {
'Type \'yes\' to continue, or \'no\' to cancel: ': 'yes'
}


First, we define path to configuration file and load data to project_settings. Then, we set env.prompts for later use when collecting static files.

Moving on to tasks:

def set_stage(stage_name='development'):
stages = project_settings['stages'].keys()
if stage_name not in stages:
raise KeyError('Stage name "{0}" is not a valid stage ({1})'.format(
','.join(stages))
)
env.stage = stage_name

@task
def stable():
set_stage('stable')
set_project_settings()

@task
def development():
set_stage('development')
set_project_settings()

def set_project_settings():
stage_settings = project_settings['stages'][env.stage]
if not all(project_settings.itervalues()):
raise KeyError('Missing values in project settings.')
env.settings = stage_settings


Tasks stable and dev select assign configuration settings from settings file to env.settings variable. Under the hood, configuration file is checked for missing values.

The main task:

@task
def deploy(tests='yes'):
'''
Deploys project to previously set stage.
'''
require('stage', provided_by=(stable, development))
require('settings', provided_by=(stable, development))
# Set env.
env.vcs_type = project_settings['vcs_type']
env.user = env.settings['user']
env.host_string = env.settings['host']

with hide('stderr', 'stdout', 'warnings', 'running'):
if tests == 'yes':
with lcd(project_settings['local']['code_src_directory']):
run_tests()
with cd(env.settings['code_src_directory']):
pull_repository()
with virtualenv(env.settings['venv_directory']):
with cd(env.settings['code_src_directory']):
collect_static()
install_requirements()
migrate_models()
restart_application()


Step by step, first we have to make sure that stage has been selected. Then some more variables are read from configuration file. Having set that, we process to fire our tasks. We supressing all output to make the output cleaner.

with hide('stderr', 'stdout', 'warnings', 'running'):


Ideally, all tasks should run smooth. In case something breaks, tasks stops and we need to run this job manualy to find the mistake. To avoid running tests, we add a variable, which allows for executing the script in the following manner:

fab development deploy:tests=no


Moving on, the functions we're running are not 'tasks' per se, as they're not supposed to be run separately from command line. To provide better output, we add a decorator:

def print_status(description):
def print_status_decorator(fn):
def print_status_wrapper():
now = datetime.now().strftime('%H:%M:%S')
fastprint('({time}) {description}{suffix}'.format(
time=now,
description=description.capitalize(),
suffix='...\n')
)
fn()
now = datetime.now().strftime('%H:%M:%S')
fastprint('({time}) {description}{suffix}'.format(
time=now,
description='...finished '+description,
suffix='.\n')
)
return print_status_wrapper
return print_status_decorator

@print_status('starting my_task')
def my_task():
# sum job


Then the tasks themselves are:

@print_status('running tests locally')
def run_tests():
'''
Runs all tests locally. Tries to use settings.test first for sqlite db.
To avoid running test, use `deploy:tests=no`.
'''
python_exec = project_settings['local']['venv_python_executable']
test_command = python_exec + ' manage.py test'
with settings(warn_only=True):
result = local(test_command + ' --settings=settings.test')
if not result.failed:
return
result = local(test_command + ' --settings=settings.dev')
if result.failed:
abort('Tests failed. Use deploy:tests=no to omit tests.')

def pull_repository():
'''
Updates local repository, selecting the vcs from configuration file.
'''
if env.vcs_type == 'mercurial':
pull_mercurial_repository
elif env.vcs_type == 'git':
pull_git_repository
else:
abort(
'vcs type must be either mercurial or git,'
' currently is: {vcs}'.format(vcs=env.vcs_type)
)

@print_status('pulling mercurial repository')
def pull_mercurial_repository():
run('hg pull -u')
run('hg up {branch}'.format(branch=env.settings['vcs_branch']))

@print_status('pulling git repository')
def pull_git_repository():
command = 'git pull {} {}'.format(
env.project_settings.get('git_repository'),
env.settings.get('vcs_branch')
)
run(command)

@print_status('collecting static files')
def collect_static():
run('python manage.py collectstatic')

@print_status('installing requirements')
def install_requirements():
with cd(env.settings['code_src_directory']):
run('pip install -r {0}'.format(env.settings['requirements_file']))

@print_status('migrating models')
def migrate_models():
run('python manage.py migrate')

@print_status('restarting application')
def restart_application():
with settings(warn_only=True):
restart_command = env.settings['restart_command']
result = run(restart_command)
if result.failed:
abort('Could not restart application.')


Last but not least, a task printing example of usage.

@task
def help():
message = '''
Remote updating application with fabric.

Usage example:

Deploy to development server:
fab development deploy

Deploy to production server with no tests:
fab stable deploy:tests=no
'''
fastprint(message)


This script is intentionally simple as each project requires some extensions and customization. Even with limited functionality, it does a great job as an entry point for automating our job.