Categories
GNU/Linux Free Software & Open Source Programming & Web Development

My Python & Django deployment workflow and tools

Yellow python

If you’re new to Python web development with Django, there are some things that tutorials don’t teach. Deploying to a server when you finish your local development can be a frustrating task. Here is a rundown of the tools and workflow I use for deploying a Django based website.

A few days ago when I launched the new version of Notasbit.com, I ended up having a discussion with a friend about deployment methods for Ruby on Rails and Django websites. My friend is used to Rails development and deployment, something I am not familiar with since Rails 1.8 (looong time ago by now) and he insisted that Ruby Gems method is way easier while Python deployment was a hassle. Yes, for people not familiar with the right tools it can be. There are different versions of Python and library dependencies and versions that can make maintaining a Django website a nightmare.

Here is what I use for deploying Django websites on a live server:

The tools

Git

You cannot be serious about programming in general these days if you’re not using a version control system. In Linus Torvald’s words: “if you’re not using Git, you’re an idiot”. I keep all my projects under version control, even if I’m the only developer on it, even when the code is for personal use only. It not only is a good practice to keep, but it also works as your backup. When I got robbed from my computers, I didn’t loose any project data because it was all backed up on several Git repositories at different places. For deployment tasks, you need to install Git on your server. The server you’re deploying to will serve as a backup as well.

Virtualenv

All that version and dependency hell can be isolated using Python’s Virtualenv tool. This is a tool that will create an encapsulated environment of a given python version and all the libraries you want to use in a project without messing with the system’s main versions. So this way, you can have an old Django 1.4 project running on the same server than another project running a later Django 1.8 or 1.9 version, as well as their respective dependencies.

Fabric

The only reason I still haven’t started using Python 3 on my projects is Fabric. This tool will automate all the project maintenance tasks needed, and that includes doing the deployment. It is similar to a Makefile but you write your tasks in Python code so you don’t need to learn a whole new syntax.

Now to the implementation details…

On your local development machine the project structure will look like this:

Project directory structure. The idea is to have the virtualenv outside the version controlled directory. You can also use virtualenvwrapper and have all your virtualenvs in a separate location. I like to have them contained in a same folder for now.

mainproject.com/
 |
 +-- env/ (virtualenv)
 |
 +-- django_project/
     |
     --- manage.py
     --- .git/
     --- fabfile.py
     --- deploy_tools/
     --- All other django project files & dirs

Deployment scripts

In the directory deploy_tools we’ll place the files necessary to configure apache, nginx, gunicorn, uwsgi or whatever other server configuration scripts needed for deployment.

Here’s an example of the Nginx configuration script I use:

upstream mywebsite {  #the upstream component nginx needs to connect to
    server unix:///tmp/mywebsite.sock; # for a file socket
}

server {    
    listen      80;   # the port your site will be served on
    server_name www.mywebsite.com;   # the domain name it will serve for
    charset     utf-8;

    client_max_body_size 250M;   # max upload size, adjust to taste

    location /static {
        alias /var/www/mywebsite/static; # your Django project's static files - amend as required
    }

    location / {     # Finally, send all non-media requests to the Django server.
        uwsgi_pass  mywebsite;
        uwsgi_param QUERY_STRING $query_string;
        uwsgi_param REQUEST_METHOD $request_method;
        uwsgi_param CONTENT_TYPE $content_type;
        uwsgi_param CONTENT_LENGTH $content_length;

        uwsgi_param REQUEST_URI $request_uri;
        uwsgi_param PATH_INFO $document_uri;
        uwsgi_param DOCUMENT_ROOT $document_root;
        uwsgi_param SERVER_PROTOCOL $server_protocol;
        uwsgi_param HTTPS $https if_not_empty;

        uwsgi_param REMOTE_ADDR $remote_addr;
        uwsgi_param REMOTE_PORT $remote_port;
        uwsgi_param SERVER_PORT $server_port;
        uwsgi_param SERVER_NAME $server_name;
    }
}

Settings handling

There are many ways to solve the problem about settings file management in Django applications. I like to have a general settings file and a local one for the different environments. To achieve this, add a local_settings.py file and add it to the git ignore list.

Then add the following at the end of your general settings.py file:

# Import local settings
try:
    from local_settings import *
except ImportError:
    pass

In your remote server, create a local_settings.py file and add DEBUG = False or override any other setting you need specifically for that server. You can have different settings for local, staging, testing or production or any other servers you need to have.

Fabric tasks

For example, you might need to download a copy of your production database to use for development tests. Instead of typing the same mysqldump command every time, you can automate it like this:

def backup_db():
    """
    Gets a database dump from remote server
    """
    production1()
    date = time.strftime('%Y-%m-%d-%H%M%S')
    dbname = 'MY-DATABASE-NAME'
    path = os.path.join(os.path.dirname(__file__), 'db_backups')
    fname = "{dbname}_backup_{date}.sql.gz".format(date=date,dbname=dbname)

    run("mysqldump -u {dbuser} -p'{password}' --add-drop-table -B {database} | gzip -9 > {filename}".format(
        database=dbname,
        dbuser='MY-DB-USER',
        password='MY-SERVER-PASSWORD',
        filename=os.path.join('/tmp', fname))
    )
    get(remote_path=os.path.join('/tmp', fname),
        local_path=os.path.join(path, fname))
    run("rm {filename}".format(filename=os.path.join('/tmp', fname)))

Likewise, you can create a deploy command to get everything in the server. Here’s a fabfile.py example:

from fabric.api import *
from fabric.colors import green, red
import os
import sys
import time
from fabric.contrib import django
import datetime

sys.path.append(os.path.join(os.path.dirname(__file__),
                             'mysite'))

django.settings_module('mysite.settings')
from django.conf import settings

# Hosts
production = '[email protected]'

# Branch to pull from
env.branch = 'master'

@production
def deploy():
    """Uploads files and runs deployment actions to a given server"""
    # path to the directory on the server where your vhost is set up
    path = "/var/www/mysite"
    # name of the application process
    process = "uwsgi"

    print green("Beginning Deploy:")
    with cd(path):
        run("pwd")
        print green("Pulling master from Git server...")
        run("git pull origin %s" % env.branch)
        # use server's virtualenv for commands
        with prefix("source %s/env/bin/activate" % path):
            print green("Installing requirements...")
            run("pip install -r requirements.txt")
            print green("Collecting static files...")
            run("python mysite/manage.py collectstatic --noinput")
            print green("Migrating the database...")
            run("python mysite/manage.py migrate")
        print green("Restart the uwsgi process")
        sudo("service %s restart" % process)
    print green("DONE!")

With these tools, all you need to do when deploying new code to the server is to run one fabric command:

fab deploy

And you’re done.

Most of these ideas I took them from the book Test-Driven Development with Django. You can read it online for free to check out more details or parts I didn’t use in this example.

This is not a perfect solution to every case, but I hope this gives you some ideas for the workflow that fits your case. Share in the comments your deployment workflow or any suggestions to improve this one.

Categories
Programming & Web Development

Select sort algorithm explained with code and dance

Select sort is a very simple algorithm but it’s very inefficient for large lists, but it is good when memory is an issue, since it is an in-place sorting algorithm. Meaning it is an algorithm that doesn’t need to use extra memory to store sorted from unsorted elements.

The way it works is the following: Take the fist element in the list and compare with the next elements to find the minimum value of the list. If found swap values. Repeat starting with second element and so on until reaching the last element in the list.

Select sort example

Big-O notation

O(n2)

Pseudo code

for each current_element in the list:
  assume_minimum = current_element
  for each element in the list:
     if assume_minimum > element[value]
        assume_minimum = element[value]
  swap current element with assume_minimum values

PHP code example

/**
 * Select sort
 *
 * @param array $list The unsorted list
 * @return array The sorted list
 */
function select_sort($list) {
  for($i = 0; $i <= count($list); $i++) {
      $min_position = $i;
      for($j = $i + 1; $j <= count($list); $j++) {
          if($list[$min_position] > $list[$j]) {
              $min_position = $j;
          }
      }
      list($list[$i], $list[$min_position]) = array($list[$min_position], $list[$i]);
  }
  return $list;
}

Python code example

def select_sort(a_list):
  """ Performs a select sort on list """
  for i in range(0,len(a_list)-1):
    min_position = i
    for j in range(i+1,len(a_list)):
      if a_list[min_position] > a_list[j]:
        min_position = j
    a_list[i], a_list[min_position] = a_list[min_position], a_list[i]
  return a_list

Gypsy folk dance example

Categories
Programming & Web Development

Insert-sort algorithm explained with code and dance

Insert sort is a fast algorithm for small lists. On some languages, insert sort is used to sort small arrays (small sometimes means less than 10) instead of more complex algorithms because it’s fast enough and doesn’t need much memory.

Insertion sort example

To perform Insert-sort: take one element from the list on each iteration, starting from the first, and compare with the next element. If the previous one is greater, move it one position ahead. Continue comparing the rest of the elements pushing them forward until you find a the place of the current value and no elements are left. Repeat with next element in list.

Big-O notation

O(n2)

Pseudo code

function insert_sort(list):
  (evaluate the list forward)
  for i from 1 to length_of_list - 1:
    value = list[i]
    (evaluate the list backward from i):
    while prev_position >= 0 and list[prev_position] > value:
      (swap values from current position to the next)
      list[prev_position+1] = list[prev_position]
      go on position back in the list
    if there was no swap, set value back in current position

PHP code example

function insert_sort($list) {
    for($i = 0; $i < count($list); $i++) {
        $value = $list[$i];
        $j = $i - 1;
        while($j >= 0 AND $list[$j] > $value) {
            $list[$j + 1] = $list[$j];
            $j--;
        }
        $list[$j + 1] = $value;
    }
    return $list;
}

Python code example

def insert_sort(unordered_list):
    for i in xrange(0,len(unordered_list)):
        value = unordered_list[i]
        j = i -1
        while (j >= 0) and (unordered_list[j] > value):
            unordered_list[j+1] = unordered_list[j]
            j -= 1
        unordered_list[j+1] = value
    return unordered_list

Insert sort algorithm can be explained better visually with a Romanian folk dance:

Categories
GNU/Linux Free Software & Open Source Programming & Web Development

10 PHP code quality tools to avoid a mess in your projects

img_2052

When programming in any language there are certain common errors that everyone makes as they mature and evolve their programming skills. In the case of PHP, I’ve seen a lot of ugly and complicated code around, since the language is very permissive.

To have a good and healthy PHP code base so it won’t resemble the unmanageable cabling mess in the picture above, your project needs to have the following:

– no spaghetti code
– code reuse (avoid code repetition)
– avoid complicated nested loops
– Encapsulation (organize code in classes)
– Clean user I/O
– Code Documentation
– Consistent code conventions
– Tests

Luckily there are some PHP code quality tools that can help us address most of these problems and give us a diagnostic of our project’s code health.

###Documenting your code

PHP Documentor: Documenting code with PHP Documentor is very simple and easy. You just need to follow a very simple markup before your functions and classes and the tool will generate all the developer documentation for you. It’s also very handy when adding new people to your project or inheriting someone else’s work. Just by seeing the documentation comments in the code you can understand what a given function does, what each of its parameters need to be and what it returns. You can also expand the documentation to add use cases and even unit tests!

###Code consistency

PHP Code Sniffer: A tool that analyses the syntax of your project’s code based on a PHP syntax standard. Known standards like PEAR or Zend syntax styles are included by default but you can also customize and define our own team’s style.

PHP Depend
Analyzes the code for common issues and potential optimizations. Some examples of those are:

– if your function is too long
– if your function has too many parameters
– variable names too long or too short
– too many nested cycles
– use of eval()
– name convention consistency
– unused methods or variables

PHP Mess detector: A fork of PHPDepend with a friendlier UI and configuration options

PHPCPD (Copy/Paste Detector): A tool that will detect copy/pasted code across your project that can be reused.

###Testing

Unit testing

PHP Unit and Simpletest: Unit testing has been very useful in my experience on recent projects where I’ve been allowed to implement it. On personal project I always tend to do unit testing before layout out all my code. It might take more time to develop your project at the beginning but it will save you tons of time debugging and tracing back errors when you think you’re done programming.

If you’re using Ubuntu and have problems running PHPUnit, here’s a post about how to solve the problem.

Here’s a quick guide on how to start using Simpletest.

Integration Testing (browser emulation)

Mink and Behat are the tools you’ll need to do browser emulation to test your interfaces and how it’s all put together and interacting for the user.

###Put it all together with Phing

Phing is a tool like GNU Make or Apache Ant but it’s in PHP and extensible via PHP classes. You can put all of these tools together with Phing, so you can automate all the checking, testing and even deployment and cleanup of files or databases.

With these tools you can have a very robust quality assurance workflow on your project. Of course, this doesn’t mean that there’s not going to be any bugs or that your project will have the best code, but at least it will reduce very common problems, will increase code maintainability and help you spot problems easier. The transitions from adding or removing people from the project (like in large teams or dynamic organizations) will be smoother, as well. Also those transitions will be even smoother if you consider using an established web framework for your project instead of writing your own.

I gave a talk at FSL Vallarta 2012 about this, you can download the slides for the talk “Herramientas de calidad en codigo PHP” here. (slides in Spanish)

Categories
GNU/Linux Free Software & Open Source Programming & Web Development

HTML5 a jump backwards in web standards?

Web standards have always been about good markup, keeping things in order and cleaned up, etc. Using transitional or strict DTDs and validating our markup with different tools was a good practice encouraged by many. With HTML5 things seem to be going a bit backwards regarding structure and markup rules. Google’s HTML styleguide shows an example of how many tags in HTML5 are now optional. The good part of it is that markup is more simple and file sizes get reduced a lot. For high traffic websites like Google’s, every byte saved can be a big difference.

Back when HTML4 was being widely used, there was a lot of messy markup, incomplete tags and many ugly things done by different tools or even made by hand. Then XHTML came in to establish good practices. Web standards were not exactly born then, but got established, more widely known and adopted. XHTML brought a lot of structure and rules on the markup, all with the promise of being forward compatible with whatever new technologies would bring. By having a very well structured document and a correct markup, there would be no ambiguity for the markup parsers of that time and the ones the future would bring.

w3c html5 valid

But now with HTML5 everything that XHTML came to establish seems abandoned. The HTML markup has always been very permissive, as browsers tend to fix any unclosed tags and minor details like that. But the fact that the specification itself is also very permissive with lots of thing as optional, makes me think that we’re going backwards and contradicting everything that was told on the XHTML “web standards days”.

Very little effort is needed to update any XHTML document into HTML5. The move with HTML5’s new tags is to have a more semantic web and it seems to work fine. The question is: will HTML5’s loose markup be as forward compatible as XHTML markup turned out to be?

Categories
GNU/Linux Free Software & Open Source Programming & Web Development

CSS3 columns in Internet Explorer and other unsupported browsers

CSS3 Columns

Recently I’ve been working on a project for a client that makes use of CSS3 columns in its layout design. The problem with CSS3 columns is that it’s not supported by any stable version of Internet Explorer so far (surprised? no? good!). At the time of this writing IE9 is the latest version.

Luckily there’s hope. The CSS3 Multi-column script helps render columns in browsers that do not have this capability. All you need to do is download the file and add it to your HTML after all your stylesheet declarations. It will read all your CSS3 column declarations and render the content accordingly.

The script uses some crossdomain ajax calls to work its magic, but IE8 does things differently. I’ve patched the library with the fixes needed for it to work correctly in IE8.

Download the patched CSS3 Multi-column script.

Thanks to Tobin Titus, Internet Explorer Performance Manager, who helped me track the bug down.