Categories
Programming & Web Development

Avoid Django’s invalid HTTP_HOST error message

Invalid HTTP_HOST error log

I have several Django projects published, and I constantly get my email inbox and log files inundated with errors of spiders and hack attempts to connect to my applications. Those error messages have the email subject: “[Django] ERROR (EXTERNAL IP): Invalid HTTP_HOST …”.

This is due to a Django’s security setting ALLOWED_HOSTS to prevent attacks. Better explained by Django’s documentation page:

This is a security measure to prevent an attacker from poisoning caches and triggering password reset emails with links to malicious hosts by submitting requests with a fake HTTP Host header, which is possible even under many seemingly-safe web server configurations.

At first, I thought of trying to configure Django’s logging to ignore those errors, but I knew that was not the right way to fix the situation. After several attempts, I found the right solution to the problem: a way to configure the web server to stop those connection attempts before they reach Django.

Here’s a configuration example for Apache web server (taken from StackOverflow):

SetEnvIfNoCase Host example\.com VALID_HOST
Order Deny,Allow
Deny from All
Allow from env=VALID_HOST

Here’s a configuration example for Nginx web server (more details here):

upstream app_server {
    server unix:/tmp/gunicorn_mydomain.com.sock fail_timeout=0;
}

server {

    ...

    ## Deny illegal Host headers
    if ($host !~* ^(mydomain.com|www.mydomain.com)$ ) {
        return 444;
    }

    location  / {
        proxy_pass               http://app_server;
        ...
    }

}
Categories
GNU/Linux Free Software & Open Source Programming & Web Development

My Python & Django deployment workflow and tools

Yellow python

If you’re new to Python web development with Django, there are some things that tutorials don’t teach. Deploying to a server when you finish your local development can be a frustrating task. Here is a rundown of the tools and workflow I use for deploying a Django based website.

A few days ago when I launched the new version of Notasbit.com, I ended up having a discussion with a friend about deployment methods for Ruby on Rails and Django websites. My friend is used to Rails development and deployment, something I am not familiar with since Rails 1.8 (looong time ago by now) and he insisted that Ruby Gems method is way easier while Python deployment was a hassle. Yes, for people not familiar with the right tools it can be. There are different versions of Python and library dependencies and versions that can make maintaining a Django website a nightmare.

Here is what I use for deploying Django websites on a live server:

The tools

Git

You cannot be serious about programming in general these days if you’re not using a version control system. In Linus Torvald’s words: “if you’re not using Git, you’re an idiot”. I keep all my projects under version control, even if I’m the only developer on it, even when the code is for personal use only. It not only is a good practice to keep, but it also works as your backup. When I got robbed from my computers, I didn’t loose any project data because it was all backed up on several Git repositories at different places. For deployment tasks, you need to install Git on your server. The server you’re deploying to will serve as a backup as well.

Virtualenv

All that version and dependency hell can be isolated using Python’s Virtualenv tool. This is a tool that will create an encapsulated environment of a given python version and all the libraries you want to use in a project without messing with the system’s main versions. So this way, you can have an old Django 1.4 project running on the same server than another project running a later Django 1.8 or 1.9 version, as well as their respective dependencies.

Fabric

The only reason I still haven’t started using Python 3 on my projects is Fabric. This tool will automate all the project maintenance tasks needed, and that includes doing the deployment. It is similar to a Makefile but you write your tasks in Python code so you don’t need to learn a whole new syntax.

Now to the implementation details…

On your local development machine the project structure will look like this:

Project directory structure. The idea is to have the virtualenv outside the version controlled directory. You can also use virtualenvwrapper and have all your virtualenvs in a separate location. I like to have them contained in a same folder for now.

mainproject.com/
 |
 +-- env/ (virtualenv)
 |
 +-- django_project/
     |
     --- manage.py
     --- .git/
     --- fabfile.py
     --- deploy_tools/
     --- All other django project files & dirs

Deployment scripts

In the directory deploy_tools we’ll place the files necessary to configure apache, nginx, gunicorn, uwsgi or whatever other server configuration scripts needed for deployment.

Here’s an example of the Nginx configuration script I use:

upstream mywebsite {  #the upstream component nginx needs to connect to
    server unix:///tmp/mywebsite.sock; # for a file socket
}

server {    
    listen      80;   # the port your site will be served on
    server_name www.mywebsite.com;   # the domain name it will serve for
    charset     utf-8;

    client_max_body_size 250M;   # max upload size, adjust to taste

    location /static {
        alias /var/www/mywebsite/static; # your Django project's static files - amend as required
    }

    location / {     # Finally, send all non-media requests to the Django server.
        uwsgi_pass  mywebsite;
        uwsgi_param QUERY_STRING $query_string;
        uwsgi_param REQUEST_METHOD $request_method;
        uwsgi_param CONTENT_TYPE $content_type;
        uwsgi_param CONTENT_LENGTH $content_length;

        uwsgi_param REQUEST_URI $request_uri;
        uwsgi_param PATH_INFO $document_uri;
        uwsgi_param DOCUMENT_ROOT $document_root;
        uwsgi_param SERVER_PROTOCOL $server_protocol;
        uwsgi_param HTTPS $https if_not_empty;

        uwsgi_param REMOTE_ADDR $remote_addr;
        uwsgi_param REMOTE_PORT $remote_port;
        uwsgi_param SERVER_PORT $server_port;
        uwsgi_param SERVER_NAME $server_name;
    }
}

Settings handling

There are many ways to solve the problem about settings file management in Django applications. I like to have a general settings file and a local one for the different environments. To achieve this, add a local_settings.py file and add it to the git ignore list.

Then add the following at the end of your general settings.py file:

# Import local settings
try:
    from local_settings import *
except ImportError:
    pass

In your remote server, create a local_settings.py file and add DEBUG = False or override any other setting you need specifically for that server. You can have different settings for local, staging, testing or production or any other servers you need to have.

Fabric tasks

For example, you might need to download a copy of your production database to use for development tests. Instead of typing the same mysqldump command every time, you can automate it like this:

def backup_db():
    """
    Gets a database dump from remote server
    """
    production1()
    date = time.strftime('%Y-%m-%d-%H%M%S')
    dbname = 'MY-DATABASE-NAME'
    path = os.path.join(os.path.dirname(__file__), 'db_backups')
    fname = "{dbname}_backup_{date}.sql.gz".format(date=date,dbname=dbname)

    run("mysqldump -u {dbuser} -p'{password}' --add-drop-table -B {database} | gzip -9 > {filename}".format(
        database=dbname,
        dbuser='MY-DB-USER',
        password='MY-SERVER-PASSWORD',
        filename=os.path.join('/tmp', fname))
    )
    get(remote_path=os.path.join('/tmp', fname),
        local_path=os.path.join(path, fname))
    run("rm {filename}".format(filename=os.path.join('/tmp', fname)))

Likewise, you can create a deploy command to get everything in the server. Here’s a fabfile.py example:

from fabric.api import *
from fabric.colors import green, red
import os
import sys
import time
from fabric.contrib import django
import datetime

sys.path.append(os.path.join(os.path.dirname(__file__),
                             'mysite'))

django.settings_module('mysite.settings')
from django.conf import settings

# Hosts
production = '[email protected]'

# Branch to pull from
env.branch = 'master'

@production
def deploy():
    """Uploads files and runs deployment actions to a given server"""
    # path to the directory on the server where your vhost is set up
    path = "/var/www/mysite"
    # name of the application process
    process = "uwsgi"

    print green("Beginning Deploy:")
    with cd(path):
        run("pwd")
        print green("Pulling master from Git server...")
        run("git pull origin %s" % env.branch)
        # use server's virtualenv for commands
        with prefix("source %s/env/bin/activate" % path):
            print green("Installing requirements...")
            run("pip install -r requirements.txt")
            print green("Collecting static files...")
            run("python mysite/manage.py collectstatic --noinput")
            print green("Migrating the database...")
            run("python mysite/manage.py migrate")
        print green("Restart the uwsgi process")
        sudo("service %s restart" % process)
    print green("DONE!")

With these tools, all you need to do when deploying new code to the server is to run one fabric command:

fab deploy

And you’re done.

Most of these ideas I took them from the book Test-Driven Development with Django. You can read it online for free to check out more details or parts I didn’t use in this example.

This is not a perfect solution to every case, but I hope this gives you some ideas for the workflow that fits your case. Share in the comments your deployment workflow or any suggestions to improve this one.

Categories
Programming & Web Development

Using Django models in external Python scripts

Since Django 1.4 the syntax to import you models in external scripts has changed. The setup_environ method has been deprecated for a while and it no longer exists in Django 1.6.

Assuming you’re using a virtualenv and your script is one directory outside the main Django project directory structure.

my_directory/
  |
  |---> virtualenv/
  |---> my_project/   <--- Django project files
  |---> my_script.py

I use this for Django 1.6 integration in my_script.py:

sys.path.append(os.path.join(os.path.dirname(__file__), ‘my_project’))
os.environ.setdefault(“DJANGO_SETTINGS_MODULE”, “my_project.settings”)
from django.conf import settings

from my_app.models import MyModel

Categories
Programming & Web Development

Select sort algorithm explained with code and dance

Select sort is a very simple algorithm but it’s very inefficient for large lists, but it is good when memory is an issue, since it is an in-place sorting algorithm. Meaning it is an algorithm that doesn’t need to use extra memory to store sorted from unsorted elements.

The way it works is the following: Take the fist element in the list and compare with the next elements to find the minimum value of the list. If found swap values. Repeat starting with second element and so on until reaching the last element in the list.

Select sort example

Big-O notation

O(n2)

Pseudo code

for each current_element in the list:
  assume_minimum = current_element
  for each element in the list:
     if assume_minimum > element[value]
        assume_minimum = element[value]
  swap current element with assume_minimum values

PHP code example

/**
 * Select sort
 *
 * @param array $list The unsorted list
 * @return array The sorted list
 */
function select_sort($list) {
  for($i = 0; $i <= count($list); $i++) {
      $min_position = $i;
      for($j = $i + 1; $j <= count($list); $j++) {
          if($list[$min_position] > $list[$j]) {
              $min_position = $j;
          }
      }
      list($list[$i], $list[$min_position]) = array($list[$min_position], $list[$i]);
  }
  return $list;
}

Python code example

def select_sort(a_list):
  """ Performs a select sort on list """
  for i in range(0,len(a_list)-1):
    min_position = i
    for j in range(i+1,len(a_list)):
      if a_list[min_position] > a_list[j]:
        min_position = j
    a_list[i], a_list[min_position] = a_list[min_position], a_list[i]
  return a_list

Gypsy folk dance example

Categories
Programming & Web Development

Insert-sort algorithm explained with code and dance

Insert sort is a fast algorithm for small lists. On some languages, insert sort is used to sort small arrays (small sometimes means less than 10) instead of more complex algorithms because it’s fast enough and doesn’t need much memory.

Insertion sort example

To perform Insert-sort: take one element from the list on each iteration, starting from the first, and compare with the next element. If the previous one is greater, move it one position ahead. Continue comparing the rest of the elements pushing them forward until you find a the place of the current value and no elements are left. Repeat with next element in list.

Big-O notation

O(n2)

Pseudo code

function insert_sort(list):
  (evaluate the list forward)
  for i from 1 to length_of_list - 1:
    value = list[i]
    (evaluate the list backward from i):
    while prev_position >= 0 and list[prev_position] > value:
      (swap values from current position to the next)
      list[prev_position+1] = list[prev_position]
      go on position back in the list
    if there was no swap, set value back in current position

PHP code example

function insert_sort($list) {
    for($i = 0; $i < count($list); $i++) {
        $value = $list[$i];
        $j = $i - 1;
        while($j >= 0 AND $list[$j] > $value) {
            $list[$j + 1] = $list[$j];
            $j--;
        }
        $list[$j + 1] = $value;
    }
    return $list;
}

Python code example

def insert_sort(unordered_list):
    for i in xrange(0,len(unordered_list)):
        value = unordered_list[i]
        j = i -1
        while (j >= 0) and (unordered_list[j] > value):
            unordered_list[j+1] = unordered_list[j]
            j -= 1
        unordered_list[j+1] = value
    return unordered_list

Insert sort algorithm can be explained better visually with a Romanian folk dance:

Categories
Programming & Web Development

Basic programming algorithm: Bubble Sort

Bubble sort is one of the most basic sorting algorithms in taught in computer science classes. It will order values in a list from smallest to largest passing several times through the list comparing two items at a time and reordering them. This algorithm is not very efficient with large lists because it has to evaluate every member of the list several times. The longer the list, the more it will take the algorithm to finish. In big-O notation, this algorithm’s efficiency is represented by: O(n2)

An explanation of bubble sort from Wikipedia:

Bubble sort, sometimes incorrectly referred to as sinking sort, is a simple sorting algorithm that works by repeatedly stepping through the list to be sorted, comparing each pair of adjacent items and swapping them if they are in the wrong order. The pass through the list is repeated until no swaps are needed, which indicates that the list is sorted. The algorithm gets its name from the way smaller elements “bubble” to the top of the list. Because it only uses comparisons to operate on elements, it is a comparison sort. Although the algorithm is simple, most of the other sorting algorithms are more efficient for large lists.

bubble sort example

While passing through the list, compare two elements, if the second one is bigger than the first, swap their positions in the list. Continue going through the list, comparing the next elements in the same way. Go through the list until no more swaps take place.

Pseudocode:

while not sorted:
  sorted = True
  for iterator = 1 to length_of_list - 1 do:
    check if pair is out of order:
    if list[iterator] is bigger than list[iterator+1] then
    swap them:
      temporal_value = list[iterator]
      list[iterator] = list[iterator+1]
      list[iterator+1] = temporal_value
      sorted = False
      skip to next element in list

PHP code example:

/**
 * Sort an array using bubble sort algorithm
 * @param array $list The list to be sorted
 * @return array The sorted array
 */
function bubbleSort($list) {
   while(!$sorted){
     $sorted = True;
     foreach($list as $key => $value) {
        if($list[$key] > $list[$key+1]) {
          $temp = $list[$key+1];
          $list[$key+1] = $list[$key];
          $list[$key] = $temp;
          $sorted = False;
        }
     }
   }
   return $list;
}

Python code example:

function bubbleSort(a_list):
  """ Sort an array using bubble sort algorithm """
  sorted = False
  length = len(a_list)
  while not sorted:
     sorted = True
     for position in range(length):
       if a_list[position] > a_list[position+1]:
         # Swap values
         a_list[position], a_list[position+1] = a_list[position+1], a_list[position]
         sorted = False
  return a_list

There is another way to write the bubble sort, by shrinking the list by one element after every pass. If you see the animated graphic above, after each pass, the last number is already in its correct place, so you don’t have to iterate over those every subsequent time. You can discard the last element each time, making the algorithm faster.

An example of this version in Python code would look like this:

function betterBubbleSort(a_list):
  """ Sort an array using bubble sort algorithm """
  length = len(a_list)
  while length:
     for position in range(length):
       if a_list[position] > a_list[position+1]:
         # Swap values
         a_list[position], a_list[position+1] = a_list[position+1],a_list[position]
     length -= 1
  return a_list

Do you have other ways of writing the bubble sort? Share them in the comments section below.