Categories
Programming & Web Development

Avoid Django’s invalid HTTP_HOST error message

Invalid HTTP_HOST error log

I have several Django projects published, and I constantly get my email inbox and log files inundated with errors of spiders and hack attempts to connect to my applications. Those error messages have the email subject: “[Django] ERROR (EXTERNAL IP): Invalid HTTP_HOST …”.

This is due to a Django’s security setting ALLOWED_HOSTS to prevent attacks. Better explained by Django’s documentation page:

This is a security measure to prevent an attacker from poisoning caches and triggering password reset emails with links to malicious hosts by submitting requests with a fake HTTP Host header, which is possible even under many seemingly-safe web server configurations.

At first, I thought of trying to configure Django’s logging to ignore those errors, but I knew that was not the right way to fix the situation. After several attempts, I found the right solution to the problem: a way to configure the web server to stop those connection attempts before they reach Django.

Here’s a configuration example for Apache web server (taken from StackOverflow):

SetEnvIfNoCase Host example\.com VALID_HOST
Order Deny,Allow
Deny from All
Allow from env=VALID_HOST

Here’s a configuration example for Nginx web server (more details here):

upstream app_server {
    server unix:/tmp/gunicorn_mydomain.com.sock fail_timeout=0;
}

server {

    ...

    ## Deny illegal Host headers
    if ($host !~* ^(mydomain.com|www.mydomain.com)$ ) {
        return 444;
    }

    location  / {
        proxy_pass               http://app_server;
        ...
    }

}
Categories
GNU/Linux Free Software & Open Source Programming & Web Development

My Python & Django deployment workflow and tools

Yellow python

If you’re new to Python web development with Django, there are some things that tutorials don’t teach. Deploying to a server when you finish your local development can be a frustrating task. Here is a rundown of the tools and workflow I use for deploying a Django based website.

A few days ago when I launched the new version of Notasbit.com, I ended up having a discussion with a friend about deployment methods for Ruby on Rails and Django websites. My friend is used to Rails development and deployment, something I am not familiar with since Rails 1.8 (looong time ago by now) and he insisted that Ruby Gems method is way easier while Python deployment was a hassle. Yes, for people not familiar with the right tools it can be. There are different versions of Python and library dependencies and versions that can make maintaining a Django website a nightmare.

Here is what I use for deploying Django websites on a live server:

The tools

Git

You cannot be serious about programming in general these days if you’re not using a version control system. In Linus Torvald’s words: “if you’re not using Git, you’re an idiot”. I keep all my projects under version control, even if I’m the only developer on it, even when the code is for personal use only. It not only is a good practice to keep, but it also works as your backup. When I got robbed from my computers, I didn’t loose any project data because it was all backed up on several Git repositories at different places. For deployment tasks, you need to install Git on your server. The server you’re deploying to will serve as a backup as well.

Virtualenv

All that version and dependency hell can be isolated using Python’s Virtualenv tool. This is a tool that will create an encapsulated environment of a given python version and all the libraries you want to use in a project without messing with the system’s main versions. So this way, you can have an old Django 1.4 project running on the same server than another project running a later Django 1.8 or 1.9 version, as well as their respective dependencies.

Fabric

The only reason I still haven’t started using Python 3 on my projects is Fabric. This tool will automate all the project maintenance tasks needed, and that includes doing the deployment. It is similar to a Makefile but you write your tasks in Python code so you don’t need to learn a whole new syntax.

Now to the implementation details…

On your local development machine the project structure will look like this:

Project directory structure. The idea is to have the virtualenv outside the version controlled directory. You can also use virtualenvwrapper and have all your virtualenvs in a separate location. I like to have them contained in a same folder for now.

mainproject.com/
 |
 +-- env/ (virtualenv)
 |
 +-- django_project/
     |
     --- manage.py
     --- .git/
     --- fabfile.py
     --- deploy_tools/
     --- All other django project files & dirs

Deployment scripts

In the directory deploy_tools we’ll place the files necessary to configure apache, nginx, gunicorn, uwsgi or whatever other server configuration scripts needed for deployment.

Here’s an example of the Nginx configuration script I use:

upstream mywebsite {  #the upstream component nginx needs to connect to
    server unix:///tmp/mywebsite.sock; # for a file socket
}

server {    
    listen      80;   # the port your site will be served on
    server_name www.mywebsite.com;   # the domain name it will serve for
    charset     utf-8;

    client_max_body_size 250M;   # max upload size, adjust to taste

    location /static {
        alias /var/www/mywebsite/static; # your Django project's static files - amend as required
    }

    location / {     # Finally, send all non-media requests to the Django server.
        uwsgi_pass  mywebsite;
        uwsgi_param QUERY_STRING $query_string;
        uwsgi_param REQUEST_METHOD $request_method;
        uwsgi_param CONTENT_TYPE $content_type;
        uwsgi_param CONTENT_LENGTH $content_length;

        uwsgi_param REQUEST_URI $request_uri;
        uwsgi_param PATH_INFO $document_uri;
        uwsgi_param DOCUMENT_ROOT $document_root;
        uwsgi_param SERVER_PROTOCOL $server_protocol;
        uwsgi_param HTTPS $https if_not_empty;

        uwsgi_param REMOTE_ADDR $remote_addr;
        uwsgi_param REMOTE_PORT $remote_port;
        uwsgi_param SERVER_PORT $server_port;
        uwsgi_param SERVER_NAME $server_name;
    }
}

Settings handling

There are many ways to solve the problem about settings file management in Django applications. I like to have a general settings file and a local one for the different environments. To achieve this, add a local_settings.py file and add it to the git ignore list.

Then add the following at the end of your general settings.py file:

# Import local settings
try:
    from local_settings import *
except ImportError:
    pass

In your remote server, create a local_settings.py file and add DEBUG = False or override any other setting you need specifically for that server. You can have different settings for local, staging, testing or production or any other servers you need to have.

Fabric tasks

For example, you might need to download a copy of your production database to use for development tests. Instead of typing the same mysqldump command every time, you can automate it like this:

def backup_db():
    """
    Gets a database dump from remote server
    """
    production1()
    date = time.strftime('%Y-%m-%d-%H%M%S')
    dbname = 'MY-DATABASE-NAME'
    path = os.path.join(os.path.dirname(__file__), 'db_backups')
    fname = "{dbname}_backup_{date}.sql.gz".format(date=date,dbname=dbname)

    run("mysqldump -u {dbuser} -p'{password}' --add-drop-table -B {database} | gzip -9 > {filename}".format(
        database=dbname,
        dbuser='MY-DB-USER',
        password='MY-SERVER-PASSWORD',
        filename=os.path.join('/tmp', fname))
    )
    get(remote_path=os.path.join('/tmp', fname),
        local_path=os.path.join(path, fname))
    run("rm {filename}".format(filename=os.path.join('/tmp', fname)))

Likewise, you can create a deploy command to get everything in the server. Here’s a fabfile.py example:

from fabric.api import *
from fabric.colors import green, red
import os
import sys
import time
from fabric.contrib import django
import datetime

sys.path.append(os.path.join(os.path.dirname(__file__),
                             'mysite'))

django.settings_module('mysite.settings')
from django.conf import settings

# Hosts
production = '[email protected]'

# Branch to pull from
env.branch = 'master'

@production
def deploy():
    """Uploads files and runs deployment actions to a given server"""
    # path to the directory on the server where your vhost is set up
    path = "/var/www/mysite"
    # name of the application process
    process = "uwsgi"

    print green("Beginning Deploy:")
    with cd(path):
        run("pwd")
        print green("Pulling master from Git server...")
        run("git pull origin %s" % env.branch)
        # use server's virtualenv for commands
        with prefix("source %s/env/bin/activate" % path):
            print green("Installing requirements...")
            run("pip install -r requirements.txt")
            print green("Collecting static files...")
            run("python mysite/manage.py collectstatic --noinput")
            print green("Migrating the database...")
            run("python mysite/manage.py migrate")
        print green("Restart the uwsgi process")
        sudo("service %s restart" % process)
    print green("DONE!")

With these tools, all you need to do when deploying new code to the server is to run one fabric command:

fab deploy

And you’re done.

Most of these ideas I took them from the book Test-Driven Development with Django. You can read it online for free to check out more details or parts I didn’t use in this example.

This is not a perfect solution to every case, but I hope this gives you some ideas for the workflow that fits your case. Share in the comments your deployment workflow or any suggestions to improve this one.

Categories
Programming & Web Development

Using Django models in external Python scripts

Since Django 1.4 the syntax to import you models in external scripts has changed. The setup_environ method has been deprecated for a while and it no longer exists in Django 1.6.

Assuming you’re using a virtualenv and your script is one directory outside the main Django project directory structure.

my_directory/
  |
  |---> virtualenv/
  |---> my_project/   <--- Django project files
  |---> my_script.py

I use this for Django 1.6 integration in my_script.py:

sys.path.append(os.path.join(os.path.dirname(__file__), ‘my_project’))
os.environ.setdefault(“DJANGO_SETTINGS_MODULE”, “my_project.settings”)
from django.conf import settings

from my_app.models import MyModel

Categories
Bookmarks Programming & Web Development

From my reading links: HTML5 forms and Django, Google API playground and Python development tips

  • Making Forms fabulous with html5 Great introduction to the new HTML5 form input types and what you can do with them.
  • HTML5 – Example Form validation and styling Form elements With the new HTML5 form input types and CSS3 properties, form validation is very quick and easy, as well as nice form element presentation.
  • Head JS A very interesting utility that will load all your js scripts (jquery, google analytics, etc) in parallel so page load times get shorter, plus a lot of other nice features like new HTML5 elements styling in browsers that don’t yet support them like some versions of Internet Explorer.
  • Google APIs code playgroud I didn’t know you could play with all the Google APIs in a single webpage, testing your code as your write it.
  • GeoDjango and google maps An interesting read if you’re trying to integrate Google Maps with Django
  • HTML5 and Django form inputs How to render the new HTML5 input types with Django form classes. It doesn’t require much, really easy and fast.
  • Emacs as Python IDE A bunch of tips to have a good Emacs Python development environment. I personally don’t prefer IDEs, so I don’t use ECB but the rest is very useful.
  • Setting up a Python Environment Good advice for those coming from a PHP world and are getting started on Python development. I didn’t know about virtualenv until I had to deploy my first Django app on a shared host web server. Then it all made sense.
Categories
GNU/Linux Free Software & Open Source Programming & Web Development

My 5 reasons to leave Ruby on Rails

I’m stopping all my ruby on rails development. Its been a while since my last rails line was written and I will not continue to develop on that framework.

Here are my reasons for this:

  1. Documentation and entry point price
  2. First of all, to get any serious documentation on how to use the framework, you have to buy Pragmatic Programmer’s Ruby on Rails book. While its a good read and all, its a $40 US dollars (ok, now its $19.95, but I bought it when it was recently published) entry price to get serious on the subject. Another good resource are the Peepcode’s videos. They are excellent reference material, but still there’s a cost. Its not that I’m against commerce and all that. I’m just making the point that there’s very little or no good documentation for rails for free. Plus, there’s a disadvantage on printed books that they’re not up to date with the latest changes. There’s a lot of “when rails2 comes out…” comments on it.

  3. Community problems
  4. Then are the community problems. Its people and people sometimes don’t get along, but the comments from Zed Shaw, the mongrel web server developer are something to pay some attention to. Besides, he’s also stopped development on mongrel, so the installation trend must, or probably by now has, shifted again.

  5. Hosting
  6. Another big problem is the hosting support. All the projects I’ve developed I’ve tried then successfully on my development computer and on some self owned servers. Hosting support for Ruby on Rails is still not very common, so if any of my clients don’t have control over their server and their hosting solution doesn’t support it, its bad.

  7. Production deployment “trends”
  8. One of the main problems I see often is production deployment “trends”. The method to succesfully deploy a rails app on any server changes constantly. First, the recommendation was fastcgi with lighttpd, then it was mongrel servers doing load balancing with an apache server, then capistrano, then nginx, now I’ve lost track of what to do. Even on the book this ambiguity is mentioned.

  9. Ruby gems
  10. And finally, now that we’re talking about owning a server and deploying it for production, the final caveat is the ruby gems mess, described on more detail on several Debian developers (Wouter, Gunnar, Luke Kanie) blogs.

So, with all this in mind, I’ll be looking to port some of my already developed rails apps to Django. I’ve seen a lot of progress on this framework and like a lot of its design concepts. As much as I’ll miss a lot of ruby’s syntax and “sugar”, python is not bad either and its also a very comfortable language to work with. There’s a lot more libraries for python than for ruby to build stuff. Also, its interpreter is a lot faster than the ruby1.8 one. I know there’s a promise that ruby1.9 will be a lot faster, but its not ready yet.

Finally, I see there’s a lot of good stuff in Rails, but for me, right now is not a good time to jump on that ship.