Categories
GNU/Linux Free Software & Open Source Programming & Web Development

My Python & Django deployment workflow and tools

Yellow python

If you’re new to Python web development with Django, there are some things that tutorials don’t teach. Deploying to a server when you finish your local development can be a frustrating task. Here is a rundown of the tools and workflow I use for deploying a Django based website.

A few days ago when I launched the new version of Notasbit.com, I ended up having a discussion with a friend about deployment methods for Ruby on Rails and Django websites. My friend is used to Rails development and deployment, something I am not familiar with since Rails 1.8 (looong time ago by now) and he insisted that Ruby Gems method is way easier while Python deployment was a hassle. Yes, for people not familiar with the right tools it can be. There are different versions of Python and library dependencies and versions that can make maintaining a Django website a nightmare.

Here is what I use for deploying Django websites on a live server:

The tools

Git

You cannot be serious about programming in general these days if you’re not using a version control system. In Linus Torvald’s words: “if you’re not using Git, you’re an idiot”. I keep all my projects under version control, even if I’m the only developer on it, even when the code is for personal use only. It not only is a good practice to keep, but it also works as your backup. When I got robbed from my computers, I didn’t loose any project data because it was all backed up on several Git repositories at different places. For deployment tasks, you need to install Git on your server. The server you’re deploying to will serve as a backup as well.

Virtualenv

All that version and dependency hell can be isolated using Python’s Virtualenv tool. This is a tool that will create an encapsulated environment of a given python version and all the libraries you want to use in a project without messing with the system’s main versions. So this way, you can have an old Django 1.4 project running on the same server than another project running a later Django 1.8 or 1.9 version, as well as their respective dependencies.

Fabric

The only reason I still haven’t started using Python 3 on my projects is Fabric. This tool will automate all the project maintenance tasks needed, and that includes doing the deployment. It is similar to a Makefile but you write your tasks in Python code so you don’t need to learn a whole new syntax.

Now to the implementation details…

On your local development machine the project structure will look like this:

Project directory structure. The idea is to have the virtualenv outside the version controlled directory. You can also use virtualenvwrapper and have all your virtualenvs in a separate location. I like to have them contained in a same folder for now.

mainproject.com/
 |
 +-- env/ (virtualenv)
 |
 +-- django_project/
     |
     --- manage.py
     --- .git/
     --- fabfile.py
     --- deploy_tools/
     --- All other django project files & dirs

Deployment scripts

In the directory deploy_tools we’ll place the files necessary to configure apache, nginx, gunicorn, uwsgi or whatever other server configuration scripts needed for deployment.

Here’s an example of the Nginx configuration script I use:

upstream mywebsite {  #the upstream component nginx needs to connect to
    server unix:///tmp/mywebsite.sock; # for a file socket
}

server {    
    listen      80;   # the port your site will be served on
    server_name www.mywebsite.com;   # the domain name it will serve for
    charset     utf-8;

    client_max_body_size 250M;   # max upload size, adjust to taste

    location /static {
        alias /var/www/mywebsite/static; # your Django project's static files - amend as required
    }

    location / {     # Finally, send all non-media requests to the Django server.
        uwsgi_pass  mywebsite;
        uwsgi_param QUERY_STRING $query_string;
        uwsgi_param REQUEST_METHOD $request_method;
        uwsgi_param CONTENT_TYPE $content_type;
        uwsgi_param CONTENT_LENGTH $content_length;

        uwsgi_param REQUEST_URI $request_uri;
        uwsgi_param PATH_INFO $document_uri;
        uwsgi_param DOCUMENT_ROOT $document_root;
        uwsgi_param SERVER_PROTOCOL $server_protocol;
        uwsgi_param HTTPS $https if_not_empty;

        uwsgi_param REMOTE_ADDR $remote_addr;
        uwsgi_param REMOTE_PORT $remote_port;
        uwsgi_param SERVER_PORT $server_port;
        uwsgi_param SERVER_NAME $server_name;
    }
}

Settings handling

There are many ways to solve the problem about settings file management in Django applications. I like to have a general settings file and a local one for the different environments. To achieve this, add a local_settings.py file and add it to the git ignore list.

Then add the following at the end of your general settings.py file:

# Import local settings
try:
    from local_settings import *
except ImportError:
    pass

In your remote server, create a local_settings.py file and add DEBUG = False or override any other setting you need specifically for that server. You can have different settings for local, staging, testing or production or any other servers you need to have.

Fabric tasks

For example, you might need to download a copy of your production database to use for development tests. Instead of typing the same mysqldump command every time, you can automate it like this:

def backup_db():
    """
    Gets a database dump from remote server
    """
    production1()
    date = time.strftime('%Y-%m-%d-%H%M%S')
    dbname = 'MY-DATABASE-NAME'
    path = os.path.join(os.path.dirname(__file__), 'db_backups')
    fname = "{dbname}_backup_{date}.sql.gz".format(date=date,dbname=dbname)

    run("mysqldump -u {dbuser} -p'{password}' --add-drop-table -B {database} | gzip -9 > {filename}".format(
        database=dbname,
        dbuser='MY-DB-USER',
        password='MY-SERVER-PASSWORD',
        filename=os.path.join('/tmp', fname))
    )
    get(remote_path=os.path.join('/tmp', fname),
        local_path=os.path.join(path, fname))
    run("rm {filename}".format(filename=os.path.join('/tmp', fname)))

Likewise, you can create a deploy command to get everything in the server. Here’s a fabfile.py example:

from fabric.api import *
from fabric.colors import green, red
import os
import sys
import time
from fabric.contrib import django
import datetime

sys.path.append(os.path.join(os.path.dirname(__file__),
                             'mysite'))

django.settings_module('mysite.settings')
from django.conf import settings

# Hosts
production = '[email protected]'

# Branch to pull from
env.branch = 'master'

@production
def deploy():
    """Uploads files and runs deployment actions to a given server"""
    # path to the directory on the server where your vhost is set up
    path = "/var/www/mysite"
    # name of the application process
    process = "uwsgi"

    print green("Beginning Deploy:")
    with cd(path):
        run("pwd")
        print green("Pulling master from Git server...")
        run("git pull origin %s" % env.branch)
        # use server's virtualenv for commands
        with prefix("source %s/env/bin/activate" % path):
            print green("Installing requirements...")
            run("pip install -r requirements.txt")
            print green("Collecting static files...")
            run("python mysite/manage.py collectstatic --noinput")
            print green("Migrating the database...")
            run("python mysite/manage.py migrate")
        print green("Restart the uwsgi process")
        sudo("service %s restart" % process)
    print green("DONE!")

With these tools, all you need to do when deploying new code to the server is to run one fabric command:

fab deploy

And you’re done.

Most of these ideas I took them from the book Test-Driven Development with Django. You can read it online for free to check out more details or parts I didn’t use in this example.

This is not a perfect solution to every case, but I hope this gives you some ideas for the workflow that fits your case. Share in the comments your deployment workflow or any suggestions to improve this one.

Categories
GNU/Linux Free Software & Open Source News

Launching Notasbit, a website for tech news in Spanish

notasbit_preview
For the past year I’ve been on a radio show on Tuesdays talking about the tech news of the week. I started by hunting down the most important stories, writing my script in Spanish and then broadcasting it.

One day I thought it would be a good idea to put my scripts as posts in a tech news blog. I started the blog and it was a place for people listening to the news and wanted to know more or check out the photos or videos about them with links to the original sources.

But later with work loads I could barely have time to hunt for the news, so the blog languished. I couldn’t do all that extra work besides my main day job.

So as an engineer, I decided to automate things. Notasbit.com is no longer a news blog. It is now an automated tech news website. Notasbit is the point of reference for the most interesting tech news in Spanish. The site links to the latest posts from many Spanish language tech and science news websites and sorts the most important ones at the top of the list. It differentiates itself from a simple RSS feed reader with this algorithm, so you don’t have to dig into your several feeds to find the best articles. You can also see a list of all news gathered sorted by the time they were published.

I’ll be adding more enhancements and features that will help improve the selection of the best news. Many experts say that the projects consumed by their makers are the best ones, since you have to “eat your own dog food”. I’ll see if that happens in my case.

I really hope people enjoy Notasbit.com. For me, it will ease my task of hunting down news for the radio show and will also serve as the point of reference for people who listen to me and want to know more.

Let me know if you like the site or if you have any suggestions in the comments below.

Categories
GNU/Linux Free Software & Open Source

Debian on Lenovo Thinkpad X240

I recently got myself a new computer, the Lenovo Thinkpad X240. It’s my first Thinkpad so I cannot compare it to previous models. My old laptop is a Dell XPS m1530 that’s about 5 years old by the time of this writing. Still very functional but I wanted an upgrade in hardware and a lighter computer to travel with.

The pros:

– weight
– matte screen and higher resolution
– keyboard
– speed (processor, RAM, SSD, USB3)

The cons:

– GNU/Linux compatibility
– wireless
– brightness controls
– FN key and FN Lock

So let’s go and review the whole thing:

### BIOS

It came with Windows 8 pre-installed, but I immediately installed Debian Testing (Jessie) on it. This machine comes with UEFI boot, but fortunately it has a Legacy mode to behave like normal BIOS. Installing like normal boot instead of UEFI boot is a lot easier and you don’t have to struggle with the disk partitions and boot options. On the boot configuration settings, disable secure boot and set it to legacy boot first instead of UEFI boot first.

### Hardware

Lenovo X240 open

The X240 is very light, even with the additional 6 cell battery instead of the default 3 cell battery. It is a 12.5″ computer, so it can be comparable to a Macbook Air or similar computer. The case is plastic, but feels very well built and durable.

X240 width

There are no indicating LEDs anywhere except for the power button LED that indicates if the computer is on, off or suspended (blinking). There are no hard disk writing indication, battery charging indicator, wireless, bluetooth or any. Just an additional led behind the screen, the dot in the “ThinkPad” logo lights following the power button LED and that’s it.

This is a bit confusing specially when charging the laptop, since the charger also lacks any LED indicator, so the only way to know if your computer is charging is with the software indicators in your desktop environment (the battery monitor icon in your system tray).

It has no HDMI port, but it has the old VGA adapter and a mini digital port for external monitors. Only 2 USB ports, both are USB 3.0, and one with power over USB. My model came with a fingerprint reader, an SD card reader, a 720p webcam and an Ethernet port.

### Keyboard

Lenovo X240 Keyboard

It seems that all new Lenovo models are coming with the new “chicklets” style keyboard and its new layout. I’ve heard some criticism about it, but since this is my first Thinkpad, I’m not biased. Comparing to other laptop keyboards, it is nice. The keys feel good and not fragile and is very silent. My only compliant was the strange placement of the Fn key where I usually expect the Ctrl key. Fortunately for me as an Emacs user I map my CapsLock key to an additional Ctrl and use that instead, so my key stroke memory doesn’t get much affected by that. What I didn’t like is that the F keys are now by default media keys and to use them as F keys you need to press Fn+key or Fn+Esc to activate ‘Function key lock’ then press the F key you need. So, for example, if you want to reload your browser and immediately lower the volume of the speakers, you have to strike additional keys.

The screen brightness control keys by default didn’t work for me, but there is a fix. You need to load the thinkpad acpi kernel module on boot. So edit /etc/modules file and add:

thinkpad_acpi

Then you’ll need to add this to your /etc/default/grub file and check that your kernel options are as follows:

RUB_CMDLINE_LINUX_DEFAULT="quiet acpi_osi=!Windows2012 acpi_backlight=vendor"

This will enable the volume, mute, brightness and wireless media keys. I haven't been able to make the microphone mute button work.

The keyboard backlight works by default using FN+spacebar and it looks nice. It has three states: dim, bright and off.

### Pointers

As many other previous ThinkPads, the X240 still keeps the nipple mouse or clit mouse or however you've heard it's called. The downside is on the trackpad, now called clickpad which has no hardware buttons but does have a larger surface area. By default on Debian Jessie + KDE the clickpad works but it's not precise. When trying to press for a click it inevitably moves from the target area, so clicking is hard. Right click worked out of the box for me by just pressing the pad on its bottom right area. Clicking can be fixed by addding:

sudo apt-get install kde-config-touchpad

and configure single tap clicking, two-finger scrolling and three finger tap for middle click. If you like to use the /nipple/ pointer and miss the buttons for it on the top of the pad, you can configure the button area to be on the top part instead of the bottom part in the X config file.

Make sure you have the following in your file /usr/share/X11/xorg.conf.d/50-synaptics.conf

    # This option enables the bottom right corner to be a right button on
    # non-synaptics clickpads.
    # This option is only interpreted by clickpads.
    Section "InputClass"
            Identifier "Default clickpad buttons"
            MatchDriver "synaptics" 
            #Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
            Option "SoftButtonAreas" "60% 0 0 5% 40% 60% 0 5%"
    #       To disable the bottom edge area so the buttons only work as buttons,
    #       not for movement, set the AreaBottomEdge
            #Option "AreaBottomEdge" "82%"
            Option "AreaTopEdge" "4%"
    EndSection

I found this solution at this blog

### Wireless

The wireless card is an Intel 7620 rev 6. After having to struggle with a Broadcom card for many years I thought my wireless card struggle days were going to be over. Well, turns out that the 7620 card is a very recent card and is not very well supported yet. By default it doesn't work with Debian. To make it work you need to enable the non-free repository and install

sudo apt-get install firmware-iwlwifi

This will enable your card and make it work, but if you suspend the computer, you'll loose bluetooth conectivity. To fix that you need to disable the 802.11n compatibility in the driver configuration. Add this to a file in a new file called: /etc/modprobe.d/wifi-disable11n.conf

options iwlwifi 11n_disable=1

I have experienced some instability with it. At random times the driver would just stop working and your card will seem working and connected but no traffic goes on. If you suspend the computer with the card in that state, it will not suspend and will hang the system. Once the wifi card is stuck the only way to bring it back is with a restart. I've tried rfkill, unloading and loading the module, but nothing works.

Another issue is that it will not connect to a wireless-n router, even when disabling n-band in the driver configurations. I had to configure my home router to only use b/g bands for it to connect.

**UPDATE**: Debian Jessie has upgraded to the Linux kernel 3.13, enabling the use of the iwlwifi driver version 22.24.8.0, which doesn't need any of the changes mentioned and doesn't crash anymore.

### Conclusion

Not being able to compare to other previous Series X Thinkpad models, I cannot say if the X240 is an improvement or not. I've seen a lot of criticism to it, and I don't blame them. Some indicating LEDs would be nice to have, and why does a new computer model in 2014 doesn't have an HDMI port and has the old VGA port instead? There are weird hardware choices in this, but overall I'm enjoying the portability and speed of the computer. I'm hoping that my GNU/Linux compatibility issues (specially the WiFi card issues) get fixed over time.

Do you have some other configuration or fix tips for Debian on the X240? What to you think of this model? Share some ideas with me on the comments.

Categories
GNU/Linux Free Software & Open Source

How to install Google Music Manager on Debian Wheezy

If you are running Debian Wheezy and have trouble installing the Google Music Manager from the official Google Music downloads page, here’s how you can fix it.

The current Google Music Manager depends on libc6 2.15 and Debian Wheezy has libc6 2.13. Changing the libc version is a mayor risk to the stability of your system. So what to do?

Install an older version that can be used with libc6 2.13. You can download the older version for amd64 or i386. Additionally, you’ll need to install libqt4-webkit dependency.

sudo apt-get install libqt4-webkit

Then install the downloaded file

sudo dpkg -i google-musicmanager-beta_current_amd64.deb

Open Google Music Manager and setup your Google account and sync folder and you’re done.

Update: Thanks to Martin for the i386 version.

Categories
GNU/Linux Free Software & Open Source Tutorials & Tips

How to fix Skype on Debian 64bits constant crashing

debian logo skype logo

If you use Debian 64bits and Skype you may run into some trouble installing it and running it, since there is no 64bit version of Skype, setup is a little bit more complicated. The debian wiki explains how to install Skype on Debian and points to some troubleshooting, but not the one I was having.

Once I upgraded to Skype 4.0.2.11 on Debian 7, I started to have lots of crashes, to the point that I couldn’t use the application anymore.

After almost two months of frustrating constant Skype crashes, I finally found the missing piece. The error I was constantly getting was the following:

*** Error in `skype': corrupted double-linked list: 0xe2ee83e8 ***

To fix it, install libpulse0:i386

sudo apt-get install libpulse0:i386

Even though the Debian wiki indicates you should install it if you have audio problems, you must install it to avoid constant crashing.

Categories
Digital rights Law & Freedom GNU/Linux Free Software & Open Source

10 apps for privacy and secure communication

Mobile security

With all the recent news about privacy violations, user data requests, gag orders and the like, it is useful to know that there are tools to communicate in safer ways. I can’t say that they are bullet-proof, as I’m not a security analyst, but at least you can add an extra layer of complexity to those trying to tap into your communications.

  1. Redphone – allows you to have encrypted phone calls
  2. TextSecure – for secure SMS/MMS communication
  3. Gibberbot – for encrtypted chat over Jabber (XMPP) or Google Talk (Hangouts), Facebook Chat, VKontakte, Yandex, Hyves, Odnoklassniki, StudiVZ, Livejournal, and more
  4. OscuraCam – helps you censor out parts of an image like a phone number, license plate, a face…or body part
  5. NoteCipher – stores encrypted notes on your mobile device
  6. GNU Privacy Guard for Android – to encrypt anything, from notes, photos.. any file and emails
  7. K-9 Mail – an open source email client with PGP support for sending and receiving encrypted emails
  8. Orbot – a free proxy app for your mobile device that encrypts your traffic using the TOR network.

    On the browser you can use:

  9. Mailvelope – for encrypting your emails through webmail.
  10. Cryptocat – for private chats within the web browser using OTR encryption

As I mentioned before in my encryption tutorial, having your privacy is not about having something to hide, it’s more about protecting yourself.

In the words of author Ayn Rand:

Civilization is the progress toward a society of privacy. The savage’s whole existence is public, ruled by the laws of his tribe. Civilization is the process of setting man free from men.