Realtime Django Using Node.js and Socket.IO (Posted on January 12th, 2013)

Our goal for today is to build a realtime chatroom using Django, Redis, and Socket.IO. While we'll be building a chatroom the concepts can be applied to almost any web app. At a high level this post will show you how you can convert your REST based app into a realtime web app. I'll be using Django to create the REST based portion but feel free to use any language/framework you're comfortable with. With that said let's jump into the code and get setup with what we need.

The Setup

  • Django 1.4+
  • Redis 2.6.x (somewhat optional, but recommended)
  • Redis-py 2.7.x (only needed if you're using Redis)
  • Node.js v0.8.x
  • Socket.IO v0.9.x
  • Cookie v0.0.5
  • Some sort of database or sqlite if you consider that a database

Your mileage may vary with other versions. I just haven't tested the code with other versions. As of writing these are the latest releases. If you have none of these technologies installed here is a quick guide I've compiled from each package's repo for Ubuntu. You can follow the commented links to learn about other operating systems.

#https://docs.djangoproject.com/en/dev/topics/install/
sudo apt-get install python-pip
sudo pip install django

#http://redis.io/download
sudo apt-get install redis-server

#https://github.com/andymccurdy/redis-py
sudo pip install redis    
    
#https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs

#https://github.com/LearnBoost/socket.io
npm install socket.io

#https://github.com/shtylman/node-cookie
npm install cookie

Django Project

Let's get started with this bad boy!

django-admin.py startproject realtime_tutorial && cd realtime_tutorial
python manage.py startapp core
mkdir nodejs

Now that our file structure is all setup lets update the settings file to include our database information. If you haven't created a blank database for this project yet go ahead and create one now. Here is a copy of my settings file for reference. I've added "core" to my installed apps and also told Django where it can find my templates and login urls. If you have a certain way you like to setup your settings feel free to do so but make sure to add the proper installed apps.

The Model

The models for this project is going to be really simple. We have a comment which contains some text and is associated with a user. If you want to make it more complex you could also add a chatroom variable. To keep things simple we'll just stick with two fields.

from django.db import models
from django.contrib.auth.models import User

class Comments(models.Model):
    user = models.ForeignKey(User)
    text = models.CharField(max_length=255)

Since this is the only model we will be using it is safe to run a syncdb and create the tables for our app. Feel free to create a few users on this step to do some testing with later on.

python manage.py syncdb
python manage.py createsuperuser

Node Server With Socket.IO

This is the part where our realtime message sending and receiving will occur. We'll use Node.js to create an app server but will then rely on Socket.IO and Redis to do the grunt work. In the nodejs directory create a file called "chat.js" and place this in there:

var http = require('http');
var server = http.createServer().listen(4000);
var io = require('socket.io').listen(server);
var cookie_reader = require('cookie');
var querystring = require('querystring');

var redis = require('socket.io/node_modules/redis');
var sub = redis.createClient();

//Subscribe to the Redis chat channel
sub.subscribe('chat');

//Configure socket.io to store cookie set by Django
io.configure(function(){
    io.set('authorization', function(data, accept){
        if(data.headers.cookie){
            data.cookie = cookie_reader.parse(data.headers.cookie);
            return accept(null, true);
        }
        return accept('error', false);
    });
    io.set('log level', 1);
});

io.sockets.on('connection', function (socket) {
    
    //Grab message from Redis and send to client
    sub.on('message', function(channel, message){
        socket.send(message);
    });
    
    //Client is sending message through socket.io
    socket.on('send_message', function (message) {
        values = querystring.stringify({
            comment: message,
            sessionid: socket.handshake.cookie['sessionid'],
        });
        
        var options = {
            host: 'localhost',
            port: 3000,
            path: '/node_api',
            method: 'POST',
            headers: {
                'Content-Type': 'application/x-www-form-urlencoded',
                'Content-Length': values.length
            }
        };
        
        //Send message to Django server
        var req = http.get(options, function(res){
            res.setEncoding('utf8');
            
            //Print out error message
            res.on('data', function(message){
                if(message != 'Everything worked :)'){
                    console.log('Message: ' + message);
                }
            });
        });
        
        req.write(values);
        req.end();
    });
});

Up top we do our imports and create an http server to listen on localhost port 4000. We then subscribe to the Redis "chat" channel. We could easily call this "rabblerabble" as long as we do the same on the publish end in our Django view.

Next we setup Socket.IO to be able to use the cookie that Django sets for the localhost domain. This enables us to access the cookie data via socket.handshake.cookie['the_key_we_want']. This is how we will get the user's sessionid.

After we setup the cookies with Socket.IO we can then handle some events. The first event is for our Redis pubsub channel. When our subscriber notices a new message has been posted it will send the message to all clients on the site.

The other event is when the client sends a message through Socket.IO. We use the querystring module to create a query that can be sent to our Django server. Our Django server will be running on localhost port 3000 but you can change that as needed. The path is set to /node_api which is a URL we will create on the Django side later on. Once we send the querystring we wait for Django to save the comment and send us back "Everything worked :)". If we don't get that back then we output the error to the Node console.

A note about not using Redis

You don't really need to use Redis for this project at all. I found it to be a good learning experience. If you want to bypass Redis you can create a route, using Express or some other library, in the above code that receives a message from Django when a comment has been saved. Then you can broadcast the comment to all clients via Socket.IO.

The Template

This is where all our HTML and client side javascript will be placed. This will allow us to display comments and interact with our Node server.

<!DOCTYPE html>
<html>
<head>
  <title>Realtime Django</title>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <script src="//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js" type="text/javascript"></script>
  <script src="http://localhost:4000/socket.io/socket.io.js"></script>
  <script>
    $(document).ready(function(){
      var socket = io.connect('localhost', {port: 4000});
      
      socket.on('connect', function(){
        console.log("connect");
      });
      
      var entry_el = $('#comment');
               
      socket.on('message', function(message) {
        //Escape HTML characters
        var data = message.replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;");
        
        //Append message to the bottom of the list
        $('#comments').append('<li>' + data + '</li>');
        window.scrollBy(0, 10000000000);
        entry_el.focus();
      });
                     
      entry_el.keypress(function(event){
        //When enter is pressed send input value to node server
        if(event.keyCode != 13) return;
        var msg = entry_el.attr('value');
        if(msg){
           socket.emit('send_message', msg, function(data){
                console.log(data);
           });
        
        //Clear input value   
        entry_el.attr('value', '');
       }
      });
    });
  </script>
</head>
<body>
    <ul id="comments">
        {% for comment in comments %}
            <li>{{comment.user}}: {{comment.text}}</li>
        {% endfor %}
    </ul>
    <input type="text" id="comment" name="comment" />
</body>
</html>

Up top we're connecting to our node server with socket.io on localhost port 4000. When we get a message from the server we do some escaping on the content and then append it to our comments list. When we want to send a message we check for a keypress of 13 (enter key) on our input box. Once that is pressed we emit the message to the server to be handled. Once it is saved to our database by Django we'll get a "message" event which will append it to our chat list.

Our Django view that we create in the next step will just be loading a "comments" variable. So we set that up and loop through them all at the bottom. This part is only used when the page is first loaded. Our javascript will append data to this list as new data comes in from our Node server.

The View

Go ahead and open up realtime_tutorial/core/views.py and edit it to look like mine:

from core.models import Comments, User

from django.shortcuts import render
from django.http import HttpResponse, HttpResponseServerError
from django.views.decorators.csrf import csrf_exempt
from django.contrib.sessions.models import Session
from django.contrib.auth.decorators import login_required

import redis

@login_required
def home(request):
    comments = Comments.objects.select_related().all()[0:100]
    return render(request, 'index.html', locals())

@csrf_exempt
def node_api(request):
    try:
        #Get User from sessionid
        session = Session.objects.get(session_key=request.POST.get('sessionid'))
        user_id = session.get_decoded().get('_auth_user_id')
        user = User.objects.get(id=user_id)

        #Create comment
        Comments.objects.create(user=user, text=request.POST.get('comment'))
        
        #Once comment has been created post it to the chat channel
        r = redis.StrictRedis(host='localhost', port=6379, db=0)
        r.publish('chat', user.username + ': ' + request.POST.get('comment'))
        
        return HttpResponse("Everything worked :)")
    except Exception, e:
        return HttpResponseServerError(str(e))

Let's breakdown what's going on here. Our "home" view is pretty standard. I'm using select_related to also grab the username for each comment rather than doing a query for each comment individually when the page is first loaded.

The second view is what our Node app is sending data to. We grab the sessionid from the POST data and decode it to grab the user id associated with it. Once we have the user and can verify that they exist we can create the comment. Now we send the username and comment to our Redis server. Since our pubsub channel name is "chat", we send our data to that channel.

The URLs

The URLs are pretty straight forward. For logging in and out we'll use the default Django views. We'll also use the default admin panel login template.

from django.conf.urls import patterns, include, url

urlpatterns = patterns('',
    url(r'^$', 'core.views.home', name='home'),
    url(r'^node_api$', 'core.views.node_api', name='node_api'),
    url(r'^login/$', 'django.contrib.auth.views.login', {'template_name': 'admin/login.html'}, name='login'),
    url(r'^logout/$', 'django.contrib.auth.views.logout', {'next_page': '/'}, name='logout'),
)

Start It Up!

That should be all we need to get this working. Let's start both of the servers.

python manage.py runserver localhost:3000

#In a new terminal tab cd into the nodejs directory we created earlier
node chat.js

I've posted the source code to github incase you want to check it out and play around with the code. If you're looking for a quick challenge go ahead and modify the code to allow users to create/join chatrooms. You could also drop the Django portion and implement a different backend such as PHP or Rails.

As always if you have any feedback or questions feel free to drop them in the comments below or contact me privately on my contact page.

Tags: Django, Redis, Node.js