Realtime Django Using Node.js and Socket.IO (Posted on January 12th, 2013)
Our goal for today is to build a realtime chatroom using Django, Redis, and Socket.IO. While we'll be building a chatroom the concepts can be applied to almost any web app. At a high level this post will show you how you can convert your REST based app into a realtime web app. I'll be using Django to create the REST based portion but feel free to use any language/framework you're comfortable with. With that said let's jump into the code and get setup with what we need.
The Setup
- Django 1.4+
- Redis 2.6.x (somewhat optional, but recommended)
- Redis-py 2.7.x (only needed if you're using Redis)
- Node.js v0.8.x
- Socket.IO v0.9.x
- Cookie v0.0.5
- Some sort of database or sqlite if you consider that a database
Your mileage may vary with other versions. I just haven't tested the code with other versions. As of writing these are the latest releases. If you have none of these technologies installed here is a quick guide I've compiled from each package's repo for Ubuntu. You can follow the commented links to learn about other operating systems.
#https://docs.djangoproject.com/en/dev/topics/install/
sudo apt-get install python-pip
sudo pip install django
#http://redis.io/download
sudo apt-get install redis-server
#https://github.com/andymccurdy/redis-py
sudo pip install redis
#https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs
#https://github.com/LearnBoost/socket.io
npm install socket.io
#https://github.com/shtylman/node-cookie
npm install cookie
Django Project
Let's get started with this bad boy!
django-admin.py startproject realtime_tutorial && cd realtime_tutorial python manage.py startapp core mkdir nodejs
Now that our file structure is all setup lets update the settings file to include our database information. If you haven't created a blank database for this project yet go ahead and create one now. Here is a copy of my settings file for reference. I've added "core" to my installed apps and also told Django where it can find my templates and login urls. If you have a certain way you like to setup your settings feel free to do so but make sure to add the proper installed apps.
The Model
The models for this project is going to be really simple. We have a comment which contains some text and is associated with a user. If you want to make it more complex you could also add a chatroom variable. To keep things simple we'll just stick with two fields.
from django.db import models
from django.contrib.auth.models import User
class Comments(models.Model):
user = models.ForeignKey(User)
text = models.CharField(max_length=255)
Since this is the only model we will be using it is safe to run a syncdb and create the tables for our app. Feel free to create a few users on this step to do some testing with later on.
python manage.py syncdb python manage.py createsuperuser
Node Server With Socket.IO
This is the part where our realtime message sending and receiving will occur. We'll use Node.js to create an app server but will then rely on Socket.IO and Redis to do the grunt work. In the nodejs directory create a file called "chat.js" and place this in there:
var http = require('http');
var server = http.createServer().listen(4000);
var io = require('socket.io').listen(server);
var cookie_reader = require('cookie');
var querystring = require('querystring');
var redis = require('socket.io/node_modules/redis');
var sub = redis.createClient();
//Subscribe to the Redis chat channel
sub.subscribe('chat');
//Configure socket.io to store cookie set by Django
io.configure(function(){
io.set('authorization', function(data, accept){
if(data.headers.cookie){
data.cookie = cookie_reader.parse(data.headers.cookie);
return accept(null, true);
}
return accept('error', false);
});
io.set('log level', 1);
});
io.sockets.on('connection', function (socket) {
//Grab message from Redis and send to client
sub.on('message', function(channel, message){
socket.send(message);
});
//Client is sending message through socket.io
socket.on('send_message', function (message) {
values = querystring.stringify({
comment: message,
sessionid: socket.handshake.cookie['sessionid'],
});
var options = {
host: 'localhost',
port: 3000,
path: '/node_api',
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': values.length
}
};
//Send message to Django server
var req = http.get(options, function(res){
res.setEncoding('utf8');
//Print out error message
res.on('data', function(message){
if(message != 'Everything worked :)'){
console.log('Message: ' + message);
}
});
});
req.write(values);
req.end();
});
});
Up top we do our imports and create an http server to listen on localhost port 4000. We then subscribe to the Redis "chat" channel. We could easily call this "rabblerabble" as long as we do the same on the publish end in our Django view.
Next we setup Socket.IO to be able to use the cookie that Django sets for the localhost domain. This enables us to access the cookie data via socket.handshake.cookie['the_key_we_want']. This is how we will get the user's sessionid.
After we setup the cookies with Socket.IO we can then handle some events. The first event is for our Redis pubsub channel. When our subscriber notices a new message has been posted it will send the message to all clients on the site.
The other event is when the client sends a message through Socket.IO. We use the querystring module to create a query that can be sent to our Django server. Our Django server will be running on localhost port 3000 but you can change that as needed. The path is set to /node_api which is a URL we will create on the Django side later on. Once we send the querystring we wait for Django to save the comment and send us back "Everything worked :)". If we don't get that back then we output the error to the Node console.
A note about not using Redis
You don't really need to use Redis for this project at all. I found it to be a good learning experience. If you want to bypass Redis you can create a route, using Express or some other library, in the above code that receives a message from Django when a comment has been saved. Then you can broadcast the comment to all clients via Socket.IO.
The Template
This is where all our HTML and client side javascript will be placed. This will allow us to display comments and interact with our Node server.
<!DOCTYPE html>
<html>
<head>
<title>Realtime Django</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js" type="text/javascript"></script>
<script src="http://localhost:4000/socket.io/socket.io.js"></script>
<script>
$(document).ready(function(){
var socket = io.connect('localhost', {port: 4000});
socket.on('connect', function(){
console.log("connect");
});
var entry_el = $('#comment');
socket.on('message', function(message) {
//Escape HTML characters
var data = message.replace(/&/g,"&").replace(/</g,"<").replace(/>/g,">");
//Append message to the bottom of the list
$('#comments').append('<li>' + data + '</li>');
window.scrollBy(0, 10000000000);
entry_el.focus();
});
entry_el.keypress(function(event){
//When enter is pressed send input value to node server
if(event.keyCode != 13) return;
var msg = entry_el.attr('value');
if(msg){
socket.emit('send_message', msg, function(data){
console.log(data);
});
//Clear input value
entry_el.attr('value', '');
}
});
});
</script>
</head>
<body>
<ul id="comments">
{% for comment in comments %}
<li>{{comment.user}}: {{comment.text}}</li>
{% endfor %}
</ul>
<input type="text" id="comment" name="comment" />
</body>
</html>
Up top we're connecting to our node server with socket.io on localhost port 4000. When we get a message from the server we do some escaping on the content and then append it to our comments list. When we want to send a message we check for a keypress of 13 (enter key) on our input box. Once that is pressed we emit the message to the server to be handled. Once it is saved to our database by Django we'll get a "message" event which will append it to our chat list.
Our Django view that we create in the next step will just be loading a "comments" variable. So we set that up and loop through them all at the bottom. This part is only used when the page is first loaded. Our javascript will append data to this list as new data comes in from our Node server.
The View
Go ahead and open up realtime_tutorial/core/views.py and edit it to look like mine:
from core.models import Comments, User
from django.shortcuts import render
from django.http import HttpResponse, HttpResponseServerError
from django.views.decorators.csrf import csrf_exempt
from django.contrib.sessions.models import Session
from django.contrib.auth.decorators import login_required
import redis
@login_required
def home(request):
comments = Comments.objects.select_related().all()[0:100]
return render(request, 'index.html', locals())
@csrf_exempt
def node_api(request):
try:
#Get User from sessionid
session = Session.objects.get(session_key=request.POST.get('sessionid'))
user_id = session.get_decoded().get('_auth_user_id')
user = User.objects.get(id=user_id)
#Create comment
Comments.objects.create(user=user, text=request.POST.get('comment'))
#Once comment has been created post it to the chat channel
r = redis.StrictRedis(host='localhost', port=6379, db=0)
r.publish('chat', user.username + ': ' + request.POST.get('comment'))
return HttpResponse("Everything worked :)")
except Exception, e:
return HttpResponseServerError(str(e))
Let's breakdown what's going on here. Our "home" view is pretty standard. I'm using select_related to also grab the username for each comment rather than doing a query for each comment individually when the page is first loaded.
The second view is what our Node app is sending data to. We grab the sessionid from the POST data and decode it to grab the user id associated with it. Once we have the user and can verify that they exist we can create the comment. Now we send the username and comment to our Redis server. Since our pubsub channel name is "chat", we send our data to that channel.
The URLs
The URLs are pretty straight forward. For logging in and out we'll use the default Django views. We'll also use the default admin panel login template.
from django.conf.urls import patterns, include, url
urlpatterns = patterns('',
url(r'^$', 'core.views.home', name='home'),
url(r'^node_api$', 'core.views.node_api', name='node_api'),
url(r'^login/$', 'django.contrib.auth.views.login', {'template_name': 'admin/login.html'}, name='login'),
url(r'^logout/$', 'django.contrib.auth.views.logout', {'next_page': '/'}, name='logout'),
)
Start It Up!
That should be all we need to get this working. Let's start both of the servers.
python manage.py runserver localhost:3000 #In a new terminal tab cd into the nodejs directory we created earlier node chat.js
I've posted the source code to github incase you want to check it out and play around with the code. If you're looking for a quick challenge go ahead and modify the code to allow users to create/join chatrooms. You could also drop the Django portion and implement a different backend such as PHP or Rails.
As always if you have any feedback or questions feel free to drop them in the comments below or contact me privately on my contact page.
Comments:
Andreas - 4 months, 1 week ago
I think Node.js isn't the best choice in this regard. What I found to be a better solution is tornado + sock.js. Socket.io is ok, but it is very stringent on the protocol it uses. Tornado can be run from a management command in Django, it can even use the ORM (sparingly, otherwise the thread blocks...). However you need two redis packages, redis-py and tornado-redis, one is blocking, one is not. Don't get me wrong, I like javascript (especially coffeescript). I just don't see a good reason to use it server side. Granted, node with coffeescript is a tad more elegant than tornado, but then there's Dieselweb and Gevent, which use a more sequential coding style than what tornado does with the callbacks or generator syntax.
muhia - 4 months ago
Hi. i usually don't comment a lot, but i just had to do it this once. i was wondering if you are really a human or a bot, coz i swear i saw this exact comment on another site a few days ago. Anyway, i think it is relevant tho, thanks
Line height ftw - 4 months, 1 week ago
please try document.body.style.lineHeight = '1.5em' within your browser console, isn't that better?
Devastator - 4 months, 1 week ago
Hey man, thanx for this, very good job! Since there is a ... api to socket.io, you must use only examples like yours. Bye.
FZambia - 4 months, 1 week ago
Hello! Thanks for the article. Recently i also tried to implement something like this, but when event(chat message) happened - I first saved it in Django - this allows to easily handle different user permissions. And then I send POST request to asynchronous server. So this is a "one-direction" realtime. I've implemented this using Tornado, Node, Twisted as async servers and Socket.IO and Sock.js as client-side libs. Here is repo with all of this: https://github.com/FZambia/django-realtime-playground
Max Burstein - 4 months ago
Just took a look through your projects in that folder. Very cool stuff. It sounds like you're indeed doing something similar to my code except cutting out Redis. Out of curiosity have you run benchmarks to compare the speed/reliability of the different backends?
FZambia - 4 months ago
Yes, similiar, but not only except Redis - new data first go to Django and then to Node or Tornado. Of course I thought about making benchmark, but have no time enough at the moment.
Peter Bengtsson - 4 months ago
I too prefer Python That's why I wrote: https://github.com/peterbe/django-sockjs-tornado
Dami O - 4 months ago
If you are hitting 500 on Django's server, you may want modify chat.js there is a pull request here https://github.com/damilare/django-realtime-tutorial/commit/3496f699ea4c8e501600f4be8c8057ca0ba3878e
Max Burstein - 4 months ago
Thank you for the pull request. I have merged it into my codebase. I think you're getting the 500 error due to something else though. The original code works for me on my dev environment. The code works since we're setting the type as POST which is basically overriding the shortcut .get method in the node http module. Nonetheless, .request is the right method to be calling as you pointed out.
Oz Katz - 3 months, 3 weeks ago
I have actually tried to take this exact approach (and stack) and turn into into something more generic that is reusable (not only for Django but possibly for other frameworks as well). it's called Announce.js - https://github.com/ozkatz/announce.js The django client is here: https://github.com/ozkatz/django-announce/
miguel - 3 months, 3 weeks ago
This implementation reminds me of what Armin Ronacher posted here: http://lucumr.pocoo.org/2012/8/5/stateless-and-proud/ It's similar but not quite how he describes it I think. Either way, thanks a lot, that is a good boilerplate code.
Johniek - 3 months, 2 weeks ago
Hi, first thanks for this example, secondly i use u code and in views.py you getting POST['comment'] - why? In me POST does not sended on view, <QueryDict: {}>... I mean this line: socket.emit('send_message', msg, function(data){})
Max Burstein - 2 months, 2 weeks ago
Once the page is loaded the client gets all the data from the node server. The messages are also sent off to the Django server so that they can be stored. This way when a new client connects they can see the chat history. socket.emit sends the data to all clients except the one that sent the message.
Rohit - 2 months, 2 weeks ago
Hi just downloaded code from github installed dependencies but I am getting an error events.js:71 throw arguments[1]; // Unhandled 'error' event ^ Error: connect ECONNREFUSED at errnoException (net.js:770:11) at Object.afterConnect [as oncomplete] (net.js:761:19) When i am trying to enter some value in the text box please help me out .
Max Burstein - 2 months, 2 weeks ago
Is the request ever getting to the Django server? Also what version of Django/Node are you using?
Rohit - 2 months, 2 weeks ago
I am using django.VERSION (1, 4, 0, 'final', 0) And my node version is v0.8.18 Yes when i am log in i am getting an input box ,But whenever I i am hitting enter button after entering some value in input box i am getting this error.
Max Burstein - 2 months, 1 week ago
Alright shoot me an e-mail through my contact form and I'll be able to help you out better. I feel like the issue is coming from the request not getting to the django server. In the console for the Django server, is the request to /node_api reporting a 200 status code?
n3rV3 - 2 months, 1 week ago
Just tried it out, this is really awesome.. but would it be possible to have a similar setup only of Python based components? Please suggest if there are any good alternatives to node.js in Python.
Max Burstein - 2 months, 1 week ago
You could utilize https://github.com/abourget/gevent-socketio or one of the other libraries that port socket.io over. I've found that the gevent one is relatively well supported.
dougvk - 1 month, 3 weeks ago
Max - thanks for writing this up. I was wracking my brain for how to use django's authentication kickassedness but ditch the request/response. Are there security vulnerabilities in ditching the CSRF token?
Max Burstein - 1 month, 2 weeks ago
You should be able to include the Django CSRF value from the cookie that it sets. When I tried it, it wasn't working so I ditched the CSRF token. One thing that CSRF tokens prevent is people from making phishing sites and then sending the post data to your server. You can read more about them here http://en.wikipedia.org/wiki/Cross-site_request_forgery
Stacy - 1 month, 2 weeks ago
Hi, I'm getting an ImportError: No module named debug_toolbar.... Why is that?
Max Burstein - 1 month, 2 weeks ago
You can install the debug toolbar which is super useful for debugging Django apps or just remove the app from the middleware and installed apps file in your settings.py. Here is where you can get it from https://github.com/django-debug-toolbar/django-debug-toolbar
Venkat - 1 month ago
Why do you get the user-id from session and not from request.user?
Sara - 1 month ago
Why do you use debug_toolbar?
Max Burstein - 3 weeks, 6 days ago
It's a super useful tool that I include in all of my projects by default. It allows me to see slow running queries and such for making optimizations. Definitely check it out if you've never used it.
Jenna - 1 month ago
I'm getting this error when I try your code: "DatabaseError: no such table: django_site"
Jenna - 1 month ago
This is at the /login/ portion. DatabaseError at /login/ no such table: django_site Request Method: GET Request URL: http://localhost:3000/login/?next=/ Django Version: 1.5.1 Exception Type: DatabaseError Exception Value: no such table: django_site Exception Location: /usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py in execute, line 362 Python Executable: /usr/bin/python Python Version: 2.7.3 Python Path: ['/home/euridice/realtime_tutorial', '/usr/local/lib/python2.7/dist-packages/pip-1.3.1-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol'] Server time: Wed, 24 Apr 2013 17:12:34 -0400
Max Burstein - 3 weeks, 6 days ago
I think you need to setup the database information in your settings.py then run "python manage.py syncdb". That should create the table for you.
Sara - 3 weeks, 3 days ago
"If you're looking for a quick challenge go ahead and modify the code to allow users to create/join chatrooms." Could you provide a quick tutorial on how to do this? Would I just be using Socket.io's documentation for implementation?
Max Burstein - 3 weeks, 2 days ago
This would be a good place to get started with https://github.com/LearnBoost/socket.io/wiki/Rooms You'll also want to add an additional column in your models for storing the data. This way you can keep a history of each channel. Then you'll also want to add the chatroom name/id to the data sent over redis or just give each chatroom it's own redis pubsub channel. If you some code that's not working or something feel free to shoot me an e-mail through my contact form and I'd be happy to take a look at it.
trojjer - 16 hours ago
That was interesting. You could've used a FormView with a custom queryset for node_api. I love Class Based Views in Django, having been introduced to them with Django itself at work and having no experience of working with the functional views. With a FormView you could just specify a ModelForm for the Chat model. You could then pass in the User ref to the form in a custom get_form_kwargs() method, and validate it separately in the ModelForm's clean_data(). You could also declare a session_id field explicitly and have the ModelForm put it in cleaned_data['session_id']. I'm going on a bit, but I saw that you left the code open to a basic SQL injection vulnerability by not sanitising user input POST data :P
trojjer - 16 hours ago
A followup for anyone puzzled by CBVs in Django. I still find myself struggling with the Django docs sometimes; every Django dev using CBVs should know about this awesome third party API listing: http://ccbv.co.uk/projects/Django/1.4/
About Me
My name is Max Burstein and I am currently a senior at the University of Central Florida majoring in Information Technology. I enjoy developing large, scalable web applications and I seek to change the world.