Tim Child

As part of my on going effort for realtime web updates on one of our applications I needed to install RabbitMQ HEAD (That is the development head) with the RabbitMQ STOMP adaptor. Talking on IRC channel it was recommended to me to install from source, but unfortunately the documentation is out of date. So the following will do it cleanly:

Updated: The following will create a Debian style package and install (replace ** with the version numbering that you wish to use:

Activating the STOMP plugin:

Configuration file:

Restart to activate (the activation script for plugins is depreciated):

And you should have a working RabbitMQ on Ubuntu with a STOMP adaptor.

Today I got my Django project working with Orbited.

By creating a twisted proxy for Orbited and Django I am able to serve a development environment for both Django and Orbited plus the builtin MorbidQ message queue.

I am going to be using this to give live notifications of events that are happening on the system in realtime, and so it should be able to handle quite a large load. To do this I will use another queue in production, but I have found it needed to sort out the development environment first of course.

This daemon I have put up at GitHub .

It even serves the static files. Much thanks goes to the HotDot project for seeing how alot of this was done, but I have taken out the authentication and other bits to make it cleaner for general use.

If you are having problems with _PQbackendPID, psycopg2 and OS X and have tried the forcing to 32 bit mentioned in other posts about the web then it might be time to try a more forceful approach. This is an example of the error that you might face:

The first thing to do is get rid of your old Psycopg2 (just in case), so locate the site-packages directory:

Then in that directory:

Then remove the old version of PostGreSQL as much as possible. Not sure what version you might have installed, but if you have pg_config on your sys path it will tell you where it is. These instructions may or may not work for you:

1. Stop the server. sudo launchctl unload /Library/LaunchDaemons/com.edb.launchd.postgresql-8.4.plist
2. Remove the plist file
   sudo rm -f /Library/LaunchDaemons/com.edb.launchd.postgresql-8.4.plist
3. Remove the Applications Menu.
   sudo rm -f “/Applications/PostgreSQL 8.4″
4. Remove the installation directory
   sudo rm -rf /Library/PostgreSQL/8.4
   (Note: You can backup your data directory in case you need it.)
5. Remove the ‘postgres’ user
   sudo dscl . delete /Users/postgres
   (Note: This step is optional)
6. Remove the ini file
   sudo rm -f /etc/postgres-reg.ini

I found a reboot worked to make sure that we are back to normal and nothing is installed.

Download psycopg2-2.2.1.tar.gz
Download PostGreSQL from http://www.enterprisedb.com/products/download.do – the version that isn’t the “Standard Server” but just “PostgreSQL 8.4 ” – you do have to register for that.

Install Postgresql – making sure that it completes, if not it means that you probably had some old files or users in there. If it went ok it should have installed and started running on port 5432. Try removing it /Library/PostgreSQL/8.4/uninstall-postgresql.app – and then removing the user or files that it complained about.

Download and install http://www.python.org/download/releases/2.6.5/ Mac Installer disk image. Restart (not sure that this restart is totally needed, but I did it anyway).

Now when you are on a new terminal window and type “python” it should say version 2.6.5. You will have to reinstall any python bits and pieces that you have installed previously – but make sure that you reinstall PIP and easy_install – they will use the old version of Python if you are not careful.

Now go into the directory of psycopg2-2.2.1 that you downloaded earlier.

Change setup.cfg so it looks like the following:

Now run the installation using the new python:

This should install. You should be able to do “import psycopg2″ from the python prompt. Now install all the libraries and python modules that you need, and create the Postgresql database that you want (pgadmin3 is the best tool for this).

Please note this was done on a OS X Snow Leopard 10.6.4 with XCode and the 10.4 libraries installed.

My latest task and one that I have been researching and playing with for some time is to integrate real time feedback in to our web application. The main goal is to be able to provide real time feedback of jobs and updates to users without Ajax style polling.

This is a simple requirement to write, but one that had many deeper issues. Firstly it needs to be able to handle many users concurrently and secondly it should do so without blocking execution of code. Our application framework was not designed for this and this is no problem, it has been designed to do certain thing well and other I am really not going to force upon it.

The main application framework will handle the standard page requests, user authentication and the rest of the standard stack, so I have been looking at technologies to complement its capabilities. This has led me to look at Comet applications and very fast non-block frameworks.

Of these I would prefer to keep the language in Python, there seem to be good alternatives in Java and .Net but I would rather keep an upgrade path by having a language that can be migrated across to these (thanks Jython and IronPython) and keep data in a backend store that is language agnostic.

Orbited and Tornado are really good candidates. I am going to stay away from more raw implementations of non-blocking event based frameworks such as Twisted because efforts are often better at this stage in building to serve a business purpose where possible. Orbited does actually use twisted, and I am hoping that I will have little need to dig that deep except for understanding its implementation.

Tornado I initially didn’t think of, but came around after reading “Building the Realtime User Experience” in which the author, Ted Roden, seems rather partial to it. In fact it can work quite well in fronting our application, but I am starting to reconsider based upon two factors. Firstly, even though it itself was built for a largish scale operation (FriendFeed – now owned by FaceBook), it has little other commercial users behind it and I haven’t found many examples of usage with integrating with Django, Pylons, Turbogears or Zope in a production environment. Secondly what really turns me off is the immaturity of its tests. Looking on GitHub even the README shows that little effort has been put into its testing environment, let alone their sporadic coverage.

Twisted on the other hand is extremely mature in both terms of numbers of deployments and its test environment, and this makes Orbited a much better candidate.

I have yet to look at certain other alternatives, such as Diesel, but I have a feeling that I will again come around to start using Orbited.

The other huge plus for me in using Orbited is that it can use the STOMP protocol, and use RabbitMQ or ActiveMQ. Other parts of our technology stack are starting to use STOMP and I wish to move more operations on to an event queue, which will open up much in terms of flexibility in the future.

Its a pity that “Building the Realtime User Experience” was so limited in terms of covering alternatives to Tornado.

Wake-on-lan is incredibly useful for those scenarios where you have more than one machine on a network, but you don’t always want to have them running but they are needed sometimes.

To setup, make sure that the Motherboard and NIC support is there from your manufacturer. Sometimes it has to be enabled, such as making sure that it uses S3 for shutdown, and that PCI wake is supported.

Next you need to check support from the operating system, I am using Ubuntu 10.04, but this should work for other operating systems. Install ethtool, and then check against the ethernet adapter (eth0 in my example).

If support is enabled, but it is turned off you will see:

So we turn it on:

Now we need to make sure that its always enabled everytime we restart:

And in that file:

Now make sure that its put into the correct runlevels:

And that is it. It should be supported that you can wake the machine from another computer/server using your favourite wakeup utility ( such as wakeonlan ).

I have been using the Yahoo YUI Javascript library for more than a couple of years now (perhaps about 4) and although I really like it, I have never felt particularly productive using it.

Over those years I have also dabbled in other frameworks, including:

• ExtJS (mainly from my old work looking at theming it – and its heavily based upon YUI.) – I dislike the licensing on ExtJS so it really isn’t an option for me any more
• Looked strongly at Cappucino, but there was no real productivity gains to be had, and I wasn’t writing my current application as a solely Javascript app
• MooTools – Seems nice but never really gelled with it and I haven’t liked the documentation
• SproutCore – Love it so far, but again I am not writing a pure Javascript App.
• JQuery – have implemented some solutions over the last 6 months, and loved the productivity

Anyway, what I am currently developing really needed some strong enhancements, such as AJAX calls, better table views, nice widgets for date and time picking – the usual enhancements and I just felt that YUI was taking too long. My wishlist was:

1. A Very productive library
2. Good support from a community of active users
3. Good documentation.
4. Doesn’t impose made CSS changes
5. Speed of implementation and download

YUI 3 was a consideration, after all the Javascript team at Yahoo are among the best in the world, but it was too new and so didn’t have the community I was looking for.

I ended up with JQuery. After spending a day or so getting to grips and training myself properly with it, sure enough my productivity using it has reached heights to which I never got near with YUI (2.7 was the latest version I was using).

Another thing that I noticed was that I am actually writing a lot more productive code, rather than implementing bits of script that I change from all over the web. All the functions I need seem to be in the core library.

So right now, I am making many progressive enhancements to our App and it feels all the better for it.

It was just announced that Django 1.2 will be released on May 17th, which is in 3 days time.

Been waiting a while for this one to go to production release so I am really glad. The new messages framework looks much better, and I also love the date i18n features – something that really helps on a project at Cantemo where we have to deal with ISO formatted datetimes.

Now the users will be able to choose how their dates are displayed on individual systems.

I always have this problem when moving VMWare images of Ubuntu, the most recent of a 10.04 LTS Server. Whenever you move or copy a VMWare image it assigns a new Mac address and Ubuntu starts without properly bringing up the interface.

Really annoying when you are using VMWare server without a console.

To make it grab the IP Address again, simply remove the rule:

Then restart.

Of course before you move the image you could remove the rule and not have them problem.

Really interesting slide show by Drew Houston on what they have learnt about starting up Dropbox.

http://www.slideshare.net/gueste94e4c/dropbox-startup-lessons-learned-3836587

There is a lot of interesting information about creating plugin architectures using Python all over the web, but its in fairly disparate places. This is an overview of the documentation that I found as of April 2010.

Firstly Dr André Roberge has some very interesting posts, as well as a talk at PyCon 2009 on Blip.tv entitled Plugins and monkeypatching: increasing flexibility, dealing with inflexibility. He is also the author of Crunchy which uses a plugin system. He goes on to write about his experiences in creating a plugin architecture on his blog, over 6 parts starting here on part 1.

Marty Alchin, the author of Pro Django has an interesting couple of pages in that same book about creating a simple plugin system, and the snippet of code is on djangosnippets.com. He also has an excellent blog post about implementing a plugin architecture, it also talks about Django but is fairly non-specific.

William E. Hart over 2009-2010 has also been researching Python Plugin Frameworks, his overview is on his blog and he is also the author of the PyUtilib Component Architecture. This was very recently released and so hasn’t gathered much momentum as of this post but looks very promising indeed.

Zope. The big one is the Zope Component Architecture. I must admit, that looking into it I think its overkill, I don’t like the configuration utility or the overhead it introduces – just to use it requires a lot of extra Zope modules. However google for it if you are interested in learning further.

Yapsy. Yapsy is interesting, looks extremely lightweight, has been around for a few years, is currently on version 1.7 and serves as a good starting point. It has no external dependancies.

SprinklesPy. SprinklesPy is also very lightweight but I am not sure that development is active on it (2006 was the last mention of it being used that I found), or that it has anyone in the community using it (a benefit I believe if you are implementing a system that you are not creating yourself.)

Trac is the one component architecture that frequently gets mentioned, and it has a very decent user base. However, ripping the plugin system out might be more work than is reasonable.

Examples of Plugin use in the wild:

• http://code.google.com/p/oohembed

Other interesting links:

• http://www.charlesleifer.com/blog/looking-registration-patterns-django/
• Stackoverflow thread on minimal plugin architectures
• http://live.gnome.org/Gedit/PythonPluginHowTo

It is clear to me that there is no correct way of implementing plugins and you have to pick based upon the needs of your project, but this should serve as a good starting point. I will add and update this post as my knowledge grows.