1. Larger desktop

    It seems that no matter how many pixels I have to play with I could always use more. Consider this number of apps that I have open a day:

    Browser window with debugging pane (1200 x 1200)
    Skype (220×1000 high) with conversation window (400×600)
    Mail (1000×800)
    Text mate or vim (typically 1600×1200)
    Terminal windows – at least two. One for Git and one for python shell. (1600×1200)
    2nd browser window for looking stuff up though I can use tabs for them.

    Could also do with having an irc client, finder and adium but I usually close them.

    It all equals too many pixels

    By timc3 on the
    September 2nd, 2011
  2. PostgreSQL on Lion

    The installer for Postgresql on Lion is supposedly “unsupported”, which means that they haven’t got it working yet. Looking at it quickly is seems to be the way that the postgres user is being created using dscl.

    To get it to work, manually create a postgres user from under System Preferences:

    Then run the installer as per normal. I used the Postgres 9 installer.

    I also upgraded python to 2.7.2 and installed psycopg2 pointing to pg_config under /Library/PostgreSQL/9.0/bin/pg_config.

    By timc3 on the
    August 27th, 2011
  3. What’s the coolness

    Looking back over the last posts I realize that I haven’t been talking much about the software and and code that I have been using every day and some of it is really quite interesting.

    Python. I use python a lot, it’s my preferred go to language to solve a lot of problems. It has great libraries, an awesome community and the language itself is compact and non verbose. We are using Django heavily in our solutions and its really enjoyable to work with, and I never feel that coding is a chore. Coupled with the fact that we can target Java servers with Jython, .Net with IronPython and have an amazing solution in PyPy for python running on Python I can see the love affair continue.

    Eventlet, Gevent and Meinheld. Python networking libraries (actually Meinheld is a WSGI server) that work in an event’d manner, all are really easy to work with, all are really fast. With these libraries its really easy to build highly performant services that scale well on standard architecture. For instance to write a websocket / comet server, I wouldn’t hesitate to use Eventlet or Gevent. Tornado looks like another nice Python alternative but I have never been happy with its lib curl dependancy.

    Celery. Asynchronous task queue/job queue written in Python for python that can use message servers such as RabbitMQ. Really easy to build strong reliable services spanning multiple machines.

    Hudson. Continuous integration server that is just seamless, I use it for building packages (python and debian), running test frameworks and automatically building documentation which gets uploaded to Dropbox. Can’t recommend it strongly enough.

    Linux KVM and Proxmox. Virtualisation has obviously been a part of most deployment and development cycles for some time now , but innovation is still being made. Having used VMWare, VMWare ESX, VirtualBox, Microsoft’s Hyper-V and others I have found what I believe to be the sweet point in Linux KVM – simple extensions to the Linux kernel which allows for running virtual machines on top of a normal linux platform. I have moved my personal servers, and our work servers to KVM as much as possible and an easy integrated way of doing this is to run ProxMox.

    Memcached. Memory caching server that I use with my goto web services stack – Nginx – Supervisord – Meinheld – Django – Memcached – Postgres. Simply just works.

    Selenium. Automated browser testing, this is a product that no matter how much I use it, I feel that I can use it more than I do. Have been using the RC version to do browser testing against Firefox and IE on Windows.

    PostgreSQL. NoSQL is all very well and good, but for most serious projects I tend to turn to Postgres for its fully featured SQL compliance, its rock solid hosting of data and its well proven ability to scale.

    RabbitMQ. RabbitMQ is a message queue that I tend to deploy with Python Celery to help perform offline processing of tasks.

    Supervisord. Currently used for daemonising a setup.

    Git. Distributed version control. I have tried most of the others, and before that been a long time user of SVN but Git installs on everything, has awesome support in GitHub, and I am really enjoying the extending merging and branching.

    Redmine. For task tracking and bug fixing, this is the best tool that I have used. Migrated everything from trac to it with no problems, and have also migrated from Redmine backed by MySQL to PostgreSQL with no problems.

    Glassfish. My goto Java Application Server. I hope to have more exposure to JBoss in the coming few months, but Glassfish has been good so far.

    nginx. Can’t say enough good things about Nginx. Its fast, easy to configure, flexible, lightweight and hosts all our software at Cantemo with little problem.

    HAProxy. When I need to load balance nginx or other services (including straight TCP/IP which nginx can’t load balance) I turn to HAProxy.

    backbone.js. Almost all the Javascript that I am now creating is built upon underscore.js and backbone.js. There are many tutorials but this deserves another space all of its own.

    In fact there are many things that I have left out that I am using everyday but these are some of the technologies, software and libraries that have been making my life easier and the systems that I have built more interesting for the past 18 months or so. I really ought to do a write up on each of them.

    By timc3 on the
    August 14th, 2011
  4. HTML5 audio support

    As a followup to my last post, and also because Portal needs to support audio as well as video, I have implemented the same kind of detection and serving of files for audio. We transcode audio files into MP3 slightly low bitrate (depends on the clients wishes, but its not a great codec for streaming lots of audio over a network), and we can also do an OGG version. Here is the current support list:

    HTML5 audio format support
    Browser MP3 Ogg WAV Notes
    Safari 5.0+ - 5.0+
    Firefox Use flash 3.6+ 3.6+
    Chrome 6.0+ 6.0+ -
    Opera Use flash 10.5+ 10.5+
    IE 9.0+ - 9.0+* IE8 and before only support using flash fallback

    Of course this doesn’t actually talk about the problems, such as MP3 always having silence at the beginning and the end of the file, and that Chrome’s support has been surprisingly bad over time. Its a pity that there are so many inconstancies and the specification doesn’t really seem to have been thought out (where is the buffering support?)

    There is an even more in-depth view here:

    http://www.phoboslab.org/log/2011/03/the-state-of-html5-audio

    By timc3 on the
    August 9th, 2011
  5. HTML5 video support

    At Cantemo for our Portal product we are working a lot with HTML5 video, its starting to get really nicely supported by all the major vendors and we have some nice technology to support frame accurate playback, frameforward and frameback on a lot of browsers (Chrome, Firefox, Safari etc).

    This puts us in a much better position than we have had in the past with h.264 playing back through the Flash player (though this is a fallback mechanism still).

    Trouble is we have to implement support for different codecs, for the different browsers. Our software supports this out of the box, but it means that we have to do some interesting detection of browsers, formats that we have, and offer a plugin solution. Today we can transcode to OGG, WebM and h.264 with no problems, and then match this to browsers:

    Here is a table of what browsers support which video formats:

    HTML5 video format support
    Browser Ogg h.264 (mp4 extension) WebM Notes
    Safari * 3.0+ * *Safari will play what Quicktime supports
    Firefox 3.5+ Use flash. 4.0+
    Chrome 5.0+ 5.0+ 6.0+ h.264 will drop soon, still works as of 12
    Opera 10.5+ Use flash 10.6+
    IE - 9.0+ 9.0+* *Only works with optional plugin

    This table assumes that h.264 is in an MP4 wrapper, with an AAC codec for sound. OGG is encoded using Theora, with a Vorbis audio codec.

    By timc3 on the
    August 6th, 2011
  6. Quick update

    Haven’t had much time recently to post anything on here, though lots has been going on that I could write about.

    Been to Cyprus for a family wedding which was really nice, pity that we had some stuff stolen including my iPhone, Nikon DSLR and video camera but otherwise it was great.

    At work we are building new HTML5 based tools, improving the look and feel and capabilities of our main product – should really write up about some of the technology being used. To quickly list – backbone.js (which I love), gunicorn, Eventlet, some more nginx tricks, been building .deb installers, Python configuration tools, Johnny Cache for Django, Celery delayed processing and some more bits and pieces.

    By timc3 on the
    June 14th, 2011
  7. Resizing ext3 or ext4 partition for virtual image

    Note: please backup and make sure that you have a copy of your data and virtual images, this can go wrong and I won’t be held responsible. This worked for me, and might not work for you. For ext4 please use the latest GParted.

    As I migrated from raw or qcow images, some of the partition sizes that I have to work with are not as big as they could be. So I need to resize.

    I generally give some of my hosts a 24Gb LVM virtual group each. But the image files can be as small as 2Gb – A lot of wasted space which the filesystem on the image might as well have. These instructions assume that there are only two partitions, a main partition, and a second linux swap partition (either on an extended partition or a primary).

    First taken the image offline, then get the name of image that you wish to resize, then the disk storage items:

    [cc lang="bash"]
    ls /etc/vm/
    lvdisplay
    [/cc]

    In my case it was “/dev/vdisk05/vm-102-disk-1″. Then you can check to see how large the volume group is set to (vgdisplay) resize:

    [cc lang="bash"]
    lvresize -L 23G /dev/vdisk05/vm-102-disk-1
    [/cc]

    Next I mount a SystemRescueCD on the CDRom, and make sure its the first item in the boot sequence. Then boot into the system, go for the first choice in SystemRescueCD, and it will give you a rootprompt (I am VNC’ing into the system so that I can access the console as if using a physical machine):
    [cc lang="bash"]
    #Change the passwd so that you can ssh in if you need to.
    passwd
    #Start the GUI so you can run Gparted (First choice again).
    wizard
    [/cc]

    From a command prompt do (check file system, repair, remove journal):

    1. fsck -n /dev/sda1
    2. e2fsck -f /dev/sda1
    3. tune2fs -O ^has_journal /dev/sda1

    Then do the following in the GUI:

    1. Choose CD > System > GParted
    2. Delete the swap partition as it will be the last
    3. Resize partition, leaving some space for a new swap partition
    4. Create a swap (type linux-swap) partition twice the size of the amount of RAM you allocate to the image
    5. Click Edit > Apply all operations
    6. Close GParted

    Back on the command prompt do (repair, check file system, journal, shutdown):

    1. e2fsck -f /dev/sda1
    2. fsck -n /dev/sda1
    3. tune2fs -j /dev/sda1
    4. shutdown -h now

    Remove the CDRom device, and change the boot order back then start the Virtual machine. Issue a df -h to see if the partition has changed. It should have done!

    By timc3 on the
    May 2nd, 2011
  8. Virtual host QCow convert to LVM

    Sometimes its better to have your virtual images running directly of LVM instead of using raw or QCow formatted files. So how do we convert? Basically you have to make sure that you have a volume group, of the right size or bigger and then dd the file into a LV on that volume group:

    This should in the directory where you have the image:

    [cc lang="bash"]
    # Get the filesize of the raw image in Kilobytes.
    ls -lk
    #Create lv on vdisk05 (the VG)
    lvcreate -L 5242880K -n vm-102-disk-1 vdisk05
    #Convert from QCow to Raw
    qemu-img convert bld-ubuntu-1004-2.qcow2 -O raw bld-ubuntu-1004-2.raw
    #Copy the image into the lv:
    dd if=bld-ubuntu-1004-2.raw of=/dev/vdisk05/vm-102-disk-1
    [/cc]

    Then change the configuration of your QEMU or whatever you are using on your virtualisation platform.

    Sorry for the briefness of the instructions, its mainly for my own benefit and if you are doing this you should really know about LVM, dd and ls.

    By timc3 on the
    May 1st, 2011
  9. Damn nice bag

    They people have some damn nice bags:

    http://shop.hardgraft.com/bags/2unfold-laptop-bag-brown

    By timc3 on the
    April 27th, 2011
  10. Webcasts to watch

    Recently I was unfortunately quite ill and away from work for an extended time, the only upside being that I had some time to spend learning some new things and watching web conferences on my Apple TV2. Of those that are of interest, I am going to note them down here:

    * LeWeb – Really quite good, european focused web event
    * PyCon 2011 – A must watch for anyone using Python.
    * JSConf – Good Javascript conference, though I got the feeling that it wasn’t quite as friendly as Pycon.
    * Webstock – New Zealand conference.
    * Future – Some interesting interviews.
    * Google Techtalks

    Of course I also watched quite a lot of Ted videos.

    I will keep adding to this list as I find more events to add, as there must be quite a few now.

    By timc3 on the
    April 15th, 2011