blog.timc3.com » Linux

Resizing ext3 or ext4 partition for virtual image

timc3 — Mon, 02 May 2011 08:41:43 +0000

Note: please backup and make sure that you have a copy of your data and virtual images, this can go wrong and I won’t be held responsible. This worked for me, and might not work for you. For ext4 please use the latest GParted.

As I migrated from raw or qcow images, some of the partition sizes that I have to work with are not as big as they could be. So I need to resize.

I generally give some of my hosts a 24Gb LVM virtual group each. But the image files can be as small as 2Gb – A lot of wasted space which the filesystem on the image might as well have. These instructions assume that there are only two partitions, a main partition, and a second Linux swap partition (either on an extended partition or a primary).

First taken the image offline, then get the name of image that you wish to resize, then the disk storage items:

ls /etc/vm/
lvdisplay

In my case it was “/dev/vdisk05/vm-102-disk-1″. Then you can check to see how large the volume group is set to (vgdisplay) resize:

lvresize -L 23G /dev/vdisk05/vm-102-disk-1

Next I mount a SystemRescueCD on the CDRom, and make sure its the first item in the boot sequence. Then boot into the system, go for the first choice in SystemRescueCD, and it will give you a rootprompt (I am VNC’ing into the system so that I can access the console as if using a physical machine):

#Change the passwd so that you can ssh in if you need to.
passwd
#Start the GUI so you can run Gparted (First choice again).
wizard

From a command prompt do (check file system, repair, remove journal):

fsck -n /dev/sda1
e2fsck -f /dev/sda1
tune2fs -O ^has_journal /dev/sda1

Then do the following in the GUI:

Choose CD > System > GParted
Delete the swap partition as it will be the last
Resize partition, leaving some space for a new swap partition
Create a swap (type Linux-swap) partition twice the size of the amount of RAM you allocate to the image
Click Edit > Apply all operations
Close GParted

Back on the command prompt do (repair, check file system, journal, shutdown):

e2fsck -f /dev/sda1
fsck -n /dev/sda1
tune2fs -j /dev/sda1
shutdown -h now

Remove the CDRom device, and change the boot order back then start the Virtual machine. Issue a df -h to see if the partition has changed. It should have done!

Virtual host QCow convert to LVM

timc3 — Sun, 01 May 2011 11:47:16 +0000

Sometimes its better to have your virtual images running directly of LVM instead of using raw or QCow formatted files. So how do we convert? Basically you have to make sure that you have a volume group, of the right size or bigger and then dd the file into a LV on that volume group:

This should in the directory where you have the image:

# Get the filesize of the raw image in Kilobytes.
ls -lk
#Create lv on vdisk05 (the VG)
lvcreate -L 5242880K -n vm-102-disk-1 vdisk05
#Convert from QCow to Raw
qemu-img convert bld-ubuntu-1004-2.qcow2 -O raw bld-ubuntu-1004-2.raw
#Copy the image into the lv:
dd if=bld-ubuntu-1004-2.raw of=/dev/vdisk05/vm-102-disk-1

Then change the configuration of your QEMU or whatever you are using on your virtualisation platform.

Sorry for the briefness of the instructions, its mainly for my own benefit and if you are doing this you should really know about LVM, dd and ls.

My Jenkins nginx proxy script

timc3 — Fri, 15 Apr 2011 12:35:11 +0000

Here is a very simple configuration for nginx to proxy Jenkins (the new name for the un-Oracle’d Hudson)

server {
listen 80 default;
server_name bld-master localhost;
root /var/lib/jenkins;

access_log /var/log/nginx/jenkins.access.log;

location / {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

}

Setting MAC Addresses on Debian and Ubuntu

timc3 — Thu, 14 Apr 2011 10:34:48 +0000

I recently had a problem with having duplicate MAC addresses on my network for a couple of machines that I was using in a Linux KVM cluster. Flashing the firmware on the motherboards seem to have reset the MAC address to a default (thanks ASUS) that was the same on each board.

Changing the MAC address back seems like quite a hard job and looking around on the internet it seems that many people are doing this the hard way with shell scripts and so forth. Instead its easy enough to do this using the scripts that are already setting up the network for Debian (and I am sure this will work for Ubuntu just as well).

Edit /etc/network/interfaces like so:

auto eth0
iface eth0 inet manual
pre-up ifconfig $IFACE hw ether 00:01:02:03:04:05
pre-up ifconfig $IFACE up
pre-down ifconfig $IFACE down

What this is doing is manually controlling the startup and shutdown of the eth0 interface. I am using bridging on my KVM boxes, so I don’t actually have to set up the IP address (albeit DHCP or manual IP configuration), but that aside the interesting parts are what happens in pre-up. Firstly I set the hardware address to a new address (and please change it from the example above – thats not the actual one I used either), then we tell it to bring the interface up.

We also use pre-down to tell it what happens when stopping the network service.

Probably better than messing around with Shell scripts.

Time saving tips for linux

timc3 — Fri, 25 Mar 2011 09:23:19 +0000

Really nice little round up on Quora:

http://www.quora.com/Linux/What-are-some-time-saving-tips-that-every-Linux-user-should-know

With a follow up on hacker news:

http://news.ycombinator.com/item?id=2361978

Probably going to be more of these.

Compiling RabbitMQ on Ubuntu 10.04

timc3 — Mon, 23 Aug 2010 15:05:33 +0000

As part of my on going effort for realtime web updates on one of our applications I needed to install RabbitMQ HEAD (That is the development head) with the RabbitMQ STOMP adaptor. Talking on IRC channel it was recommended to me to install from source, but unfortunately the documentation is out of date. So the following will do it cleanly:

sudo apt-get install erlang-crypto erlang-snmp erlang-syntax-tools libsctp1 lksctp-tools erlang-runtime-tools erlang-mnesia erlang-public-key erlang-os-mon erlang-ssl erlang-base erlang-parsetools mercurial git-core build-essential erlang-dev zip erlang-tools erlang-src python-simplejson erlang-edoc
cd ~
hg clone http://hg.rabbitmq.com/rabbitmq-public-umbrella
cd rabbitmq-public-umbrella
make co
make

Updated: The following will create a Debian style package and install (replace ** with the version numbering that you wish to use:

sudo apt-get install cdbs debhelper xmlto
cd rabbitmq-server
make VERSION=1.8.** srcdist
make -C packaging/debs/Debian UNOFFICIAL_RELEASE=true package
sudo dpkg -i packaging/debs/Debian/rabbitmq-server_1.8.**-1_all.deb

Activating the STOMP plugin:

cd rabbitmq-public-umbrella/rabbitmq-stomp/dist
sudo cp amqp_client.ez /usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/plugins
sudo cp rabbit_stomp.ez /usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/plugins

Configuration file:

RABBITMQ_PLUGINS_DIR=/usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/plugins
SERVER_START_ARGS='-rabbit_stomp listeners [{"0.0.0.0",61613}]'

Restart to activate (the activation script for plugins is depreciated):

sudo /etc/init.d/rabbitmq-server restart

And you should have a working RabbitMQ on Ubuntu with a STOMP adaptor.

Wake On Lan with Ubuntu

timc3 — Fri, 06 Aug 2010 12:08:22 +0000

Wake-on-lan is incredibly useful for those scenarios where you have more than one machine on a network, but you don’t always want to have them running but they are needed sometimes.

To setup, make sure that the Motherboard and NIC support is there from your manufacturer. Sometimes it has to be enabled, such as making sure that it uses S3 for shutdown, and that PCI wake is supported.

Next you need to check support from the operating system, I am using Ubuntu 10.04, but this should work for other operating systems. Install ethtool, and then check against the ethernet adapter (eth0 in my example).

sudo apt-get install ethtool
sudo ethtool eth0

If support is enabled, but it is turned off you will see:

Supports Wake-on: g
Wake-on: d

So we turn it on:

sudo ethtool -s eth0 wol g

Now we need to make sure that its always enabled everytime we restart:

cd /etc/init.d/
sudo vim wakeonlan

And in that file:

#!/bin/bash
ethtool -s eth0 wol g
exit

Now make sure that its put into the correct runlevels:

sudo chmod a+x wakeonlan
sudo update-rc.d -f wakeonlan defaults

And that is it. It should be supported that you can wake the machine from another computer/server using your favourite wakeup utility ( such as wakeonlan ).

VMWare Ubuntu IP address change

timc3 — Tue, 04 May 2010 16:00:11 +0000

I always have this problem when moving VMWare images of Ubuntu, the most recent of a 10.04 LTS Server. Whenever you move or copy a VMWare image it assigns a new Mac address and Ubuntu starts without properly bringing up the interface.

Really annoying when you are using VMWare server without a console.

To make it grab the IP Address again, simply remove the rule:

sudo rm /etc/udev/rules.d/70-persistent-net.rules

Then restart.

Of course before you move the image you could remove the rule and not have them problem.

Django on RedHat or CentOS

timc3 — Fri, 26 Mar 2010 10:07:21 +0000

I am normally using Debian, Ubuntu or even Suse for deploying Django, but a recent customer needed to deploy on Red Hat Enterprise Server 5.

We decided beforehand to test deployment on RHEL5 and also on CentOS 5.4 and so these instructions should work for both environments. NginX will be used as the webserver.

The first thing I like to do is to upgrade the repository and add EPEL. This wasn’t needed I found on the RedHat box I was using, but was needed on CentOS but your mileage might vary. It should be noted that we made the system as up to date as possible with patches and yum updates.

I won’t be covering the database setup, it is assumed that this is already setup and done. I also won’t be covering the actually placing of the Django application anywhere.

cd ~
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
yum repolist
yum update
yum search zlib
yum install zlib-devel*
yum install gcc make

Python 2.4 ships with both RHEL5 and CentOS 5 and is needed for the packaging environment YUM so I would advise not to replace it but just compile and install 2.5.5 (or 2.6) next to it.

mkdir src
cd src
wget http://www.python.org/ftp/python/2.5.5/Python-2.5.5.tgz
tar fxz Python-2.5.5.tgz
cd Python-2.5.5
./configure
make
make install
cd ~/src
wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg#md5=64c94f3bf7a72a13ec83e0b24f2749b2
sh setuptools-0.6c11-py2.5.egg

Now we need to build and install NginX. I choose this primarily because of my experience with it, (I have used Apache2 a lot but its archaic ) and because of its speed.

yum install pcre-devel.i386 openssl-devel.i386
wget http://nginx.org/download/nginx-0.7.65.tar.gz
tar fxz nginx-0.7.65.tar.gz
./configure --sbin-path=/usr/local/sbin --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_ssl_module --with-http_flv_module --with-cc-opt="-I /usr/include/pcre"
make
make install
vim /etc/init.d/nginx
sudo chmod +x /etc/init.d/nginx
/sbin/chkconfig nginx on
/sbin/chkconfig --list nginx

There is a decent RedHat init.d script here: http://wiki.nginx.org/RedHatNginxInitScript, but remember that I have changed the installation location above so you will need to change the location in that script of nginx from /usr/sbin/nginx to /usr/local/sbin/nginx

You will also need a nginx.conf configuration script for /etc/nginx/ or you could roll your own using sites-available, sites-enabled pattern. I will attach the script soon.

Now with Python 2.5 installed we can install Django and other requirements:

easy_install-2.5 Django
easy_install-2.5 django-nose
easy_install-2.5 http://dist.repoze.org/PIL-1.1.6.tar.gz
easy_install-2.5 flup

I am using easy_install here as that is the most widely used tool, but pip with a requirements file is a better solution.

Now we need a database adapter. I am using PostgreSQL 8.4.2, if you are using MySQL, Oracle or some other database you will need to follow the recommended route for the Python adapter.

yum search psycopg2
yum install python-psycopg2.i386 (This will install for the old Python)
yum install postgresql-devel
cd ~/src
wget http://initd.org/pub/software/psycopg/psycopg2-2.0.13.tar.gz
tar xvfz psycopg2-2.0.13.tar.gz
cd psycopg2-2.0.13
vim setup.cfg

Change the location of pg_config:

pg_config=/usr/bin/pg_config

And now install:

python2.5 setup.py build
python2.5 setup.py install

Then you need to install your application. This is really specific to your needs. I choose to install the files in /opt/cantemo/application-name as that is specific to our solutions, but RedHat style might mean that you wish to install somewhere else.

Lastly you will want to have your application start easily and also make sure that it starts if the machine ever reboots. To do this I am going to install the start-stop-daemon that is used on Debian based Linux distros on Redhat. I read some threads that it will but there is no ETA on that.

You should download the initd script for Django from http://code.djangoproject.com/wiki/InitdScriptForLinux. Change the RUN_AS user to the user on RedHat that you are going to use for running Nginx. By default this is apache.

cd ~/src
wget http://developer.axis.com/download/distribution/apps-sys-utils-start-stop-daemon-IR1_9_18-2.tar.gz
tar -xzf apps-sys-utils-start-stop-daemon-IR1_9_18-2.tar.gz
cd ~/src/apps/sys-utils/start-stop-daemon-IR1_9_18-2
gcc start-stop-daemon.c -o start-stop-daemon
cp start-stop-daemon /usr/local/sbin/
chmod +x /usr/local/sbin/start-stop-daemon
vim /etc/init.id/fastcgi
chmod +x /etc/init.d/fastcgi
/etc/init.d/fastcgi start
ps ax
/etc/init.d/fastcgi stop
/sbin/chkconfig --add fastcgi
/sbin/chkconfig --list fastcgi

Lastly make sure that your application directories have the correct user permissions but you should be able to start everything by issuing the following commands:

/etc/init.d/nginx start
/etc/init.d/fastcgi start

Eric Florenzano posted a nice tutorial on deploying django using FastCGI so also check that out, but I didn’t have much luck with daemontools on CentOS, and didn’t bother trying on RedHat. I suggest instead using start-stop-daemon. Check it out for other hints and tips though, particularly on prefork vs. threaded, pip, and more nginx configuration tips.

Postfix and Gmail

timc3 — Sat, 20 Mar 2010 12:07:03 +0000

I have been using Hudson as a continuous integration server for a short while now and I am super impressed. I have it running on Debian and is really quite feature-full.

One thing that I did have a problem with was with the standard Debian 5 Java environment there was a problem using TLS for mail it seemed, and I didn’t just want to send stuff to gmail’s SMTP server all the time so I configured a local Postfix SMTP relay on another server.

This guide, Send Mail Postfix Through Gmail’s SMTP On A Ubuntu LTS Server helped a lot – but be sure to read the comments as there are some helpful hints.

So now I have it. Hudson running and sending email through my relay to my gmail accounts.

More information on Hudson soon.