A Million Monkeys

I installed Ubuntu 10.04 on a few servers a while back, and instantly noticed a lot of problems with a few of our sites (that’s usually what happens when you do a major upgrade without reading any release notes!). I fairly quickly found out that the cause was that in Ubuntu 10.04, PHP is now running version 5.3 (it was 5.2 in previous releases).

Unfortunately, PHP 5.3 deprecates a few common features (see here for more info) which causes a lot of sites to just stop working, or act strangely.

I found that the best way to move forward was to downgrade PHP back to 5.2 while we could work out what the problem was with our sites (as is always the case in a fast moving development cycle, this was months ago and we haven’t got around to it yet). I found a really useful step by step guide to do this, and within 10 minutes the server was running 5.2 using the Ubuntu Karmic sources.

The guide can be found at http://mrkandy.wordpress.com/2010/04/16/install-php-5-2-x-in-ubuntu-10-04-lucid/ and I have to give all credit to the writer of that article. None of this was my own work, I just stumbled across the post above and it fixed all my problems (I’m hoping that it’ll help out some of my readers as well).

Anyway, here are the steps. It basically does the following: Remove all PHP packages, pin all PHP packages to karmic, add karmic to sources, and then reinstall the PHP packages which were previously installed.

php_installed=`dpkg -l | grep php| awk '{print $2}' |tr "\n" " "`

# remove all php packge
sudo aptitude purge $php_installed

# use karmic for php pakage
# pin-params:  a (archive), c (components), v (version), o (origin) and l (label).
echo -e "Package: php5\nPin: release a=karmic\nPin-Priority: 991\n"  | sudo tee /etc/apt/preferences.d/php > /dev/null
apt-cache search php5-|grep php5-|awk '{print "Package:", $1,"\nPin: release a=karmic\nPin-Priority: 991\n"}'|sudo tee -a /etc/apt/preferences.d/php > /dev/null
apt-cache search -n libapache2-mod-php5 |awk '{print "Package:", $1,"\nPin: release a=karmic\nPin-Priority: 991\n"}'| sudo tee -a /etc/apt/preferences.d/php > /dev/null
echo -e "Package: php-pear\nPin: release a=karmic\nPin-Priority: 991\n"  | sudo tee -a /etc/apt/preferences.d/php > /dev/null

# add karmic to source list
egrep '(main restricted|universe|multiverse)' /etc/apt/sources.list|grep -v "#"| sed s/lucid/karmic/g | sudo tee /etc/apt/sources.list.d/karmic.list > /dev/null

# update package database (use apt-get if aptitude crash)
sudo apt-get update

# install php
sudo apt-get install $php_installed
# or sudo aptitude install -t karmic php5-cli php5-cgi //for fcgi
# or  sudo apt-get install -t karmic  libapache2-mod-php5 //for apache module

sudo aptitude hold `dpkg -l | grep php5| awk '{print $2}' |tr "\n" " "`
#done

And that’s it. As I said previously, all credit should go to the author of this post.

Bookmark It

Having been bitten a few times by not backing data up, I’m now pretty anal about making sure my servers are backed up.

First, a little bit of background. My network consists of a large number of linux boxes (mostly debian and ubuntu), and various windows servers running anything from XP, to 2000 and 2003. I have one dedicated linux machine which is a backup server running debian. Some backups are ‘put’ onto the backup server at predefined intervals over samba, nfs, and ftp, and others are ‘retrieved’ by scripts on the backup server. Every day these backups are copied to 2 usb drives (both encrypted with truecrypt) and taken off site.

As rebuilding a server is a fairly quick task (especially ubuntu, which can be installed and configured from scratch in under 20 minutes), I only back up the necessary configuration files and data, as doing full images of every server would need a huge amount of storage.

Firstly, the backup server itself needs to be configured to run samba, nfs, and ftpd. Setting these up is fairly trivial and won’t be covered here but I will write guides to setting these up in the very near future. As a side note, we have recently switched from a backup ‘server’ to a QNAS network storage device. As this runs linux it’s more than capable of running the necessary cron jobs for the ‘get’ server backups. Plus it also comes preconfigured with samba, nfs, timecapsule, and ftp.

So, on to the backing up. All of my windows machines are part of the ‘get’ backups, where the backup server itself connects to the client and fetches the backup. To achieve this, I simply create network shares on the client machines, then do something similar to the following:

mount -t smbfs //svr2/share /mnt/backup
tar -zcvf /home/backups/svr2.tgz /mnt/backup/* -R
umount /mnt/backup

I have this saved as a shell script which runs every night to fetch a backup of the necessary data from the windows machine. Note that this method can be used for any windows server where the necessary data is shared on the network, or any linux server where the data is shared over samba. I have a number of these shell scripts, all of which connect to different servers and save their vital data.

Second, onto backing up our client machines which are all running windows XP. We’ve ‘trained’ all our users to store their important documents inside the ‘My Documents’ folder. I then have samba shares on the backup server for each user, and re-map their ‘My Documents’ folder to the network drive. To achieve this, I map the drive on their machine, and then right click on ‘My Documents’, and change the location it points to.

To backup these folders, all I need to do is create a tar archive of each of the samba shares.

Finally, backing up the linux servers. As I said above, some of these are backed up by just making a tar archive of their samba shares. For servers which don’t have samba shares, I run cron jobs on each machine which creates a local backup and then copies it to the backup server through ftp.

An example of how to back up a standard debian/ubuntu apache install:

tar -zcvf /home/backups/apache_configs.tgz /etc/apache2/* -R
tar -zcvf /home/backups/website_data.tgz /var/www/* -R

An example of how to back up a MySQL server. There are 2 ways to do this. The simple way to back up all data is to make a tar archive of the entire MySQL data directory. This is fine for many small sites but may not be suitable for sites where the data is constantly changing, or the tables are using innodb (innodb doesn’t always backup 100% successfully by simply copying the data files). For a standard debian/ubuntu install, this can be achieved by:

tar -zcvf /home/backups/mysql_data.tgz /var/lib/mysql/* -R

Another method of backing up mysql is to use the ‘mysqldump’ command line utility, which will create a dump of the database as a .sql file, and can be re imported at a later date. This is seen as a more secure backup because the sql files can usually be imported across mysql versions easily, but has the downside of a slower backup, slower restore, and occasionally you will need to manually edit the dump files to get them to reimport properly. A mysqldump can be created by running a command similar to the following:

mysqldump -uUSER -pPASSWORD -hHOST DB_NAME > /home/backups/file.sql
mysqldump -uUSER -pPASSWORD -hHOST ANOTHER_DB_NAME > /home/backups/file2.sql
tar -zcvf /home/backups/databases.tgz /home/backups/*.sql

Backing up a DNS server is also very simple, and can be done in much the same way as the apache backup above, making sure that the directory including the configs, and zonefiles are included in the archive.

Mail servers can be a bit more tricky since they rely on configs for different services which are usually found in different locations. Also, all user email must be backed up. The way I achieve this is to have the shell script create a tar file for all user email, and then manually copy the important configs into a directory before creating a tar file containing them all.

Once you’ve backed up the data for the servers, you will need to move it to the backup server for safe keeping. I usually do this over FTP, by adding something similar to the following to the bottom of the backup shell script:

# FTP to backup server

HOST=’backup_ip’
USER=’backup’
PASS=’password’
cd /home/backups/

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASS
binary
quote CWD backup/SERVER_NAME/
put backup_file.tgz
quit
END_SCRIPT

That’s about all there is to know. It’s all fairly simple to do, but ensures that you have backups of all your important data. If you wish to perform incremental backups then it’s advisable to look into rsync.

Once all of the servers have been backed up, I have a script on the ‘backup server’, which does some tidying up, and copies all of the daily backups to a directory with todays date. This means I can store backups for as long as I wish without accidently overwriting them.

Bookmark It

We recently got some Dell Poweredge R410 servers, and I thought it would be quite useful to change the LCD text to something meaningful. Unfortunately, this turned out to be a bit more of a pain than I was expecting, but I have it working, and a PHP script to update the display.

First, you’ll need to go into the BIOS and change the display to ‘custom’.

The updates will be done using ‘ipmitool’, so we need to install it and add the necessary modules to /etc/modules

apt-get install ipmitool
echo “ipmi_devintf” >> /etc/modules
echo “ipmi_si” >> /etc/modules
echo “ipmi_msghandler” >> /etc/modules

You should now be able to view the contents of the LCD by running:

ipmitool delloem lcd

I didn’t get anywhere trying to write new values to the LCD using the delloem command, so resorted to sending raw commands to ipmitool.

I wrote the following php script, which can be used by first making it executable, then running it from the shell

chmod +x updateLCD.php
./updateLCD.php “test string”

The php script is pretty simple and can be found here

It’s also fairly easy to extend this script to update the LCD to display stats such as disk usage, server load, cpu usage etc..

Bookmark It

This is something which took me quite a while to figure out, but it can be done fairly easily in a few minutes. This guide will install Dell OMSA and use omreport to view the status of the RAID.

First, add the following line to /etc/apt/sources.list:

deb ftp://ftp.sara.nl/pub/sara-omsa dell sara

Then, if you try ‘apt-get update’, you will get a key error, so do the following:

wget -O – http://ftp.sara.nl/debian_sara.asc | apt-key add  -
apt-get update
apt-get install dellomsa

To use omreport, you need to install libstdc++5 to fix the dependancy problems:

For 32 bit systems:
cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_i386.deb
dpkg -i libstdc++5_3.3.6-17ubuntu1_i386.deb

For 64 bit systems (you will need the 32 bit version as well):
cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_i386.deb
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_amd64.deb
dpkg -i libstdc++5_3.3.6-17ubuntu1_amd64.deb
dpkg-deb -x libstdc++5_3.3.6-17ubuntu1_i386.deb ./extracts
cp ./extracts/usr/lib/* /usr/lib32/
rm -rf ./extracts

Now start the dataeng service:

/etc/init.d/dataeng start

And you should be able to view information about the RAID using omreport:

omreport system summary
omreport storage connector controller=0
omreport storage pdisk controller=0
omreport storage vdisk controller=0

You can then make a simple shell/php/perl script cron which checks the status of the RAID every x minutes and notifies you of any problems.

Bookmark It

If you don’t know what memcache is, the official site says the following:

“memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.”

Basically, what this means is that memcached is a daemon running on a server which allows you to save and retrieve variables in the server memory. This is primarily used to ease database load on websites and applications. The logic for doing something like this is…

- Before running a query, check if the result is available in memcache.
- If the result is in memcache, return the cached result.
- If not, run the query on the database, and store the result in memcache.

To install memcached, simply run the following command:

sudo apt-get install memcached

Once it’s installed, edit /etc/memcached.conf and change the line beginning ‘-m’ which is the amount of memory (in megabytes) to allocate to the server. You can also change the IP address that the server listens on  in the line beginning ‘-l’.

Now restart the daemon by running

/etc/init.d/memcached restart

Now you have memcache set up and running on your server. A great feature of memcached is that you can easily cluster servers. If you want to do this, simply install memcache on your other servers before continuing.

At this point, I’d recommend downloading memcache.php which is a php script showing you a lot of useful information about your memcached servers. Once downloaded, put it in a web facing directory on your server, and modify the $MEMCACHE_SERVERS array with your server addresses.

Using memcache in php scripts is very easy. There are either procedural, or object oriented functions already available. Here is an example of a script which will store a simple variable and then retrieve it and display it.

<?php
$memcache = new Memcache;

$memcache->addServer(’10.2.0.245′, 11211) or die (“Could not connect”);
$memcache->addServer(’10.2.0.249′, 11211) or die (“Could not connect”);

$memcache->set(‘mytestvariable’, “this is the data in my test variable”, false, 60) or die (“Unable to save the data to the server”);
echo “Data has been stored in the cache<br />”;

$result = $memcache->get(‘mytestvariable’);
echo “Retrieved data from the server:<br/>”;

var_dump($result);

?>

Obviously if you only have one server then only use one addServer line.

The script above will store the test variable in the cache for 60 seconds. If you drop the timeout down to something like 1 second, and then sleep for 3 seconds before attempting to get the data, you will find that the data has expired and nothing will be returned.

It’s also useful to know that you can store anything which can be serialized in memcache. This means it’s safe to store things like arrays in the cache without having to ‘flatten’ the data beforehand.

Obviously how and where you implement a cache is entirely dependant on how your system works. If you have a lot of intense database usage then you will find that even caching with a short timeout will vastly reduce the amount of database queries. You should find that memcache is considerably faster than normal database access, and even faster than the mysql query cache. You also have total control over what data is cached, and how long it is cached for before your queries fall back to checking a database.

Some further reading:
Facebook engineering blog – How facebook use memcache
Fotolog case study

Also, it’s worth checking out the PHP manual page.

Bookmark It

This guide explains how to turn standard debian or ubuntu mysql-server installs into a full ndb cluster. As we’re using the standard mysql-server package, you won’t need to download any non .deb binaries or do any compiling. If everything goes well, this should take under 10 minutes to get working. I don’t plan to cover much theory here, as it is just a guide to getting the server up and running.

Please note that clustering works differently in MySQL 5.1, so this guide may not be relevant for that.

I also recommend the book ‘MySQL Clustering’ by Alex Davies and Harrison Fisk (ISBN 0-672-32855-0) as it explains how the cluster works in a lot more detail than I plan to here.

I will explain how to set up 3 servers, with the following roles:

server-a = management node (IP 10.1.0.10)
server-b = storage and sql node (IP 10.1.0.11)
server-c = storage and sql node (IP 10.1.0.12)

It is easy to have storage and sql nodes on separate servers, but to make this guide easier to follow (and so we don’t need 5 machines), I’ll use 3 servers.

All servers will need to have mysql-server installed. If you need help doing this, have a look at this guide.

Setting up the management node (10.1.0.10)

Default debian/ubuntu installs look for /etc/mysql/ndb_mgmd.cnf for the management node, so we need to create this file, with the following contents:

[NDBD DEFAULT]
NoOfReplicas=2
DataDir= /var/lib/mysql-cluster

# Management Node
[NDB_MGMD]
HostName=10.1.0.10
DataDir=/var/lib/mysql-cluster

# Storage Nodes (one for each node)
[NDBD]
HostName=10.1.0.11
DataDir=/var/lib/mysql-cluster
[NDBD]
HostName=10.1.0.12
DataDir=/var/lib/mysql-cluster

# SQL Nodes (one for each node)
[MYSQLD]
HostName=10.1.0.11
[MYSQLD]
HostName=10.1.0.12

The line beginning NoOfReplicas tells the cluster how many copies of data should be kept.

Now we start the management node, and it will sit waiting for connections from the storage and SQL nodes:

/etc/init.d/mysql-ndb-mgm start

Setting up SQL and data nodes

All we need to do here is make a few changes to /etc/mysql/my.cnf (the mysql config file)

First, add the following 2 lines (using the IP of your management node) inside the [mysqld] section of the config file:

ndbcluster
ndb-connectstring=10.1.0.10

And near the bottom of the file there will be a section for [MYSQL_CLUSTER] which you will need to uncomment, and change the ndb-connectstring line to your management nodes IP.

If you are using separate data and sql nodes, the [mysqld] part is relevant to the sql nodes only, and the [MYSQL_CLUSTER] part is relevant to the data nodes only.

Before we start the services, we have to create the /var/lib/mysql-cluster directory and set it to be owned by the mysql user:

mkdir /var/lib/mysql-cluster
chown mysql:mysql /var/lib/mysql-cluster

Now we need to start the node services:

/etc/init.d/mysql restart
(which starts the sql node)
/etc/init.d/mysql-ndb restart
(which starts the data node)

By now, everything should be running, so we connect to the management node (by running ndb_mgm from the command line) and check that the other nodes have connected properly (using the show command):

ndb_mgm> show;
Cluster Configuration
———————
[ndbd(NDB)]     2 node(s)
id=2    @10.1.0.11  (Version: 5.0.51, Nodegroup: 0)
id=3    @10.1.0.12  (Version: 5.0.51, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @10.1.0.10  (Version: 5.0.51)

[mysqld(API)]   2 node(s)
id=4    @10.1.0.11  (Version: 5.0.51)
id=5    @10.1.0.12  (Version: 5.0.51)

This shows that everything has connected properly. If connections are missing, then it’s worth checking /var/log/syslog on the affected server to see if there are any error messages.

Using the cluster

It’s worth noting that any databases already on the servers will continue to work as before. Tables only become part of the cluster when their engine type is changed to ‘ndbcluster’ by issuing this command (from a mysql prompt):

alter table tablename engine=ndbcluster;

But for now we will create a new database and clustered table, and test that the data is clustered. The cluster setup applies to tables only, not databases, so we first need to create the database on both sql nodes:

create database cluster;

Now, when we create a table inside the cluster database, as long as the engine is ndbcluster, the data will be synced across data nodes, which we can test by doing the following (from a mysql prompt on either sql node):

create table clustertest (i int) engine=ndbcluster;
insert into clustertest () values (1);
select i from clustertest;

Which should return the single row with the value of 1. Now connect to the other SQL node and try…

insert into clustertest () values (2);
select i from clustertest;

Which should return both rows, which will happen whichever SQL node you connect to as the table is now stored in the cluster.

If you receive an error like:

ERROR 1015 (HY000): Can’t lock file (errno: 4009)

Then it is likely that some of your cluster nodes haven’t started correctly, so its worth checking the management interface again.

Shutting down and restarting the cluster

To shutdown the data nodes and management node, all you need to do is enter the command ‘shutdown’ in the management interface.

To restart, simply run

/etc/init.d/mysql-ndb-mgm start

On the management node, and

/etc/init.d/mysql-ndb start

On the data nodes. The SQL nodes continue running, and can be stopped/started using the standard mysql init script.

That’s it, you’ve now should have a working MySQL cluster. As you have NoOfReplicas=2 in the management config, you should be able to unplug either data node at any time and still have access to all of the clustered tables.

Bookmark It

If you install and uninstall a lot of packages using apt, aptitude, or dpkg, you may find that you end up with a lot of redundant packages, or ones which have been uninstalled but the config files are left behind.

You can find packages which have been removed but still have config files by running (basically the status field will be ‘rc’):

dpkg -l | grep “^rc”

Or, suppose you installed mysql 5.1 from my previous post using the dotdeb repositories and wanted to see a list of what was installed:

dpkg -l | grep dotdeb

Now, removing those packages is easy, all you need to do is pass the list to dpkg –purge using awk in a similar way to grep above, so the 2 examples above would become:

dpkg --purge `dpkg -l | awk '/^rc/{print $2}'`
dpkg --purge `dpkg -l | awk '/dotdeb/{print $2}'`

(note that you may need to delete and retype the backticks depending on how cut and paste is handled)

Using the methods above, it should be easy to clean up your system.

Bookmark It

For debian/ubuntu users who want to use MySQL 5.1, there aren’t really many options available apart from compiling from source.

Whilst this is probably the best solution, a far simpler and easier (no need to worry about things like the client) method is using the repositories from dotdeb.org. Although these are made for debian installs, I tried it earlier on ubuntu 8.10 and it worked without any problems. As usual, back up your data first!

Assuming you’ve got mysql-server-5.0 installed (although if you don’t have the server installed at all it should still work), here are the steps to get 5.1.

First, you need to edit /etc/apt/sources.list and add the dotdeb repository. This is the line you need to add for Lenny, but just change it to Etch if you haven’t upgraded yet (see my earlier post if you want to know how to do that)

deb http://packages.dotdeb.org lenny all

Then, update…

apt-get update

And then install 5.1

apt-get install mysql-server-5.1

This should also install the dependancies: libmysqlclient16, mysql-client-5.1, and update mysql-common. If you already have 5.0 installed, this will also remove the server and client for it.

If you get a message saying that packages can’t be authenticated, it isn’t anything to worry about. Just select ‘y’ and continue. After the install is complete, you should have MySQL 5.1 installed with the most useful engines:

Server version: 5.1.32-0.dotdeb.0 (Debian)

You can also use the dotdeb repositories to upgrade to the latest versions of apache, and php.

It’s probably worth reading this guide on apt pinning, which is the best way to maintain a system using repositories with different versions of the same packages.

Bookmark It

My main reason for posting this is completeness (so I’ve got guides for everything you need for a basic hosting server here), not because it’s difficult to do.

Getting mysql server installed and running is simple to do:

apt-get install mysql-server

Once this is done, you will probably want to set the root password, or add an admin user:

mysql
grant all privileges on *.* to ‘username’@'host’ identified by ‘password’ with grant option;

Obviously use your own username, host, and password. To allow connections from any host, you can use ‘%’

Next, I usually change the config so MySQL is listening on all available IP addresses (instead of just localhost). This is done by commenting out the ‘bind-address’ line in /etc/mysql/my.cnf and restarting the server:

/etc/init.d/mysql restart

That’s it, you’ve now got a working MySQL server.

Bookmark It

This guide assumes that you’ve already got a server running either debian or ubuntu, and you want to make it serve web pages.

First, install apache2, php, and some other useful modules

apt-get install apache2 libapache2-mod-php5 php5-cli php5-curl php5-gd php5-imap php5-common php5-mysql

That’s it, you should now be able to browse to http://localhost and see the default apache page!

By default, the webroot is /var/www/ but if you’re planning to host a domain you’ll probably want to point it somewhere else. This is how you configure apache to listen for the domain www.example.com and set the webroot as /home/www/www.example.com/ (it’s assumed that you’ve already got an A record in the sites DNS pointing to this server)

/etc/init.d apache2 stop

Then create the apache zone file in /etc/apache2/sites-available/www.example.com/ with the following content:

<VirtualHost x.x.x.x:80>
ServerAdmin me@example.com
DocumentRoot /home/www/www.example.com
ServerName www.example.com
ErrorLog /var/log/apache2/error_log
CustomLog /var/log/apache2/access_log combined
EnableSendfile Off
EnableMMap Off
</VirtualHost>

Replacing x.x.x.x with your servers IP address.

Next, you need to enable the site, and start apache.

a2ensite www.example.com
/etc/init.d/apache2 start

That’s it!

Bookmark It