I’ll try to explain the most common methods here, and give examples and benchmarks.

1, Simple compression and encrypted transmission

This is probably the most common way to copy data, and it’s simply tar and gzip the data and then send it to another machine over scp:

machine1$ tar -zcvf backup.tgz /home/backupdata/
machine1$ scp backup.tgz root@machine2:/home/backups/

machine2$ tar -zxvf backup.tgz

The problem with this method is that disk reads and writes can be intensive because you need to read from machine1′s hard drive and then write to it during the gzip process, then read from it and write to machine2′s hard drive during the scp. This can slow down the process. Also you’re required to log in to machine2 and unzip the resulting file.

2, rsync

rsync is good for syncing data between two machines, and for incremental copying (ie if a copy is interrupted, it can resume). I won’t go into the actual details here because there are many ways of using rsync and this article is mainly aimed at one shot copying of data. It’s definitely worth checking ‘man rsync’ to see if it will be relevant for your needs though.

I tend to use rsync for backups where I want to keep an exact copy of a directory on another server, and keep it up to date. rsync is useful here because any time I run the rsync command it will only copy changes over between the servers. An example of this usage is:

machine1$ rsync -Ravr –delete –delete-after ./backupdata/ root@machine2:/home/backups/

The flags I’ve passed are: R – relative, r – recursive, a – archive mode, v – verbose.

3, Gzip and SCP in one command

This will usually perform better than option 1 simply because you aren’t writing the gzip data to the disk on machine1 before sending it to machine2. Basically, this is the commands from option 1 in one line, so the data is gzipped and piped to ssh on machine2 rather than writing to the disk on machine1 and then copying over.

machine1$ tar -zcvf – /home/backupdata/* | ssh root@machine2 “cd /home/backups/; tar -zxvf -”

4, Netcat

In theory, this should be the best solution because the data isn’t encrypted or decrypted as it is in ssh/scp (a little less cpu overhead), and there isn’t any needless IO activity as in option 1. First you will need to tell machine2 to ‘listen’ on a specific port (98765 in this example), and uncompress anything which arrives on that port:

machine2$ nc -l -p 98765 | tar -zxvf -

Then gzip and send the data from machine1 to the specified port on machine2:

machine1$ tar -zcvf – /home/backupdata/ | nc -q 1 machine2 98765

As we’re using the verbose (-v) option in tar, you will see the output on both machines. I find netcat more convenient if I need to send a lot of data from one server to another as I can just leave netcat running on machine2 and send data to it multiple times from server1 (or any other server).

5, SMB or NFS

These are also decent alternatives, and if you already have NFS or SMB shares set up then it may be worth just compressing the data and copying it to a mounted share using ‘cp’. An example would be (assuming the share name is ‘backups’).

machine1$ tar -zcvf backup.tgz /home/backupdata/
(SMB) machine1$ mount -t smbfs //machine2/backups /mnt/machine2
(NFS) machine1$ mount machine2:/home/backups /mnt/machine2
machine1$ cp backup.tgz /mnt/machine2/

Benchmarks

For my tests, I’m using 2 servers with the following basic specs:
Dual CPU Intel E5520
16gb RAM
Hardware RAID1.

I’ll be sending a few log files which total 7GB of data.

1 – The initial gzip took 114 seconds and scp took 6 seconds for a total of 120 seconds.
2 – This took a total of around 140 seconds, so was the slowest in this test. You would notice the benefits of rsync when doing incremental backups so only differences are copied over rather than redoing the whole copy.
3 – Total of 119 seconds (so not really any faster but you don’t need to log in to machine2 to unzip).
4 – Total of 109  seconds, so this is the fastest option but not by a huge margin.

Bookmark It

“memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.”

Basically, what this means is that memcached is a daemon running on a server which allows you to save and retrieve variables in the server memory. This is primarily used to ease database load on websites and applications. The logic for doing something like this is…

- Before running a query, check if the result is available in memcache.
- If the result is in memcache, return the cached result.
- If not, run the query on the database, and store the result in memcache.

To install memcached, simply run the following command:

sudo apt-get install memcached

Once it’s installed, edit /etc/memcached.conf and change the line beginning ‘-m’ which is the amount of memory (in megabytes) to allocate to the server. You can also change the IP address that the server listens on  in the line beginning ‘-l’.

Now restart the daemon by running

/etc/init.d/memcached restart

Now you have memcache set up and running on your server. A great feature of memcached is that you can easily cluster servers. If you want to do this, simply install memcache on your other servers before continuing.

At this point, I’d recommend downloading memcache.php which is a php script showing you a lot of useful information about your memcached servers. Once downloaded, put it in a web facing directory on your server, and modify the $MEMCACHE_SERVERS array with your server addresses.

Using memcache in php scripts is very easy. There are either procedural, or object oriented functions already available. Here is an example of a script which will store a simple variable and then retrieve it and display it.

<?php
$memcache = new Memcache;

$memcache->addServer(’10.2.0.245′, 11211) or die (“Could not connect”);
$memcache->addServer(’10.2.0.249′, 11211) or die (“Could not connect”);

$memcache->set(‘mytestvariable’, “this is the data in my test variable”, false, 60) or die (“Unable to save the data to the server”);
echo “Data has been stored in the cache<br />”;

$result = $memcache->get(‘mytestvariable’);
echo “Retrieved data from the server:<br/>”;

var_dump($result);

?>

Obviously if you only have one server then only use one addServer line.

The script above will store the test variable in the cache for 60 seconds. If you drop the timeout down to something like 1 second, and then sleep for 3 seconds before attempting to get the data, you will find that the data has expired and nothing will be returned.

It’s also useful to know that you can store anything which can be serialized in memcache. This means it’s safe to store things like arrays in the cache without having to ‘flatten’ the data beforehand.

Obviously how and where you implement a cache is entirely dependant on how your system works. If you have a lot of intense database usage then you will find that even caching with a short timeout will vastly reduce the amount of database queries. You should find that memcache is considerably faster than normal database access, and even faster than the mysql query cache. You also have total control over what data is cached, and how long it is cached for before your queries fall back to checking a database.

Some further reading:
Facebook engineering blog – How facebook use memcache
Fotolog case study

Also, it’s worth checking out the PHP manual page.

Bookmark It

Please note that clustering works differently in MySQL 5.1, so this guide may not be relevant for that.

I also recommend the book ‘MySQL Clustering’ by Alex Davies and Harrison Fisk (ISBN 0-672-32855-0) as it explains how the cluster works in a lot more detail than I plan to here.

I will explain how to set up 3 servers, with the following roles:

server-a = management node (IP 10.1.0.10)
server-b = storage and sql node (IP 10.1.0.11)
server-c = storage and sql node (IP 10.1.0.12)

It is easy to have storage and sql nodes on separate servers, but to make this guide easier to follow (and so we don’t need 5 machines), I’ll use 3 servers.

All servers will need to have mysql-server installed. If you need help doing this, have a look at this guide.

Setting up the management node (10.1.0.10)

Default debian/ubuntu installs look for /etc/mysql/ndb_mgmd.cnf for the management node, so we need to create this file, with the following contents:

[NDBD DEFAULT]
NoOfReplicas=2
DataDir= /var/lib/mysql-cluster

# Management Node
[NDB_MGMD]
HostName=10.1.0.10
DataDir=/var/lib/mysql-cluster

# Storage Nodes (one for each node)
[NDBD]
HostName=10.1.0.11
DataDir=/var/lib/mysql-cluster
[NDBD]
HostName=10.1.0.12
DataDir=/var/lib/mysql-cluster

# SQL Nodes (one for each node)
[MYSQLD]
HostName=10.1.0.11
[MYSQLD]
HostName=10.1.0.12

The line beginning NoOfReplicas tells the cluster how many copies of data should be kept.

Now we start the management node, and it will sit waiting for connections from the storage and SQL nodes:

/etc/init.d/mysql-ndb-mgm start

Setting up SQL and data nodes

All we need to do here is make a few changes to /etc/mysql/my.cnf (the mysql config file)

First, add the following 2 lines (using the IP of your management node) inside the [mysqld] section of the config file:

ndbcluster
ndb-connectstring=10.1.0.10

And near the bottom of the file there will be a section for [MYSQL_CLUSTER] which you will need to uncomment, and change the ndb-connectstring line to your management nodes IP.

If you are using separate data and sql nodes, the [mysqld] part is relevant to the sql nodes only, and the [MYSQL_CLUSTER] part is relevant to the data nodes only.

Before we start the services, we have to create the /var/lib/mysql-cluster directory and set it to be owned by the mysql user:

mkdir /var/lib/mysql-cluster
chown mysql:mysql /var/lib/mysql-cluster

Now we need to start the node services:

/etc/init.d/mysql restart
(which starts the sql node)
/etc/init.d/mysql-ndb restart
(which starts the data node)

By now, everything should be running, so we connect to the management node (by running ndb_mgm from the command line) and check that the other nodes have connected properly (using the show command):

ndb_mgm> show;
Cluster Configuration
———————
[ndbd(NDB)]     2 node(s)
id=2    @10.1.0.11  (Version: 5.0.51, Nodegroup: 0)
id=3    @10.1.0.12  (Version: 5.0.51, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @10.1.0.10  (Version: 5.0.51)

[mysqld(API)]   2 node(s)
id=4    @10.1.0.11  (Version: 5.0.51)
id=5    @10.1.0.12  (Version: 5.0.51)

This shows that everything has connected properly. If connections are missing, then it’s worth checking /var/log/syslog on the affected server to see if there are any error messages.

Using the cluster

It’s worth noting that any databases already on the servers will continue to work as before. Tables only become part of the cluster when their engine type is changed to ‘ndbcluster’ by issuing this command (from a mysql prompt):

alter table tablename engine=ndbcluster;

But for now we will create a new database and clustered table, and test that the data is clustered. The cluster setup applies to tables only, not databases, so we first need to create the database on both sql nodes:

create database cluster;

Now, when we create a table inside the cluster database, as long as the engine is ndbcluster, the data will be synced across data nodes, which we can test by doing the following (from a mysql prompt on either sql node):

create table clustertest (i int) engine=ndbcluster;
insert into clustertest () values (1);
select i from clustertest;

Which should return the single row with the value of 1. Now connect to the other SQL node and try…

insert into clustertest () values (2);
select i from clustertest;

Which should return both rows, which will happen whichever SQL node you connect to as the table is now stored in the cluster.

If you receive an error like:

ERROR 1015 (HY000): Can’t lock file (errno: 4009)

Then it is likely that some of your cluster nodes haven’t started correctly, so its worth checking the management interface again.

Shutting down and restarting the cluster

To shutdown the data nodes and management node, all you need to do is enter the command ‘shutdown’ in the management interface.

To restart, simply run

/etc/init.d/mysql-ndb-mgm start

On the management node, and

/etc/init.d/mysql-ndb start

On the data nodes. The SQL nodes continue running, and can be stopped/started using the standard mysql init script.

That’s it, you’ve now should have a working MySQL cluster. As you have NoOfReplicas=2 in the management config, you should be able to unplug either data node at any time and still have access to all of the clustered tables.

Bookmark It

These are basically the steps from the official manual, but I cant guarantee that your system will upgrade as easily as mine – as always when  upgrading you should backup your data and be prepared for the worst.

Usually I prefer apt-get over aptitude, but the official documentation recommends aptitude to do the upgrade, so heres what you need to do (as root)

First, edit /etc/apt/sources.list and change all mentions of etch to lenny. If you use vim, you can simply do the following:

vim /etc/apt/sources.list
(then, in vim)
:%s/etch/lenny/g
:wq

Then update…

aptitude update

At this point, you might receive the following error (I did on every server I upgraded)

W: There is no public key available for the following key IDs:
4D270D06F42584E6

W: You may want to run apt-get update to correct these problems

To fix this, you need to install the following:

aptitude install debian-archive-keyring
aptitude update

Now you’re ready to do the upgrade. The safest way to do this is in 3 parts

aptitude install aptitude
(updating apt first is a safe way of doing things)
aptitude upgrade
(when this finishes)
aptitude dist-upgrade

When this has finished, your system is upgraded to Lenny! All you need to do now is reboot, and hopefully the system will come back up and be running Lenny.

Bookmark It