A Million Monkeys

Having been bitten a few times by not backing data up, I’m now pretty anal about making sure my servers are backed up.

First, a little bit of background. My network consists of a large number of linux boxes (mostly debian and ubuntu), and various windows servers running anything from XP, to 2000 and 2003. I have one dedicated linux machine which is a backup server running debian. Some backups are ‘put’ onto the backup server at predefined intervals over samba, nfs, and ftp, and others are ‘retrieved’ by scripts on the backup server. Every day these backups are copied to 2 usb drives (both encrypted with truecrypt) and taken off site.

As rebuilding a server is a fairly quick task (especially ubuntu, which can be installed and configured from scratch in under 20 minutes), I only back up the necessary configuration files and data, as doing full images of every server would need a huge amount of storage.

Firstly, the backup server itself needs to be configured to run samba, nfs, and ftpd. Setting these up is fairly trivial and won’t be covered here but I will write guides to setting these up in the very near future. As a side note, we have recently switched from a backup ‘server’ to a QNAS network storage device. As this runs linux it’s more than capable of running the necessary cron jobs for the ‘get’ server backups. Plus it also comes preconfigured with samba, nfs, timecapsule, and ftp.

So, on to the backing up. All of my windows machines are part of the ‘get’ backups, where the backup server itself connects to the client and fetches the backup. To achieve this, I simply create network shares on the client machines, then do something similar to the following:

mount -t smbfs //svr2/share /mnt/backup
tar -zcvf /home/backups/svr2.tgz /mnt/backup/* -R
umount /mnt/backup

I have this saved as a shell script which runs every night to fetch a backup of the necessary data from the windows machine. Note that this method can be used for any windows server where the necessary data is shared on the network, or any linux server where the data is shared over samba. I have a number of these shell scripts, all of which connect to different servers and save their vital data.

Second, onto backing up our client machines which are all running windows XP. We’ve ‘trained’ all our users to store their important documents inside the ‘My Documents’ folder. I then have samba shares on the backup server for each user, and re-map their ‘My Documents’ folder to the network drive. To achieve this, I map the drive on their machine, and then right click on ‘My Documents’, and change the location it points to.

To backup these folders, all I need to do is create a tar archive of each of the samba shares.

Finally, backing up the linux servers. As I said above, some of these are backed up by just making a tar archive of their samba shares. For servers which don’t have samba shares, I run cron jobs on each machine which creates a local backup and then copies it to the backup server through ftp.

An example of how to back up a standard debian/ubuntu apache install:

tar -zcvf /home/backups/apache_configs.tgz /etc/apache2/* -R
tar -zcvf /home/backups/website_data.tgz /var/www/* -R

An example of how to back up a MySQL server. There are 2 ways to do this. The simple way to back up all data is to make a tar archive of the entire MySQL data directory. This is fine for many small sites but may not be suitable for sites where the data is constantly changing, or the tables are using innodb (innodb doesn’t always backup 100% successfully by simply copying the data files). For a standard debian/ubuntu install, this can be achieved by:

tar -zcvf /home/backups/mysql_data.tgz /var/lib/mysql/* -R

Another method of backing up mysql is to use the ‘mysqldump’ command line utility, which will create a dump of the database as a .sql file, and can be re imported at a later date. This is seen as a more secure backup because the sql files can usually be imported across mysql versions easily, but has the downside of a slower backup, slower restore, and occasionally you will need to manually edit the dump files to get them to reimport properly. A mysqldump can be created by running a command similar to the following:

mysqldump -uUSER -pPASSWORD -hHOST DB_NAME > /home/backups/file.sql
mysqldump -uUSER -pPASSWORD -hHOST ANOTHER_DB_NAME > /home/backups/file2.sql
tar -zcvf /home/backups/databases.tgz /home/backups/*.sql

Backing up a DNS server is also very simple, and can be done in much the same way as the apache backup above, making sure that the directory including the configs, and zonefiles are included in the archive.

Mail servers can be a bit more tricky since they rely on configs for different services which are usually found in different locations. Also, all user email must be backed up. The way I achieve this is to have the shell script create a tar file for all user email, and then manually copy the important configs into a directory before creating a tar file containing them all.

Once you’ve backed up the data for the servers, you will need to move it to the backup server for safe keeping. I usually do this over FTP, by adding something similar to the following to the bottom of the backup shell script:

# FTP to backup server

HOST=’backup_ip’
USER=’backup’
PASS=’password’
cd /home/backups/

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASS
binary
quote CWD backup/SERVER_NAME/
put backup_file.tgz
quit
END_SCRIPT

That’s about all there is to know. It’s all fairly simple to do, but ensures that you have backups of all your important data. If you wish to perform incremental backups then it’s advisable to look into rsync.

Once all of the servers have been backed up, I have a script on the ‘backup server’, which does some tidying up, and copies all of the daily backups to a directory with todays date. This means I can store backups for as long as I wish without accidently overwriting them.

Bookmark It

As a sysadmin, I often have to copy large amounts of data between servers, and there are a few ways of doing this (each with their own pros and cons).

I’ll try to explain the most common methods here, and give examples and benchmarks.

1, Simple compression and encrypted transmission

This is probably the most common way to copy data, and it’s simply tar and gzip the data and then send it to another machine over scp:

machine1$ tar -zcvf backup.tgz /home/backupdata/
machine1$ scp backup.tgz root@machine2:/home/backups/

machine2$ tar -zxvf backup.tgz

The problem with this method is that disk reads and writes can be intensive because you need to read from machine1′s hard drive and then write to it during the gzip process, then read from it and write to machine2′s hard drive during the scp. This can slow down the process. Also you’re required to log in to machine2 and unzip the resulting file.

2, rsync

rsync is good for syncing data between two machines, and for incremental copying (ie if a copy is interrupted, it can resume). I won’t go into the actual details here because there are many ways of using rsync and this article is mainly aimed at one shot copying of data. It’s definitely worth checking ‘man rsync’ to see if it will be relevant for your needs though.

I tend to use rsync for backups where I want to keep an exact copy of a directory on another server, and keep it up to date. rsync is useful here because any time I run the rsync command it will only copy changes over between the servers. An example of this usage is:

machine1$ rsync -Ravr –delete –delete-after ./backupdata/ root@machine2:/home/backups/

The flags I’ve passed are: R – relative, r – recursive, a – archive mode, v – verbose.

3, Gzip and SCP in one command

This will usually perform better than option 1 simply because you aren’t writing the gzip data to the disk on machine1 before sending it to machine2. Basically, this is the commands from option 1 in one line, so the data is gzipped and piped to ssh on machine2 rather than writing to the disk on machine1 and then copying over.

machine1$ tar -zcvf – /home/backupdata/* | ssh root@machine2 “cd /home/backups/; tar -zxvf -”

4, Netcat

In theory, this should be the best solution because the data isn’t encrypted or decrypted as it is in ssh/scp (a little less cpu overhead), and there isn’t any needless IO activity as in option 1. First you will need to tell machine2 to ‘listen’ on a specific port (98765 in this example), and uncompress anything which arrives on that port:

machine2$ nc -l -p 98765 | tar -zxvf -

Then gzip and send the data from machine1 to the specified port on machine2:

machine1$ tar -zcvf – /home/backupdata/ | nc -q 1 machine2 98765

As we’re using the verbose (-v) option in tar, you will see the output on both machines. I find netcat more convenient if I need to send a lot of data from one server to another as I can just leave netcat running on machine2 and send data to it multiple times from server1 (or any other server).

5, SMB or NFS

These are also decent alternatives, and if you already have NFS or SMB shares set up then it may be worth just compressing the data and copying it to a mounted share using ‘cp’. An example would be (assuming the share name is ‘backups’).

machine1$ tar -zcvf backup.tgz /home/backupdata/
(SMB) machine1$ mount -t smbfs //machine2/backups /mnt/machine2
(NFS) machine1$ mount machine2:/home/backups /mnt/machine2
machine1$ cp backup.tgz /mnt/machine2/

Benchmarks

For my tests, I’m using 2 servers with the following basic specs:
Dual CPU Intel E5520
16gb RAM
Hardware RAID1.

I’ll be sending a few log files which total 7GB of data.

1 – The initial gzip took 114 seconds and scp took 6 seconds for a total of 120 seconds.
2 – This took a total of around 140 seconds, so was the slowest in this test. You would notice the benefits of rsync when doing incremental backups so only differences are copied over rather than redoing the whole copy.
3 – Total of 119 seconds (so not really any faster but you don’t need to log in to machine2 to unzip).
4 – Total of 109  seconds, so this is the fastest option but not by a huge margin.

Bookmark It

This is something I always forget how to do, so I’ll post it here.

When copying or connecting between Linux servers, the most straightforward solution is to use SSH or SCP. The only problem is that you’ll need to enter the password for the remote machine every time you connect, making this not very useful for scheduled scripts such as backups.

The easiest way to do this is to use public/private keys. To create a key on the local machine, do the following:

ssh-keygen -t rsa

Then just press enter at all of the prompts. This will create a keyfile called ~/.ssh/id_rsa.pub which you will need to copy to the remote machine.

ssh user@host “cat >> .ssh/authorized_keys” < ~/.ssh/id_rsa.pub

If the file ~/.ssh/authorized_keys doesn’t exist, you’ll need to create it, and ensure that it’s permissions are correct:

-rw-r–r– 1 root root 1412 2007-04-25 08:36 authorized_keys

Once this is done, you should be able to SSH and SCP to the remote machine without a password.

Obviously, from a security point of view this is a bad idea (especially if you’re doing it as root), but there are a lot of occasions where it can be useful.

Bookmark It