A Million Monkeys

Just a quick post explaining a problem I found when upgrading my jailbroken (using spirit) 3.1.3 iPhone 3GS to iOS4.

I did the upgrade last night, and noticed this morning that my data network didn’t work at all. WiFi was working fine, but network data wasn’t at all.

After a bit of googling, I came across this post which states that it is due to the jailbreak

So, to anyone running 3.1.3 jailbroken, here’s how to upgrade

First, do a restore rather than an upgrade (click restore in itunes and it will basically do a clean install of iOS4 then resync everything) – you won’t lose any data. This should work fine and you won’t have any data issues.

If you did a normal upgrade rather than a restore, you must have a backup pre iOS4 to be able to get it working again. In this instance, simply restore the device and then restore the backup pre iOS4. For more information on how to save multiple backups at any time, check out my earlier blog post.

If you did an ‘upgrade’ and don’t have a backup pre iOS4 any more, it looks like you’re going to have to do a clean restore and not use a backup. You can get some of your data (SMS etc..) back by using the advice in the link above.

Good Luck!

Bookmark It

This is a guide to setting up Windows 7 Media Center, and configuring it to use 3rd party codecs such as CoreAVC and AC3Filter.

I’ve been using an old Vista machine as a media center for a long time. It’s not great spec, so I use CoreAVC to playback HD content. Firstly, here’s a little background on the setup:

AMD Athlon FX57 processor (2.8ghz)
2GB RAM
ATI Radeon x850
1TB external drive with all my content

I plug it into my TV using a VGA cable and stereo audio (my TV is a Samsung LCD) as the graphics card doesn’t support HDMI sound bridging. I decided that I’d upgrade to Windows 7 to hopefully get a bit more performance and get the latest version of windows media center. I know that as this is a ‘linux’ blog, I should probably go for MythTV or XBMC on linux, but as the pc is used to play back things for my wife and 2 year old daughter, I wanted it to be as easy as possible.

So, I’d expected the windows 7 set up to be easy – I know a bit about codecs etc.. and usually just install the K-Lite pack (say what you want about codec packs, I’ve heard it all before and know what I’m doing – it works!) and then CoreAVC, and set ffdshow to not decode HD streams and leave it to coreavc. This works fine in Vista, so I expected it to work just as easily in Win7 – I was wrong!

Installing windows 7 was easy, it was a clean install. I then made sure I had drivers for everything, and installed all my usual ‘media oriented’ apps.

I know that Windows 7 includes decoders for h264 ,xvid, divx etc.. so decided firstly to give it a try and see if it would work without any extra codecs. Normal avi’s (xvid) worked flawlessley and I was happy to see that Win7 media center is capable of rewinding and fastforwarding them! I went to play an HD file to be told that the file type isn’t supported!

It seems that whilst h264 does natively work, a splitter is needed to play mkv files. I installed haali for this purpose and tried again. HD playback dropped frames and was choppy, even on 720p.

Now, I know that Windows 7 likes to lock playback down to the native decoders and doesn’t play nicely with CoreAVC. I found a brilliant guide to tweak the registry to force CoreAVC to be the preferred decoder here. This is a direct cut and paste of the steps to get CoreAVC to be set as the preferred decoder:

1. Open REGEDIT as an Administrator
2. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DirectShow\Preferred
3. Right click on ‘Preferred’ and select ‘Permissions’.
4. Click on ‘Advanced’ -> ‘Owner’ tab. ‘Administrators’ , check ‘Replace Owner on subcontainers and objects’ checkbox, OK.
5. Now back to the first window – click ‘Administrators’ and check ‘Full Control’. Press OK.
6. In the left pane look for {e06d8026-db46-11cf-b4d1-00805f6cbbea}
{31435641-0000-0010-8000-00AA00389B71}.
7. Edit these entries with a double click and replace the value data entry with that for CoreAVC – {09571A4B-F1FE-4C60-9760-DE6D310C7C31}.
8. Reboot. This is important.

That’s it, now windows media player and media center will use CoreAVC as the decoder for mkv’s. Everything worked fine until I noticed that some of my media didn’t have any sound. I tested it in media player and had no sound so realised it was obviously a codec issue. I opened the file with GSpot and found that the audio codec on the silent files was DTS. As I hadn’t installed K-Lite on this install, I decided to install AC3Filter and as soon as it was installed, everything worked fine. Windows 7 also had ‘real’ support for my TV (automatically setting the desktop to the native resolution and in device manager it shows as a Samsung LCD), and has noticably better picture quality, especially on HD. I’d always found VGA to be a bit washed out on Vista and just assumed it was because I was using VGA instead of a digital connection. On Windows 7, the quality is comparable to Sky+HD over HDMI.

Now that it’s all working, I’ll recommend a few useful addons for media center:

Media Center Studio – Allows you to customise the interface, including the ability to add/remove icons.
BBC iPlayer addon – Lets you watch iPlayer content, and it looks like a native media center app.
Heatwave - I haven’t actually used this, but I’ve been recommended it, and it looks like a great weather app.
My Movies – The best movie catalogue app for media center, bar none!

Bookmark It

One thing I found slightly irritating about the way iTunes backs up iPhones is the fact that it only keeps the most recent backup. This means that if you accidently delete a text, photo, or something else that’s backed up and only realise after you’ve backed up again, it’s too late.

Also, the fact that iTunes backs up as soon as you plug the phone in is annoying. If you plug in to restore something you’ve removed and aren’t quick enough to ‘slide to cancel’, a new backup will be created and overwrite your old one.

It’s actually fairly easy to keep old backups – it’s not something you can do automatically, but is worth knowing. All you need to do is make a copy of your current back up, and rename it to something else in the filesystem. You can also change the description of it so that it shows a more friendly name when you attempt a restore.

First, browse to the folder containing the most recent backup:

Windows XP:
C:\Documents and Settings\USERNAME\Application Data\Apple Computer\MobileSync\Backup

Windows 7 and Vista:
C:\Users\YOURUSERNAME\AppData\Roaming\Apple Computer\MobileSync\Backup

Mac OSX:
~/Library/Application Support/MobileSync/Backup/

Inside this folder, you will see another folder with a long string of numbers and letters, something like…

To keep this backup, all you need to do is make a copy of the folder, and rename it to something like ’20100311 – 8deabb43……’.

Next, enter this folder, and open the file ‘Info.plist’ in your favourite text editor. To give the backup a meaningful name, you need to find the key ‘Display Name’ and change the <string> tag to anything you want.

Once this is done, open up iTunes and plug your phone in. Now, in the future when you want to restore one of your previously saved backups, just right click the iPhone in iTunes and click ‘Restore from Backup’.
Now you will notice that you have multiple backups to choose from, and can restore back to any of the backups which you’ve saved. Note that the display name I edited above is the one which shows in the drop down list.

Apologies for this last one being a photo rather than a screenshot, but I’m writing this on a macbook pro (running windows 7!) and there’s no print screen key, and when trying to use snipping tool, the drop down box closes when it loses focus.

There you go, that’s all you need to be able to manage multiple backups on an iPhone. I assume this will work for iPods as well but I don’t have one to test it on.

Bookmark It

I recently changed from Windows Mobile to an iPhone, after getting tired of Windows Mobiles sluggishness and general lack of innovation.

I don’t like iTunes, and generally prefer to manage my music manually in explorer, and use winamp to play songs/albums. I know of alternatives like SharePod which allow you to manage music on your iPhone or iPod, but you still need to use iTunes for things like backing up etc..

The way I get around this is fairly simple, I still do everything in iTunes but manage my music manually. Firstly, I use a great free app called mp3tag to make sure that the music has the correct id3 tags, album art etc..

Then it’s just a case of putting the music I want on my iPhone into iTunes, and nothing else. First, set iTunes to sync all music with the iPhone, and then simply drag and drop folders or individual songs from explorer into iTunes and sync with the iPhone. To remove music, just delete it from iTunes and sync again.

I find this an easy way to make sure that I don’t have to use iTunes to actually listen to music or manage my collection, but I can keep my phone happy by using iTunes to sync music to it. It’s much easier than having all your music in iTunes and telling it to only sync certain artists/playlists.

I have some friends who either use the ‘selective sync’ option, or just use alternatives like SharePod or Media Monkey to copy music, but after I show them the simplicity of this method, they make the switch. Hopefully a couple of others will find this useful.

Bookmark It

Having been bitten a few times by not backing data up, I’m now pretty anal about making sure my servers are backed up.

First, a little bit of background. My network consists of a large number of linux boxes (mostly debian and ubuntu), and various windows servers running anything from XP, to 2000 and 2003. I have one dedicated linux machine which is a backup server running debian. Some backups are ‘put’ onto the backup server at predefined intervals over samba, nfs, and ftp, and others are ‘retrieved’ by scripts on the backup server. Every day these backups are copied to 2 usb drives (both encrypted with truecrypt) and taken off site.

As rebuilding a server is a fairly quick task (especially ubuntu, which can be installed and configured from scratch in under 20 minutes), I only back up the necessary configuration files and data, as doing full images of every server would need a huge amount of storage.

Firstly, the backup server itself needs to be configured to run samba, nfs, and ftpd. Setting these up is fairly trivial and won’t be covered here but I will write guides to setting these up in the very near future. As a side note, we have recently switched from a backup ‘server’ to a QNAS network storage device. As this runs linux it’s more than capable of running the necessary cron jobs for the ‘get’ server backups. Plus it also comes preconfigured with samba, nfs, timecapsule, and ftp.

So, on to the backing up. All of my windows machines are part of the ‘get’ backups, where the backup server itself connects to the client and fetches the backup. To achieve this, I simply create network shares on the client machines, then do something similar to the following:

mount -t smbfs //svr2/share /mnt/backup
tar -zcvf /home/backups/svr2.tgz /mnt/backup/* -R
umount /mnt/backup

I have this saved as a shell script which runs every night to fetch a backup of the necessary data from the windows machine. Note that this method can be used for any windows server where the necessary data is shared on the network, or any linux server where the data is shared over samba. I have a number of these shell scripts, all of which connect to different servers and save their vital data.

Second, onto backing up our client machines which are all running windows XP. We’ve ‘trained’ all our users to store their important documents inside the ‘My Documents’ folder. I then have samba shares on the backup server for each user, and re-map their ‘My Documents’ folder to the network drive. To achieve this, I map the drive on their machine, and then right click on ‘My Documents’, and change the location it points to.

To backup these folders, all I need to do is create a tar archive of each of the samba shares.

Finally, backing up the linux servers. As I said above, some of these are backed up by just making a tar archive of their samba shares. For servers which don’t have samba shares, I run cron jobs on each machine which creates a local backup and then copies it to the backup server through ftp.

An example of how to back up a standard debian/ubuntu apache install:

tar -zcvf /home/backups/apache_configs.tgz /etc/apache2/* -R
tar -zcvf /home/backups/website_data.tgz /var/www/* -R

An example of how to back up a MySQL server. There are 2 ways to do this. The simple way to back up all data is to make a tar archive of the entire MySQL data directory. This is fine for many small sites but may not be suitable for sites where the data is constantly changing, or the tables are using innodb (innodb doesn’t always backup 100% successfully by simply copying the data files). For a standard debian/ubuntu install, this can be achieved by:

tar -zcvf /home/backups/mysql_data.tgz /var/lib/mysql/* -R

Another method of backing up mysql is to use the ‘mysqldump’ command line utility, which will create a dump of the database as a .sql file, and can be re imported at a later date. This is seen as a more secure backup because the sql files can usually be imported across mysql versions easily, but has the downside of a slower backup, slower restore, and occasionally you will need to manually edit the dump files to get them to reimport properly. A mysqldump can be created by running a command similar to the following:

mysqldump -uUSER -pPASSWORD -hHOST DB_NAME > /home/backups/file.sql
mysqldump -uUSER -pPASSWORD -hHOST ANOTHER_DB_NAME > /home/backups/file2.sql
tar -zcvf /home/backups/databases.tgz /home/backups/*.sql

Backing up a DNS server is also very simple, and can be done in much the same way as the apache backup above, making sure that the directory including the configs, and zonefiles are included in the archive.

Mail servers can be a bit more tricky since they rely on configs for different services which are usually found in different locations. Also, all user email must be backed up. The way I achieve this is to have the shell script create a tar file for all user email, and then manually copy the important configs into a directory before creating a tar file containing them all.

Once you’ve backed up the data for the servers, you will need to move it to the backup server for safe keeping. I usually do this over FTP, by adding something similar to the following to the bottom of the backup shell script:

# FTP to backup server

HOST=’backup_ip’
USER=’backup’
PASS=’password’
cd /home/backups/

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASS
binary
quote CWD backup/SERVER_NAME/
put backup_file.tgz
quit
END_SCRIPT

That’s about all there is to know. It’s all fairly simple to do, but ensures that you have backups of all your important data. If you wish to perform incremental backups then it’s advisable to look into rsync.

Once all of the servers have been backed up, I have a script on the ‘backup server’, which does some tidying up, and copies all of the daily backups to a directory with todays date. This means I can store backups for as long as I wish without accidently overwriting them.

Bookmark It

As a sysadmin, I often have to copy large amounts of data between servers, and there are a few ways of doing this (each with their own pros and cons).

I’ll try to explain the most common methods here, and give examples and benchmarks.

1, Simple compression and encrypted transmission

This is probably the most common way to copy data, and it’s simply tar and gzip the data and then send it to another machine over scp:

machine1$ tar -zcvf backup.tgz /home/backupdata/
machine1$ scp backup.tgz root@machine2:/home/backups/

machine2$ tar -zxvf backup.tgz

The problem with this method is that disk reads and writes can be intensive because you need to read from machine1′s hard drive and then write to it during the gzip process, then read from it and write to machine2′s hard drive during the scp. This can slow down the process. Also you’re required to log in to machine2 and unzip the resulting file.

2, rsync

rsync is good for syncing data between two machines, and for incremental copying (ie if a copy is interrupted, it can resume). I won’t go into the actual details here because there are many ways of using rsync and this article is mainly aimed at one shot copying of data. It’s definitely worth checking ‘man rsync’ to see if it will be relevant for your needs though.

I tend to use rsync for backups where I want to keep an exact copy of a directory on another server, and keep it up to date. rsync is useful here because any time I run the rsync command it will only copy changes over between the servers. An example of this usage is:

machine1$ rsync -Ravr –delete –delete-after ./backupdata/ root@machine2:/home/backups/

The flags I’ve passed are: R – relative, r – recursive, a – archive mode, v – verbose.

3, Gzip and SCP in one command

This will usually perform better than option 1 simply because you aren’t writing the gzip data to the disk on machine1 before sending it to machine2. Basically, this is the commands from option 1 in one line, so the data is gzipped and piped to ssh on machine2 rather than writing to the disk on machine1 and then copying over.

machine1$ tar -zcvf – /home/backupdata/* | ssh root@machine2 “cd /home/backups/; tar -zxvf -”

4, Netcat

In theory, this should be the best solution because the data isn’t encrypted or decrypted as it is in ssh/scp (a little less cpu overhead), and there isn’t any needless IO activity as in option 1. First you will need to tell machine2 to ‘listen’ on a specific port (98765 in this example), and uncompress anything which arrives on that port:

machine2$ nc -l -p 98765 | tar -zxvf -

Then gzip and send the data from machine1 to the specified port on machine2:

machine1$ tar -zcvf – /home/backupdata/ | nc -q 1 machine2 98765

As we’re using the verbose (-v) option in tar, you will see the output on both machines. I find netcat more convenient if I need to send a lot of data from one server to another as I can just leave netcat running on machine2 and send data to it multiple times from server1 (or any other server).

5, SMB or NFS

These are also decent alternatives, and if you already have NFS or SMB shares set up then it may be worth just compressing the data and copying it to a mounted share using ‘cp’. An example would be (assuming the share name is ‘backups’).

machine1$ tar -zcvf backup.tgz /home/backupdata/
(SMB) machine1$ mount -t smbfs //machine2/backups /mnt/machine2
(NFS) machine1$ mount machine2:/home/backups /mnt/machine2
machine1$ cp backup.tgz /mnt/machine2/

Benchmarks

For my tests, I’m using 2 servers with the following basic specs:
Dual CPU Intel E5520
16gb RAM
Hardware RAID1.

I’ll be sending a few log files which total 7GB of data.

1 – The initial gzip took 114 seconds and scp took 6 seconds for a total of 120 seconds.
2 – This took a total of around 140 seconds, so was the slowest in this test. You would notice the benefits of rsync when doing incremental backups so only differences are copied over rather than redoing the whole copy.
3 – Total of 119 seconds (so not really any faster but you don’t need to log in to machine2 to unzip).
4 – Total of 109  seconds, so this is the fastest option but not by a huge margin.

Bookmark It

We recently got some Dell Poweredge R410 servers, and I thought it would be quite useful to change the LCD text to something meaningful. Unfortunately, this turned out to be a bit more of a pain than I was expecting, but I have it working, and a PHP script to update the display.

First, you’ll need to go into the BIOS and change the display to ‘custom’.

The updates will be done using ‘ipmitool’, so we need to install it and add the necessary modules to /etc/modules

apt-get install ipmitool
echo “ipmi_devintf” >> /etc/modules
echo “ipmi_si” >> /etc/modules
echo “ipmi_msghandler” >> /etc/modules

You should now be able to view the contents of the LCD by running:

ipmitool delloem lcd

I didn’t get anywhere trying to write new values to the LCD using the delloem command, so resorted to sending raw commands to ipmitool.

I wrote the following php script, which can be used by first making it executable, then running it from the shell

chmod +x updateLCD.php
./updateLCD.php “test string”

The php script is pretty simple and can be found here

It’s also fairly easy to extend this script to update the LCD to display stats such as disk usage, server load, cpu usage etc..

Bookmark It

This is something which took me quite a while to figure out, but it can be done fairly easily in a few minutes. This guide will install Dell OMSA and use omreport to view the status of the RAID.

First, add the following line to /etc/apt/sources.list:

deb ftp://ftp.sara.nl/pub/sara-omsa dell sara

Then, if you try ‘apt-get update’, you will get a key error, so do the following:

wget -O – http://ftp.sara.nl/debian_sara.asc | apt-key add  -
apt-get update
apt-get install dellomsa

To use omreport, you need to install libstdc++5 to fix the dependancy problems:

For 32 bit systems:
cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_i386.deb
dpkg -i libstdc++5_3.3.6-17ubuntu1_i386.deb

For 64 bit systems (you will need the 32 bit version as well):
cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_i386.deb
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_amd64.deb
dpkg -i libstdc++5_3.3.6-17ubuntu1_amd64.deb
dpkg-deb -x libstdc++5_3.3.6-17ubuntu1_i386.deb ./extracts
cp ./extracts/usr/lib/* /usr/lib32/
rm -rf ./extracts

Now start the dataeng service:

/etc/init.d/dataeng start

And you should be able to view information about the RAID using omreport:

omreport system summary
omreport storage connector controller=0
omreport storage pdisk controller=0
omreport storage vdisk controller=0

You can then make a simple shell/php/perl script cron which checks the status of the RAID every x minutes and notifies you of any problems.

Bookmark It

If you don’t know what memcache is, the official site says the following:

“memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.”

Basically, what this means is that memcached is a daemon running on a server which allows you to save and retrieve variables in the server memory. This is primarily used to ease database load on websites and applications. The logic for doing something like this is…

- Before running a query, check if the result is available in memcache.
- If the result is in memcache, return the cached result.
- If not, run the query on the database, and store the result in memcache.

To install memcached, simply run the following command:

sudo apt-get install memcached

Once it’s installed, edit /etc/memcached.conf and change the line beginning ‘-m’ which is the amount of memory (in megabytes) to allocate to the server. You can also change the IP address that the server listens on  in the line beginning ‘-l’.

Now restart the daemon by running

/etc/init.d/memcached restart

Now you have memcache set up and running on your server. A great feature of memcached is that you can easily cluster servers. If you want to do this, simply install memcache on your other servers before continuing.

At this point, I’d recommend downloading memcache.php which is a php script showing you a lot of useful information about your memcached servers. Once downloaded, put it in a web facing directory on your server, and modify the $MEMCACHE_SERVERS array with your server addresses.

Using memcache in php scripts is very easy. There are either procedural, or object oriented functions already available. Here is an example of a script which will store a simple variable and then retrieve it and display it.

<?php
$memcache = new Memcache;

$memcache->addServer(’10.2.0.245′, 11211) or die (“Could not connect”);
$memcache->addServer(’10.2.0.249′, 11211) or die (“Could not connect”);

$memcache->set(‘mytestvariable’, “this is the data in my test variable”, false, 60) or die (“Unable to save the data to the server”);
echo “Data has been stored in the cache<br />”;

$result = $memcache->get(‘mytestvariable’);
echo “Retrieved data from the server:<br/>”;

var_dump($result);

?>

Obviously if you only have one server then only use one addServer line.

The script above will store the test variable in the cache for 60 seconds. If you drop the timeout down to something like 1 second, and then sleep for 3 seconds before attempting to get the data, you will find that the data has expired and nothing will be returned.

It’s also useful to know that you can store anything which can be serialized in memcache. This means it’s safe to store things like arrays in the cache without having to ‘flatten’ the data beforehand.

Obviously how and where you implement a cache is entirely dependant on how your system works. If you have a lot of intense database usage then you will find that even caching with a short timeout will vastly reduce the amount of database queries. You should find that memcache is considerably faster than normal database access, and even faster than the mysql query cache. You also have total control over what data is cached, and how long it is cached for before your queries fall back to checking a database.

Some further reading:
Facebook engineering blog – How facebook use memcache
Fotolog case study

Also, it’s worth checking out the PHP manual page.

Bookmark It

I’m posting this mainly because I forget the exact syntax, but it might be useful for others.

This applies to all services on all ports, but for this example I’ll use the standard IMAP port. For example, if I have a mail server (listening for IMAP connections on port 143) which isn’t accessible from outside it’s LAN, but I have another server on the LAN which is accessible from outside – I could forward the port on the open server to give myself access to the mail server.

Here’s the example setup:
Mail server: 10.1.0.2, listening on port 143 which isn’t accessible from outside.
Other server 10.1.0.3, accessible from outside.

By running these 2 commands, I can forward all requests to 10.1.0.3:143 to 10.1.0.2:143 and connect to the IMAP server from anywhere:

iptables -t nat -I PREROUTING 1 -i eth0 -p tcp –dport 143 -j DNAT –to-destination 10.1.0.2:143
iptables -t nat -I POSTROUTING 1 -o eth0 -p tcp -d 10.1.0.2 –dport 143 -j SNAT –to-source 10.1.0.3

For this to work, you will need to have IP forwarding enabled, which can be done temporarily (which will reset on reboot), or permanently:

echo 1 > /proc/sys/net/ipv4/ip_forward
(temporary)

vim /etc/sysctl.conf
// uncomment the line ‘net.ipv4.ip_forward=1′
(permanent)

Now if you telnet to 10.1.0.3:143, the connection should be forwarded to 10.1.0.2:143

Bookmark It