As an avid user of the Hipstamatic iPhone app, I soon noticed that it started taking up a lot of disk space.

I had a look around the filesystem (my iPhone is jailbroken, so I can access the whole filesystem) and found that inside the Documents folder for the app, there are 2 folders which hold any photos you take (FilmRoll for the thumbnail images, and HiResPrints for the full sized ones).

When you delete a photo from the film roll, the thumbnail and full size image aren’t removed from these folders, meaning that after you’ve taken a lot of photos, you can end up with the app taking up a huge amount of space and have no way of freeing the space up.

If you’re jailbroken, you can remove old images from the filesystem fairly easily, but you need to take care not to delete ones which havent been deleted from the app.

The file HipstaData.sqlite is a sqlite database holding info about all of the photos which are still in the app, so you could correlate the records in that with what’s in the folders above to decide what to delete, but it’s a pain to do and you need a sqlite browser and basic understanding of the filesystem and databases to do.

Rather than the hassle of having to do this manually, I decided to write a shell script to do it for me. I achieved it by doing the following.

1, Locate the hipstamatic app folder on the device (I believe this is different for all installs, so I made the script find the app itself rather than hardcoding it). On a 3GS, this takes a few seconds.

DIR_NAME=`find /var/mobile/Applications/ -name “Hipstamatic.app” | cut -d \/ -f 5`

2, Kill the running hipstamatic process (this might not be needed, but it’s safer to not have the app running while you’re editing the contents of its Documents folder).

killall -v Hipstamatic

3, Loop through all files in the FilmRoll directory (I assume that every file in this directory will have a corresponding entry in HiResPrints). For each file, query the sqlite database and if the filename exists in the ZRECENTPRINT table then it’s still part of the film roll inside the app so needs to be kept.

The full script is available here (removed). To use it, copy it to your device using something like cyberduck or i-funbox, then make it executable and run it through ssh…

mv hipsta.txt hipsta.sh
chmod +x hipsta.sh
./hipsta.sh

Now a quick disclaimer! This script works for me. I’ve only used it myself on my own iPhone so whilst I think it will work correctly, I can’t guarantee that it won’t brick your device and empty your bank accounts. Please make sure you have a backup (I’d recommend AppBackup or Chronus) before you attempt to use it. Also, I’m not an expert at shell scripting so if anyone who reads this is and notices any issues then please let me know and I’ll fix it as soon as possible. I don’t want anyone to suffer any data loss because of my script!

Update 14th May 2011 – It appears that this problem has been fixed as of Hipstamatic v210, so this script isn’t needed any more!

Bookmark It

Unfortunately, PHP 5.3 deprecates a few common features (see here for more info) which causes a lot of sites to just stop working, or act strangely.

I found that the best way to move forward was to downgrade PHP back to 5.2 while we could work out what the problem was with our sites (as is always the case in a fast moving development cycle, this was months ago and we haven’t got around to it yet). I found a really useful step by step guide to do this, and within 10 minutes the server was running 5.2 using the Ubuntu Karmic sources.

The guide can be found at http://mrkandy.wordpress.com/2010/04/16/install-php-5-2-x-in-ubuntu-10-04-lucid/ and I have to give all credit to the writer of that article. None of this was my own work, I just stumbled across the post above and it fixed all my problems (I’m hoping that it’ll help out some of my readers as well).

Anyway, here are the steps. It basically does the following: Remove all PHP packages, pin all PHP packages to karmic, add karmic to sources, and then reinstall the PHP packages which were previously installed.

php_installed=`dpkg -l | grep php| awk '{print $2}' |tr "\n" " "`

# remove all php packge
sudo aptitude purge $php_installed

# use karmic for php pakage
# pin-params:  a (archive), c (components), v (version), o (origin) and l (label).
echo -e "Package: php5\nPin: release a=karmic\nPin-Priority: 991\n"  | sudo tee /etc/apt/preferences.d/php > /dev/null
apt-cache search php5-|grep php5-|awk '{print "Package:", $1,"\nPin: release a=karmic\nPin-Priority: 991\n"}'|sudo tee -a /etc/apt/preferences.d/php > /dev/null
apt-cache search -n libapache2-mod-php5 |awk '{print "Package:", $1,"\nPin: release a=karmic\nPin-Priority: 991\n"}'| sudo tee -a /etc/apt/preferences.d/php > /dev/null
echo -e "Package: php-pear\nPin: release a=karmic\nPin-Priority: 991\n"  | sudo tee -a /etc/apt/preferences.d/php > /dev/null

# add karmic to source list
egrep '(main restricted|universe|multiverse)' /etc/apt/sources.list|grep -v "#"| sed s/lucid/karmic/g | sudo tee /etc/apt/sources.list.d/karmic.list > /dev/null

# update package database (use apt-get if aptitude crash)
sudo apt-get update

# install php
sudo apt-get install $php_installed
# or sudo aptitude install -t karmic php5-cli php5-cgi //for fcgi
# or  sudo apt-get install -t karmic  libapache2-mod-php5 //for apache module

sudo aptitude hold `dpkg -l | grep php5| awk '{print $2}' |tr "\n" " "`
#done

And that’s it. As I said previously, all credit should go to the author of this post.

Bookmark It

I did the upgrade last night, and noticed this morning that my data network didn’t work at all. WiFi was working fine, but network data wasn’t at all.

After a bit of googling, I came across this post which states that it is due to the jailbreak

So, to anyone running 3.1.3 jailbroken, here’s how to upgrade

First, do a restore rather than an upgrade (click restore in itunes and it will basically do a clean install of iOS4 then resync everything) – you won’t lose any data. This should work fine and you won’t have any data issues.

If you did a normal upgrade rather than a restore, you must have a backup pre iOS4 to be able to get it working again. In this instance, simply restore the device and then restore the backup pre iOS4. For more information on how to save multiple backups at any time, check out my earlier blog post.

If you did an ‘upgrade’ and don’t have a backup pre iOS4 any more, it looks like you’re going to have to do a clean restore and not use a backup. You can get some of your data (SMS etc..) back by using the advice in the link above.

Good Luck!

Bookmark It

I’ve been using an old Vista machine as a media center for a long time. It’s not great spec, so I use CoreAVC to playback HD content. Firstly, here’s a little background on the setup:

AMD Athlon FX57 processor (2.8ghz)
2GB RAM
ATI Radeon x850
1TB external drive with all my content

I plug it into my TV using a VGA cable and stereo audio (my TV is a Samsung LCD) as the graphics card doesn’t support HDMI sound bridging. I decided that I’d upgrade to Windows 7 to hopefully get a bit more performance and get the latest version of windows media center. I know that as this is a ‘linux’ blog, I should probably go for MythTV or XBMC on linux, but as the pc is used to play back things for my wife and 2 year old daughter, I wanted it to be as easy as possible.

So, I’d expected the windows 7 set up to be easy – I know a bit about codecs etc.. and usually just install the K-Lite pack (say what you want about codec packs, I’ve heard it all before and know what I’m doing – it works!) and then CoreAVC, and set ffdshow to not decode HD streams and leave it to coreavc. This works fine in Vista, so I expected it to work just as easily in Win7 – I was wrong!

Installing windows 7 was easy, it was a clean install. I then made sure I had drivers for everything, and installed all my usual ‘media oriented’ apps.

I know that Windows 7 includes decoders for h264 ,xvid, divx etc.. so decided firstly to give it a try and see if it would work without any extra codecs. Normal avi’s (xvid) worked flawlessley and I was happy to see that Win7 media center is capable of rewinding and fastforwarding them! I went to play an HD file to be told that the file type isn’t supported!

It seems that whilst h264 does natively work, a splitter is needed to play mkv files. I installed haali for this purpose and tried again. HD playback dropped frames and was choppy, even on 720p.

Now, I know that Windows 7 likes to lock playback down to the native decoders and doesn’t play nicely with CoreAVC. I found a brilliant guide to tweak the registry to force CoreAVC to be the preferred decoder here. This is a direct cut and paste of the steps to get CoreAVC to be set as the preferred decoder:

1. Open REGEDIT as an Administrator
2. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DirectShow\Preferred
3. Right click on ‘Preferred’ and select ‘Permissions’.
4. Click on ‘Advanced’ -> ‘Owner’ tab. ‘Administrators’ , check ‘Replace Owner on subcontainers and objects’ checkbox, OK.
5. Now back to the first window – click ‘Administrators’ and check ‘Full Control’. Press OK.
6. In the left pane look for {e06d8026-db46-11cf-b4d1-00805f6cbbea}
{31435641-0000-0010-8000-00AA00389B71}.
7. Edit these entries with a double click and replace the value data entry with that for CoreAVC – {09571A4B-F1FE-4C60-9760-DE6D310C7C31}.
8. Reboot. This is important.

That’s it, now windows media player and media center will use CoreAVC as the decoder for mkv’s. Everything worked fine until I noticed that some of my media didn’t have any sound. I tested it in media player and had no sound so realised it was obviously a codec issue. I opened the file with GSpot and found that the audio codec on the silent files was DTS. As I hadn’t installed K-Lite on this install, I decided to install AC3Filter and as soon as it was installed, everything worked fine. Windows 7 also had ‘real’ support for my TV (automatically setting the desktop to the native resolution and in device manager it shows as a Samsung LCD), and has noticably better picture quality, especially on HD. I’d always found VGA to be a bit washed out on Vista and just assumed it was because I was using VGA instead of a digital connection. On Windows 7, the quality is comparable to Sky+HD over HDMI.

Now that it’s all working, I’ll recommend a few useful addons for media center:

Media Center Studio – Allows you to customise the interface, including the ability to add/remove icons.
BBC iPlayer addon – Lets you watch iPlayer content, and it looks like a native media center app.
Heatwave - I haven’t actually used this, but I’ve been recommended it, and it looks like a great weather app.
My Movies – The best movie catalogue app for media center, bar none!

Bookmark It

Also, the fact that iTunes backs up as soon as you plug the phone in is annoying. If you plug in to restore something you’ve removed and aren’t quick enough to ‘slide to cancel’, a new backup will be created and overwrite your old one.

It’s actually fairly easy to keep old backups – it’s not something you can do automatically, but is worth knowing. All you need to do is make a copy of your current back up, and rename it to something else in the filesystem. You can also change the description of it so that it shows a more friendly name when you attempt a restore.

First, browse to the folder containing the most recent backup:

Windows XP:
C:\Documents and Settings\USERNAME\Application Data\Apple Computer\MobileSync\Backup

Windows 7 and Vista:
C:\Users\YOURUSERNAME\AppData\Roaming\Apple Computer\MobileSync\Backup

Mac OSX:
~/Library/Application Support/MobileSync/Backup/

Inside this folder, you will see another folder with a long string of numbers and letters, something like…

To keep this backup, all you need to do is make a copy of the folder, and rename it to something like ’20100311 – 8deabb43……’.

Next, enter this folder, and open the file ‘Info.plist’ in your favourite text editor. To give the backup a meaningful name, you need to find the key ‘Display Name’ and change the <string> tag to anything you want.

Once this is done, open up iTunes and plug your phone in. Now, in the future when you want to restore one of your previously saved backups, just right click the iPhone in iTunes and click ‘Restore from Backup’.
Now you will notice that you have multiple backups to choose from, and can restore back to any of the backups which you’ve saved. Note that the display name I edited above is the one which shows in the drop down list.

Apologies for this last one being a photo rather than a screenshot, but I’m writing this on a macbook pro (running windows 7!) and there’s no print screen key, and when trying to use snipping tool, the drop down box closes when it loses focus.

There you go, that’s all you need to be able to manage multiple backups on an iPhone. I assume this will work for iPods as well but I don’t have one to test it on.

Bookmark It

I don’t like iTunes, and generally prefer to manage my music manually in explorer, and use winamp to play songs/albums. I know of alternatives like SharePod which allow you to manage music on your iPhone or iPod, but you still need to use iTunes for things like backing up etc..

The way I get around this is fairly simple, I still do everything in iTunes but manage my music manually. Firstly, I use a great free app called mp3tag to make sure that the music has the correct id3 tags, album art etc..

Then it’s just a case of putting the music I want on my iPhone into iTunes, and nothing else. First, set iTunes to sync all music with the iPhone, and then simply drag and drop folders or individual songs from explorer into iTunes and sync with the iPhone. To remove music, just delete it from iTunes and sync again.

I find this an easy way to make sure that I don’t have to use iTunes to actually listen to music or manage my collection, but I can keep my phone happy by using iTunes to sync music to it. It’s much easier than having all your music in iTunes and telling it to only sync certain artists/playlists.

I have some friends who either use the ‘selective sync’ option, or just use alternatives like SharePod or Media Monkey to copy music, but after I show them the simplicity of this method, they make the switch. Hopefully a couple of others will find this useful.

Bookmark It

First, a little bit of background. My network consists of a large number of linux boxes (mostly debian and ubuntu), and various windows servers running anything from XP, to 2000 and 2003. I have one dedicated linux machine which is a backup server running debian. Some backups are ‘put’ onto the backup server at predefined intervals over samba, nfs, and ftp, and others are ‘retrieved’ by scripts on the backup server. Every day these backups are copied to 2 usb drives (both encrypted with truecrypt) and taken off site.

As rebuilding a server is a fairly quick task (especially ubuntu, which can be installed and configured from scratch in under 20 minutes), I only back up the necessary configuration files and data, as doing full images of every server would need a huge amount of storage.

Firstly, the backup server itself needs to be configured to run samba, nfs, and ftpd. Setting these up is fairly trivial and won’t be covered here but I will write guides to setting these up in the very near future. As a side note, we have recently switched from a backup ‘server’ to a QNAS network storage device. As this runs linux it’s more than capable of running the necessary cron jobs for the ‘get’ server backups. Plus it also comes preconfigured with samba, nfs, timecapsule, and ftp.

So, on to the backing up. All of my windows machines are part of the ‘get’ backups, where the backup server itself connects to the client and fetches the backup. To achieve this, I simply create network shares on the client machines, then do something similar to the following:

mount -t smbfs //svr2/share /mnt/backup
tar -zcvf /home/backups/svr2.tgz /mnt/backup/* -R
umount /mnt/backup

I have this saved as a shell script which runs every night to fetch a backup of the necessary data from the windows machine. Note that this method can be used for any windows server where the necessary data is shared on the network, or any linux server where the data is shared over samba. I have a number of these shell scripts, all of which connect to different servers and save their vital data.

Second, onto backing up our client machines which are all running windows XP. We’ve ‘trained’ all our users to store their important documents inside the ‘My Documents’ folder. I then have samba shares on the backup server for each user, and re-map their ‘My Documents’ folder to the network drive. To achieve this, I map the drive on their machine, and then right click on ‘My Documents’, and change the location it points to.

To backup these folders, all I need to do is create a tar archive of each of the samba shares.

Finally, backing up the linux servers. As I said above, some of these are backed up by just making a tar archive of their samba shares. For servers which don’t have samba shares, I run cron jobs on each machine which creates a local backup and then copies it to the backup server through ftp.

An example of how to back up a standard debian/ubuntu apache install:

tar -zcvf /home/backups/apache_configs.tgz /etc/apache2/* -R
tar -zcvf /home/backups/website_data.tgz /var/www/* -R

An example of how to back up a MySQL server. There are 2 ways to do this. The simple way to back up all data is to make a tar archive of the entire MySQL data directory. This is fine for many small sites but may not be suitable for sites where the data is constantly changing, or the tables are using innodb (innodb doesn’t always backup 100% successfully by simply copying the data files). For a standard debian/ubuntu install, this can be achieved by:

tar -zcvf /home/backups/mysql_data.tgz /var/lib/mysql/* -R

Another method of backing up mysql is to use the ‘mysqldump’ command line utility, which will create a dump of the database as a .sql file, and can be re imported at a later date. This is seen as a more secure backup because the sql files can usually be imported across mysql versions easily, but has the downside of a slower backup, slower restore, and occasionally you will need to manually edit the dump files to get them to reimport properly. A mysqldump can be created by running a command similar to the following:

mysqldump -uUSER -pPASSWORD -hHOST DB_NAME > /home/backups/file.sql
mysqldump -uUSER -pPASSWORD -hHOST ANOTHER_DB_NAME > /home/backups/file2.sql
tar -zcvf /home/backups/databases.tgz /home/backups/*.sql

Backing up a DNS server is also very simple, and can be done in much the same way as the apache backup above, making sure that the directory including the configs, and zonefiles are included in the archive.

Mail servers can be a bit more tricky since they rely on configs for different services which are usually found in different locations. Also, all user email must be backed up. The way I achieve this is to have the shell script create a tar file for all user email, and then manually copy the important configs into a directory before creating a tar file containing them all.

Once you’ve backed up the data for the servers, you will need to move it to the backup server for safe keeping. I usually do this over FTP, by adding something similar to the following to the bottom of the backup shell script:

# FTP to backup server

HOST=’backup_ip’
USER=’backup’
PASS=’password’
cd /home/backups/

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASS
binary
quote CWD backup/SERVER_NAME/
put backup_file.tgz
quit
END_SCRIPT

That’s about all there is to know. It’s all fairly simple to do, but ensures that you have backups of all your important data. If you wish to perform incremental backups then it’s advisable to look into rsync.

Once all of the servers have been backed up, I have a script on the ‘backup server’, which does some tidying up, and copies all of the daily backups to a directory with todays date. This means I can store backups for as long as I wish without accidently overwriting them.

Bookmark It

I’ll try to explain the most common methods here, and give examples and benchmarks.

1, Simple compression and encrypted transmission

This is probably the most common way to copy data, and it’s simply tar and gzip the data and then send it to another machine over scp:

machine1$ tar -zcvf backup.tgz /home/backupdata/
machine1$ scp backup.tgz root@machine2:/home/backups/

machine2$ tar -zxvf backup.tgz

The problem with this method is that disk reads and writes can be intensive because you need to read from machine1′s hard drive and then write to it during the gzip process, then read from it and write to machine2′s hard drive during the scp. This can slow down the process. Also you’re required to log in to machine2 and unzip the resulting file.

2, rsync

rsync is good for syncing data between two machines, and for incremental copying (ie if a copy is interrupted, it can resume). I won’t go into the actual details here because there are many ways of using rsync and this article is mainly aimed at one shot copying of data. It’s definitely worth checking ‘man rsync’ to see if it will be relevant for your needs though.

I tend to use rsync for backups where I want to keep an exact copy of a directory on another server, and keep it up to date. rsync is useful here because any time I run the rsync command it will only copy changes over between the servers. An example of this usage is:

machine1$ rsync -Ravr –delete –delete-after ./backupdata/ root@machine2:/home/backups/

The flags I’ve passed are: R – relative, r – recursive, a – archive mode, v – verbose.

3, Gzip and SCP in one command

This will usually perform better than option 1 simply because you aren’t writing the gzip data to the disk on machine1 before sending it to machine2. Basically, this is the commands from option 1 in one line, so the data is gzipped and piped to ssh on machine2 rather than writing to the disk on machine1 and then copying over.

machine1$ tar -zcvf – /home/backupdata/* | ssh root@machine2 “cd /home/backups/; tar -zxvf -”

4, Netcat

In theory, this should be the best solution because the data isn’t encrypted or decrypted as it is in ssh/scp (a little less cpu overhead), and there isn’t any needless IO activity as in option 1. First you will need to tell machine2 to ‘listen’ on a specific port (98765 in this example), and uncompress anything which arrives on that port:

machine2$ nc -l -p 98765 | tar -zxvf -

Then gzip and send the data from machine1 to the specified port on machine2:

machine1$ tar -zcvf – /home/backupdata/ | nc -q 1 machine2 98765

As we’re using the verbose (-v) option in tar, you will see the output on both machines. I find netcat more convenient if I need to send a lot of data from one server to another as I can just leave netcat running on machine2 and send data to it multiple times from server1 (or any other server).

5, SMB or NFS

These are also decent alternatives, and if you already have NFS or SMB shares set up then it may be worth just compressing the data and copying it to a mounted share using ‘cp’. An example would be (assuming the share name is ‘backups’).

machine1$ tar -zcvf backup.tgz /home/backupdata/
(SMB) machine1$ mount -t smbfs //machine2/backups /mnt/machine2
(NFS) machine1$ mount machine2:/home/backups /mnt/machine2
machine1$ cp backup.tgz /mnt/machine2/

Benchmarks

For my tests, I’m using 2 servers with the following basic specs:
Dual CPU Intel E5520
16gb RAM
Hardware RAID1.

I’ll be sending a few log files which total 7GB of data.

1 – The initial gzip took 114 seconds and scp took 6 seconds for a total of 120 seconds.
2 – This took a total of around 140 seconds, so was the slowest in this test. You would notice the benefits of rsync when doing incremental backups so only differences are copied over rather than redoing the whole copy.
3 – Total of 119 seconds (so not really any faster but you don’t need to log in to machine2 to unzip).
4 – Total of 109  seconds, so this is the fastest option but not by a huge margin.

Bookmark It

First, you’ll need to go into the BIOS and change the display to ‘custom’.

The updates will be done using ‘ipmitool’, so we need to install it and add the necessary modules to /etc/modules

apt-get install ipmitool
echo “ipmi_devintf” >> /etc/modules
echo “ipmi_si” >> /etc/modules
echo “ipmi_msghandler” >> /etc/modules

You should now be able to view the contents of the LCD by running:

ipmitool delloem lcd

I didn’t get anywhere trying to write new values to the LCD using the delloem command, so resorted to sending raw commands to ipmitool.

I wrote the following php script, which can be used by first making it executable, then running it from the shell

chmod +x updateLCD.php
./updateLCD.php “test string”

The php script is pretty simple and can be found here

It’s also fairly easy to extend this script to update the LCD to display stats such as disk usage, server load, cpu usage etc..

Bookmark It

First, add the following line to /etc/apt/sources.list:

deb ftp://ftp.sara.nl/pub/sara-omsa dell sara

Then, if you try ‘apt-get update’, you will get a key error, so do the following:

wget -O – http://ftp.sara.nl/debian_sara.asc | apt-key add  -
apt-get update
apt-get install dellomsa

To use omreport, you need to install libstdc++5 to fix the dependancy problems:

For 32 bit systems:
cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_i386.deb
dpkg -i libstdc++5_3.3.6-17ubuntu1_i386.deb

For 64 bit systems (you will need the 32 bit version as well):
cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_i386.deb
wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_amd64.deb
dpkg -i libstdc++5_3.3.6-17ubuntu1_amd64.deb
dpkg-deb -x libstdc++5_3.3.6-17ubuntu1_i386.deb ./extracts
cp ./extracts/usr/lib/* /usr/lib32/
rm -rf ./extracts

Now start the dataeng service:

/etc/init.d/dataeng start

And you should be able to view information about the RAID using omreport:

omreport system summary
omreport storage connector controller=0
omreport storage pdisk controller=0
omreport storage vdisk controller=0

You can then make a simple shell/php/perl script cron which checks the status of the RAID every x minutes and notifies you of any problems.

Bookmark It