Setting up Auto-Backup for Digital Pictures In Linux

Well, it’s a bit like spring cleaning, and while it did involve cleaning (mostly to dig for my Ruby on Rails book… did I lend this to anyone or has it been timewarped out of my office completely?), it was mostly about cleanup. I finally got around to installing the new monitor I got at a boxing day sale, the exact twin of my current gorgeous 20.1″ flat panel LCD goodness, and in it cleaned off a lot of space on my desk, and figured it was time to implement a backup strategy I had decided to do approximately one year ago, when I got a great boxing day deal on an external USB hard drive.

The idea was that I could have a constant nightly back up of all my digital pictures and scans done to an external drive. This would give me an extra layer of protection against multiple hard drive crashes (currently they are stored on a RAID 5 array, but it’s also using EVMS and LVM, which occassionally makes me nervous, so this is mostly for my piece of mind. It also lets me grab something that I know will contain all my images if the house is burning down or something.

Read on for more…

First step, the physical setup. Pretty easy here, the drive is one of those ones with a bolt on bottom plate to let it stand upright, which I simply removed and then I could lay the drive on it’s site on top of the fileserver. A couple of zap straps to secure the extra power cable length and plugging it into the USB on the back of the server and it’s all done.

Second step was preparing the drive. It was already an NTFS partition that I was using for image backup already, so after being really careful (and umounting the current image store just in case) I re-partitioned and formatted it to use the Linux EXT3 filesystem, as it appears that Linux writing to NTFS isn’t 100% yet. This will be an extention of the fileserver itself, and will be accessed via SAMBA share by any windows machines, so that was all right.

Other than a minor hiccup as I got frustrated by the super slow format, tried to kill it, wedged the process and had to reboot, the format went well.

Next step, seeding the external drive and writing the backup script.

I had the choice of using something that does incremental backups, like rdiff-backup, or a more pure straight duplication of what’s on the hard drive at the time. I opted (at least for now), to go with the incremental backup using rdiff-backup. My reasoning was first that it gives some added protection against me accidentally deleting something I need and not noticing until after the duplication is done, and second that since generally I’ll only be adding to the image directories, the size of the backed up files will be roughly proportional to the size of the directory that it’s backing up. This is because if I was doing a lot of adding and deleting of files, those deleted files would live on in the incremental backups, at least for a time, increasing the size of the backups. As the external drive I’m using is only 160G, and the size of my backup folder is 112G, this is acceptable, at least for now.

So with that done, I created the shell of my backup script while I let rdiff-backup “seed” the external drive.

naked ~# rdiff-backup /mnt/storage/pictures/ /mnt/imagebackup/pictures/

Sadly, the drive is connected via a USB 1.0 connection, ie: painfully slow compared to USB 2.0. Very painful, so it’s still doing the initial seeding, fully a day after I started. At this rate it’ll take approximately 35 hours to seed the 113G of data.

I think it’s time to source up a USB 2.0 card from somewhere 🙂 I probably have one in my collection of spare PCI cards.

As with all backup scripts, it’s one line (the one above) wrapped in a butt-load of checks and balances, creating something far too complicated, but the owner (ie: me) still refuses to install some commercial backup solution that’ll do it all in one or two mouse clicks.

Currently, my backup script looks something like this:

#!/bin/sh
# (c) 2007 Alan Bailward <alan awy ufies dawt org>
BACKUP_TO=/mnt/imagebackup/pictures/
BACKUP_FROM=/mnt/storage/pictures/
# log status
echo "Backup script started at: $(date)"
# spin up the drive and pause to ensure it's up
echo "Ensuring drive is running"
sdparm --command=start /dev/sdc
sleep 5
# mount the backup directory
if [[ "$(df | grep /mnt/imagebackup)" = "" ]]; then
mount /mnt/imagebackup
if [ $? -ne 1 ]; then
BACKUP_MOUNTED="y"
echo "Mounted image backup directory"
else
echo "Unable to mount /mnt/imagebackup!"
echo "Mount status:"
df | grep "/imagebackup"
echo "Bailing out"
exit
fi
else
echo "Image Backup directory already mounted"
BACKUP_MOUNTED="y"
fi
# do the backup and grab the return code
nice rdiff-backup --print-statistics --no-hard-links $BACKUP_FROM $BACKUP_TO
RET1=$?
# if the backup was ok, remove older increments
if [ "$RET1" = 0 ] ; then
echo "Removing old local backups..."
nice rdiff-backup --print-statistics --remove-older-than 10D --force $BACKUP_TO
else
echo "Bad return code ($RET1) for local backup, not removing old copies!!"
fi
# sync the disks for safety sake
sleep 5
sync
echo "Synced disks"
sleep 5
# unmount /backup if we mounted it
if [[ "$BACKUP_MOUNTED" = "y" ]]; then
umount /mnt/imagebackup
if [ $? -ne 1 ]; then
echo "Unmounted image backup directory"
else
echo "Unable to umount, already unmounted?"
fi
fi
# spin down the drive and pause to ensure it's down and synced
sdparm --command=sync /dev/sdc
sleep 5
sdparm --command=stop /dev/sdc
sleep 5
# log status
echo "Image Backup script finished successfully at: $(date)"

(Fancy HTML color by colorer.sf.net)

Probably not the most beautiful thing in the world, but it seems to do the job. Basically:

• Start the drive spinning up
  
• Ensure that the script isn’t already running (actually done automagically by rdiff-backup)
  
• Mount the external drive
  
• Backup <- The important part!
  
• Remove any backups that are older than N days (in this case 10 days)
  
• Sync to ensure that all data is flushed to disk
  
• Unmount the directory
  
• Stop the disk spinning
  

Ok, script is fleshed out so test it, make sure it runs and is debugged (I can assure you that the script listed above has changed since I first started writing this article on January 1^st 🙂

Last step, make sure it runs nightly. In /etc/crontab I entered the following:

MAILTO=my email address here

0 23 * * * root /usr/local/sbin/backup-pictures.sh

This ensures any output of the script will be emailed to me, and runs the script itself at 11pm every night. I want to see the output for the first few days to ensure that everything is running properly. After I know it’s going well I’ll add a “> /dev/null” to it to suppress any informational messages that it’s going to spit out.

And that’s about it. Other than the waiting for the slow USB transfers, setting it up was relatively painless. At the end of the day I know that the most I’m going to lose is 1 days worth of images, and I can recover from any bad deletions. I also have the piece of mind that if my machine is blown from a power surge or something that nukes all it’s hard drives, I have a backup 🙂