Tuttology

I finally set up (again) an automatic off-site incremental backup of my dreamhost account.

Last time, after a disk crash, I set up a new linux box and restored everything from the backup; indeed almost everything. I forgot to restore just one thing: the backup scripts

The idea is to sync the remote server files with a local directory and then make an incremental backup on the local computer. Subsequently copy back the archive, having so an historically memory of every changes both on local and remote machine.

Thanks to fullo for checking this stuff.

Step 0: Directory and users setup

Let’s say you want to backup stuff located on foo.com and the user assigned by provider on the remote server is user.

I created a backup user of my home linux box; this isn’t required, but I prefer to not mix my own stuff and the backup files.

So the directories layout is:

~/foo.com
~/foo.com/copy
~/foo.com/archives

Step 1: SSH setup

To automate all things, you need to be able to connect through SSH without using a password. The steps required are describe here and are valid in general, not just for dreamhost accounts. Anyway, I’m copying just a list of commands here, without explanations.

On your home box, login as debug user and write these lines; you will be prompted for password twice:

ssh-keygen -t rsa -f foo.com.rsa -N ''
scp foo.com.rsa.pub user@foo.com:~/
ssh user@foo.com

mkdir .ssh
cat foo.com.rsa.pub >> .ssh/authorized_keys
rm foo.com.rsa.pub

chmod go-w ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Logout and test whether SSH is working:

ssh user@foo.com -i foo.com.rsa

It should open an ssh session without asking you the password.

Step 2: MySQL dump

While logged on the remote server, do:

mkdir -p ~/backup/archives
cd ~/backup

With your preferred (or available) editor write into ~/backup/dumpmysql.sh on the remote server:

#!/bin/sh
mysqldump -u yourmysqluser -pyourmysqlpassword \\
     -h mysqlhost databasename \\
     | bzip2 > ~/backup/databasename.sql.bz2

If you have more databases, you could just add a line for each database to backup, or may be you prefer to store more database backup in one single file.

chmod u+x ~/backup/dumpmysql.sh
~/backup/dumpmysql.sh
crontab -e

The last command will open a text editor; write a new line like this:

0 0 * * * ~/backup/dumpmysql.sh

Starting by now, this command will be executed every day at midnight. Further details and examples can be found here.

Step 4: Syncing local copy with remote server files

Before syncing the files, make a list of files you don’t want to backup: for example I don’t want cache files in the archives. So create two files; call the first ~/foo.com/exclude.lst and the latter ~/foo.com/include.lst. For example, in the first I wrote:

backup/archives
logs
*cache

In the other:

wp-cache*

The first line of the exclude file is very important, as it excludes the backup copied back to the server; without it you’ll end up in multiple self-replicating copies of your backup, one inside the other! The last line of include file is useful for preventing the exclusion of the wp-cache wordpress plugin from backups, which would otherwise match with the *cache line in the exclude file.

Now you’re ready for the first sync:

sync -az -e "ssh -l user -i /home/backup/foo.com.rsa" \\
     --exclude-from=~/foo.com/exclude.lst \\
     --include-from=~/foo.com/include.lst \\
     user@foo.com: ~/foo.com/copy

Step 5: Make a base archive

Create a list of file to not compress: it’s not mandatory, but it’ll speed up a lot the process whilst increasing very few the actual archive size. Create a file called nocompress.lst, containing the masks matching the files to archive uncompressed, for example (pay attention to write everything on a single line):

-Z *.jp*g -Z *.gif -Z *.swf -Z *.avi -Z *.flv -Z *.mov
-Z *.wmv -Z *.wma -Z *.mp*g -Z *.jar -Z *.zip
-Z *.gz -Z *.bz2 -Z *.Z -Z *.rar -Z *.ace -Z *.png
-Z *.7z -Z *.JP*G -Z *.GIF -Z *.SWF -Z *.AVI
-Z *.FLV -Z *.MOV -Z *.WMV -Z *.WMA -Z *.MP*G
-Z *.JAR -Z *.ZIP -Z *.GZ -Z *.BZ2 -Z *.RAR -Z *.ACE
-Z *.PNG -Z *.7Z -Z *.msi -Z *.dmg -Z *.iso

You are now ready to create the base archive and transfer it on the server:

ARCHIVE=~/foo.com/archives/`date +%Y-%m-%d`_base
dar -y -c $ARCHIVE \\
     -R ~/foo.com/copy/ \\
     `cat ~/foo.com/nocompress.lst`
scp -i ~/foo.com/foo.com.rsa \\
     $ARCHIVE user@foo.com:~/backup/archives

Step 6: Schedule syncing and backup

Create the script ~/foo.com/sync.sh:

#!/bin/sh
USER=user
HOST=foo.com
BASEDIR=~/foo.com
SSHKEY=$BASEDIR/foo.com.rsa
SYNCDIR=$BASEDIR/copy
ARCHIVESDIR=$BASEDIR/archives
ARCHIVE=$ARCHIVESDIR/`date +%Y-%m-%d`

pushd $BASEDIR > /dev/null

rsync -az -e "ssh -l $USER -i $SSHKEY" --delete \\
     --exclude-from=$BASEDIR/exclude.lst \\
     --include-from=$BASEDIR/include.lst \\
     $USER@$HOST: $SYNCDIR

LASTARCHIVE=`ls -t $BASEDIR/archives/* \\
     | head -n1 \\
     | sed -e 's/.[0-9]*.dar$//'`
dar -y -c $ARCHIVE -X *.dar \\
     -A $LASTARCHIVE \\
     -R $SYNCDIR \\
     `cat $BASEDIR/nocompress.lst`
rsync -az -e "ssh -l $USER -i $SSHKEY" --delete \\
     $ARCHIVESDIR/ $USER@$HOST:~/backup/archives

popd > /dev/null

Last things to do: copy these files on server for future reference and schedule the backup:

chmod u+x ~/foo.com/sync.sh
scp -i ~/foo.com/foo.com.rsa ~/foo.com/* user@foo.com:~/backup
crontab -e

In the text editor write a line like the following:

0 2 * * 7 ~/foo.com/sync.sh

Backup will now run every Sunday at 2:00AM.