HOWTO: The poor man's differential backup

[VinzC]

Hi.

Just wanted to share this. As I sought the Internet for backup solutions, especially about differential backups, I found lots of scripts and tools all over the place.

The context

I have a big drive that I must backup every day. No complicated solution is involved, just the good old tar command, it does wonders. Until a few days ago I was doing only full backups. Each archive file is named after the date of the backup, allowing me to go back in time for a certain amount of days. But the storage space on the backup drive is fixed and as the full backups grew in size I needed to decrease the history length. I ended up keeping only a few days and I wanted more. Another issue is the backup process takes so much time that it still runs during work hours.

Logically, I had to create full backups less frequently, outside work hours or during week-ends. Every other day, just do either differential or incremental backups.

A little history.

Differential backups are easier to manage than incremental: only restore the latest full backup then the latest differential backup. With incrementals, you need to restore *all* of them in sequence after the last full.

Typically differentials and incrementals do rely on the fact that a file was changed after the last backup. That's the purpose of the archive bit on Windows. Only full and incremental backups do reset this bit. The catch is — someone prove me wrong there — there's no such bit in GNU/Linux filesystems. But that's no big deal, really.

So the question is how do I create a differential backup?

There are scripts. There are tools. I want neither .

The [easy] solution

Of course, tar cannot create differential backups. But it can restore and skip newer files (tar -xp --keep-newer-files)! It's all we want, isn't it? Say we want to do full backups every week. All we need to do is find all files that have been changed since the last full backup. Simply put: we just need to find the files that have changed in the last 7 days. The restore process will take care not to overwrite the newer files, i.e. files in the full backup that are newer than those in the differential. Better more than not enough files in the archive, right?

So, in this scenario, here's the principle:

[x22]

[VinzC]

[Sven Vermeulen]

I use rsync with the --link-dest option. It allows you to do rsync's, but when files are already available at the link-dest location, it uses a hardlink rather than a copy, saving you the space of the files that aren't modified (although even hardlinks have file system impact).
_________________
Please add "[solved]" to the initial topic title when it is solved.

[x22]

[John R. Graham]

There's a neat Perl script that drives the traditional *nix archiving tools (find, tar, cpio, and their brethren) called flexbackup. It provides a management layer on top of those tools that creates full, incremental, or differential backups and supports a strong regular expression based exclusion mechanism.

[depontius]

[XQYZ]

[SlashBeast]

you guys should check rsnapshot and rdiff-backup.

[VinzC]