Backup Best Practices

[Nicias]

Hello,

I have a home media server that I would like to be backing up regularly. My plan is to back it up to a veracrypt encrypted drive which I store at my workplace. I have a total of about 3TB of data to store, and I'll be using a 5TB drive. All of the data has par2 checkfiles.

My plan for a procedure for doing my backups was something like this:

1. Connect drive to server.
2. Run smartctl -t conveyance
3. Mount veracrypt container
4. Run fsck on /dev/mapper/veracrypt1 (unmounting before and remounting after)?
5. rsync over changes
6. Run fsck again?
7. Run dd if=/dev/sdX of=/dev/null to force the drive to read all sectors.?
8. Run smartctl -t long?

But now I think that seems like overkill.

Do I need steps 4, 6, 7, 8? Is there something else I should be including?

Thanks!

[szatox]

It absolutely is an overkill in terms of work you're doing, and sub-par in terms of results.

Mount, decrypt, rsync, umount is OK, there is no need for the other steps.
Also, consider running rsync with hardlinks ( --link-dest= <previous_backup_path> ) or using snapshots* for incremental backups, so you can have multiple versions of your files.
Finally, clone your backup to another drive.

Whether or not this step is necessary depends on how important that data is for you. You're really planning to go out of your way there, so I assume you want to be very safe. So just have a second disk with the same data instead.
Preferably a different brand or age (so they don't fail at the same time due to manufacturing flaws), in another location (theft, fire, floods, power surges, meteors etc), synchronized over the internet.

Running long test on a monthly schedule or so is fine. There is no need to do that every time. Long test attempts to read the whole disk, so dd is not necessary either. Rsync reads destination files too.


* Both options have their flaws and can fail spectacularly under certain conditions, know the limitations of your chosen solution so you don't shoot your foot. Also, having a clone helps recover from mistakes as well as hardware failures.

[Nicias]

Thanks for the quick reply!

I actually was going to have two backups, in two different locations (I work in two different buildings).

I will look into --link-dest.

[figueroa]

In my opinion, all the checking is overkill, and it's not very efficient. I make automatic backups nightly (crontab) to secondary hard drive. The backups are made using tar with compression but not encryption.

When I make my off-site backups, I use tar with gpg encryption but no compression. Since they are already compressed, I don't compress again. But, it would be easy to add compression when the script reads "--compress-algo none". Compression reduces I/O and writes to media.

Here is one of my actual scripts just for user data with USERNAME changed to protect the guilty.

[szatox]

Oh, one more thing:

[pietinger]

[psycho]

Nicias, your "two different locations" plan is sensible. You've achieved nothing, backing up your data to your NAS or whatever, if a burglar steals the desktop *and* the NAS. Some people might see it as overkill, but if your data's important to you, yes, you need at least one backup offsite so that even if the building burns down, your data isn't lost. Of course cloud backups can achieve that (and you can copy a whole encrypted volume to them, so that they're pretty safe with a strong enough password, even if the provider doesn't offer end-to-end encryption, although obviously that's better as another layer), but even an occasional manual physical copy to a USB drive or something else that's not stored onsite is much better than having all your eggs in one basket. Keeping a laptop somewhere else and syncing it whenever it's onsite can be another casual solution, so you're not counting entirely on your local backup.

You can add the local backup to your shutdown script so you don't even have to think about it. I use a process similar to the one you're outlining: I decrypt a volume on a NAS and rsync my local encrypted data to that volume. It's got enough redundancy that I could lose a drive (on the NAS) without losing my data...and whenever I think of it (which, admittedly, is not often enough...sometimes I'll forget for a week or more) I back it up to a USB stick on my keyring too. The keyring drive not only stores the encrypted data, but also the tools (via live USB OS) to access it. Then in addition to the regular stuff I've got old backups in other weird places. As everyone learns the hard way eventually, overkill is better than underkill when it comes to backups.

[figueroa]

Yes, I also have two large (but physically small - Samsung Bar Plus) USB flash drives on my keyring with a copy of my monthly backup set. I also get email reminders from my calendar to make the weekly and monthly off-site backups. On site backups are automatic and made while I sleep, desktop and server running 24/7.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

[psycho]

[Hund]

I can highly recommend Back In Time for backups. It's based on rsync and it simplifies everything a lot.

https://packages.gentoo.org/packages/app-backup/backintime
_________________
Collect memories, not things.

[user]

3-2-1 Backup Strategy

1) rdiff-backup hourly for non-binary content, daily for binary content (one copy data locally)
daily incremental tar archive (incremental state monthly reseted)
asymmetric gpg encryption of daily tar archive (no online access to secret gpg key)
2) daily encrypted archive copy to local NAS (second copy of data on-site but different media)
3) daily encrypted archive copy to different geolocated remote destinations with write-only access (N copies of data off-site)

- content always encrypted outside of local host (NAS or off-site)
- restore need secret gpg key and initial monthly tar archive with latest incremental tar archive
- not usable for petabyte of content

[figueroa]

Since the subject is "Best Practices," I'll toss this out for consideration. For users and administrators, this is a burning issue. Apparently one-size does not fit all.

Best Practices
1. Do make backups.
2. Make backups often.
3. Make backups automatically.
4. Store some backups off-site.

Considerations
1. Location
2. Compression
3. User data versus operating system
4. Tolerance for loss
5. Physical security
6. Network security
7. Encryption
8. Automation
9. Scheduling
10. Recovery/Restoration

Discussion Starters
1. Tolerance for loss drives consideration of using real-time/near-real-time backups, versus hourly/nightly/weekly/monthly.
2. Some users backup none of their operating system, saying they will just re-install if they loose the system. Others backup a few critical operating system files to help them duplicate the OS upon re-installation, i.e. /var/lib/portage/world, /etc/portage/, /etc/fstab, /etc/hosts, /boot/config~. Still others backup their entire OS with third-party programs (i.e. backintime) or usually built-in tools (i.e. rsync, tar, dar, gzip, gpg, etc. for stage4 or equivalent).
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

[pa4wdh]

To chime in on the "best practices": You may have heard of the 3-2-1 rule: Make 3 backups on 2 different media types and store 1 of them on a different location.

For me personally:
I make (automated) daily backups of unique and changing data. Things like homedir, mail, etc. This backup is encrypted with gpg and stored on a VPS. The decryption key is stored on a (encrypted) USB Stick and on paper.
I make quarterly full backups of all systems usually after i updated them. This data doesn't change much so i don't lose much if disasters happen just before i make my next backup. This is stored on USB attached HDD which is only attached when i make the backups and is encrypted with LUKS.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com

[psycho]

Re "Some users backup none of their operating system, saying they will just re-install", yes, this is an interesting discussion point, and a question of how much additional time this will take. The time to "just re-install" is always going to be longer than the time to simply restore the fully installed and configured system from a backup...but even with a very quickly installed binary distro, how much longer it takes depends upon how much customisation is done, and whether the customisation itself has been backed up somehow.

I used to invest many hours in customising my installations (as root, i.e. to parts of the system that would not be backed up with the normal user customisations of GUI setups and so on), all of which I did manually without taking the time to document it in the form of shell scripts...so this made "just re-install" an unreasonable option, as to re-install from scratch would have cost me many, many hours: backing up user data alone was certainly not viable. Eventually though, I made the effort to track all of these manual tweaks in shell scripts, so that I can transform a fresh out-of-the-box installation into a fully configured and customised system very quickly and easily...theoretically with a single instruction, although for ease of managing the process I prefer to do it in stages with a few different scripts. So, the clean-install-plus-data-restore approach is viable now and takes maybe half an hour longer (much of which is the actual installation) on e.g. systems using apt with binary repositories.

Gentoo is different though: "just install" is a comical notion if you're in the middle of something important and need to "just install" Gentoo from scratch to the point where you can get back to restoring and using your data! Unless you've got a binary package server to accelerate things, reinstalling Gentoo is a very time-consuming process (even assuming you've meticulously backed up or scripted all your edits to /etc/* and your kernel .config and so on, so that none of that needs to be re-done)...I would never consider a Gentoo system with only its user data backed up to be properly backed up. Some kind of full system backup or snapshot is essential, unless you have lots of time on your hands and are only concerned about being able *eventually* to restore things, and not concerned about the time involved in doing so.

[pietinger]

[figueroa]

In the few times I've had a primary hard drive suddenly die on a Gentoo system, I've been very happy to be able to restore a full stage4 tarball or stage4-equivalent backup set that was no more than a week old. I make these automatically on the following crontab schedule.

[A.S. Pushkin]

One more thank you for all your suggestions.

It's more evidence why Gentoo is such a great distro.

I have gotten such great education as I use Linux.


Real democracy for computer users.
_________________
ASPushkin

"In a time of universal deceit - telling the truth is a revolutionary act." -- George Orwell