View previous topic :: View next topic |
Author |
Message |
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Wed Apr 10, 2024 7:05 pm Post subject: online backup f2fs? |
|
|
The title basically says it, anyone has a good way to get a somewhat consistent backup a mounted f2fs?
There is no lvm underneath, so can't snapshot it this way, and it doesn't seem like it supports snapshotting at fs level either.
I found one thing I might be able to abuse for a block level backup, but honestly I hope someone can suggest a better option.
Also, I'd prefer file-level backup, so I can easily drop caches worth like 50GB of transient data I could just re-download if I really needed to. _________________ Make Computing Fun Again |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Thu Apr 11, 2024 3:11 am Post subject: |
|
|
I suggest making a compressed tar archive. Look for my contribution for making a stage4 backup in the following thread. The thread is worth looking looking through for it's on sake since the subject was backing up Gentoo.
https://forums.gentoo.org/viewtopic-t-1162924-highlight-stage4.html _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Thu Apr 11, 2024 11:05 am Post subject: |
|
|
Pushing it _somewhere_ in whatever format is easy and I can handle it just fine; I still need to get the data off of the original device though, in a consistent state, while the device is in use. That is the tricky part, and I probably need something specific to F2FS that I'm currently not aware of.
Like, can I somehow double mount it to get a frozen view of the data? As stated earlier, unfortunately I don't have LVM underneath. _________________ Make Computing Fun Again |
|
Back to top |
|
|
tholin Apprentice
Joined: 04 Oct 2008 Posts: 204
|
Posted: Thu Apr 11, 2024 1:54 pm Post subject: |
|
|
I am aware of two possible solutions but neither is ideal.
1. You could use the blksnap module. This tool enables the creation of writable snapshots of any Linux block devices even without LVM. The code is not part of the mainline kernel. The author of blksnap is working on getting it merged into the upstream kernel but that is still a work in progress. I've never used blksnap myself and I am only familiar with it from an older lwn.net article.
https://github.com/veeam/blksnap
https://lore.kernel.org/all/20240209160204.1471421-1-sergei.shtepa@linux.dev/
2. Another option is to use /sbin/fsfreeze to freeze the file system while the backup is running. The fsfreeze manpage states that f2fs is supported. Any program attempting to write to a frozen file system will halt until the file system is unfrozen. Some programs do not respond well when write() calls stall for several minutes during a backup. Fsfreeze is primarily designed to be used by LVM or blksnap to force the on-disk layout into a consistent state for the brief period of time it takes to create a snapshot. Even so I use fsfreeze in my backup scripts with an ext4 rootfs and have never encountered any issues.
If you decide to use fsfreeze make sure that you enable sysrq so you can execute "sysrq + j" to unfreeze the file system if your entire system becomes unresponsive. You may also want to include "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" in your backup script because the kernel will begin spamming warnings in the system log about hung tasks otherwise. |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Thu Apr 11, 2024 4:16 pm Post subject: |
|
|
Yes, not ideal, but fsfreeze might actually be acceptable. I didn't want to remount=ro, because it would make writes fail, but delaying them instead doesn't seem that bad. Even if some applications become unresponsive for a few minutes, it's still OK-ish as long as they resume afterwards.
I like this much more than remounting with checkpoints disabled (should keep inconsistencies caused by writes during backup to free blocks)
Definitely worth testing. Thanks _________________ Make Computing Fun Again |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Thu Apr 11, 2024 9:59 pm Post subject: |
|
|
So I tried to use fsfreeze and run borg backup... And it didn't start. Strace showed its trying to create a file in /tmp, so I converted it to tmpfs. Launched borg again, and it didn't start. This time trying to create a file in ~/.cache/something...
Seriously... When I read docs, it looked like locks would be created inside repositories.
Damn. Looking for more ideas. (I suppose it is possible to do a workaround for those locks, but since I already know about 2 locks at the beginning of a single job, I'm reluctant to try and discover 3rd, 4th and 5th too) _________________ Make Computing Fun Again |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Thu Apr 11, 2024 10:51 pm Post subject: |
|
|
What's wrong with the compressed tar archive (stage4)? _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Fri Apr 12, 2024 10:02 am Post subject: |
|
|
It doesn't come with deduplication across multiple backups of the same machine (and multiple machines), so I'd have to store much more data than I really need. Also, I'd have to manage pruning myself. (Which I _can_ do and have in the past, but I hoped to move on to something better than rsync rather than worse)
So nothing wrong with it by itself, but there are better tools for the job. _________________ Make Computing Fun Again |
|
Back to top |
|
|
tholin Apprentice
Joined: 04 Oct 2008 Posts: 204
|
Posted: Fri Apr 12, 2024 10:59 am Post subject: |
|
|
szatox wrote: | Strace showed its trying to create a file in /tmp, so I converted it to tmpfs. |
Borg writes stuff to a tmp folder but that can be changed by exporting TMPDIR=something_else
szatox wrote: | This time trying to create a file in ~/.cache/something... |
That is the borg cache folder. It can be changed to something else by exporting BORG_BASE_DIR=/folder and the cache dir will then be /folder/.cache/borg. Borg will also create a .config folder in BORG_BASE_DIR.
szatox wrote: | When I read docs, it looked like locks would be created inside repositories. |
Well, the borg repo is where the backups are stored so that folder must always be writable.
For reference here is a bash script I use to backup my root fs to an external drive:
Code: | #!/bin/bash
if [ ! -e "/dev/disk/by-id/usb-INTEL_SS_DSC2CW120A3_DB1234567895AA-0:0" ]; then
echo "backup usb drive not found"
exit 1
fi
if [ -e "/dev/mapper/usb-backup" ]; then
echo "encryption already unlocked"
exit 1
fi
cryptsetup --cipher=aes-xts-plain64 --key-size=512 --hash=sha256 create usb-backup "/dev/disk/by-id/usb-INTEL_SS_DSC2CW120A3_DB12345
67895AA-0:0-part1"
mount -o ro /dev/mapper/usb-backup /mnt/borg-tmp-usbbackup/
if [ ! -d "/mnt/borg-tmp-usbbackup/borg" ]; then
echo "failed to open encrypted container"
exit 1
fi
fsck.ext4 -n -f /dev/mapper/usb-backup
fsck_exit=$?
if [ ${fsck_exit} -gt 0 ]; then
echo "fsck.ext exited with errors"
exit 1
fi
mount -o remount,rw /mnt/borg-tmp-usbbackup/
touch /mnt/borg-tmp-usbbackup/writable_testfile
if [ ! -f "/mnt/borg-tmp-usbbackup/writable_testfile" ]; then
echo "remounting fs rw failed"
exit 1
fi
rm /mnt/borg-tmp-usbbackup/writable_testfile
if [ -d "/mnt/borg-tmp-usb-root/bin" ]; then
echo "bind root fs already mounted"
exit 1
fi
mount --bind / /mnt/borg-tmp-usb-root/
if [ ! -d "/mnt/borg-tmp-usb-root/bin" ]; then
echo "failed to bind mount root fs"
exit 1
fi
export BORG_BASE_DIR=/mnt/borg-tmp-usbbackup/borg
export TMPDIR=/mnt/borg-tmp-usbbackup/borg/tmpdir
export BORG_REPO=/mnt/borg-tmp-usbbackup/borg/repo
echo -n "Freeze root FS [y/n]? "
read FREEZE
if [ $FREEZE = "y" ]
then
#tasks will hang so silence the warnings
echo 0 > /proc/sys/kernel/hung_task_timeout_secs
fsfreeze --freeze /
echo "Root fs frozen (sysrq + j to force unfreeze)"
fi
echo "Running backup..."
borg create \
--stats \
--one-file-system \
--atime \
--compression "auto,lzma" \
\
--exclude 'mnt/borg-tmp-usb-root/tmp/' \
--exclude 'mnt/borg-tmp-usb-root/var/tmp/ccache/' \
--exclude 'mnt/borg-tmp-usb-root/var/tmp/portage/' \
--exclude 'mnt/borg-tmp-usb-root/usr/portage/' \
--exclude 'mnt/borg-tmp-usb-root/usr/src/linux*' \
\
::'usb-fs-{now}' /mnt/borg-tmp-usb-root
borg_exit=$?
echo "unfreezing root fs"
fsfreeze --unfreeze /
if [ ${borg_exit} -gt 0 ]; then
echo "WARNING: borg finished with an error"
fi
echo 120 > /proc/sys/kernel/hung_task_timeout_secs
echo -n "Borg check [y/n]? "
read CHECK
if [ $CHECK = "y" ]
then
echo "Running check..."
borg check --verbose --progress ::
fi
umount /mnt/borg-tmp-usb-root
umount /mnt/borg-tmp-usbbackup/
cryptsetup remove usb-backup
sync
echo "safe to unplug drive"
|
|
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Sat Apr 13, 2024 4:13 am Post subject: |
|
|
szatox wrote: | It doesn't come with deduplication across multiple backups of the same machine (and multiple machines), so I'd have to store much more data than I really need. ... |
I trade deduplication for compression and redundancy that I find desirable in a backup system. Each of my system backup compressed archive files (~.tar.zst) is about 5.5 GB (no personal files) and takes about 10 minutes to create. Each is a stand-alone full-system backup that can be made and restored with commonly included Linux tools (tar and zstd or gzip), so no 3rd party program needed. It suits my preferences. I never have more than six-eight of those tarballs present on a secondary hard drive at any one time as they are replaced (overwritten) on a daily/weekly/monthly schedule from crontab. Encrypted copies are also in the cloud and on my keyring flash drive.
My personal files are similarly backed up but with a different set of scripts; and the collection of these tar archives are huge. But storage is cheap these days.
ADDED: Did I mention that I'm an evangelist (also a fanatic) for backups, whatever method is used. Backups should be systematic, regular, and automatic, with consideration for disaster recovery from all possible hazards. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
wjb l33t
Joined: 10 Jul 2005 Posts: 629 Location: Fife, Scotland
|
Posted: Sat Apr 13, 2024 3:28 pm Post subject: |
|
|
The type of backup you do depends on what you expect to do with it. In my case, it's file/directory recovery or comparison, so borg's ability to mount a backup set is ideal. If the PC/disk blows up, I expect to install new Gentoo and recover my files/config to the new installation eventually while being able access them in the interim using any (linux) pc capable of running borg. I have done this once for real (three weeks before retiring, which was very inconvenient but not a disaster) so I know it works.
There is currently about 4TB of backup data for 4 PCs taking up <300GB on a 2T drive.
There is one database I care about, and I just have my backup script pause the server while the backup is in progress - takes about 5min which is fine for my usage. I generally try to avoid doing installs while the backup is running, but if it happens it happens - it'll sort itself out the following day, and although its nice to have the built/installed files safe it's not strictly necessary.
I am careful to keep up-to-date & step-by-step recovery instructions on the backup media, because googling stuff on a phone while in a tizz never seems to end well. |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Sat Apr 13, 2024 5:34 pm Post subject: |
|
|
Guys, you're really focusing on the wrong end of the stick now. The point was specifically to get the data off of f2fs, while its in use, without corrupting it.
So far fsfreeze is the winner, and I'll most likely use. I think it fits the bill well enough to be the missing piece from my first post.
Backup target wasn't really a part of that question. Worst case, I'll just dust my old rsync script, which is not perfect, but worked without issues for a number of years.
I wanted to upgrade to borg to make things easier for myself, but I'm second-guessing that now. Dropping lockfiles into the backup target is perfectly fine; using the FS being backed up for processing said backup is not. I mentioned it because it caught me off-guard and I consider this part a broken design.
With additional information about borg's variables and so on, I'll weight my options and priorities and pick a suitable solution. This is still not really a part of getting the data off off that source device though
Anyway... Thanks for tips, everyone! _________________ Make Computing Fun Again |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Mon Apr 22, 2024 4:23 pm Post subject: |
|
|
Just to wrap it up in case anyone was interested in the final result:
I ended up going with rdiff-backup instead of borg, and fsfreeze remains an important part of the whole setup.
Rdiff-backup is not the fastest thing in the world (io-wait on spinning plates hits it hard), can't deduplicate data across multiple nodes, and overall is only a minor upgrade compared to rsync, but it is available on both Gentoo and Debian, seems to have decent backwards compatibility, doesn't clash with fsfreeze, and doesn't require hacks to pull data from remote clients (borg only works in push mode and I don't want to leave ssh keys to my home network on a VPS).
So, while it's not perfect, it's probably the best ready-to-use option for me right now. I might improve its compression ratio by converting underlying storage to btrfs one day, but it won't be a priority for some time. _________________ Make Computing Fun Again |
|
Back to top |
|
|
wjb l33t
Joined: 10 Jul 2005 Posts: 629 Location: Fife, Scotland
|
Posted: Mon Apr 22, 2024 7:09 pm Post subject: |
|
|
(the ssh key can be restricted so a client is only able to access its own repository on the borg server, though sadly that does not provide any way to prevent a hijacked client from wiping its archives) |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Mon Apr 22, 2024 7:29 pm Post subject: |
|
|
Well, to be fair borg can be launched in append-only mode which according to its docs would prevent the client from removing _anything_ from its repo, but forcing command on a key is waaay more hackish than just running rdiff-backup user@remote::/path /local/mirror without having any key to local machine on the remote node (no sharing via ssh_agent either, so no window of opportunity for re-using the exposed key for other purposes), and adds more places where I can make mistakes setting things up. _________________ Make Computing Fun Again |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3678 Location: Rasi, Finland
|
Posted: Mon Apr 22, 2024 9:19 pm Post subject: |
|
|
szatox wrote: | Just to wrap it up in case anyone was interested in the final result:
I ended up going with rdiff-backup instead of borg, and fsfreeze remains an important part of the whole setup. |
I use rdiff-backup when I can't take snapshots. One example is at my work, where I need to make sure the backups disks are easily readable by "vanilla" windows. So, the solution is to use rdiff-backup and use ntfs on the destination.
Otherwise I'd maybe use btrfs and snapshots on the backup drive. Although XFS can refcopy nowdays... _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
wjb l33t
Joined: 10 Jul 2005 Posts: 629 Location: Fife, Scotland
|
Posted: Mon Apr 22, 2024 11:15 pm Post subject: |
|
|
Ok I'm interested, two things - why do you consider setting a command on a key 'hackish' (I'd call it annoyingly whitespace-sensitive), and it's not exactly expensive to set up a separate ssh key just for the purpose of backing up (other than maybe having to add it to /etc/ssh/ssh_config). |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3404
|
Posted: Tue Apr 23, 2024 12:41 pm Post subject: |
|
|
Quote: | why do you consider setting a command on a key 'hackish' (I'd call it annoyingly whitespace-sensitive) | Well, you named one reason already
It creates room for mistakes and those keys need to be managed in some way that will prevent me from forgetting about them. Like, if I forget to force command, backup will still run just fine, but now the remote node has a key which allows it to execute arbitrary code. Which would be fine if it was physically under my control, but that control happens to be imaginary and I don't trust it to the same extent.
Also, I'm behind a NAT, so need some way to open connection from the internet to home, which translates to running stuff through a vpn, so now I have another layer of encapsulation and encryption in the mix, OR i have to use some tricks with socat and remote commands to spawn both ends of the pipeline in the most convoluted way possible, and this method is actually described in borg's official docs. They know that pulling data from a remote node is an important feature, but somehow it's easier to document this monstrosity than add an "if" inside borg binary.
Kinda reminds me a very old rant comparing php to a weird toolbox, where everything kinda works, but it really shouldn't. Like, you want to fix a nail. So you grab a hammer from your toolbox. The hammer happens to have a pointy end and a swallow's tail end. You can still get the job done if you hold it side-ways though.... _________________ Make Computing Fun Again |
|
Back to top |
|
|
|