View previous topic :: View next topic |
Author |
Message |
idrk n00b
Joined: 23 Apr 2024 Posts: 14
|
Posted: Thu Aug 15, 2024 4:27 pm Post subject: How to backup Gentoo correctly? |
|
|
Hi, I have some questions and doubts about how I can backup my Gentoo installation.
1. How to be able to deploy my Gentoo installation on another machine or the same machine for reinstallation
I read you can make a stage4 with some scripts like mkstage4(which is deprecated that's another story)
How do I do that? Do I make a stage4 then put in(and consistently sync it) on a external hard drive an I can use it to install Gentoo on other machines?
also I can use that if my Gentoo ever breaks after an update?
2. Is the above solution useful for the time that a single file(like a config file) is corrupted on my system and I want to recover that?
3. I also read that there's a good piece of software that's called 'Restic' which is also used for backup. Is that any useful if I wanted to use it for my current situation?
4. I asked some people and there are lots of scripts(like gentoo-install, gentoo-sync) I'm kinda confused which one to use here if I wanted to deploy on other machines quickly too, like just extract the stage4 and use the scripts to setup my gentoo installation like in the original pc
5. Is using backup system on the current machine(or even current disk) any useful? like using restic on the same drive. Is it stupid or it can get handy at times? like files missing(maybe gentoo breakage too?)
6. What things should I consider for my personal/private data? should I include my /home dir + other thingn in the backup? |
|
Back to top |
|
|
alamahant Advocate
Joined: 23 Mar 2019 Posts: 3916
|
Posted: Thu Aug 15, 2024 5:47 pm Post subject: |
|
|
I use rsync to backup my entire / to a different partition before each update.
Plz see
https://wiki.archlinux.org/title/Rsync#Full_system_backup
Please add --delete also to above rsync command
Code: |
rsync -aAXHv --delete --exclude='/dev/*' --exclude='/proc/*' --exclude='/sys/*' --exclude='/tmp/*' --exclude='/run/*' --exclude='/mnt/*' --exclude='/media/*' --exclude='/lost+found/' / /path/to/backup
|
You can also use tar.
If you wish to deploy to another system then you will need to chroot into the new system, reinstall grub, modify fstab and /etc/default/grub then update grub and regenerate initramfs. _________________
|
|
Back to top |
|
|
idrk n00b
Joined: 23 Apr 2024 Posts: 14
|
Posted: Fri Aug 16, 2024 9:55 pm Post subject: |
|
|
Hi, I was recommended rsync too. But I would like to get answers to my questions because I can understand backup fully. |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Tue Aug 20, 2024 3:28 am Post subject: |
|
|
Do a search here on the forum for stage4 and separately for backup. There are abundant electrons already been spent on those topics. I use my own stage4 script (stage4.scr) which you can also find in these forums. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
Kethreveris n00b
Joined: 16 Feb 2009 Posts: 20 Location: Rethwallen
|
Posted: Thu Sep 05, 2024 7:05 am Post subject: |
|
|
One thing I found early on when testing RSync backups and full restores, was that I needed a basic "/dev" populated. So my RSync backup script does that first;
Code: | rsync -aAHSXx --delete --stats /dev/ ${MY_BACKUP_PATH}/dev/ |
I don't know if that is needed today, since I have not performed a full restore in a long time. But, I do perform the occasional restore of a file. Or backup set verification. However, since "/dev" is tiny in amount of space used, it's pretty fast to backup and is meaningless for amount of space used in the backup.
By backing up "/dev", I can:
- Take a blank disk
- Partition and make file systems as needed
- Re-lay the file system(s) on it
- Prepare the restored OS for "chroot"ing
- "chroot" to the newly restored OS
- Install Grub on the boot device
- Exit and reboot
This absolutely worked in the past. And the thing that killed me before was lacking a minimal "/dev" at boot.
I found that doing the Grub from inside the newly restored OS solves some odd problems. _________________ Kethreveris, (aka Kethrery) |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22618
|
Posted: Thu Sep 05, 2024 12:37 pm Post subject: |
|
|
The current advice in most cases is not to archive /dev at all, and instead to rely on the kernel's devtmpfs, which is automatically maintained and represents devices the kernel knows how to operate. This avoids issues if you restore the backup onto different hardware, such as creating the backup from a SATA SSD (/dev/sda) and restoring it to an NVMe (/dev/nvme0n1).
You are correct though, that a system without a valid /dev will have strange problems. |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Thu Sep 05, 2024 2:15 pm Post subject: |
|
|
With regard to /dev, I only backup/restore the the empty directory. I do the same for other system directories that don't require the content to be backed up. Since my backup process uses tar to build the compressed tarball, I use the --no-recursion switch for those directories. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
gentoo_ram Guru
Joined: 25 Oct 2007 Posts: 502 Location: San Diego, California USA
|
Posted: Thu Sep 05, 2024 10:46 pm Post subject: |
|
|
During my backups, I use a bind-mount of the root filesystem to backup the default stage3 contents of /dev. As in "mount -o bind / /mnt/root", then backup the contents of /mnt/root/dev. In other words, reveal the contents of /dev as installed on the root filesystem before the devtmpfs is mounted. I believe the initial contents of /dev are useful to the boot process before devtmpfs is mounted. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22618
|
Posted: Fri Sep 06, 2024 12:04 am Post subject: |
|
|
For systems without an initramfs, enabling the Kconfig option DEVTMPFS_MOUNT directs the kernel to pre-mount devtmpfs, so that it is available at the very beginning of init. For systems with an initramfs, that Kconfig option is ignored, but the initramfs could provide a stub /dev (often just null, console, and tty), which is sufficient for the initramfs to explicitly mount devtmpfs. In both cases, the system can get by with a minimal and host-independent static /dev. It can rely on the kernel to provide hardware-specific device nodes. |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Fri Sep 06, 2024 12:59 am Post subject: |
|
|
This is what I exclude when making a full system backup as a stage4 tarball, using tar's --exclude-from=FILE switch:
Code: | $ cat ~/bin/exclude.stage4
/dev/*
/home/*/*
lost+found
/media/*
/mnt/*/*
/proc/*
/root/.cache/*
/root/.thumbnails/*
/root/.dbus/session-bus/*
/run/*
/scratch/*
/sys/*
/tmp/*
/var/cache/distfiles/*
/var/log/portage/*
/var/tmp/* |
The resulting stage4 compressed tarball restores and runs without error or trouble of any kinds. Yes, I don't have a static /dev and I do have DEVTMPFS_MOUNT in my kernel. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
AprilGrimoire n00b
Joined: 05 Jun 2020 Posts: 57
|
Posted: Fri Sep 06, 2024 1:10 am Post subject: Re: How to backup Gentoo correctly? |
|
|
If you are considering a backup, you can try dumping the whole disk or partitions into an image. This avoids the problem of needing to recreate filesystems, which is particularly useful in scenarios with complicated disk schemes like LVM over LUKS. Simply recreating the partitions and copying all the files might not work since UUID could be different. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22618
|
Posted: Fri Sep 06, 2024 1:28 am Post subject: |
|
|
Creating an image is wasteful, since you copy all the free space, you cannot create incremental backups, and a restore must be done to a drive with equal or greater total capacity. Additionally, if the restore drive is an SSD, you will needlessly write the old drive's free sectors to the new drive.
An image may be appropriate for forensics. For ordinary backup, I advise against it. |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Fri Sep 06, 2024 1:35 am Post subject: |
|
|
AprilGrimoire;
It's up to the user to edit any necessary files for UUID and/or network changes. A compressed archive of the files is much more portable than a disk image owing to the reduced size of the compressed archive. My current compressed archives run < 6 GB, easily stored on-line or flash drive. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
nicop Tux's lil' helper
Joined: 10 Apr 2014 Posts: 88
|
Posted: Fri Sep 06, 2024 7:09 am Post subject: |
|
|
An often forgotten but important step before backup : do a snapshot or unmount / freeze the filesystem. Otherwise you may have corrupted files. |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 3005 Location: Edge of marsh USA
|
Posted: Fri Sep 06, 2024 3:36 pm Post subject: |
|
|
nicop wrote: | An often forgotten but important step before backup : do a snapshot or unmount / freeze the filesystem. Otherwise you may have corrupted files. |
That's not an issue with my stage4 script because it neither backs up personal files or those live system directory contents (because they are freshly repopulated with each boot). You can search for my stage4 script in these forums (easy to find) as well as see: https://wiki.archlinux.org/index.php/rsync#Full_system_backup for similar with rsync. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3416 Location: Canada
|
Posted: Fri Sep 06, 2024 4:17 pm Post subject: |
|
|
Administering only my own machines, I do not backup system. I back up only /etc and /home. The rest I can reinstall |
|
Back to top |
|
|
simplycorbett n00b
Joined: 14 Mar 2024 Posts: 27
|
Posted: Tue Sep 10, 2024 3:00 am Post subject: |
|
|
Assuming you have three partitions (/ /boot /boot/efi)
Step 1(backup):
1) Boot live CD
2) Mount your installed / partition onto /mnt/gentoo
3) CD to the folder
4) Run Code: | tar -cvpzf backup.tar.gz --exclude=backup.tar.gz --one-file-system . |
5) Copy file to safe location when done.
6) remove the tar.gz backup from folder.
Step 2 (restore):
1) Boot live CD
2) re-create partitions
3) Mount empty partitions
4) CD to the folder your gentoo / is mounted on (/mnt/gentoo typically)
5) Copy the file to the /mnt/gentoo folder
6) Run Code: | sudo tar -xvpzf backup.tar.gz -C . --numeric-owner |
7) remove the tar.gz backup from folder.
8.1) Assuming you have boot mounted on /mnt/gentoo/boot and efi mounted on /mnt/gentoo/boot/efi
8.2) chroot into the install. Run source /etc/profile
8.2) Run Code: | grub-install --target=x86_64-efi --efi-directory=/boot/efi --removable |
8.3) Run emerge gentoo-kernel-bin
8.4) Run emerge @module-rebuild
8.5) Run Code: | grub-mkconfig -o /boot/grub/grub.cfg | (EDIT: Make sure /etc/fstab is setup!)
You are done. Reboot. |
|
Back to top |
|
|
sabayonino Veteran
Joined: 03 Jan 2012 Posts: 1036
|
Posted: Wed Sep 11, 2024 11:17 am Post subject: |
|
|
Hi.
There are a lot GNU-Linux tools for Backup (System Backup and/or Personal Data backup)
All depend on your own needs
My backup solutions are as follows :
Differential/Backups for System and Personal Data
Rsnapshot
Code: | [I] app-backup/rsnapshot
Available versions: 1.4.4-r1 ~1.4.5-r1
Installed versions: 1.4.4-r1(16:42:33 29/03/2024)
Homepage: https://www.rsnapshot.org
Description: A filesystem backup utility based on rsync |
Snapshots (Hourly,daily,weekly,monthly snapshots) are stored on my Local Drive.
Easy to access in realtme to restore something.
FSArchiver (as filesystem level) (System Backup)
Code: | app-backup/fsarchiver
Available versions: 0.8.7 {debug lz4 lzma lzo static +zstd}
Installed versions: 0.8.7(11:00:49 29/03/2024)(lz4 lzma lzo zstd -debug -static)
Homepage: https://www.fsarchiver.org
Description: Flexible filesystem archiver for backup and deployment tool |
Filesystem archives are stored on my external harddrive
Data Archives are stored on my external harddrive (Backup as Directory-Level not filesystem level)
My personal DOCs are also sincronyzed using Unison tool , stored on my LAN-NAS deck (You can also opt for your own rsync script)
Code: | net-misc/unison
Available versions: (2.53) 2.53.3-r1 ~2.53.5-r1
{debug doc gui +ocamlopt +threads}
Installed versions: 2.53.3-r1(2.53)(20:42:31 10/07/2024)(ocamlopt threads -debug -doc -gui)
Homepage: https://www.seas.upenn.edu/~bcpierce/unison/ https://github.com/bcpierce00/unison/
Description: Two-way cross-platform file synchronizer |
Raw Docs and last system weekly snapshot are stored on a External RAID storage system.
For my User data directory and misc apps configurations , I run my own script with DAR Differential/Incremental Backup
All data are stored on my local Harddrive and External RAID solution.
Code: | [I] app-backup/dar
Available versions: 2.7.15^t {argon2 curl dar32 dar64 doc gcrypt gpg lz4 lzo nls rsync xattr}
Installed versions: 2.7.15^t(10:25:01 15/08/2024)(curl dar64 gcrypt lz4 lzo nls rsync xattr -argon2 -dar32 -doc -gpg)
Homepage: http://dar.linux.free.fr/
Description: A full featured backup tool, aimed for disks
|
If you are just interested in archiving the operating system ,
TAR/RSYNC and/or FSArchiver may be your solutions
FSArchiver is very usefull _________________ LRS i586 on G.Drive
LRS x86-64 EFI on MEGA |
|
Back to top |
|
|
lyallp Veteran
Joined: 15 Jul 2004 Posts: 1599 Location: Adelaide/Australia
|
Posted: Wed Oct 09, 2024 5:47 am Post subject: |
|
|
Given my system uses btrfs, I take advantage of this and use the following script, via Cron, to backup to my NAS every week
/etc/cron.weekly/SystemBackup.sh
Code: |
#!/bin/bash
# Mount NAS drives if not mounted
if [ ! -d /mnt/nas/backups ]
then
# keep backups off-line until we use them.
mount /mnt/nas/backups || { echo "$0:Failed to mount /mnt/nas/backups" >&2 ; exit 1 ; }
fi
# check to see if the backup directory exists after mounting
if [ -d /mnt/nas/backups ]
then
/root/SystemBackup.sh /mnt/nas/backups/
# keep backups off-line until we use them.
umount /mnt/nas/backups
else
echo "$0:Backup directory /mnt/nas/backups not found" >&2
fi
|
Suggestions or corrections welcomed
Code: |
#!/bin/bash
backupFilesystem="${1:-/mnt/usb/Backup}"
if [ ! -d "${backupFilesystem}" ]
then
echo "$0: ${backupFilesystem} is not a directory"
exit 1
fi
echo "Starting backup at $(date) to ${backupFilesystem}"
backupDir="${backupFilesystem}/Backups"
SNAPSHOT="$(date +'%Y-%m-%d')"
echo "Snapshot = ${SNAPSHOT}"
snapshotsDir="/.snapshots/${SNAPSHOT}"
function CreateSnapshots
{
# Want to backup boot, which is not normally mounted
/bin/mount /boot
mkdir -p "${snapshotsDir}"
# create a read only snapshot
/sbin/btrfs subvolume snapshot -r / ${snapshotsDir}/root
/sbin/btrfs subvolume snapshot -r /home ${snapshotsDir}/home
/sbin/btrfs subvolume snapshot -r /var ${snapshotsDir}/var
}
function RemoveSnapshots
{
/sbin/btrfs subvolume delete ${snapshotsDir}/root
/sbin/btrfs subvolume delete ${snapshotsDir}/home
/sbin/btrfs subvolume delete ${snapshotsDir}/var
rmdir "${snapshotsDir}"
# no longer need /boot mounted
/bin/umount /boot
}
CreateSnapshots
# Remove snapshots regardless of exit status
trap "RemoveSnapshots" EXIT
snapshots="/.snapshots/${SNAPSHOT}"
echo "Starting backup of ${fileSystem} at $(date)"
for fileSystemName in root home var boot
do
case "${fileSystemName}" in
( "root" ) fileSystem="${snapshotsDir}/root" ;;
( "home" ) fileSystem="${snapshotsDir}/home" ;;
( "var" ) fileSystem="${snapshotsDir}/var" ;;
( "boot" ) fileSystem="/boot" ;;
( * ) echo "$0:Invalid Filesystem '${fileSystemName}'" >&2
exit 1
;;
esac
backupFile="${backupDir}/${fileSystemName}.tar.bz2"
previousBackupDir="${backupDir}/${fileSystemName}.prev"
#
## #########################################################################
## Backup previous backups in a directory - we don't really know how many
## files there are.
## #########################################################################
#
/bin/mkdir --parents "${previousBackupDir}"
#
## #########################################################################
## Remove any previous backups
## #########################################################################
#
/bin/rm --force "${previousBackupDir}"/*
#
## #########################################################################
## Save the current ones away
## #########################################################################
#
/bin/mv --force "${backupDir}"/${fileSystemName}.tar.bz2* "${previousBackupDir}"
#
## #########################################################################
## Backup the filesystem into 1gb chunks
## #########################################################################
#
/bin/tar --one-file-system --create --file - "${fileSystem}" | /bin/bzip2 --stdout | /usr/bin/split --bytes=1000m - "${backupFile}."
echo "Verifying backup of ${fileSystem} at $(date)"
/bin/cat "${backupFile}".* | /bin/tar jtf - > /dev/null 2>&1
rc=$?
if [ "$rc" -eq 0 ]
then
echo "Completed backup of ${fileSystem} at $(date)"
else
echo "**FAILED* backup of ${fileSystem} at $(date) - Did not verify!"
fi
done
# unmount usb disks as they seem to freeze after a period of disuse
/bin/mount -l | /bin/grep '/mnt/usb' | while read device on dir desc ; do /bin/umount -l ${dir} ; done
echo "Completed backup at $(date)"
|
_________________ ...Lyall |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 517
|
Posted: Wed Oct 09, 2024 7:24 am Post subject: |
|
|
Hu wrote: | Creating an image is wasteful, since you copy all the free space,... |
That's what compression is for
But I agree that it produces a rather unmanageable blob of data, once it's compressed. And you have to have some place to store that blob. And it's only going to be remotely useful if you have well-organized partitions. Still, I've found this process (dumping the raw filesystem) a life-saver from time to time. I don't believe there is a more accurate way to back up a partition. It's only going to be practicable in a small number of situations, but I wouldn't dismiss it out of hand.
BR, Lars. |
|
Back to top |
|
|
sabayonino Veteran
Joined: 03 Jan 2012 Posts: 1036
|
Posted: Wed Oct 09, 2024 2:01 pm Post subject: |
|
|
lyallp wrote: |
/etc/cron.weekly/SystemBackup.sh
Code: |
#!/bin/bash
# Mount NAS drives if not mounted
if [ ! -d /mnt/nas/backups ]
then
# keep backups off-line until we use them.
mount /mnt/nas/backups || { echo "$0:Failed to mount /mnt/nas/backups" >&2 ; exit 1 ; }
fi
# check to see if the backup directory exists after mounting
if [ -d /mnt/nas/backups ]
then
/root/SystemBackup.sh /mnt/nas/backups/
# keep backups off-line until we use them.
umount /mnt/nas/backups
else
echo "$0:Backup directory /mnt/nas/backups not found" >&2
fi
|
Suggestions or corrections welcomed
|
Look at "mountpoint" command to test if a mountpoint exists (for remote mountpoint or local mounpoint)
If /mnt/nas is your mountpoint , then test it
Code: | $ mountpoint -q /mnt/nas && echo "NAS Mounted" || echo "Nas not mounted" |
or
Code: | if [[ mountpoint -q /mnt/nas ]] ; then
echo "Nas mounted"
# do somthing else
else
echo "NAS not mounted"
# do something else
fi
|
see mounpoint manual or help comand
Code: | man mountpoint
mounpoint --help
|
_________________ LRS i586 on G.Drive
LRS x86-64 EFI on MEGA |
|
Back to top |
|
|
lyallp Veteran
Joined: 15 Jul 2004 Posts: 1599 Location: Adelaide/Australia
|
Posted: Wed Oct 09, 2024 10:09 pm Post subject: |
|
|
Thanks!
Will integrate the suggestion. _________________ ...Lyall |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22618
|
Posted: Thu Oct 10, 2024 5:59 pm Post subject: |
|
|
sabayonino wrote: | Code: | if [[ mountpoint -q /mnt/nas ]] ; then |
| Your example has a slight error: Code: | $ if [[ mountpoint /tmp ]]; then echo OK; fi
-bash: conditional binary operator expected
-bash: syntax error near `/tmp' | For using an external command here, omit the [[: Code: | $ if mountpoint /tmp; then echo OK; fi
/tmp is a mountpoint
OK |
|
|
Back to top |
|
|
sabayonino Veteran
Joined: 03 Jan 2012 Posts: 1036
|
Posted: Thu Oct 10, 2024 6:11 pm Post subject: |
|
|
sorry .remove the -q flag
[[ ]] expect true or false ( 0 or 1 ) result
Code: |
#!/bin/bash
if [[ $(mountpoint /mnt/nas) ]] ; then
echo "mounted"
else
echo "not mounted"
fi
|
or
Code: | #!/bin/bash
mountpoint -q /mnt/nas && echo "mounted" || echo "not mounted"
|
you can run the scipt and analyze the script syntax adding "-x" flag to the bash command
Code: |
$ bash -x /my/script.sh
|
####### simple script sample #######
Code: |
#!/bin/bash
if [[ $(mountpoint /mnt/gentoo) ]] ; then
echo "mounted"
else
echo "not mounted"
fi
#### or
mountpoint -q /mnt/gentoo && echo "mounted" || echo "not mounted"
|
Code: | └─$ bash -x ./test.sh
++ mountpoint /mnt/gentoo
+ [[ -n /mnt/gentoo is a mountpoint ]]
+ echo mounted
mounted
+ mountpoint -q /mnt/gentoo
+ echo mounted
mounted |
we're going off-topic _________________ LRS i586 on G.Drive
LRS x86-64 EFI on MEGA |
|
Back to top |
|
|
idrk n00b
Joined: 23 Apr 2024 Posts: 14
|
Posted: Wed Oct 16, 2024 7:29 am Post subject: |
|
|
so, is using Catalyst software any better than using this command provided by archwiki:
Code: | # rsync -aAXHv --exclude='/dev/*' --exclude='/proc/*' --exclude='/sys/*' --exclude='/tmp/*' --exclude='/run/*' --exclude='/mnt/*' --exclude='/media/*' --exclude='/lost+found/' / /path/to/backup |
and doing compression and stuff manually |
|
Back to top |
|
|
|