Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Clone an entire filesystem
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
superdeez
n00b
n00b


Joined: 10 May 2015
Posts: 61

PostPosted: Sat Mar 18, 2023 8:55 am    Post subject: Clone an entire filesystem Reply with quote

When I built my current system for my root drive, I used an SSD that (while free) was on the small side and finally I ran out of space.

Not sure if dd would work if the two drives are of different sizes. When I've used cp before I've found it doesn't copy over *everything* (though it's been good enough when I've moved my home dir around between different drives). What's the best way to attempt to copy over my entire / and /boot filesystems?

root is ext4, boot is ext2.
Back to top
View user's profile Send private message
sdauth
l33t
l33t


Joined: 19 Sep 2018
Posts: 659
Location: Ásgarðr

PostPosted: Sat Mar 18, 2023 10:46 am    Post subject: Reply with quote

rsync -avHAX /mnt/oldssd_root/ /mnt/newssd_root/
rsync -avHAX /mnt/oldssd_boot/ /mnt/newssd_boot/
should do to the job. I used that a few weeks ago when moving from ext4 to btrfs.

then restore UUID of your old boot & root partition to the new one. (With tune2fs -U *UUID* ..)

Otherwise, since the old one is smaller, you could simply dd the old one to the new one and then expand the root partition.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54644
Location: 56N 3W

PostPosted: Sat Mar 18, 2023 12:12 pm    Post subject: Reply with quote

superdeez,

dd copies everything, including the empty space. That's a bad thing for an SSD because even if its erased empty space on the source, it will incur a write on the destination.
rsync is good, it only copies the content of the filesystems.

You must do the copy as root, while the source filesystem is not mounted. If its in use, files open for writing will be corrupt in the copy.

You can update the UUIDs but they are supposed to be Universally Unique ... the kernel will hate you for two identical UUIDs in the same system at the same time, so I advise against it.
Chroot into the copy, once its done and update all the UUID consumers. That's future proof.

BIOS booting users will need to reinstall grub to the MBR as that is outside any filesystems.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
superdeez
n00b
n00b


Joined: 10 May 2015
Posts: 61

PostPosted: Sat Apr 15, 2023 3:41 pm    Post subject: Reply with quote

I will come back and update this thread with the (apparently) successful conclusion.

I had never ever thought of using rsync to mirror a filesystem like that. I'd always used some variation of cp -R.

It worked like a charm. I was replacing a 128GiB ssd that was unused and free to me. Since my early Linux days I've used one hdd for root and another, larger (4TiB, in this case) ssd for my home dir. Though I may get flak for this, I used a low hours 500GiB mechanical hdd I had in a laptop that bit the dust last year. Very slightly slower, but I already own it and it only has around 2500 hours on it.

No UUIDs or anything here, my machines all still use lilo as a bootloader.

But rsync -avHAX worked perfectly, then I just chrooted into the new HDD's root, ran lilo and it all works.

I only had to run one srync command as I mounted both /boots before running it and it just copied that all as well.

Thank you to all who replied.
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3461
Location: Canada

PostPosted: Sat Aug 24, 2024 5:31 pm    Post subject: Reply with quote

NeddySeagoon wrote:
superdeez,

dd copies everything, including the empty space. That's a bad thing for an SSD because even if its erased empty space on the source, it will incur a write on the destination.
rsync is good, it only copies the content of the filesystems.

You must do the copy as root, while the source filesystem is not mounted. If its in use, files open for writing will be corrupt in the copy.

You can update the UUIDs but they are supposed to be Universally Unique ... the kernel will hate you for two identical UUIDs in the same system at the same time, so I advise against it.
Chroot into the copy, once its done and update all the UUID consumers. That's future proof.

BIOS booting users will need to reinstall grub to the MBR as that is outside any filesystems.


Can you clarify what "You must do the copy as root, while the source filesystem is not mounted. If its in use, files open for writing will be corrupt in the copy." means, I.e. how I can use rsync on unmounted source ?
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22877

PostPosted: Sat Aug 24, 2024 5:53 pm    Post subject: Reply with quote

I think that was a confusing phrasing. If the filesystem is mounted read-write, then its superblock may be inconsistent (bad when using dd; irrelevant when using rsync). Additionally, if it is mounted read-write and any applications have files open for write, then those files may also be inconsistent. To guard against both cases, the source filesystem should be mounted -o ro (read-only), so that the kernel enforces that no application can write to any file in that filesystem. In my experience, though I am not aware of a hard guarantee in this regard, the superblock tends to be consistent for a read-only filesystem, which benefits dd. As Neddy says though, dd is not a good choice if your filesystem can be copied by rsync. You might use dd if the filesystem is damaged and you are trying to create a clone for recovery purposes.
Back to top
View user's profile Send private message
C5ace
Guru
Guru


Joined: 23 Dec 2013
Posts: 488
Location: Brisbane, Australia

PostPosted: Sat Aug 24, 2024 10:34 pm    Post subject: Reply with quote

I use this batch file to backup a running laptop with MBR and grub2 via WIFI to my desktop"

Code:
#!/bin/sh

echo TOSH-Backup start
date | tee /var/log/TOSH-Backup.log

/usr/bin/rsync -crtpogvlHDisuz --rsh=ssh --progress --delete --stats --exclude=/dev/* --exclude=/tmp/* --exclude=/proc/* --exclude=/run/* --exclude=/sys/* --exclude=/home/*/.cache/mozilla/* --exclude=/home/*/.cache/thumbnails/* --exclude=/home/*/.cache/thunderbird/* --exclude=/root/.cache/mozilla/* --exclude=/root/.cache/thumbnails/* --exclude=/root/.cache/thunderbird/* --exclude=/var/cache/distfiles/* --exclude=/var/db/repos/gentoo/*  root@tosh.itw.lan:/ /home/backups/TOSH-FULL-Backup-21-08-2024/ | tee -a /var/log/TOSH-Backup.log

date | tee -a /var/log/TOSH-Backup.log

echo TOSH-Backup end


To restore to the running laptop:
Code:
#!/bin/sh

echo TOSH-Restore start
date | tee /var/log/TOSH-Restore.log

/usr/bin/rsync -crtpogvlHDisuz --rsh=ssh --progress --delete --stats --exclude=/dev/* --exclude=/tmp/* --exclude=/proc/* --exclude=/run/* --exclude=/sys/* --exclude=/home/*/.cache/mozilla/* --exclude=/home/*/.cache/thumbnails/* --exclude=/home/*/.cache/thunderbird/* --exclude=/root/.cache/mozilla/* --exclude=/root/.cache/thumbnails/* --exclude=/root/.cache/thunderbird/* --exclude=/var/cache/distfiles/* --exclude=/var/db/repos/gentoo/*  /home/backups/TOSH-FULL-Backup-21-08-2024/ root@tosh.itw.lan:/ | tee -a /var/log/TOSH-Backup.log

date | tee -a /var/log/TOSH-Restore.log

echo TOSH-Restore end


To restore to a new drive, I boot the laptop using "Minimum Installation CD", create, format and mount the required partitions, then run TOSH-Restore. When completed, I 'chroot' and run 'grub-install /dev/sda' and 'grub-mkconfig -o /boot/grub/grub.cfg'.
_________________
Observation after 30 years working with computers:
All software has known and unknown bugs and vulnerabilities. Especially software written in complex, unstable and object oriented languages such as perl, python, C++, C#, Rust and the likes.
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2187

PostPosted: Sun Aug 25, 2024 10:56 am    Post subject: Reply with quote

FWIW I take backups and copy disks and so forth in Single User mode:
Code:
openrc single

IIUC that should mean the only source of processes that can update things is the terminal on which I'm running (as root), so if there is background I/O activity, it's my fault.

I presume/hope there's a systemd equivalent.
_________________
Greybeard
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3461
Location: Canada

PostPosted: Sun Aug 25, 2024 5:42 pm    Post subject: Reply with quote

Sorry for somewhat hijacking the thread, but I just resized/moved partitions and /home filesystem around to make extra space for root / . (*) Now I am in position to delete and recreate larger root partition and expand the filesystem.
What stops is a question whether I can do it on life partition (should be, second step is definitely doable), but more that I will get a new PARTUUID and that is how root is build in into my EFI stub kernel parameters. So I can rebuild kernel with new parameter, but is there a way to change manually PARTUUID the way filesystem UUID can be changed with tune2fs ? Or are some advance versions of fdisk ( I tend to use this primitive one) allow setting PARTUUID when creating the partition ?

The danger is of course that if anything goes wrong I'll be unbootable

(*) having experienced remarkable hardware problems on the way, but that is another story
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22877

PostPosted: Sun Aug 25, 2024 7:10 pm    Post subject: Reply with quote

As I read the interactive help of fdisk, you can enter expert mode, then use u to change the partition UUID.
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3461
Location: Canada

PostPosted: Sun Aug 25, 2024 9:50 pm    Post subject: Reply with quote

Hu wrote:
As I read the interactive help of fdisk, you can enter expert mode, then use u to change the partition UUID.


Right, thanks ! Never new there is something interesting behind 'x' command :)
Last thing, to convince oneself that I can delete and recreate the increased partition on the mounted drive despite the warnings :) (yes, with the same starting block, of course)
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3477

PostPosted: Mon Aug 26, 2024 8:31 am    Post subject: Reply with quote

You can.
Partition table is already loaded to memory, your system wont even see the change until you rescan (though fdisk should trigger it automatically after you write changes; if it fails, partprobe is your friend).
_________________
Make Computing Fun Again
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3461
Location: Canada

PostPosted: Mon Aug 26, 2024 10:51 pm    Post subject: Reply with quote

szatox wrote:
You can.
Partition table is already loaded to memory, your system wont even see the change until you rescan (though fdisk should trigger it automatically after you write changes; if it fails, partprobe is your friend).


Thanks for confirmation, I thought that it should be fine, but ... It worked fine at the end. I even did not mistype PARTUUID I needed to have :) Did not know about partprobe, useful info
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum