| View previous topic :: View next topic |
| Author |
Message |
Vieri
l33t
Joined: 18 Dec 2005
Posts: 921
|
 Posted: Tue Mar 18, 2025 12:58 pm Post subject: [SOLVED] understanding rsync and incremental backups |
 |
|
Hi,
If I rsync from one dir to another on the same ext4 fs I get the following expected behavior:
| Code: | # mkdir data
# touch data/file1
# touch data/file2
# touch data/file3
# mkdir backup
# rsync -avi data backup/
sending incremental file list
cd+++++++++ data/
>f+++++++++ data/file1
>f+++++++++ data/file2
>f+++++++++ data/file3
sent 250 bytes received 77 bytes 654.00 bytes/sec
total size is 0 speedup is 0.00
# echo "change" >> data/file2
# rsync -avi data backup/
sending incremental file list
>f.st...... data/file2
sent 180 bytes received 36 bytes 432.00 bytes/sec
total size is 7 speedup is 0.03 |
So the second run of rsync only copies file2. Good.
Now, if I do the same operation but the backup target is in another fs:
| Code: | # mkdir /mnt/KINGSTON/backup
# rsync -avi data /mnt/KINGSTON/backup/
sending incremental file list
cd+++++++++ data/
>f+++++++++ data/file1
>f+++++++++ data/file2
>f+++++++++ data/file3
sent 261 bytes received 77 bytes 676.00 bytes/sec
total size is 7 speedup is 0.02
# echo "change" >> data/file2
# rsync -avi data /mnt/KINGSTON/backup/
sending incremental file list
.d..tp..... data/
>f..tp..... data/file1
>f.stp..... data/file2
>f..tp..... data/file3
sent 274 bytes received 77 bytes 702.00 bytes/sec
total size is 21 speedup is 0.06
# rsync -avi data /mnt/KINGSTON/backup/
sending incremental file list
.d..tp..... data/
>f..tp..... data/file1
.f...p..... data/file2
>f..tp..... data/file3
sent 213 bytes received 61 bytes 548.00 bytes/sec
total size is 21 speedup is 0.08 |
Even the unchanged files are copied over again.
| Code: | The device contains 'vfat' signature and it will be removed by a write command. See fdisk(8) man page and --wipe option for more details.
Command (m for help): p
Disk /dev/sdf1: 461.03 GiB, 495025979392 bytes, 966847616 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2c6b7369
Device Boot Start End Sectors Size Id Type
/dev/sdf1p1 1936028272 3787887330 1851859059 883G 68 unknown
/dev/sdf1p2 1330184192 1869160479 538976288 257G 79 unknown
/dev/sdf1p3 538989391 1937352302 1398362912 666.8G 53 OnTrack DM6 Aux3
/dev/sdf1p4 1394627663 1394648999 21337 10.4M 49 unknown
Partition table entries are not in disk order.
# findmnt /mnt/KINGSTON
TARGET SOURCE FSTYPE OPTIONS
/mnt/KINGSTON
/dev/sdf1
vfat rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,fl
|
So, instead of -a I tried to use -c:
| Code: | # rsync -rcvi data /media/vieri/KINGSTON/backup/
sending incremental file list
sent 161 bytes received 17 bytes 356.00 bytes/sec
total size is 21 speedup is 0.12
# echo "change" >> data/file2
# rsync -rcvi data /media/vieri/KINGSTON/backup/
sending incremental file list
>fcsT...... data/file2
sent 232 bytes received 36 bytes 536.00 bytes/sec
total size is 28 speedup is 0.10
# rsync -rcvi data /media/vieri/KINGSTON/backup/
sending incremental file list
sent 161 bytes received 17 bytes 356.00 bytes/sec
total size is 28 speedup is 0.16
|
That seems to do the trick, but why?
Could be a timestamp or permission/ownership issue as the target fs is different.
The problem with -rc in rsync is that it takes longer to back up a really big directory.
What are my options here?
Should I stick with -c, or should I change the fs in the backup device so -a will "work" for incremental backups?
In the latter case, which fs is more suitable for different operating systems and devices (such as TVs, etc.)?
Unfortunately, I need compatibility with Windows and Android.
Would ExFAT be the best fs for the backup device?
As a side note, I don't understand fdisk's output above. So I ran parted instead, and I got this output which makes more sense to me:
| Code: | (parted) print
Model: Kingston DataTraveler 3.0 (scsi)
Disk /dev/sdf: 495GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 237MB 495GB 495GB primary fat32 boot, lba |
Thanks
Last edited by Vieri on Tue Mar 18, 2025 4:32 pm; edited 1 time in total |
|
| Back to top |
|
 |
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23327
|
| Posted: Tue Mar 18, 2025 1:12 pm Post subject: |
|
|
The problem is not that you are crossing filesystems. The problem is that your destination filesystem is incapable of accurately representing the source metadata, so despite rsync's best efforts, the destination filesystem is not fully synced with the source. Specifically, since you used a FAT-based filesystem as the destination, the modification times are not necessarily right. This is mentioned in the rsync manual page, along with a workaround: | man rsync: | --modify-window=NUM, -@
When comparing two timestamps, rsync treats the timestamps as
being equal if they differ by no more than the modify-window
value. The default is 0, which matches just integer seconds.
If you specify a negative value (and the receiver is at least
version 3.1.3) then nanoseconds will also be taken into account.
Specifying 1 is useful for copies to/from MS Windows FAT
filesystems, because FAT represents times with a 2-second reso‐
lution (allowing times to differ from the original by up to 1
second). |
If you used any of the major Linux filesystems (ext4, xfs, btrfs, etc.), the metadata would be correct and rsync would not repeat the transfer.
Since FAT also does not have proper ownership tracking, you may need to remove -a and use the subset of a's options that cover only the features FAT can provide.
If you need compatibility with Microsoft Windows, then I think there are no good choices. |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 921
|
| Posted: Tue Mar 18, 2025 4:32 pm Post subject: |
|
|
seems to do the job in my case.
Thank you very much. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
