Compressing filesystems with squashfs and squashmount

[Massimo B.]

[mv]

[Massimo B.]

[Massimo B.]

[Lomaxx]

Hello.

I've been using squashmount for quite some while by now. Since i did a world-update (including kernel-update) yesterday, i first broke my aufs-support, then repaired it and now have got it working again, BUT with an annoying delay while booting that I didn't experience before. The delay is altogether roughly 5-10 seconds long for 3 squashed directories (portage/db/layman), i do not receive any errors while booting and i wonder what might be causing it. Before there was no delay at all.
If someone got a close guess it would help me somewhat. Otherwise I have to try to solve it by trial and error.

Some info on my system:

- sys-kernel/aufs-sources-3.16.1
- sys-fs/aufs-headers-3.16_p20140818:0
- sys-fs/aufs-util-3.16_p20140818:0
- sys-fs/squashfs-tools-4.2-r1:0
- sys-fs/squashmount-6.0a:0

Kernel-Config regarding aufs:

CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
# CONFIG_AUFS_EXPORT is not set
# CONFIG_AUFS_FHSM is not set
# CONFIG_AUFS_RDU is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set

extracts from squashmount.pl:

#!/usr/bin/perl (this is only for editors)
@order = qw(aufs! unionfs-fuse! unionfs??# funionfs??#);
my $defaults = {
COMPRESSION => 'xz', # We could omit this line as xz is default.
# However, this might change in the future
COMPOPT_LZ4 => '-Xhc', # We could omit this line as -Xhc is default
COMPOPT_XZ => ['-Xbcj', 'x86'] # Use this in case COMPRESSION => 'xz'
};
my $non_binary = {
COMPOPT_XZ => undef # "-Xbcj x86" is slower for pure text archives
};

@mounts = (
added_hash($defaults, $non_binary, {
TAG => 'db',
DIR => '/var/db',
FILE => '/var/db.sqfs',
BACKUP => '.bak', # keep a backup in /var/db.sqfs.bak
# For an absolute path, we could have written:
# BACKUP => '/backup-disk/db.sqfs'
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
THRESHOLD => '30m',
BLOCKSIZE => 65536
}),
added_hash($defaults, $non_binary, {
TAG => 'layman',
DIR => '/var/lib/layman',
FILE => '/var/lib/layman.sqfs',
BACKUP => '.bak', # keep a backup in /var/db.sqfs.bak
# For an absolute path, we could have written:
# BACKUP => '/backup-disk/db.sqfs'
CHANGES => '/var/lib/layman.changes',
READONLY => '/var/lib/.readonly',
THRESHOLD => '30m',
BLOCKSIZE => 65536
}),

standard_mount('portage', '/usr/portage', $defaults, $non_binary, {
THRESHOLD => '80m',
FILL => qr{^local/(?!(\.git|profiles|metadata)(/|$))}
}),
);

As a solution I thought about rebuilding the directories freshly or enabling the userspace-option in the kernel, but haven't tried that yet. Any help would be appreciated.

[Lomaxx]

Out of curiosity i measured the times and noticed that my 5-10-seconds-guess was a bit underestimated:

Each directory takes 15 seconds to mount although their sizes differ.
So in total it takes 45 seconds.

$ ls -l /usr/portage.sqfs
-rw-r--r-- 1 root root 76918784 22. Aug 15:49 /usr/portage.sqfs
$ ls -l /var/db.sqfs
-rw-r--r-- 1 root root 31797248 22. Aug 15:48 /var/db.sqfs
$ ls -l /var/lib/layman.sqfs
-rw-r--r-- 1 root root 11046912 29. Mai 14:19 /var/lib/layman.sqfs

CPU is a Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz with 16GB ram.

[mv]

Massimo B., thanks for testing. There was a regression in an apparently trivial code change in v7.6 (a negation was missing) so that the detection of IO::Uncompress::Gunzip was inverted.

[mv]

[Lomaxx]

@mv: Good to know, bad to suffer. Thanks for the feedback. If no cure is known, then I will fiddle around and report back in case i find a solution.

[Massimo B.]

Beside all the patching around compression like LZ4 and overlayfs/aufs, can you give a short overview what is supported by main line kernels, what is easy to patch / add as userspace tools, and what is known to be broken or fiddly.

BTW. looking for backup strategies I got to know one more time btrfs, supporting transparent compression, incrementing snapshots, subvolumes etc. This would ease a lot of single mounts with squashmount, however btrfs is said to be still experimental although some enterprise linux has enabled this officially.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[mv]

[Massimo B.]

Thanks for your opinion about btrfs making things clearer.

Adding new squashes still does not work:

[mv]

[Massimo B.]

[Massimo B.]

Feature request: Is there a way to have THRESHOLD as relative levels like "0.2" (20%) or something? For binary only like /usr/lib/icedtea7 a change usually means recompression, so the default 0 is ok. But for dirs that are often changing like /var/db I usually like to have some relative level of difference instead of hard thresholds like 20 MB, no?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[mv]

[Massimo B.]

[mv]

[Massimo B.]

True, those were hard compiled.
Weird, I got the mount delay problem fixed now , be resquashing everything, and using unionfs-fuse only. This should have been some aufs3 related issue though it did not look like...
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[mv]

[Massimo B.]

The delay at mount -t squashfs is back again. It appeared again when switching around with COMPRESSION lzo and gzip and remounting.
I also got to know that some "forget" and forced create also may drop the existing squashfs and loosing all data. So I should be careful with important data such as /var/db.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[mv]

[Massimo B.]

How do I preserve permissions in the squashfs? There is something wrong. I did chmod -R texlive:texlive /usr/local/texlive. Then after resquashing, the permissions are reset to root, and some updates I did before are lost. I guess the resquashing is failing somehow and some old squashfs is mounted. This time the mounting did take more than 10 minutes.

[mv]

[Massimo B.]