| View previous topic :: View next topic |
| Author |
Message |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
 Posted: Sun Aug 18, 2013 6:10 pm Post subject: Compressing filesystems with squashfs and squashmount |
 |
|
Admin edit: Split from TIP: Compressing portage using squashfs: initscript method --pjp
Proudly announcing the successor of squash_dir - written by the maintainer of squash_dir in one weekend... 
squashmount works generically for all init-systems. Its user interface was developed from the beginning to support multiple mount-points.
It does not rely on the init-system to manage the mounts but records the mounts on its own.
Special runtime configuration features of the mounted directories are not managed by "magic files" as in squash_dir but by a clean control interface which is able to save the states until the next reboot.
squashmount is available for gentoo users from the mv overlay or for other users from github. |
|
| Back to top |
|
 |
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Mon Aug 19, 2013 8:56 am Post subject: |
|
|
what is the status of overlayfs? is that still out of kernel code?
i got a bit fed up with aufs holding me back all the time (and unionfs being a bit unreliable with tmpfs branches), so i stopped using the script for a while.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Aug 19, 2013 1:22 pm Post subject: |
|
|
| yoshi314 wrote: | | what is the status of overlayfs? is that still out of kernel code? |
It is unclear to me: Linus has announced once that it should be included, but it seems nothing has happened since then. According to this page it has been mainlined in Linux kernel 3.11 (and contains an officially looking link), but it is not clear to me what this means: The usual announcements in the german Heise forum did not mention it (and these announcements are usually very complete), and also when browsing the kernel git repository, I could not see it. |
|
| Back to top |
|
|
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Mon Aug 19, 2013 7:13 pm Post subject: |
|
|
there are syntax errors when using example ,e.g. the example has to be:
| Code: | push(@mounts, {
TAG => 'db',
DIR => '/var/db',
FILE => '/var/db.sqfs',
BACKUP => '/var/db.sqfs.bak', # keep a backup
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
|
also, i cannot get it to work with unionfs-fuse package :
| Quote: | [root@box ~]# squashmount -v start
[db]: mounting...
Failed to open /root/allow_other/: No such file or directory. Aborting!
squashmount: warning:[db]: unionfs-fuse failed
squashmount: error:[db]: fallback to mount --bind
[portage]: mounting...
Failed to open /root/allow_other/: No such file or directory. Aborting!
squashmount: warning:[portage]: unionfs-fuse failed
squashmount: error:[portage]: fallback to mount --bind
|
my config :
| Code: | @order = ( 'unionfs-fuse' );
push(@mounts, {
TAG => 'db',
DIR => '/var/db',
FILE => '/var/db.sqfs',
BACKUP => '/var/db.sqfs.bak', # keep a backup
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
THRESHOLD => '30m',
BLOCKSIZE => 65536
}, {
TAG => 'portage',
DIR => '/usr/portage',
FILE => '/usr/portage.sqfs',
CHANGES => '/usr/portage.changes',
READONLY => '/usr/portage.readonly',
THRESHOLD => '40m' # resquash on umount if 40 megabytes changed
}); |
i have unionfs-fuse installed, perhaps i need to enable unionfs then ?
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Aug 20, 2013 8:10 pm Post subject: |
|
|
Thanks for the report.
The example file had already been fixed. Note that the provided helper functions have changed in name and functionality; meanwhile also BACKUP => 1 suffices.
The unionfs-fuse bug has been fixed in squashmount-2.2 (unionfs-fuse had not yet been tested, and there were trivial typos in the call; the same holds for unionfs [which is still untested. Does it still exist yet?]). |
|
| Back to top |
|
|
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Wed Aug 21, 2013 6:10 am Post subject: |
|
|
thanks, will test later on today.
edit: works fine now. the initial example looks a bit intimidating, but it's manageable eventually 
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Aug 22, 2013 9:01 pm Post subject: |
|
|
There were some serious issues with <=squashmount-2.4 (first restart was buggy, and the fix had a serious typo concerning umount).
However, the current squashmount-2.5 should be ok. |
|
| Back to top |
|
|
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Mon Aug 26, 2013 10:36 am Post subject: |
|
|
how do i disable automatic rebuilds of squashfs images on shutdown? will commenting out the threshold do, or do i have to set it to 0 or really high value?
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Aug 26, 2013 8:05 pm Post subject: |
|
|
| yoshi314 wrote: | | how do i disable automatic rebuilds of squashfs images on shutdown? will commenting out the threshold do, or do i have to set it to 0 or really high value? |
A negative threshold means "infinity" (even more, it overrides "kill"). If you only mean for the next reboot, you can use something like | Code: | | squashmount -n set # -n is the short option for --nosquash |
which does nothing else than setting temporarily (in /run/squashmount) the threshold to -1. |
|
| Back to top |
|
|
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Mon Sep 02, 2013 10:23 am Post subject: |
|
|
after recent update to 2.8 ( i think ) i could not restart existing mounts.
after manually undoing all union and squashfs mounts, script mounted everything successfully. not sure if that meant i experienced some potential data loss or not.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 04, 2013 6:11 am Post subject: |
|
|
| yoshi314 wrote: | | after recent update to 2.8 ( i think ) i could not restart existing mounts. |
I had this also once, but I was unable to reproduce it. If you know how to reproduce it, please post and the bug can be fixed
(Maybe it happened from an update of an much earlier version? In this case it could be related due to the change of the format in /var/run).
| Quote: | | after manually undoing all union and squashfs mounts, script mounted everything successfully. not sure if that meant i experienced some potential data loss or not. |
It does not mean a data loss; squashmount.2.8 contains options -I and -f to help you in such an emergency (when it can). For instance | Code: | | squashmount -I umount |
will try to umount even if it "thinks" that the partition is not mounted; it might be necessary to combine this with -f to try more aggressively. |
|
| Back to top |
|
|
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Wed Sep 04, 2013 3:57 pm Post subject: |
|
|
seems to be happening again :
| Code: | [root@box ~]# squashmount status
* [db]: not mounted
* [portage]: not mounted
* [overlays]: not mounted
|
| Code: | mount
(....)
/mnt/storage/db.sqfs on /var/db.readonly type squashfs (ro,noatime)
unionfs on /var/db type fuse.unionfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
/mnt/storage/portage.sqfs on /mnt/storage/portage.readonly type squashfs (ro,noatime)
unionfs on /usr/portage type fuse.unionfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
/mnt/storage/overlays.sqfs on /mnt/storage/overlays.readonly type squashfs (ro,noatime)
unionfs on /mnt/storage/overlays type fuse.unionfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
|
my config is as it used to be :
| Code: | @order = ( 'unionfs-fuse' );
push(@mounts, {
TAG => 'db',
DIR => '/var/db',
FILE => '/mnt/storage/db.sqfs',
BACKUP => '/var/db.sqfs.bak', # keep a backup
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
# Do not resquash everytime but only after 30 megabytes of fresh data:
THRESHOLD => '30m',
# Since this directory contains only very small files, we cheat with
# this size by using that each file takes at least a full block:
# Hence, the number of files is more important for THRESHOLD than
# their size. In gentoo, one installed package thus "counts" about
# 2m in size (although it is actually only 20 very short files):
BLOCKSIZE => 65536
}, {
TAG => 'portage',
DIR => '/usr/portage',
FILE => '/mnt/storage/portage.sqfs',
CHANGES => '/mnt/storage/portage.changes',
READONLY => '/mnt/storage/portage.readonly',
THRESHOLD => '20m', # resquash on umount if 40 megabytes changed
COMPRESSION => 'lzo'
}, {
TAG => 'overlays',
DIR => '/mnt/storage/overlays',
FILE => '/mnt/storage/overlays.sqfs',
CHANGES => '/mnt/storage/overlays.changes',
READONLY => '/mnt/storage/overlays.readonly',
THRESHOLD => '10m', # resquash on umount if 40 megabytes changed
COMPRESSION => 'lzo'
});
# If /var is on a separate partition, we want that the
# squashfile is first generated in /var/tmp:
#TEMPDIR => '/var/tmp',
|
hm, something's odd :
| Code: | squashmount check -f
* [db]: error: DIR is nonempty!
* [portage]: error: DIR is nonempty!
* [overlays]: error: DIR is nonempty!
|
after manual unmounting it reports back fine.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 04, 2013 5:02 pm Post subject: |
|
|
| yoshi314 wrote: | | seems to be happening again: |
Can you reproduce what you did to make it happen?
| Quote: | hm, something's odd :
| Code: | | squashmount check -f |
|
This is not surprising: squashmount thinks that your original directory is not mounted; since it contains data (namely that from the mount which squashmount is not aware of), it reports the directory as nonempty. The bug happened before: squashmount should not have flagged the directory as umounted (in /run/squashmount) when actually it is mounted. |
|
| Back to top |
|
|
yoshi314
l33t
Joined: 30 Dec 2004
Posts: 850
Location: PL
|
| Posted: Wed Sep 04, 2013 7:09 pm Post subject: |
|
|
| mv wrote: | | yoshi314 wrote: | | seems to be happening again: |
Can you reproduce what you did to make it happen?
|
i just booted the system. journalctl logs say it started up fine, i have squashfs and unionfs mounted. only script seems to be having a different opinion on that.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 04, 2013 7:14 pm Post subject: |
|
|
| yoshi314 wrote: | | i just booted the system. journalctl logs say it started up fine, i have squashfs and unionfs mounted. only script seems to be having a different opinion on that. |
Ah, systemd. (Normally I run openrc). Then I guess that it is a dependency issue; I will report back. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 04, 2013 9:38 pm Post subject: |
|
|
The bug is finally fixed: systemd was the correct hint for reproducing (now I remembered that I was testing with systemd when the bug occured in my case).
It was a stupid issue: tmpfiles.d was configured to clean /run/squashmount. So if systemd cleaned this directory after starting squashmount (which could happen by accident since there was no corresponding dependency specified in the systemd service file), it cleaned all information which squashmount had stored about mounted dirs.
The issue is fixed in the current squashmount-2.9, and also the emergency behaviour was improved and explicitly documented. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1819
Location: PB, Germany
|
| Posted: Wed Sep 11, 2013 10:41 am Post subject: |
|
|
I took a look at the new squashmount. I really appreciate the rewrite in Perl. If things are getting bigger, bash and similars have their restrictions. Even if Perl is not the problem for me, are you going to keep the configuration /etc/squashmount.pl in plain Perl syntax? I think this is quite unusual even for projects written in Perl, at least I know Perl-only users like that, but the configuration is the user interface for non-developers.
So since migrating is not that easy as replacing squash_dir with squashmount, I stay with squash_dir at the moment, trying the switch some time later and watching your squashmount.
I had no issues with different init systems as I just did switch to OpenRC when Gentoo did announce the new default.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 11, 2013 6:34 pm Post subject: |
|
|
| Massimo B. wrote: | | going to keep the configuration /etc/squashmount.pl in plain Perl syntax? |
Yes, because it is more appropriate: I also thought first about a "plain" configuration file, but this way you can easily make conditions on something (e.g. I use the same config-file on different machines depending on the existence of some symlinks), you can make certain values (like the compression algorithm) easily the default for all your mount points etc. Moreover, for configuring several mount-points in one file, the syntax of an init-file would not be much simpler anyway.
| Quote: | | So since migrating is not that easy as replacing squash_dir with squashmount |
If you want, it can be almost that easy: Write
| Code: | push(@mounts, {
TAG => 'portage',
...
}); # do not forget these braces and semicolon!
|
and put into "..." your old configuration, essentially just replacing "=" by "=>", finishing each line except the last with a comma and putting the names into '...' or "..." (as you perhaps already did in squash_dir); repeat this for every mount point (replacing "portage" by a different name of your choice).
Only take care that some variable names have slightly changed (e.g. previous DIRECTORY is now called simpler DIR) and that some global variables have become perl variables; for a "normal" user this involves only the ORDER variable which must now be set differently: | Code: | | @order = ('overlayfs', 'aufs', 'unionfs-fuse'); # Do not forget the semicolon |
Everything else is just syntactic sougar which becomes useful if you have more mount-points (e.g. I have 12 mount-points, and such a huge configuration file becomes much shorter and easier to maintain using perl-syntax shortcuts than if I would repeat the above task 12 times, although the latter would be possible).
| Quote: | | I stay with squash_dir at the moment, |
I strongly recommend to switch, especially if you ever wanted to omit resquashing for the next boot or force squashing immediately. All this could be done with squash_dir, but is now much simpler and much more convenient. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1819
Location: PB, Germany
|
| Posted: Mon Sep 16, 2013 7:32 am Post subject: |
|
|
Hi. I was wondering about the sub added_hash{} as it only returns the hash it gets. Ok it merges sub_hashes, maybe meant to ease some default like sub standard_mount{}.
Having some little Perl knowledge I asked in #perl about this approach. Hash::Merge::Simple could be more appropriate.
But anyway, having configuration of executable code and calling functions from there also sounds unusual and PHPish to #perl people. Maybe the gained flexibility is not the most save way and not necessarily desirable. This even made me reading the code and ask Perl coders to understand the configuration example, which should not be the point of a user application. I stay tuned, I liked the squash_dir and I appreciate the new port.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Sep 16, 2013 8:15 pm Post subject: |
|
|
| Massimo B. wrote: | | Hi. I was wondering about the sub added_hash{} as it only returns the hash it gets. Ok it merges sub_hashes, maybe meant to ease some default like sub standard_mount{}. |
This is exactly the idea: That you can merge some "defaults" with other data (like standard_mount()). This is also the way it is used in the examples.
| Quote: | | Having some little Perl knowledge I asked in #perl about this approach. Hash::Merge::Simple could be more appropriate. |
This function is neither in perl-5.18.1 nor in Gentoo at all. Moreover, recursion and other tricks of that function are not needed. added_has() is practically a simplified (non-recursive) variant of Hash::Merge::Simple
| Quote: | | having configuration of executable code and calling functions from there also sounds unusual |
Nobody forces you to use these features. As I have explained (and hopefully also made clear on the manpage), you can use just a "plain" configuration. If you have perl knowledge you can use perl tricks to avoid the repetition or use other perl code for setup (e.g. source other setup files or whatever). The two functions are there to support you when doing this, but there is no need to do it.
It is a bit comparable with writing LaTeX: If you have to type similar texts several times you either can do this, or you setup some macros which avoid the repetition and allow for easy changing in the repetitions. The two provided functions are just IMHO useful "macros" in this context, but you can completely ignore them and just write the repetitions or write your own functions (e.g. use Hash::Merge::Simple if you have installed it). |
|
| Back to top |
|
|
Yamakuzure
Advocate
Joined: 21 Jun 2006
Posts: 2300
Location: Adendorf, Germany
|
| Posted: Tue Sep 17, 2013 10:31 am Post subject: |
|
|
| mv wrote: | | Code: | | @order = ('overlayfs', 'aufs', 'unionfs-fuse'); # Do not forget the semicolon |
|
Is there a reason to place overlayfs before aufs? Or in other words, are there reasons to prefer overlayfs over aufs? I am currently using aufs and hadto change that particular line, so squashmount won't complain every time that overlayfs isn't available. AFAIK overlayfs is a bit like an AUFS-light, right?
_________________
Edited 220,176 times by Yamakuzure |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Sep 17, 2013 12:31 pm Post subject: |
|
|
overlayfs is first, because it appeared when writing squashmount that it would be in the kernel, soon.
Both are more or less equivalent, although there are some known cases where directories from overlayfs do not behave exactly as required by POSIX - aufs contains a lot of code to cover all these (rare) cases correctly. |
|
| Back to top |
|
|
Yamakuzure
Advocate
Joined: 21 Jun 2006
Posts: 2300
Location: Adendorf, Germany
|
| Posted: Tue Sep 17, 2013 1:17 pm Post subject: |
|
|
Thank you very much!
_________________
Edited 220,176 times by Yamakuzure |
|
| Back to top |
|
|
188562
Apprentice
Joined: 22 Jun 2008
Posts: 186
|
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 18, 2013 9:44 pm Post subject: |
|
|
| init_6 wrote: | | bash script with systemd support. |
This is fine, but already squash_dir with systemd support could do much more (squashing only if threshold is reached, ignoring unimportant files, support for multiple mount points, fallbacks to alternatives to aufs, rather secure error handling, etc). The new implementation squashmount has even a lot more features, especially concerning the user interface (conveniently forcing/omitting resquashing immediately/later on, killing previous modification currently/next time/forever, ...) and is (after the birth problems have been fixed) probably also the safest choice concerning various error handing. The price for the features is a somewhat more complex setup of the mount-points, but I hope that after the setup the actual user interface is rather intuitive.
BTW, here is another argument, why it is useful to have a setup file in perl: You can use it to set the order of the tools depending on a test what your current kernel does support. Examples will probably be given in the next release of squashmount. Here just an untested one: | /etc/squasmount.pl wrote: | if(system('modprobe aufs >/dev/null 2>&1')) {
@order = ('aufs', 'unionfs-fuse')
} elsif(system('modprobe overlayfs >/dev/null 2>&1'))
@order = ('overlayfs', 'unionfs-fuse')
} else {
@order = ('unionfs-fuse')
} |
|
|
| Back to top |
|
|
|
 |
Gentoo Forums Forum Index
 Documentation, Tips & Tricks |
All times are GMT
Goto page 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 Next
|
| Page 1 of 10 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
