| View previous topic :: View next topic |
| Author |
Message |
costel78
Guru
Joined: 20 Apr 2007
Posts: 407
|
 Posted: Sun Jul 27, 2014 8:45 pm Post subject: |
 |
|
When I try to stop squashmount service I get:
| Code: | iul 27 23:32:16 gentoo squashmount[999]: * [portage]: umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error: non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error: lazy umount failed: /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: * [db]: umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]: error: non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /var/db
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]: error: lazy umount failed: /var/db |
Packages version and use flags:
| Code: | [ebuild R ] sys-fs/aufs-headers-3.15_p20140728 0 kB
[ebuild R ] sys-fs/aufs-util-3.15_p20140728 0 kB
[ebuild R ] sys-kernel/aufs-sources-3.15.6:3.15.6 USE="experimental symlink -build -deblob -module -vanilla" 0 kB
[ebuild R ] sys-fs/squashmount-7.7::added 0 kB
[ebuild R ] sys-fs/squashfs-tools-4.3::added USE="xz -lz4 -lzma -lzo -xattr" 0 kB |
Config file:
| Code: | #!/usr/bin/perl (this is only for editors)
# The tools which we have installed; if possible only the first in this list
# is used, but the others are a fallback if that fails.
@order = ('aufs', 'overlayfs', 'unionfs-fuse', 'unionfs', 'funionfs');
# Even if we define following is empty it is convenient to use
# this local variable throughout, so that we can simply change it:
my $defaults = {
COMPRESSION => 'xz'
};
push(@mounts, {
TAG => 'portage',
DIR => '/usr/portage',
FILE => '/usr/portage.sqfs',
CHANGES => '/usr/portage.changes',
READONLY => '/usr/portage.readonly',
THRESHOLD => '50m' # resquash on umount if 40 megabytes changed
}, {
TAG => 'db',
DIR => '/var/db',
FILE => '/var/db.sqfs',
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
THRESHOLD => '30m' # resquash on umount if 40 megabytes changed
}
); |
Obviously I use systemd and I didn't notice the error but during shutdown/reboot system take unusually long and culprit seems to be squashmount. It recompress despite the *.changes are well bellow threshold.
I tried to start from scratch with same results: with squashmount started compress portage and db directories, disable squashmount service, reboot, put data back and start squashmount. It does not recompress the whole thing unless threshold is reached but systemctl stop squashmount fail with above errors. Kernel config didn't change recently.
Any help is really appreciated. Thank you!
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
 |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Jul 28, 2014 6:16 am Post subject: |
|
|
[quote="costel78"]When I try to stop squashmount service I get:
| Code: | iul 27 23:32:16 gentoo squashmount[999]: * [portage]: umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error: non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error: lazy umount failed: /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: * [db]: umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]: error: non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /var/db
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]: error: lazy umount failed: /var/db |
/quote]
This is an aufs problem in your installation; perhaps recompiling aufs-util helps.
I had some problems without /sbin/umount.aufs, too, and have decided to remove it (Some hardlink information on the filesystems might get lost, but this is not needed for the portage tree). Perhaps, a variable should be added to squashmount by which one can add umount options like -I for each mountpoint separately.
Note that squashmount has no sane way to recover from this error: Removing the data in CHANGES might even crash the system after such a bug, and keeping it, leads to inconsistent states.
| Quote: | | It recompress despite the *.changes are well bellow threshold. |
Did you take into account that if you set a huge BLOCKSIZE the threshold is reached much earlier if you have many small files (as in the portage directory?)
Does the recompression happen even if shows that it should not happen? (This would be rather strange) |
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 407
|
| Posted: Mon Jul 28, 2014 8:11 am Post subject: |
|
|
Recompiling sys-fs/aufs-util-3.15_p20140728 did not help.
Recompressing on every stop did not occurred since starting from scratch.
gentoo costel # squashmount list
* [portage]: aufs (50m), modified, but will not resquash
* [db]: aufs (30m), modified, but will not resquash
To be on the safe side I'll stop using squashmount for a while (yes, it's clear now for me that IT IS NOT his fault) on portage and db. I'll keep a small directory on squashmount to monitor when problem will disappear.
Thank for support!
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Jul 28, 2014 2:37 pm Post subject: |
|
|
| costel78 wrote: | | Recompiling sys-fs/aufs-util-3.15_p20140728 did not help. |
I cannot help here.
In squashmount-7.8 there is support added for umount options (one of the few things squashmount was missing compared to squash_dir).
In your situation, it might help to add to your | /etc/squashmount.pl wrote: | | @umount = ('-i'); |
(with squashmount-7.8).
This is somewhat a hack, but AFAIK the worst thing which can happen by this is that some hardlink information gets lost, i.e. files are stored in duplicate instead of once. Except for very special directories this is not an issue, and for the /usr/portage and /var/db directories, hardlinks are not used, anyway. |
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 407
|
| Posted: Mon Jul 28, 2014 8:32 pm Post subject: |
|
|
I had this evening more time to investigate. The error appear with >=sys-fs/aufs-util-3.15_p20140721. I reported bug 518418.
Unfortunately, umount options didn't change the behaviour. It still failed to umount, but I masked | Code: | >=sys-fs/aufs-headers-3.15_p20140721
>=sys-fs/aufs-util-3.15_p20140721 |
and squashmount it running flawless again 
I put a squashmount restart on my update script, so will be more easy to discover if something goes wrong from now on.
Thank you for working on squashmount! Nice piece of software.
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
lost+found
Guru
Joined: 15 Nov 2004
Posts: 509
Location: North~Sea~Coa~s~~t~~~
|
| Posted: Tue Jul 29, 2014 6:24 am Post subject: |
|
|
| I can recommend using unionfs-fuse with gzip, if file size doesn't matter. It has proven to be fast and reliable to me. I'm using squashmount mainly to speed up emerge/portage/kernel source operations on JFS filesystems. I think it prevents file system fragmentation too. A very useful tool indeed! |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Jul 29, 2014 7:09 am Post subject: |
|
|
| costel78 wrote: | | It still failed to umount |
But I hope, not because of complaining about a problem with /sbin/umount.aufs; it would be a bug in either squashmount or umount (or a typo in your configuration...) if this tool would have been called. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Jul 29, 2014 7:21 am Post subject: |
|
|
| lost+found wrote: | | I can recommend using unionfs-fuse with gzip, if file size doesn't matter |
If file size does not matter, you should use lz4: You get an enormous speed increase, see the file compression.txt in current versions of squashmount.
However, be aware that this cannot be used with <linux-3.16; I hope that it will be included in linux-3.16, finally.
When a kernel supporting it is out for quite while, it might become default method for squashmount. (Take this a warning for those who need to use older kernels: For new releases, be aware to read the ChangeLog if you need to set something) |
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 407
|
| Posted: Tue Jul 29, 2014 7:23 pm Post subject: |
|
|
| mv wrote: | | costel78 wrote: | | It still failed to umount |
But I hope, not because of complaining about a problem with /sbin/umount.aufs; it would be a bug in either squashmount or umount (or a typo in your configuration...) if this tool would have been called. |
Yes, exactly same error. Maybe config file is wrong ?
| Code: | #!/usr/bin/perl (this is only for editors)
# The tools which we have installed; if possible only the first in this list
# is used, but the others are a fallback if that fails.
@order = ('aufs', 'overlayfs', 'unionfs-fuse', 'unionfs', 'funionfs');
# Even if we define following is empty it is convenient to use
# this local variable throughout, so that we can simply change it:
my $defaults = {
COMPRESSION => 'xz'
};
push(@mounts, {
TAG => 'portage',
DIR => '/usr/portage',
FILE => '/usr/portage.sqfs',
CHANGES => '/usr/portage.changes',
READONLY => '/usr/portage.readonly',
THRESHOLD => '50m' # resquash on umount if 40 megabytes changed
}, {
TAG => 'db',
DIR => '/var/db',
FILE => '/var/db.sqfs',
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
THRESHOLD => '50m' # resquash on umount if 40 megabytes changed
}
);
@umount = ('-i'); |
Anyway, manually using squashfs and aufs result in the same result with laters aufs-utils.
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Jul 30, 2014 8:28 am Post subject: |
|
|
There's a bug in this new feature of squashmount :oops:
@umount and @umount_ro are used oppositely as intended/documented in 7.8 and 7.9. This is fixed in 7.10.
Thus (temporarily for testing until you install 7.10) you should replace @umount by @umount_ro.
| Quote: | | Anyway, manually using squashfs and aufs result in the same result with laters aufs-utils |
Did you use option -i with umount when doing things manually? |
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 407
|
| Posted: Wed Jul 30, 2014 11:28 am Post subject: |
|
|
| Code: | [ebuild R ] sys-fs/aufs-headers-3.15_p20140728 0 kB
[ebuild R ] sys-fs/aufs-util-3.15_p20140728 0 kB
[ebuild R ] sys-fs/squashmount-7.10::added 0 kB |
Everything is working just fine. 
Manually, error occurred without -i. With it set, it worked.
Thank you very much!
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Fri Aug 15, 2014 6:15 am Post subject: |
|
|
It feels like since the latest update | Code: | | Wed Aug 13 12:04:32 2014 >>> sys-fs/aufs3-3_p20140811 |
the mounting takes very long. It does not fail but takes around 20 seconds. That makes booting up take very long for around 10 squash directories, while there are no errors in logfiles. Is there any verbose mode?
| Code: | $ squash_dir start portage
Starting portage...
* Mounting /usr/portage.sqfs as /usr/portage ... [ ok ] |
| Code: | $ mount |grep "/usr/portage"
/usr/portage.sqfs on /usr/portage.readonly type squashfs (ro,noatime)
aufs on /usr/portage type aufs (rw,noatime,si=782a68c3173c92b0) |
Yes, I'm still using squash_dir.
While there was no failure now for a long time but I remember hassle when forced to use a boot-cd to re-emerge some things, is there some emergency start of squash_dir or your new squashmount, that usually is able to mount all directories on well stuffed live-cd environment from inside Gentoo-CHROOT?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Fri Aug 15, 2014 7:48 am Post subject: |
|
|
| Massimo B. wrote: | It feels like since the latest update | Code: | | Wed Aug 13 12:04:32 2014 >>> sys-fs/aufs3-3_p20140811 |
the mounting takes very long. |
Since aufs3 was lagging behind for a while and perhaps only overlayfs will go into upstream kernel, I have currently only kernels with overlayfs for testing.
(However, squash_dir currently does not support latest changes in overlayfs, and unless someone sends patches, I will not update it anymore - squash_dir is now really abandoned...)
How does it work if you send the commands manually?
Something like | Code: | modprobe squashfs
mount -t squashfs -o loop,ro,noatime -- /path/to/SQUASHFILE /path/to/READONLY
modprobe aufs3
mount -t aufs [your options from MOUNT_AUFS] -o noatime -o br:/path/to/CHANGES=rw:/path/to/READONLY=rr -- aufs /path/to/DIRECTORY |
Probably, you want to use -v as an additional option to the last mount command.
If the latter command really just takes 20 seconds and writes nothing out (into systemlog or stdout/stderr), I am afraid that I cannot help: You have to ask the aufs author.
| Quote: | | Is there any verbose mode? |
Not in squash_dir. In squashmount, there is: If you specify -v sufficiently often, you will see the above commands. But this won't help either, if these commands do not produce any usable output...
| Quote: | | is there some emergency start of squash_dir or your new squashmount, that usually is able to mount all directories on well stuffed live-cd environment from inside Gentoo-CHROOT? |
No. If you hav a suggestion for a corresponding configuration file for squashmount (or perhaps need some functionality added for this), please send it per PM or report a bug on the squashmount bugtracker on github (or directly send a pull request if you have patches ready  ) |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Aug 16, 2014 4:03 pm Post subject: |
|
|
| Due to the recent discussion, squashmount has now obtained some enhanced configuration possibilities (e.g. suitable for non-permanent mount-points like for CDs). |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Wed Aug 20, 2014 11:21 am Post subject: |
|
|
Ok, I started migration to squashmount:
It seems /etc/squashmount.pl is the one and only configuration. Please make the config file configurable or distribute the example squashmount.pl additionally in /usr/share/doc/squashmount*.
Because lot of the example is enabled which I don't need, it is harder to keep the example and merging new changes. You could also place some /etc/squashmount.pl.example beside. Now that I compressed my final version of config I lost the useful comments.
As you mention there "do not use it unchanged" is a but unusual, so it would better to start with a working minimal example and the rest commented... but ok, let's start:
My compressed version: | Code: | use Sys::Hostname;
my $hostname = ($ENV{'HOSTNAME'} // hostname());
@order = qw(overlayfs aufs! unionfs-fuse! unionfs??# funionfs??#);
my $defaults = {
COMPRESSION => 'lz4',
COMPOPT_LZ4 => '',
};
my $non_binary = {
COMPOPT_XZ => undef # "-Xbcj x86" is slower for pure text archives
};
@mounts = (
standard_mount('adobe', '/opt/Adobe', $defaults),
standard_mount('firefox', '/usr/lib/firefox', $defaults),
standard_mount('icedtea6', '/usr/lib/icedtea6', $defaults),
standard_mount('icedtea7', '/usr/lib/icedtea7', $defaults),
standard_mount('layman', '/var/lib/layman', $defaults),
standard_mount('libreoffice', '/usr/lib/libreoffice', $defaults),
standard_mount('local_portage', '/usr/local/portage', $defaults),
standard_mount('vmware', '/opt/vmware', $defaults),
standard_mount('db', '/var/db', $defaults),
standard_mount('tex', '/usr/share/texmf-dist', $defaults, $non_binary, {
DIFF => [
qr{^ls-R$},
qr{^tex(?:/generic(?:/config(?:/language(?:\.(?:dat(?:\.lua)?|def)))?)?)?$}
]
}),
standard_mount('portage', '/usr/portage', $defaults, $non_binary, {
UMOUNT => ((@umount) ? undef : '-i'),
THRESHOLD => '80m',
FILL => qr{^local/(?!(?:\.git|profiles|metadata)(?:/|$))}
})
);
'EOF'; |
I disabled the old squash_dir setup by copying all squashed data back, so starting from scratch. But squashmount start fails for all items: | Code: | $ squashmount start -v
* [adobe]: It seems this is mounted for the first time:
The squashed file /opt/Adobe.mount/Adobe.sfs does not exist yet;
it will be initialized now from /opt/Adobe
Could not create destination file: No such file or directory
* [adobe]: error: failed: /usr/bin/mksquashfs /opt/Adobe /opt/Adobe.mount/Adobe.sfs -noappend -quiet -comp lz4 |
PS.: It seems that app-arch/lz4 is there but Kernel support was missing. Now I have lz4 modules loaded but start is still failing for mksquashfs.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Aug 20, 2014 5:50 pm Post subject: |
|
|
| Massimo B. wrote: | | Please make the config file configurable |
I do not know what you mean by that. There are options to add additional config files, but they cannot be used reasonably by the default /etc/init.d files (or the default systemd units, respectively).
| Quote: | | Because lot of the example is enabled which I don't need, it is harder to keep the example and merging new changes. |
Yes, but this is normal for complex configuration files: You have similar issues with cups, wwwoffle, ssh, mtools, tor, privoxy, nvidia, ... even for some files in /etc/cron.*, /etc/modules.load.d, /etc/logrotate.d
| Quote: | | You could also place some /etc/squashmount.pl.example beside. |
You can just make a copy for yourself.
There is now a USE=example which installs into /etc/squashmount-example.pl, but I am not sure whether this is a good idea. Maybe this will be removed, again.
| Quote: | | COMPRESSION => 'lz4' |
Unless you patched the kernel manually, you will not be happy with that: Mounting will fail...
[quote]error: failed: /usr/bin/mksquashfs /opt/Adobe /opt/Adobe.mount/Adobe.sfs -noappend -quiet -comp lz4[/code]
Ah, I see: This command is "correct", but it fails because /opt/Adobe.mount does not exist. Of course, such things usually happen only on the first installation and thus are not found in a "normal" debugging procedure.
squashmount-8.4 now generates the corresponding parent files in advance.
| Quote: | | It seems that app-arch/lz4 is there but Kernel support was missing. |
That's the problem: You can squash and (manually) unsquash with lz4, but you cannot use lz4 "regularly" unless you patch the kernel.
The kernel patch exists since almost a year (since lz4 is in the kernel), and it is only a few lines, but some kernel developer refused it with the stupid claim that it be not necessary that squashfs can compress quickly. I hope that the patch will be included, finally, but I am not very optimistic, meanwhile. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Thu Aug 21, 2014 9:11 am Post subject: |
|
|
| mv wrote: | | Massimo B. wrote: | | Please make the config file configurable |
I do not know what you mean by that. |
I meant an option to choose the config file, maybe via /etc/conf.d/squashmount. But maybe this is not the best idea either. For now I need to copy a new /etc/squashmount.pl behind of etc-update to keep my config and still get some new examples.
| mv wrote: | | Quote: | | Because lot of the example is enabled which I don't need, it is harder to keep the example and merging new changes. |
Yes, but this is normal for complex configuration files: You have similar issues with cups, wwwoffle, ssh, mtools, tor, privoxy, nvidia, ... even for some files in /etc/cron.*, /etc/modules.load.d, /etc/logrotate.d |
I know. But the thing with squashmount.pl is a bit different. The default and almost commented configurations like dnsmasq, privoxy etc. are working like this. But the squashmount.pl is absolutely required to be edited for not breaking things (it would start to squash otherwise..).
Then as I mentioned some time ago it feels a bit unusual to have real Perl code as config. Even some Perl colleagues on #perl have been surprised. But I see it is very powerful but also dangerous. No config will be comparable to others as Perl code can have very different styles. But leave it like this, it is powerful and I don't have a different idea..
| mv wrote: | | There is now a USE=example which installs into /etc/squashmount-example.pl, but I am not sure whether this is a good idea. Maybe this will be removed, again. |
Maybe just leave it in /usr/share/doc/squashmount-*.
| mv wrote: | | Unless you patched the kernel manually, you will not be happy with that: Mounting will fail... |
I don't prefer too much kernel patching as there already are some patches I'm using, like sys-fs/aufs3. Then I'm using sys-kernel/ck-sources. Is there an ebuild to do the patching? I'm quite interested in plain lz4 after reading your compress.txt benchmarks. Performance is more important than size. Even on recent platforms Portage is still very slow due to its architecture and physical drives (no SSD).
BTW, how did you measure the time, like 4 (lz4) vs. 3:50 (xz) for /var/db on Core2?
So for testing I switched all to old gzip now:
| mv wrote: | Ah, I see: This command is "correct", but it fails because /opt/Adobe.mount does not exist. Of course, such things usually happen only on the first installation and thus are not found in a "normal" debugging procedure.
squashmount-8.4 now generates the corresponding parent files in advance. |
Thanks for the bugfix, pulled and working better, but still not successful. Starting from scratch with only one item:
| Code: | $ squashmount -vvv start
* squashmount: reading config file /etc/squashmount.pl
* [adobe]: It seems this is mounted for the first time:
The squashed file /opt/Adobe.mount/Adobe.sfs does not exist yet;
it will be initialized now from /opt/Adobe
/usr/bin/mksquashfs /opt/Adobe /opt/Adobe.mount/Adobe.sfs -noappend -quiet -comp gzip
[=======================================================================================================
=====================================/] 1285/1285 100%
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /root/tmp/EgZI8WpycJ
/bin/umount -- /root/tmp/EgZI8WpycJ
* [adobe]: cleaning original DIR
* [adobe]: cleaning /opt/Adobe
* [adobe]: mounting...
* [adobe]: error: no directory /opt/Adobe.mount/readonly |
Admin edit: Added line break to long line of "=" characters. --pjp
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Aug 21, 2014 12:09 pm Post subject: |
|
|
| Massimo B. wrote: | | I meant an option to choose the config file, maybe via /etc/conf.d/squashmount. |
Installing a config-file to get the location of the config-file? This sounds very wrong. Moreover, for systemd you would have to modify the unit, if you want to have this (unless you make the unit very non-systemd-ish). But of course, nobody prevents you from adding the option to your path to the systemd unit or to the init-file; such customizations are nothing which should be covered by an ebuild.
| Quote: | | But the thing with squashmount.pl is a bit different. The default and almost commented configurations like dnsmasq, privoxy etc. are working like this. But the squashmount.pl is absolutely required to be edited for not breaking things (it would start to squash otherwise..). |
No, dnsmasq, privoxy, tor etc will not do anything useful if you do not edit the config first (sshd will, but in a rather insecure way, so leaving the default is no good idea, either), mtools will even just print an error message (at least, it used to - not checked recently). Also squashmount will just print an error message and not start squashing. So I do not see much difference.
| Quote: | | Then as I mentioned some time ago it feels a bit unusual to have real Perl code as config. Even some Perl colleagues on #perl have been surprised. |
Sourcing in a separate namespace is one of the official perl recommendations for config files for perl-projects. (I read it at several places, among them some FAQ about why there is no analogue for python's ConfigParser.)
Also, it seems very clean and not hackish to me. It would be different, if the sourcing would happen in the main:: namespace (or if you need compatibility with a non-perl-tool, of course).
Also /etc/conf.d is arbitrary shell code, and I consider this as one of the main advantages of openrc, so that you can have the identical configurations for several machines and different situations - the configuration itself being conditional with user-defined conditions. (Also systemd provides such conditions, but they are much more limited and cannot be used everywhere).
Just now when the wish for a runtime-defined path (for mounting a CD into a chroot) was mentioed, I see that this decision is absolutely correct.
| Quote: | | No config will be comparable to others as Perl code can have very different styles. |
It is not necessary to compare configs. If currently chosen data must be mailed, you can use | Code: | | squashmount -vv list |
| Quote: | | Maybe just leave it in /usr/share/doc/squashmount-*. |
At first, I also thought that this is the better idea, but then you would not see changes (not only etc-update will fail, but the previous config-file is not there at all).
| Quote: | | Is there an ebuild to do the patching? |
I don't know, and I will not spent my time to manage one. However, once that patch is in upstream kernel (or at least gentoo-sources or hardened-souces), probably the defaults in squashmount will change to lz4; this was announced here already several times...
| Quote: | | Performance is more important than size. |
Note that these benchmarks only involve the compression speed. Benchmarking decompression speed is not so easy, and this is what counts during using such a filesystem. I would guess that concerning decompression speed, the difference is not so much. Actually, xz might be superior here, since a shorter size means less disk acces and better caching possibilities.
| Quote: | | BTW, how did you measure the time, like 4 (lz4) vs. 3:50 (xz) for /var/db on Core2? |
Not too much effort was spent to make it very reliable: After verifying that no cron job will happen, essentially just
| Code: | | time mksquash /var/db ... |
on the corresponding system, usually averaged over some calls to eliminate effects of caching/daemons. It was mainly to get an idea which options to use as default and whether perhaps on some systems there is a different order of the algorithms concerning speed (it seems not). However, once the list existed, why not make it available for eveybody.
| Quote: | | Starting from scratch with only one item: |
I did the same for testing, and it worked flawlessly. The only thing which I can imagine that might have happened in your case is that you once mounted (or almost mounted) and then removed the */readonly directory again: In this case, squashmount might have stored the existence of that directory for the mount-point adobe in /run/squashmount and thus did not check its existence again. Perhaps there should be a (usually redundant) check in squashmount for such a case. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Thu Aug 21, 2014 12:27 pm Post subject: |
|
|
| mv wrote: | | Also /etc/conf.d is arbitrary shell code, and I consider this as one of the main advantages of openrc, so that you can have the identical configurations for several machines and different situations - the configuration itself being conditional with user-defined conditions. (Also systemd provides such conditions, but they are much more limited and cannot be used everywhere). |
OT: That I agree, I definitly like having the same configurations over my machines to be able to merge changes. Currently I only have that for my large /etc/vim/vimrc.local with hostname conditions. But how do you do these conditions for other conf.d stuff?
Hm, again I start from scratch. The caching of old config data should be reset by squashmount reset, no? Where are they stored and why? The .new is just my backup:
| Code: | $ ls -ald /opt/Adobe*
drwxr-xr-x 3 root root 4,0K Aug 21 09:37 /opt/Adobe/
drwxr-xr-x 3 root root 4,0K Aug 21 09:37 /opt/Adobe.new/
$ squashmount -vv list ## I like that
* [adobe]: not mounted
DIR: /opt/Adobe
READONLY: /opt/Adobe.mount/readonly
CHANGES: /opt/Adobe.mount/changes
WORKDIR: /opt/Adobe.mount/workdir
FILE: /opt/Adobe.mount/Adobe.sfs
mksquashfs options: -noappend -quiet -comp gzip
CHMOD: 0644
CHOWN: unchanged (0:0)
$ squashmount reset
* [adobe]: resetting configuration
$ squashmount start
* [adobe]: It seems this is mounted for the first time:
The squashed file /opt/Adobe.mount/Adobe.sfs does not exist yet;
it will be initialized now from /opt/Adobe
[=======================================================================================/] 1285/1285 100%
* [adobe]: cleaning original DIR
* [adobe]: mounting...
* [adobe]: error: no directory /opt/Adobe.mount/readonly |
What I'm doing wrong here?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Thu Aug 21, 2014 12:59 pm Post subject: |
|
|
As for the benchmarks, here is a decompression only benchmark. However in times when there is enough CPU but the physical drives are still slow, it could be a completely different result in real-life. I guess I will check with some Portage scenarios and different compressions. Speeding up portage is highest priority. I still remember best results with /var/db on some ancient reiserfs.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Aug 21, 2014 1:37 pm Post subject: |
|
|
| Massimo B. wrote: | | The caching of old config data should be reset by squashmount reset, no? |
No: You are looking for "squashmount forget". "squashmount reset" essentially just "undo's" the effect of some "squsahmount set" - in your current stage, this is irrelevant.
| Quote: | | Where are they stored and why? |
They are stored in /run/squashmount/adobe (in your case), but do not rely on that for future version of squashmount.
"squashmount forget" will remove this file if it is safe to do this.
They must be stored, because you might use a temporary directory (with a random, i.e. non-predictable name): once that directory is initialized, it should be reused, of course. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Aug 21, 2014 3:08 pm Post subject: |
|
|
In squashmount-8.5 a nonexistent stored directory is now considered as non-stored, that is it becomes recreated (possibly under a new temporary name): This is secure and is probably what users expect to happen in such a case.
In particular, this should fix the previous issue where "invalid" directories were stored "by mistake". |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Aug 21, 2014 7:31 pm Post subject: |
|
|
Oh, I missed this question:
| Massimo B. wrote: | | But how do you do these conditions for other conf.d stuff? |
As I said, it is just usual (POSIX) shell code. For instance, I have in my net configuration stuff like | Code: | if ! test -r /etc/firewall.d/dhcp-client
then
config_lan0="...."
fi
...
if test -f /etc/ppp/my-username
then
read username_ppp0 </etc/ppp/my-username
fi
... |
Here, I use special "magic" files to switch between certain configuration settings or to keep my "secrets" off the configuration, respectively.
Of course, you can also make conditions based on hostname or on the content of a single file or whatever is appropriate for you to distinguish your configurations. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Aug 21, 2014 7:37 pm Post subject: |
|
|
| Massimo B. wrote: | | As for the benchmarks, here is a decompression only benchmark |
Such benchmarks have to be read with care: They cannot measure the actual disk access needed for squashfs. In fact, the latter can heavily vary with the type of directory you are using with squashfs and, even more, depends on your RAM and cache size, especially if you use memory (due to cache) hungry tools like portage. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Fri Aug 22, 2014 8:38 am Post subject: |
|
|
| mv wrote: | "squashmount forget" will remove this file if it is safe to do this.
They must be stored, because you might use a temporary directory (with a random, i.e. non-predictable name): once that directory is initialized, it should be reused, of course. |
I don't understand that exactly, what is a temporary directory with a non-predictable name? Target dirs such as /opt/Adobe or /usr/portage won't change. Then I see that squashmount stop also runs a forget, so the cached information is only while squashmount is up, but with all squashfs mounted the directories won't change at all in the background, no?
It is still not working: | Code: | $ squashmount forget
* [adobe]: forgetting settings
$ squashmount -vvv start
* squashmount: reading config file /etc/squashmount.pl
* [adobe]: mounting...
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /opt/Adobe.mount/readonly
/sbin/modprobe overlayfs
modprobe: FATAL: Module overlayfs not found.
/bin/mount -t overlayfs -o noatime -o 'upperdir=/opt/Adobe.mount/changes' -o 'lowerdir=/opt/Adobe.mount/readonly' -o 'workdir=/opt/Adobe.mount/workdir' -- overlayfs /opt/Adobe
mount: unknown filesystem type 'overlayfs'
* [adobe]: warning: overlayfs failed
/sbin/modprobe aufs
/bin/mount -t aufs -o noatime -o 'br:/opt/Adobe.mount/changes=rw:/opt/Adobe.mount/readonly=rr' -- aufs /opt/Adobe
$ squashmount -vv list
* [adobe]: aufs
unmodified
THRESHOLD: 0
DIR: /opt/Adobe
READONLY: /opt/Adobe.mount/readonly
CHANGES: /opt/Adobe.mount/changes
FILE: /opt/Adobe.mount/Adobe.sfs
mksquashfs options: -noappend -quiet -comp gzip
CHMOD: 0644
CHOWN: unchanged (0:0) |
It tries to use overlayfs (how do I get that, can't find in the kernel, some secret patches too?). But I still have aufs3 working. squashmount tries overlayfs anyway and fails.
BTW. sometime in the future we could start a squashmount only thread and link here. This thread is living since the original approach, then came squash_dir, and now squashmount. On a dedicated squashmount thread you could maintain the first #1 post to have a recent overwiew about the project.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770
Last edited by Massimo B. on Fri Aug 22, 2014 8:43 am; edited 1 time in total |
|
| Back to top |
|
|
|
 |
Gentoo Forums Forum Index
 Documentation, Tips & Tricks |
All times are GMT
Goto page Previous 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 Next
|
| Page 3 of 10 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
