TIP: Compressing portage using squashfs: initscript method

[mv]

The new version 9.0 of squash_dir now also supports sys-fs/unionfs-fuse which is new in the portage tree.

One advantage of this is that now you do not get any problems with new kernels if the aufs2 patch is not yet ported: The existing in-kernel fuse support suffices for unionfs-fuse (this was already the case for funionfs, but funionfs is not in the official portage tree).

Since I could not find the download address in this script immediately, here it is once more: initscripts

[NaiL]

It's nice to see that you are still working on it. (almost 4 years)

Thanks for your work.

[js08]

Hi,
I tried your script for my portage tree(squashfs + unionfs-fuse) and I'm a little bit surprised about the fact that rsync tries to delete a lot of files with the "_HIDDEN~"-ending. Each run of emerge --sync / eix-sync produces more of this file deletions....

[mv]

This is a side effect of unionfs-fuse: This tool seems to mirror the directory .unionfs from the writable branch. This is not good: If somebody knows a way to avoid this, please let me know. I checked the unionfs-fuse manpage for a corresponding option but did not find one (I did not yet inspect the source or other documents).

So far, the only workaround which I know is to add --exclude=/.unionfs to the PORTAGE_RSYNC_OPTS.

[mv]

squash_dir now has a new name and has become a package of its own: You can find the current tarball on the same webpage as previously (but under a new name reflecting the current version). Please copy that tarball into your $DISTDIR so that the ebuild will later not download it again. In the tarball you will find in addition a corresponding ebuild and INSTALL instructions (not matter whether you want to use the ebuild or not).

Please also read the ChangeLog:

The most important change is that squash_dir will now create the squashfile and clear the original directory when the squashfile did not exist! This feature required previously in the thread is now implemented, but please be aware that it is a dangerous feature if you start with a wrong configuration (although several sanity checks are made, of course).

[js08]

ebuild works fine.

The only problem I have in my environment (AMD64 with 32bit-chroot using the same /usr/portage) is that unionfs-fuse doesn't work very reliable. in such a case /usr/portage is then no longer accessible and I have to restart
/etc/init.d/squash_portage. This happens one or two times per day without any error messages in syslog. It looks like nfs when the server disappears.
_________________
Train Hard Or Don't Train At All

[mv]

[js08]

Yes, I use mount --bind for /usr/portage. It works fine during boot or shutdown but then after several hours something strange happens and /usr/portage is no longer accessible.

I know when it happens (due to a cronjob checking /usr/portage periodically) but at the moment I have absolutely no idea why it happens.
_________________
Train Hard Or Don't Train At All

[mv]

[john.newman]

hello, I am trying to do this as per the directions and the code on the wiki (http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree).

It probably would work, except I get

[mv]

[john.newman]

thanks mv, i am making progress. I have emerged your squash_dir package and am following the README here. I actually did get portage to squash and mount, but I got an error regarding aufs. So I need to patch the kernel for aufs2 and load the module. Correct?

And the masked packages, like autoconf etc, those are OK right?

I will followup with results, and a tested install script for the impatient. (if results are 100%)

I am rolling this up through a few clean kvms to test and verify. Kernel patching does not comfort me. Ultimately it will live on a certain kvm, that will export the dir over NFS for every machine on the network. as described http://en.gentoo-wiki.com/wiki/Sharing_Portage_over_NFS

(6 copies of portage is killing me.. 1 copy, squashed, and stored in one file for archiving will be a big improvement .)

[mv]

[john.newman]

thanks mv, it is working well. I have one 45 meg file, mounted over loopback and it is fast. I say this should be built in as the default for /usr/portage once aufs etc gets stable. Thanks for all the hours you spent on this, I can tell.
For reference and the impatient, here are the exact commands I used to install mv's script for /usr/portage. I've ran through these 3 times so it should work fine for a 2.6.31 install as of 2010-01-30. I am not saying paste these in and go, read through them and step one line at a time. set -e and set -u are also your friends.

[mv]

The ebuild to squash_dir is now available on the mv overlay which can be installed with layman: You might have to do

[Dont Panic]

Thanks mv.

I've found this to be a helpful tool on my partitions that are slightly low on space.

[js08]

Hi I have sometimes a problem with the squash_dir (portage) runscript. The stopping fails with an error ... rc-status shows that squash_portage is still running.. so it is no longer possible to start or restart the squashfs portage tree without manual intervention. It is also not possible to stop because it fails again and again...

in detail:

1) first try (stop)

[mv]

[js08]

thanx for your fast reply!

I use the "bleeding edge" from your overlay and have currently version 10.5 (2010/05/20) installed together with two slightly different vanilla 2.6.34 64bit kernels on two different hosts. And it's reproducable on both! And It happens nearly everytime when I stop it.

I get always this "umount: /usr/portage.readonly: device is busy." message and when I execute "umount /usr/portage.readonly" immediately after "squash_portage stop" it works and /usr/portage.readonly is no longer mounted. I have absolutely no idea why /usr/portage.readonlly is always busy in my environments....



PS:
And it's not the mount problem ( https://bugs.gentoo.org/show_bug.cgi?id=304443 ) because I no longer do a bind-mount for my 32bit-chroot. My workaround for the 32bit-chroot is to copy the whole 40MB portage.sqfs-file to the chroot directory in the chroot32-startscript. This is at the moment the only way to avoid the mentioned file system corruption...
_________________
Train Hard Or Don't Train At All

[js08]

that the sys-fs/unionfs-fuse-0.25_alpha9999 doesn't run stable in my environment.

It seems that high load (e.g. using eix-sync) kills the unionfs-fuse stuff and then /usr/portage is empty....
As a consequence of this I ended several times with an 4kb portage.sqfs file his evening....
and a busy /usr/portage.readonly.
_________________
Train Hard Or Don't Train At All

[mv]

[aakef]

Upstream is just reading here and would like to see a detailed problem discription. [/quote]

[js08]

[aakef]

Are you sure its 0.24 and not 0.25alpha? 0.24 is mostly a bugfix release to 0.23. But 0.25 simplifies path building (unionfs internal) and handles MAX_PATH_LEN better. But this change might have introduced bugs.
However, I'm a Debian user and only found that thread here, as I had been curious what Gentoo is using unionfs-fuse for. So that means I am not familiar with what you are doing.
If you want to help me to debug it, I'm sure we will have have a solution immediately as soon as I know what is going on.

So could please desribe what you are doing and what is exactly the failure?

From mv's post I can see how you run unionfs, but then it starts to become unclear. There "busy" problem on umount? And/or you cannot access the union anymore?

If possible, please compile unionfs-fuse with "-DDEBUG" and then run it in the forground using the "-d" flag and capure the output.

Presently there are not syslogs, as it would deadlock on writing syslogs, if /var is a union served by unionfs-fuse. So syslogs are disabled until the ring-buffer+syslog threads branch is ready (almost done and will be merged with 0.25 final).

Thanks in advance,
Bernd

[mv]