TIP: Compressing portage using squashfs: initscript method

[synss]

Admin Edit:
squashmount related information has been split from this topic. It is now here:
Compressing filesystems with squashfs and squashmount
--pjp

Tue Nov 28 12:10:10 JST 2006, v. 0.1.5(4)

I have left gentoo and have not updated this script for a while, though it probably still works, mv has taken over and proposed new features and improvements, aufs is one good thing among others. And his script can be used on any directory (I would recommend /usr/src/linux and the docs) You should probably have a look at the thread and check mv's version, now. Last version is available here --- I was sure I had left a notice like this one here but only realize now I must have forgotten...

Abstract The portage tree takes typically 600MB on /usr/portage, some people have chosen to have it on a separate partition with a small block size to

decrease its global size
decrease fragmentation
increase the speed during searches and update

However that also means that the free space on that partition is lost. Another, better, solution is to use a stack file containing the filesystem with small block size to avoid the shortcomings of the separate partition. Following the discussion in this post, the best solution found is to compress the portage tree using the compressed, read-only squashfs filesystem and mount a read-write partition on top of it using unionfs. Second, for convenience, the portage tree should be mountable on boot-up and updated on shutdown, hence I have written an initscript to automate this process. I will publish and update this program here.

Assumptions I will assume that you have

read the original idea as it details what you need to get started and that you are familiar with the bootup/shutdown sequence of your computer,
read the Gentoo doc on initscripting.
DISTDIR should be out of the tree, etc.

Please just read the original post as everything is well documented there and come back once you have done that, unless you do understand what you are doing!

Introduction In brief, you will need to load the relevant kernel modules and emerge the userland utilities

[yoshi314]

th...this is great! i would never think of implementing it via initscript

nice job.

although there is one drawback with this overlay - when you upgrade your kernel you might get stuck with no portage tree at all, since not every kernel might have squashfs and unionfs modules integrated. there could be a solution in form of an mini-overlay that would contain only ebuilds for squashfs and unionfs (and required eclasses) for forgetful folks. (would it be difficult to make such a backup overlay from an existing portage tree via an bash/sh script?)

sorry for writing that here, but i thought is fits this topic more - it's a more elegant solution.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs

[synss]

Thank you!!!

No problem for writing here either...

I do not think there should be any problem:

1. squashfs is a standard kernel module, which does not require the userland utilities, if you have a recent kernel, you just enable squashfs and loop and that is enough for mounting the squashed image. Even "forgetful folks" can compile the necessary modules and modprobe them (no need to reboot into the new kernel).
2. unionfs and sys-fs/squashfs-tools are only needed for the updates because the squashed image is read only. Hence, it is not required at all as long as you do not try to sync your tree.

I have had a problem with union that was acting weird during unmount, but that was a mistake from my side (a non-existing locale when compiling glibc, I think) So you might be unable to sync, but you can then compile and load the modules and go on. In any case, it is always possible to download an image of the tree from a nearby ftp server yes, that is exactly for avoiding that that I implemented the incremental timestamped backups while debugging. (Then I leave it cause it is a good idea and you can easily clean the backups via cron if you want to.) So the only risk you are taking, AFAIK, is that you might have to mount the squashed image by hand after recompiling the modules.

Thank you again!
_________________
Compress portage tree
Elog viewer
Autodetect swap

[lazy_bum]

Thanks man, this is great! Fast, small and working! (-:

I'm using sys-fs/squashfs-tools-2.2_p2, sys-fs/unionfs-1.1.4-r2 on a sys-kernel/gentoo-sources-2.6.14-r5
_________________
roslin uberlay | grubelek

[MxxCon]

have you guys considered using http://www.miio.net/fusecompress/ or http://parallel.vub.ac.be/~johan/compFUSEd/ instead of squashfs?
from what i understand for squashfs you still need to have enough free space to unpack the whole thing, while with these 2 it's like working with compressed .tar file.

[Gergan Penkov]

[MxxCon]

[Gergan Penkov]

Well I though the same in the beginning, even with such implementation the things would have been just ok if there was some sort of file-container ala tar around the files, of course the compression ratio would not have been very good but there would have been virtually not that much fragmentation and loss of free space
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack

[lazy_bum]

Something is wrong with the new version:

[synss]

"date" creates incremental backups

What about moving the squashed tree to, ie /usr/portage_tree.sqfs and creating a symlink to it, ln -s /usr/portage_tree.sqfs /usr/portage.sqfs the second one should only be a symlink as it is overwritten on updates to point to the actual tree.

I do not know whether that helps... Could you make sure it is the first mount that is broken? and check that something is mounted by echoing the variables for example. I am sorry for the inconvenience, it works at home...

I am in a cybercafe on XP with a French kbd so debugging is not easy... But I do not see what else but the first mount can be a problem.

you can also try to mount the tree by hand, sync and copy-paste

[lazy_bum]

[gAzo0o]

I had the same problem. Fixed it by setting the PORTDIR (/usr/portage by default) varible in /etc/make.conf

[lazy_bum]

Looks like it was my "kernel panic" problem.
The only problem left is 5-6 portage-$(date +%F).sqfs after 5-6 days. (-;

::edit::
And i will try again version 0.14 of the script.
_________________
roslin uberlay | grubelek

[Mousee]

Not working at all here...

[synss]

[adsmith]

I'm glad to see you've found a way to make unionfs work with it reliably. I'll give it a try shortly.
(My current solution is still to have my main server make the squahsed image for me on my weekly emerge --sync)

By the way, it is good to combine the new metadata system with squahsfs, so you don't just duplicate the data yet again.

To do this, add "-metadata-transfer" to FEATURES and add the line "portdbapi.auxdbmodule = cache.metadata_overlay.database" to /etc/portage/modules.

This avoids the /var/cache/edb metadata cache from being created, as that data already lives in /usr/portage/metadata in the squashed image.

[dfy]

I am considering this at the moment, but even creating the initial portage.sqfs images takes more than 10 minutes. Am I right to think that every time I shutdown my system and the initscript resyncs the squashfs, it will take just as long?

[gerardo]

I've created a BASH script to cleanup the portage squash files older than one month:

[jpmayer87]

This modification to the init script removes the squashfs file that was in use before the unionfs was modified. It also handles the case where the new file is the same name as the old one.

[B.marc]

Just wanted to say, that this works for me very good. Now I'm wondering why Gentoo does not provide portage also as squashfs-files. Downloading 40MB to 50 MB and mounting it is much faster than a standard sync!

[synss]

[mv]

I like the idea to realize the task as an init-script very much. However, the current script has an security issue, and several enhancements are possible:

• synss wrote:

  Code: PORTAGE_RW="/dev/shm/portage"

  
  Using a fixed name in a world-writable directory is a serious security issue. In this case, it might be an acceptable risk, because the initscript is hopefully started before any local attack might have a chance to get started, but even then: Don't better even think about calling the script later on with a restart option.
  
  
• There are two other disadvantages of keeping PORTAGE_RW in a RAM disk: First, your previous changes to the portage tree are lost when your system crashes and the script cannot shutdown properly. The second disadvantage is that - surprise - it costs valuable RAM; the point is that you get practically no speed improvement from this ramdisk here (at least in normal cases), because the newly read data from a portage tree will remain in your disk ram anyway. So, it seems better not to use a ramdisk at all for portage_rw. In contrast, during shutdown, if you do not want to keep a backup of your .sqfs file or are short of disk space, it might be a good idea to build the new .sqfs file in a ramdisk.
  
  
• It was already mentioned in this thread that keeping backups of older .sqfs files without limiting their number automatically is not necessarily the best idea. Morever, it seems better to use .tar.bz2 as a backup anyway for two reasons: First, these files need less space than .sqfs and, more importantly, if you are forced to boot from a kernel without squashfs support (e.g. from some rescue disk), you can still unpack the .tar.bz2 file if necessary. On the other hand, the disadvantage of using .tar.bz2 files as backup is of course the double packing time.
  
  
• If you build a new kernel, you can simply compile in squashfs and loop support (or build the modules). However, to build in the unionfs support, you need access to the portage tree (unless you made a copy of the ebuild in advance). It was already mentioned that in this case you still can mount the portage tree at least readonly. However, the current script does not mount the tree in this case to the "usual" place readonly.

In the initscript of http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html there is a different attempt of a corresponding squashfs_portage with the above fixes/enhancements. Concerning the backups, you may choose in the configuration between keeping .sqfs or .tar.bz2, and you may pass options to mv to define the method and number of backups.

Edit: Fix some typos.

[synss]

[synss]

Interesting, yes, but you could mention me in the header for example...

if you want to make the script more complex, you definitely should have something like

[mv]