Squashfs with LZMA compression [ebuild]

cyrillic · Last edited by cyrillic on Wed Nov 14, 2007 3:43 am; edited 4 times in total

Updated ebuild several posts down

I'm not sure how much interest there is in squashfs these days, and there doesn't seem to be too much activity upstream ...

But hey, I use squashfs on a regular basis, so I cooked up some kernel patches (because I prefer not to build stuff outside the kernel tree), and fixed (butchered) the squashfs-tools ebuild to include LZMA support.

These patches are for 2.6.22 and 2.6.23 vanilla kernels respectively.
sqlzma.22.patch 162KB
sqlzma.23.patch 162KB
If your kernel (gentoo-sources, etc.) already includes partial squashfs support, then these may not apply cleanly.

sys-fs/squashfs-tools/squashfs-tools-3.2_p2.ebuild

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

thanks a lot for making it trivial! shall try it sometime

.
_________________
Helium Sources || Gentoo Minimal Livecd

likewhoa · Posted: Mon Oct 15, 2007 8:22 am Post subject:

sweet, time to test on my livecd.

yoshi314 · l33t Joined: 30 Dec 2004 Posts: 850 Location: PL

so there is support for lzma as an unofficial patch for squashfs. it packs pretty good, but *very* slow at packing - i keep my portage and overlays on squashfs loopback, and i saw a benefit of ~5-10% size-wise. and a huge increase in compression time :/

can i somehow use other compressiom methods with squashfs, other than the default one (what is the default method btw?) or lzma?
_________________
~amd64
shrink your /usr/portage with squashfs+aufs

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

I guess the default squashfs is pretty fast at packing too. I think it uses gzip or lzo (whatever is supported by kernel) and compresses well too. (I get livecd images of 1.9 gb from my full blown system of size 6.4 gb with default squashfs, all in under 25 minutes on an athlon64x2 and sata hdd @ 63 MB/s.)

Although I didn't try sqlzma as yet, but from my experiences of lzma, I would expect it to be slow at compressing. (Typically, 'tar -czf ...' would take 10-15 minutes, while '... lzma e -si -so ...' would take 2 hours for compressing my 6 gb system. The output tar.gz used to be 1.8 gb, while tar.lzma used to be 1.3 gb. In contrast, bzip would take 35 minutes to give out tar.bz2 of 1.6 gb. And hence, as it would anyway require a dvd to store, I always stuck to default squashfs / tar.gz for backup.)

But there are two bright points. One is that if you make the image(s) seldom, lzma is pretty good choice - we can compress a further lot more by increasing dictionary sizes (I haven't tried this for my impatience). Second is that decompression is lightning fast - as fast or better than gzip. This is where bzip2 is bad imo - it takes larger time to even decompress. Especially the small sizes in conjunction with slow i/o speed of dvd-drives, high cpu power (and fast hard-drive writes) makes lzma look even better than what it is, compared to others.

Aside, I would go on to add that as of now, I am extremely happy with regular squashfs combined with my usb flash drive. I use a usb flash drive of > 2gb (its a sandisk cruzer 4gb which gives a read speed of about 17 MB/s) to store my 1.9 gb image and its screaming faster than booting from my hard drive! (Must also add that the true random access nature of flash drives makes it more lethal than what its already impressive 17 MB/s seems!) Not to mention the no rotating parts noise. Wonder what would sqlzma make it look like

.
_________________
Helium Sources || Gentoo Minimal Livecd

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

Finally tried and tested it. Many thanks for making it work easily! (Used the patch for 2.6.22.10 .)

Although the overall result has been a bit disappointing for me compared to the much hype I had come across. My 6.4 gb of system was compressed to 1.7 gb by lzma instead of 1.95 gb with default zlib. With lzma, I was expecting something less than 1.5 gb at least

. And it took 2 full hrs to do that instead of 25 minutes with zlib.

Of course, the brighter side is that its completely backward compatible. (Not only did the old squashfs images work with the new kernel, even the new ones created with -nolzma option worked fine with old kernel. So just happy with -nolzma at the moment

.)
_________________
Helium Sources || Gentoo Minimal Livecd

baeksu · l33t Joined: 26 Sep 2004 Posts: 609 Location: Seoul, Korea

I can't access the kernel patches (sharedzilla is down?). Could someone give another source for them, or at least for .23 kernels?
_________________
Gnome:
1. A legendary being.
2. A never ending quest to make unix friendly to people who don't want unix and excruciating for those that do.

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

http://hirakendu.mooo.com/sqlzma-temp/
_________________
Helium Sources || Gentoo Minimal Livecd

baeksu · l33t Joined: 26 Sep 2004 Posts: 609 Location: Seoul, Korea

yoshi314 · l33t Joined: 30 Dec 2004 Posts: 850 Location: PL

let's hope squashfs3.3 ebuilds get posted quickly, as it's already out.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

Thats nice. Especially I wanted to try out blocksizes of 128K and more. But I don't think it will lead to any drastic improvements. And as mentioned earlier, lzma didn't turn out to be dramatic (but the time taken to compress will definitely surprise 1st timers

).
_________________
Helium Sources || Gentoo Minimal Livecd

yoshi314 · l33t Joined: 30 Dec 2004 Posts: 850 Location: PL

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

Yep, I had made that comment after reading that blog only

. That 167 mb is using 1mb + lzma. The result with 1mb + gzip is hardly impressive, compared to 64kb. Besides I don't know the increase (hopefully decrease) in time that would incur.
_________________
Helium Sources || Gentoo Minimal Livecd

cyrillic · Posted: Wed Nov 14, 2007 3:40 am Post subject:

I normally compress the portage tree, and it looks like 1MB blocks really help LZMA to compress better :
44MB with zlib + default 128K blocks
32MB with lzma + 1M blocks

This is the next generation of sqlzma, based on squashfs-3.3 and the latest LZMA patches that just came out a couple of days ago.

You can find the ebuild and kernel patches here : ftp://71.184.254.182
shh ... don't tell my ISP that I am hosting files at home. :wink:

sys-fs/squashfs-tools/squashfs-tools-3.3.ebuild

uli · n00b Joined: 08 Feb 2007 Posts: 58

Hi cyrillic,

I'm not an advanced Gentoo user. So up to now I have simply emerged packages but also managed to make an overlay.

But I have not tried to use just an ebuild. So I'm interested in squashfs/lzma together with 2.6.24-rc2-rt1 (patched with rt1). You ftp-link now shows the ebuild 3.3 and then some kernel patches and also some squashfs/lzma files.

Can you please describe the steps to patch the kernel and to run the ebuild in the proper way?

Thx

Uli

cyrillic · Posted: Mon Nov 26, 2007 2:25 pm Post subject:

This is how you patch your 2.6.24-* kernel.

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

Seems that sqlzma-3.3 is borked for me. I am guessing only the kernel part. The mksquashfs part works fine (even with lzma I believe). Even the mounting part works fine. The trouble is when I chroot into the mounted system, the filesystem is all messed up. Namely, files are spuriously linked to each other. Example, things are as crazy as /etc/mtab linked to /usr/bin/make. And I believe it happens only from within chroot. So those using it for just squashing their portage-tree etc may not notice it. However, its required for livedisks. Has anyone else noticed this. Btw, this happens even with the old squashfs images created with squashfs-tools-3.2 and using kernel with new sqlzma patches. And using vanilla squashfs-3.3 with the kernel, things are back to normal. So I guess its very much something with sqlzma 3.3 patches for kernel. I have noticed this 'bug' with both sqlzma patch in this thread for 2.6.24 and zen-sources (2.6.24-zen1). Is there possibly any workaround or mount options to get around this?

As of now, using plain vanilla squashfs-3.3 for the kernel part and squashfs-tools with lzma (using ebuild in this thread). Using '-nolzma' and '-b 1024KB' and things look pretty good.
_________________
Helium Sources || Gentoo Minimal Livecd

prymitive · Apprentice Joined: 13 Jun 2004 Posts: 260

I'm using squashfs with lzma to make initrd images for booting servers from network using PXE, to allow kernel to load lzma squashfs at boot You need this simple patch:
http://prymitive.com/files/lzma_ramdisk_boot.diff

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

I am guessing you didn't get my problem. I don't have issues with booting and/or mounting. Just that the filesystem is borked - many files are wrongly linked to each other.

EDIT : Found the bug filed with debian : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=456489.
_________________
Helium Sources || Gentoo Minimal Livecd

prymitive · Apprentice Joined: 13 Jun 2004 Posts: 260

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

Apologies

. I thought 'You' = 'me'

.
_________________
Helium Sources || Gentoo Minimal Livecd

cyrillic · Posted: Tue Feb 12, 2008 8:49 pm Post subject:

hirakendu · Guru Joined: 24 Jan 2007 Posts: 386 Location: san diego

hmm, i thought corruption was only when i chroot. externally, all seemed to look ok. yeah, i too thought of rev-engg the slax cd, but short on time. i heard a big one is coming on wednesday

.
_________________
Helium Sources || Gentoo Minimal Livecd

avx · Advocate Joined: 21 Jun 2004 Posts: 2152

Thanks very much, it's working fine for me :)

Compressed portage is now down to 4.7MB, has been 5.5MB before. If only it weren't that slow, but that's the price.

mikegpitt · Advocate Joined: 22 May 2004 Posts: 3224

Do you still have the patch for the 2.6.23 sources? It doesn't appear to be available on the sqashlzma site anymore, and the d/l links you posted seem to be dead.