Compressed filesystem : need some testers.

neuron · Advocate Joined: 28 May 2002 Posts: 2371

Deprecated, see https://forums.gentoo.org/viewtopic-t-405996.html

Hi, I've been making a compressed filesystem overlay since.. monday, and I'm on a level where I need some testers. So I figured which group of people have a huge amount of io reads which they can afford loosing... hmm... /usr/portage

It's a fuse module called lzofs, and works as an overlay (ie, it's not a filesystem, just a compressed directory on your filesystem).

To get it working:

i92guboj

neuron · Advocate Joined: 28 May 2002 Posts: 2371

, updated original post with that info, completely forgot about it

i92guboj · Posted: Fri Oct 07, 2005 5:43 pm Post subject:

My first experience:

First, I launched the sample file, and left it running in one xterm, so I can post the output later, in the bottom.

Second, in another xterm:

i92guboj · Posted: Fri Oct 07, 2005 6:52 pm Post subject:

Ok, some things, I moved /usr/portage into /home/portage. I created the raw dir into /usr/portage_raw and created /usr/portage again, to hold the mount point for the portage_raw lzo filesystem. Then I "lzofs /usr/portage_raw /usr/portage".

neuron · Advocate Joined: 28 May 2002 Posts: 2371

first of all, ls -l/du on a compressed directory will show it's uncompressed size. And ls'ing raw isn't accurate because of the cache, stuff might not be flushed to disk.

To get accurate info unpack something to x and fusermount -u x, that'll flush it to disk and you can ls -l on raw.

And I'm using LZO compression, level 1 for now (hardcoded, sorry), so I'd be surprised if it compressed a .gz file at all.

i92guboj · Posted: Fri Oct 07, 2005 7:49 pm Post subject:

neuron · Advocate Joined: 28 May 2002 Posts: 2371

trim_database was "funky" and hardlocked because of some thread magic, should work now.

Oh, and if your a bit too spammed by THREAD messages you can open lzofs.h and comment out DEFINE THREADDEBUG

//edit, oh, and keep in mind rsync is hitting a huge performance bug, as every time it does rename the cache isn't renamed, it's flushed (which means it has to read the entire file again to get the new filesize)., that will be fixed

//edit 2, debug code should be cleaned up quite a bit now, there are defines for more/less debug output in lzofs.h

neuron · Advocate Joined: 28 May 2002 Posts: 2371

heh, those original benchmarks posted by me are way off, it was the debug output + gnome terminal slowing it down.

doing a rsync checksum on files now I get:

i92guboj · Posted: Sun Oct 09, 2005 5:45 pm Post subject:

I'm giving it another shot today (sunday evening and no will to do any other thing but stay at home

). I just downloaded and compiled rev91 from svn repository and im copying the portage tree right now.

The first attemp was unsuccessfull. Right now (second attempt) it seems to work right (still copying, ill tell you later). In the first attempt I got some bad file descriptor errors after 62 megabytes copied, I accessed the files from another terminal with du, maybe that was the reason why it gave me that errors... Not sure.

Because of that, I will stay away from keyboard untill it ends the copy process.

neuron · Advocate Joined: 28 May 2002 Posts: 2371

I found some more issues that I haven't quite figured out how to solve yet :/, still working on rsync'ing the whole portage tree over without it dying on me.

neuron · Advocate Joined: 28 May 2002 Posts: 2371

I think/hope I've solved the problem for now. Rsync opened files in RDWR mode and I didn't have that implemented, so I had to restructure the code a bit, on a bright side that restructure was on my todo list and has improved performance a lot.

Note that if performance testing undefine DEBUG or you'll be slowed down a LOT by the debug output, also if you like debug output use xterm (or another fast term) and minimize it on both rsync and lzofs, but atleast lzofs

.

// edit, oh, and I've looked into how much work would be needed to support other ciphers... and.. not much at all

.

neuron · Advocate Joined: 28 May 2002 Posts: 2371

whoho, just did a full rsync of my old portage dir + a huge amount of distfiles with no errors whatsoever.

and then did an rsync -c (checksum) of it again, and it transfered 810 files, which happened to all be 0byte, no idea why it chose to send those again, but fixing it should be trivial, and it's not exactly a critical problem

The overhead is actually far smaller than I thought it'd be, I'm thinking this might be worth putting on a livecd/usb root atleast for smaller amount of reads from the disk

, that's in the future though

lazx888 · Tux's lil' helper Joined: 13 Sep 2005 Posts: 118

found you neuron

neuron · Advocate Joined: 28 May 2002 Posts: 2371

iphitus · Apprentice Joined: 03 Aug 2005 Posts: 226

neuron · Advocate Joined: 28 May 2002 Posts: 2371

neuron · Advocate Joined: 28 May 2002 Posts: 2371

small update, checkout revision 99, on rev 100 I've started work to make the ciphers into something dynamic, and it's in very active development.

i92guboj · Posted: Mon Oct 10, 2005 5:57 pm Post subject:

Hi again, I'm using rev99 and at last managed to copy back a functional portage tree without errors. All seems to operate well so far within this revision for now. I also sync'ed and im playing a bit with portage.

I haven't tracked any noticeable performance issue, but I would do some tests in the next few days (I still have the old portage copy). I have to test some things, for example, I dont care if there is any speed issue as long as it is not a very noticeable thing, but I do care about the cpu activity. Although lz compression should not be a big issue for a modern machine I will do some tests on it, to see if the compressed filesystem influences too much the system interactivity while retrieving the portage metadata (which is the hardest activity that I can think of on this partition that I use exclusively for portage).

The copy of distfiles suceeded flawlesly, and I did not notice a very big impact on my system responsivity, that is a good sign. Another parameter that worries me is the impact in ram memory. I will report my results in some days.

neuron · Advocate Joined: 28 May 2002 Posts: 2371

memory usage should be tiny, somethingl like 4kb per file until it's closed, which is almost instant anyway.

i92guboj · Posted: Mon Oct 10, 2005 6:49 pm Post subject:

Hi again

I have news. I was checking another thing (totally unrelated to lzofs), and I ran into this:

neuron · Advocate Joined: 28 May 2002 Posts: 2371

.... that is extreamly weird.

could you try to reproduce this with another fuse filesystem?

also, this could be because I'm keeping a lot of file handles open I suppose, looking into how many I should have open is on my todo list.

Does this happen when mounting only, or when mounting + using it?

(using it will make it fill up the cache with open filehandles)

i92guboj · Posted: Mon Oct 10, 2005 7:53 pm Post subject:

Rainmaker · Posted: Mon Oct 10, 2005 9:26 pm Post subject:

looks very interesting. Would love to try this, but I can't get this to compile. I get the same error message as in the 2nd message of this thread.

I'm building against 2.6.14-rc3-nitro1, which is patched with fuse by default

I tried to emerge sys-fs/fuse and compile it then (with the built module from portage), but still getting the same erro.
_________________
If you can't dazzle them with brilliance, baffle them with bullshit.

i92guboj · Posted: Mon Oct 10, 2005 9:53 pm Post subject:

That error was due to gcc being unable to find fuse.h, that, if you installed the package fuse, should reside under /usr/include/fuse.h or /usr/include/fuse/fuse.h (not sure). If that is there all should work. Are you sure that you emerged sys-fs/fuse? There is another package called app-emulation/fuse, maybe you emerge that instead...