| View previous topic :: View next topic |
| Author |
Message |
nato.Otan
n00b
Joined: 23 Oct 2006
Posts: 12
|
 Posted: Tue Oct 24, 2006 12:34 am Post subject: great idea!! |
 |
|
This sound an excelent alternative to run portage lean and swift. The method used here reminds me about puppyos. PuppyOS is extremely small distro that runs off RAM, and when you click on the programs they start instantly in the range of nano-seconds. Pretty cool.
Here I give the link to the descriptions of the bowels of puppyOS, and perhaps it could be implemented on this script to make it even Faster Than Light!!! And rename portage to Tachyon!!!
How Puppy Works |
|
| Back to top |
|
 |
synss
Apprentice
Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
| Posted: Wed Nov 08, 2006 11:29 am Post subject: |
|
|
Well, now things like compFUSEd claim they do support mounting + read-write of archives in .tar.bz2 if that would work that would obsolete these scripts (for the best I must say) however, I cannot compile compFUSEd. And I do not know how to deal with that kind of problems. If anybody is successful with that, I would be very interested.
Being able to mount a freshly downloaded tarball and update it would be great IMHO. The author uses it with his kernel sources and says he can compile into the archive and install his kernel. There are many things I can think of with these FUSE-related fs.
_________________
Compress portage tree
Elog viewer
Autodetect swap |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Nov 08, 2006 7:50 pm Post subject: |
|
|
| synss wrote: | | Well, now things like compFUSEd claim they do support mounting + read-write of archives in .tar.bz2 |
As I understood, they write in their own archive format (based on lzo). I have doubts that any writable filesystem can be fast and simultaneously compress many small files effectively. However, I would like that somebody proves that I am wrong with this conjecture. 
| Quote: | | I cannot compile compFUSEd. [...]There are many things I can think of with these FUSE-related fs |
Yes, it is a pity that for most FUSE filesystems there are no .ebuilds in the portage tree.
Actually, I would also prefer to use an FUSE-based unionfs http://freshmeat.net/projects/unionfs-fuse/, because it needs no kernel module. However, I am too lazy to write an .ebuild. |
|
| Back to top |
|
|
synss
Apprentice
Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
| Posted: Thu Nov 09, 2006 4:10 am Post subject: |
|
|
Agreed, however, there is an ebuild for funionfs in the overlay sunrise, I will see whether it works here and try to use it.
As of compFUSEd, I believe the author says he does support tar.bz2 without restrictions. There are other compressed fs who only work on lzo/gz but this one claims it can deal with .tar.bz2, so it would be worth trying. I contacted the author yesterday asking for support, I also suggested him to commit an ebuild (he is a gentoo user) to bugzy, that would earn him feedback and support, and if it turns out to work well enough, well, that would be really nice.
As of performance, I do not think it matters so much here: you have eix for fast searches and the tree is actually small and cached pretty fast if you use emerge, I think. Only size on the HD matters. And for compressing other things, well then you can chose speed or compression.
BTW, I do not really understand why you have implemented tarred backups in the script: you can just download a new image of the tree from the ftp servers around here if ever something breaks badly, you do not need to do these time-consuming backups, do you? That's why experimenting new fs is fun with the tree, if you break it, just download a new one! Risk-free.
My point above you did not understand (probably was not clear) was that, in your check_config() function, if [[ ! -e "{PORTAGE_SQFS} ]]; blah, instead of echoing a message, you should actually squash the tree to a new image and then mount it. That would allow complete newbies to just download the script and run it and compression would be done automatically upon the first use.
Thank you for mentioning me and I think you've changed the identation and now it is easier to read 
[EDIT] well, funionfs compiled flawlessly (x86) and did not complain when I asked it to mount my tree, thus | Code: | emerge layman
layman -a sunrise
/etc/init.d/squash_portage stop
modprobe -r unionfs
echo "~sys-fs/funionfs-0.4.2" >> /etc/portage/package.keywords
emerge funionfs |
| /etc/init.d/squash_portage: | #mount -t funionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR
funionfs -o dirs=$PORTAGE_RW: $PORTAGE_RO=RO -o allow_other NONE $PORTDIR |
that is in the start() thing.
| Code: | | /etc/init.d/squash_portage start |
Does not complain, I will try syncing later though, for I just did and syncing several times a day is bad practice. But it does look good!  If syncing works I will update my HOWTO, chmod'ing as you do is also a good idea, obviously, I will probably add a few of your things in mine, but I'll leave the tarred backups out...
[SECOND EDIT]sys-fs/fuse-2.6.0_rc1
stable, from emerge (not the one in the kernel) although form their website, they say funionfs should work with older versions of fuse, + my kernel is stable gentoo-source-2.6.17-r8
[THIRD EDIT]https://bugs.gentoo.org/show_bug.cgi?id=143026
an ebuild for UnionFsFuse don't ask me what the difference is or which is best between funionfs and UnionFsFuse (but give me the answer)
_________________
Compress portage tree
Elog viewer
Autodetect swap |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Nov 09, 2006 5:26 pm Post subject: |
|
|
| synss wrote: | | As of performance, I do not think it matters so much here: you have eix for fast searches and the tree is actually small and cached pretty fast if you use emerge, I think. |
The point where I expect an enormous speed loss for .tar.bz2 files (if this would really be supported) is when the data is written to the tree, i.e. for emerge --sync. Just to have some numbers (for emerge --sync with a fast server and a fast connection):Standard (without squash_portage): 3-6 minutes, depending on the filesystem fragmentation.
With squash_portage: <1 minute (that's why I like it so much: It saves not only space but also time, even if one includes the compression time at the shutdown).
With pure .tar.bz2: ??? (I expect 30 minutes or more unless you have very much ram/swap, because every time the (uncompressed!) data cannot be held in ram anymore, practically the whole tree has to be compressed, since any change in the data in the middle of a .tar.bz2 file implies changes until the end of the file by the structure of .tar.bz2 files).
| Quote: | | you can just download a new image of the tree from the ftp servers around here |
Not every computer has easy access to the internet. Moreover, the portage tree is rather large (compared to the few data for a typical emerge --sync): It can really cost money if you have to pay for the bandwith.
| Quote: | | you do not need to do these time-consuming backups, do you? |
A backup can be also handy if the previous emerge --sync dropped some ebuild/patch which you wanted. Yes, there is a cvs of the tree, but a backup is more convenient in such cases.
| Quote: | | instead of echoing a message, you should actually squash the tree to a new image and then mount it. |
An automatic action for such a fundamental change is very dangerous: Once the new tree is mounted, a complete newbie will not be able to remove the old tree anymore. The script could of course also remove the old tree, but then if something goes wrong with the mounting (wrong kernel, insufficient disk space, ...) the user is left without a portage tree. I think the printed instruction is clear enough, and in this way the user can understand what is going on and why, and he has no bad surprise just by trying the script. |
|
| Back to top |
|
|
synss
Apprentice
Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
| Posted: Fri Nov 24, 2006 4:56 am Post subject: |
|
|
Update!
I tried to include the feedback posted in this thread (including some that were posted a long time ago...) plus a few cosmetic changes and a bit more informative messages.
Changelog:
2006-11-19
Security enhancements thanks to mv
Only two sqfs images will now reside on the system: portage-old.sqfs and portage-current.sqfs
changed name of sqfs-related variables
more informational messages (it is made clear that the read-only image is mounted first and that it exists even if unionfs fails, that has always been the way it works but it seems not everybody got it)
correct usage of eend (i.e. !! when mount fails instead of OK)
tried to keep it short and simple without too many bashisms
The updated scripts are in the first post. Comments and bugs welcome.
_________________
Compress portage tree
Elog viewer
Autodetect swap |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Dec 17, 2006 4:51 pm Post subject: |
|
|
Update also for the other script.
It allows now to choose alternatively funionfs instead of unionfs, see funionfs.ebuild.
funionfs is based on fuse and thus works with all new kernels.
Since, in contrast, unionfs has not yet been ported for the stable kernel 2.8.19, funionfs is currently the only way to use this approach with the newest kernels (and it has the advantage that probably newer kernel releases won't break it again). The disadvantage is that funionfs is way slower than unionfs. |
|
| Back to top |
|
|
NaiL
Apprentice
Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN
|
| Posted: Thu Dec 21, 2006 10:34 pm Post subject: |
|
|
| Maybe the next thing is to build the squash_portage.ebuild that install all the config files and check for modules and with use flag to select from unionfs or funionfs |
|
| Back to top |
|
|
NaiL
Apprentice
Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN
|
| Posted: Sat Dec 23, 2006 12:51 am Post subject: |
|
|
Why don't do the same with:
/var/cache/edb
/var/db/pkg
Maybe it's a noob question  |
|
| Back to top |
|
|
synss
Apprentice
Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
| Posted: Sat Dec 23, 2006 8:42 am Post subject: |
|
|
| NaiL wrote: | Why don't do the same with:
/var/cache/edb
/var/db/pkg
Maybe it's a noob question |
No, it should work and that is one idea I had (also add /usr/src/linux) but no time for it. It would not be very diffictult to implement either, like only another for loop. Later, maybe...
And mery christmas!
_________________
Compress portage tree
Elog viewer
Autodetect swap |
|
| Back to top |
|
|
NaiL
Apprentice
Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN
|
| Posted: Sat Dec 23, 2006 2:47 pm Post subject: |
|
|
| synss wrote: |
No, it should work and that is one idea I had (also add /usr/src/linux) but no time for it. It would not be very diffictult to implement either, like only another for loop. Later, maybe...
And mery christmas! |
Why with /usr/src/linux? because there are lots of files?
mery christmas to you! too :p |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Dec 24, 2006 2:01 am Post subject: |
|
|
| NaiL wrote: | Why don't do the same with:
/var/cache/edb
/var/db/pkg |
These two are not worth the trouble, because they do not take too much space. It would probably be more interesting to use it for /usr/share/doc or /usr/src/linux, because these two should compress very well. In principle, you can simply use the same script repeatedly only with different directory/filename settings. Only the previous variable names are then misleading...
Christmas Present: The new version of the init-script on http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html is now called squash_dir and has no misleadings variable names.
Moreover, the new script has some new configuration options which are handy for some applications. The most important ones are: 1. You can explicitly mount a directory read-only (which needs no unionfs/funionfs).
2. You can explicitly avoid recompression during shutdown (by creating a certain magic file). In particular, if you use this script to compress /usr/src/linux, it is usually not worth to recompress /usr/src/linux after a kernel recompilation: Instead it is more reasonable to leave the modified files in a separate directory and simply use this directory on the next start (thus never do any recompression during shutdown). However, despite these changes, I am not sure whether it is a good idea to compress /usr/src/linux: If you compiled a new kernel but e.g. unionfs does not run on it yet, you might get troubles to access your original kernel read-writable so that you are able to compile unionfs...
Anyway, /usr/share/doc should cause less troubles (but it may also cause some, because e.g. the ebuild for unionfs might want to install something there). |
|
| Back to top |
|
|
NaiL
Apprentice
Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN
|
| Posted: Sun Dec 24, 2006 3:41 pm Post subject: |
|
|
For me the first important thing is to avoid disk fragmentation in the measure of possible, secondly speed up the portage metadata updates.
Thats why i think to do that with all portage related directories.
The fact of gain disk space, for me is a welcomed secondary effect. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Dec 24, 2006 4:51 pm Post subject: |
|
|
| NaiL wrote: | | secondly speed up the portage metadata updates |
Do you have a particular reason to do metadata updates? For most users, FEATURES='-metadata-transfer' and choosing the metadata_overlay cache is sufficient.
Anyway, if you really need metadata updates, you should use the script with PREFER=unionfs (in contrast to funionfs) - funionfs will probably slow down things instead of speeding them up. |
|
| Back to top |
|
|
NaiL
Apprentice
Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN
|
| Posted: Mon Dec 25, 2006 1:33 am Post subject: |
|
|
| mv wrote: | | NaiL wrote: | | secondly speed up the portage metadata updates |
Do you have a particular reason to do metadata updates? For most users, FEATURES='-metadata-transfer' and choosing the metadata_overlay cache is sufficient. |
I didn't know this feature, I'm using it now. thanks. but it works with portage overlays as well?
| mv wrote: | | Anyway, if you really need metadata updates, you should use the script with PREFER=unionfs (in contrast to funionfs) - funionfs will probably slow down things instead of speeding them up. |
I'm currently only using unionfs with the synss script. I'll try your script when i have some time!  (It looks very well. "all about choice") |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Dec 25, 2006 11:48 am Post subject: |
|
|
| NaiL wrote: | | I didn't know this feature, I'm using it now. thanks. but it works with portage overlays as well? |
For most users it works. Some things like eclasses in overlays can cause troubles. If you need the latter, an alternative is to use the sqlite cache method.
| Quote: | | "all about choice" |
You will finally probably end up writing your own script - after all, these are all just very simple scripts with a very limited purpose. It's only the idea which is important... |
|
| Back to top |
|
|
kuku
Tux's lil' helper
Joined: 23 Dec 2004
Posts: 142
|
| Posted: Tue Jan 02, 2007 7:30 am Post subject: |
|
|
| mv wrote: |
Christmas Present: The new version of the init-script on http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html is now called squash_dir and has no misleadings variable names.
Moreover, the new script has some new configuration options which are handy for some applications. The most important ones are: [list]1. You can explicitly mount a directory read-only (which needs no unionfs/funionfs). |
Hi - I've tried your script and i got some errors: | Code: | ./squash_portage restart
* Caching service dependencies ... [ ok ]
* Mounting /var/portage/portage.sqfs as /usr/portage ...
fusermount: mountpoint is not empty
fusermount: if you are sure this is safe, use the 'nonempty' mount option
* Failed mounting /var/portage/change with funionfs [exit with 1]
mount: unknown filesystem type 'unionfs'
* Failed mounting /var/portage/change with unionfs [exit with 32]
mount: wrong fs type, bad option, bad superblock on ,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
* Failed even rbinding /usr/portage [exit with 32] |
this is my /etc/conf.d/squash_portage | Code: | DIRECTORY="$(. /etc/make.conf ; echo "${PORTDIR}")"
FILE_SQFS="/var/portage/portage.sqfs"
FILE_SQFS_OLD="/var/portage/portage.old.sqfs"
PREFER="funionfs"
DIR_CHANGE="/var/portage/change"
|
any hint ? |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Jan 02, 2007 10:56 am Post subject: |
|
|
Thanks for the feedback. It seems that I introduced an error briefly before release but after testing the special case of empty DIR_SQUASH (which I do not recommend, anyway). Setting DIR_SQUASH to some empty directory should fix your problem.
Today's version of the script should hopefully also fix the bug.
I must say that I had some bad experience with funionfs: It seems that sometimes(?) portage cannot "see" the files mounted by funionfs, e.g. it occassionally complains about missing eclasses or even a wrong profile link (although an ls shows that the files are there): I guess some directory function is not always properly mapped by funionfs.
Does anybody have an idea what might be wrong?
So, in the moment, I can recommend only unionfs which you apparently haven't installed:
| kuku wrote: | | Code: | | mount: unknown filesystem type 'unionfs' |
|
BTW: If there is a need of unionfs for kernel 2.6.19, I might publish some quick-and-dirty hacks with instructions how to make them work... |
|
| Back to top |
|
|
kuku
Tux's lil' helper
Joined: 23 Dec 2004
Posts: 142
|
| Posted: Tue Jan 02, 2007 1:48 pm Post subject: |
|
|
| mv wrote: |
BTW: If there is a need of unionfs for kernel 2.6.19, I might publish some quick-and-dirty hacks with instructions how to make them work... |
I only use funionfs of 2.6.19 so it might be helpfull |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
|
| Back to top |
|
|
Maxwell
Tux's lil' helper
Joined: 10 Dec 2003
Posts: 97
|
| Posted: Fri Jan 12, 2007 1:08 am Post subject: |
|
|
Hi!
I'm trying to use an old pc as an host for some network services and i wanted to use its (64M) ram to speed things up. So when i searched the forums i found this.
As you all have tried several ways of doing this (sort of), how can i put something in ram when i boot from grub and then use the ram as the root filesystem? How much can i compress the data? 20% or 30%?
Thanks for the help!
_________________
Freedom works. Use it!
Linux, by Gentoo |
|
| Back to top |
|
|
synss
Apprentice
Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
| Posted: Fri Jan 12, 2007 12:46 pm Post subject: |
|
|
| Maxwell wrote: | | As you all have tried several ways of doing this (sort of), how can i put something in ram when i boot from grub and then use the ram as the root filesystem? How much can i compress the data? 20% or 30%? |
This is going seriously off-topic so you might as well start a new thread, but what you are looking for is initramfs. Data compression can be achieved in there with squashfs or cramfs, look for live CDs too, some of them (knoppix for example) do that.
_________________
Compress portage tree
Elog viewer
Autodetect swap |
|
| Back to top |
|
|
Maxwell
Tux's lil' helper
Joined: 10 Dec 2003
Posts: 97
|
| Posted: Fri Jan 12, 2007 3:04 pm Post subject: |
|
|
Thanks for the tip!
| Quote: | | This is going seriously off-topic |
Yeah, i know, but i've already started a thread, but no replies were found. Sometimes i just prefer to ask the right guy
| Quote: | | but what you are looking for is initramfs. |
Ok then!
BTW, how much compression can you get from the portage tree? how much ram is being used?
thanks for the help
_________________
Freedom works. Use it!
Linux, by Gentoo |
|
| Back to top |
|
|
jpmayer87
n00b
Joined: 19 Mar 2006
Posts: 51
Location: Troy, NY
|
| Posted: Fri Jan 12, 2007 3:16 pm Post subject: |
|
|
With squashfs I reduced the entire portage tree to about 44 Meg, that includes overlays as well.
JP |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
