Shrinking Portage

[avx]

First of all, I didn't find an existing topic for this. If there's one, please merge or delete this one, thx!

So today I did my regular system-update via 'emerge --sync && update-eix && emerge -atuvD world' and watched the files being checked for updates when it (again) came to my mind "why the heck is portage that big?". I remebered that I once stumbled across the exclude feature of rsync(and with that also portage) and so read up about how to do it.
I did an 'du -hs /usr/portage' and it returned ~620mb(not including distfiles and packages!). For me, portage isn't readable via the filesystem and I'm not really a fan of GUIs for such tasks, so I browsed portage on http://gentoo-portage.com to see, what's really in portage which I need. It's been frustrating I use only <= 5% of what's in portage, but I download the ebuilds/patches/digests/etc for everything in portage, which of course isn't good for me nor for myriad of mirrors out there.

So far my system is running well and has everything I need, so there's no real reason to keep ebuilds on my box/sync them, if I would never use it. So I quickly hacked up a little script to generate me an exclude-file based on what I've installed on my system, let it run, rm'd portage an sync'ed again. Well, after again doing an 'du -hs /usr/portage' I was quite buffled, seeing that it has only 117mb left by now and still includes all I need. Since I'm using a compress portage as described in this thread my squashed-down portage has less than 10mb in size and is fast enough for me.

So what's this thread all about? Well, why do I have to do the work? My script's far from perfect, but at least it works - thinking about doing this task per hand makes me really shudder, 'cause the file now has nearly 1k of lines(could be made more compact, but I prefer it being readle just in case).
So I think, the benefits are clear. I/the user saves bandwidth and diskspace and on top I get a faster portage(syncing/searching) and the mirrors would save a whole lot of bandwidth, if anybody would make an exclude-file.

So the question is, wouldn't it be better/easier to have an portage_include instead of portage_exclude? With it, the user would only net to answer a set of questions and get a portage as needed.

To make it clear what I was thinking about, check this little questions, one could be asked:

1) Desktop or Server?
In case of desktop, categories like www-servers, www-apps, sys-cluster, net-www, net-zope are unneeded.

2) Architecure?
Let it be x86/x_64, so you could drop anything related to for example ppc,sparc,etc.

3) Desktop or Notebook?
If desktop, most things related to acpi, powersaving, wireless, pcmcia, etc could be dropped.

4) X or not?
If not, any X-program could be dropped

5) If X, Gnome, KDE, XFCE, other?
Drop all which are not $answer, if $answer = other, then ask for the prefered toolkit and drop the others.

6) Gaming?
Drop all games-* if not

There could be a lot more of this easy questions, but I think you get the idea.

So, what do you think?

cheers,
ph

[anello]

Gentoo is all about choices, but these choices aren't supposed to limit.

Your questions are limiting the user. For example, what if I want to have a desktop, but also want to have trac since I'm a developer ...

There are always exceptions and the beautiful thing with gentoo is that I can configure the system just as I need it to be.
_________________
Antonino Catinello | http://catinello.eu

[avx]

I see your point since I'm also running an apache for testing on my desktop-machine, but still I find it easier to include packages I really want instead of throwing out everything I won't need. A little grep/sed-magic would make it easy to include packages and there dependencies.

Sure, Gentoo is about choice and I don't want to force anybody to use an exclude/include-file, but it would be good if it would be used.

[think4urs11]

[Q-collective]

Portage has a number of issues but you name two very important ones:
1. its use of rsync
2. it's huge

Problem 1 can be devided into several other problems: it's slow and the tree is always current or out of date which can create problems if users depend on specific versions of ebuilds or something screwed their system and they don't exactly know what it was. You can solve this by using subversion instead of rsync which is much faster and has the ability to pull in older versions of portage if needed. The common argument against subversion is that it would put a higher load on the mirrors. I personally think that this is not really an issue since we're talking about the same volume of data.

Problem 2 is trickier. Portage consists of about 150 000 files that are all a few kilobyte big, this does create a huge tree of several hundreds of MB. You already mentioned the answer: using squashfs by default would immensely reduce the tree. This does create a little slower tree, but all in all squashfs is well known for its good performance.

I think this would be two very good, yet relatively simple, improvements to portage.

[pjp]

This has been discussed quite a bit. Here's one issue:

[Genone]

[Q-collective]

[Genone]

[ciaranm]

Please stop confusing 'the tree' and 'Portage'. It's very annoying.

[Hauser]

[Archangel1]

I'm not sure it'd work in practice particularly well. I could disagree with a couple of the examples you gave,

[boniek]

You can always put portage tree in a sparse file, formatted with filesystem using small block size to make it somewhat smaller.

[avx]

For my examples, they are *just* examples - haven't thought about them too much

For the squashfs-part...have you read the whole thread I linked in the first post? It's not only about squashfs, but also about unionfs to make squashfs writable in RAM and write it the changes back squashed directly, which works pretty good.

My 'tree' is now only 117mb unsquashed and *less* then 10mb squashed, not noticibly (do I write it like this?) slower and it wasn't too much work. As I said, my systems are build up finished I have what I want, nothing more or less, so it's pretty easy to find out what parts of the tree I don't need. If I really need or want to try out a new application I hear/read about it somewhere on the net so it ain't a big problem to open up gentoo-portage.com and/or b.g.o./$overlay and check if there are already ebuilds available. If there's no ebuild the I may write one myself(seeing the deps on the authors page) or build it by hand - in both cases I need to know if/what dependencies I need and therefor it's pretty easy to include them in my list. Right now I'm working on a script to add (recursive) deps to the list and it works so far. I'm no script-guru at all, but if I can do it one of our beloved devs can do it, too.

To make it clear once again...I *don't* want to force anybody to do it and I know (by my own experience) that it can be a little difficult to setup, but also I see the benefits for my self and also for Gentoo as a whole - after all, it's one easy way to give something back to the community AND make my own system better.

So please, if you're only hanging up on my examples, think about some other examples, there's definetly no need to have the whole tree at home if:

a) you know what you want
b) your system already has everything you want/need
c) are willing to spend some time

cheers,
ph

[AllenJB]

You can exclude categories (and even individual packages) from portgae with rsync's exclude_from. See http://gentoo-wiki.com/TIP_Exclude_categories_from_emerge_sync