Binary packages for stable users [long]

Lazlo

It is a fairly long post and I took some time to think about it and write it. So please read it to the end before trolling me. Thanks.

I'd like to see the possiblity for users that don't want to build every package to get packages off a binary repository. This would typically be people who has actual work to be done and can't afford the system to be unavailable for an extended period. Servers and average workstations users come to mind.

Some server administrators are already building their own binary repositories to cover their own needs. Some even open and contribute their repositories to the community.
https://forums.gentoo.org/viewtopic.php?t=87151

We would need to introduce a concept I call a PROFILE. Your gentoo profile would consist of your ARCH, CHOST, CFLAGS and USE variables.
Now if you had the same profile as some other user should be able to exchange binary packages without any problems. So obviously users who want to take advantage of my little scheme would have to gravitate around common profiles. With the users I have in mind I'd guess that you'd have to stick to more conservative settings. The settings used to produce the gentoo stage 3 system, could be used as the common ground.

Now you'd use the -k flag in emerge as it was intended. If a binary package for a matching profile exists, emerge would just download that and install it. If no matching package exist you'd just build it locally as you do now.

There are of course some technical issues that should be resolved before this could become a reality.

1) Unlimited number of profiles makes space requirements explode on server

We might have to limit the profiles to a small select handful. I don't belive that there will be a noticable difference between the performance of "all cflags i could find mentioned at the forums" and best practise flags. Of course we would have to have a profile for each processor family. Athlon-XP, P4, IA-64, Athlon64 and so forth, but apart from that someone insightful would have to deside what flags can stay and which will have to go from the cflags. For the USE flags I think it is pretty obvious that we would have to go with just one for each machine arch. However there shouldn't be any limits. If you change your profile to something clever, and can get enough people to do the same you have a community in your own right.
Plus 200GB discs are more than reasonably prized and can fit an amazing amount of data.

2) Binary packages would fill the server over time

A program would have to analize the hits the binary packages get on the server. Once a package hasn't gotten a hit for a while it should be deleted. So if somebody would force install e.g. gnome 2.0 they'd have to wait for a manual build.

3) With all the packages on the server they'd run out of bandwidth. Trying to serve all the users and sync each other at the same time.

They would if we relied on ftp or rsync technology. For large files something like BitTorrent would be a much better match. We would just need one central mirror to host the torrents. Then all who downloads would benefit from each others bandwidth. Gentoo users like to be up to date, so there would be a fair chance that they'd all download the same stuff at the same time (e.g. after the gnome 2.6 release) which is just the kind of situation that BT is created for.

4) Who would maintain all these binary packages.

I am open to suggestions. You and me, mostly. ArchLinux has an open ftp server where you can upload binary packages for review. Maybe gentoo could have the same thing. Perhaps emerge could be expanded to generate the binary packages (it already does this part), couple it with the profile metadata and upload it to your preferred upload mirror. Later a maintainer could approve the package and publish the torrent to the portage tree. Placements in the portage tree and naming conventions should leverage existing portage mechanisms for torrent discovery.

5) What if you want binary packages, but don't have a matching profile.

Like you do now, you could temporarily alter your USE and CFLAGS to adhere to a certain profil of your choice -- at your own risk of course!
A "fallback" i686 or even i586 repository could also be maintained, I don't know if that is really needed though.

6) How could profiles evolve ?

If the profile flag was adopted as a standard part of the make.conf you could have it point to a file with the flags in. Like athlon_xp_server.gentoo.profile.conf pentium_4_workstation.gentoo.profile.conf and so forth. Those files should be updated as part of the standard emerge update process then.

Thanks for taking the time to read my post. I am looking forward to hear your feedback

Regards
Lasse

klieber

madchaz · Posted: Sat Jan 03, 2004 12:40 pm Post subject:

Hello Lazlo
Binary packadges have been a "hot button" topic for a bit on the forum here. Some people are agains it, some are for. However, I think you might want to look at a few things that are comming.

Some of the talk about binaries in portage is going on from the dev side as well, from what I understand. I think portage-ng that is comming will have the capabilities to introduce binary packadges.

As you pointed out, the bigest draw back of binaries is optimisation. You can't have ALL the possible use flags and cflags for everyone. Standards will have to be set in order for this to become possible.

Personaly, I find the idea of bin packadges interesting, but it as to remain an option, not something you have to use. Check out the portage dev page to see what's comming for this

_________________
Someone asked me once if I suffered from mental illness. I told him I enjoyed every second of it.
www.madchaz.com A small candle of a website. As my lab specs on it.

klieber · Posted: Sat Jan 03, 2004 12:51 pm Post subject:

Lazlo · Posted: Sat Jan 03, 2004 1:10 pm Post subject:

I want to make absolutly clear, that this is an enhancement, not a replacement of emerge.

As kurt mentions emerge already has the -k flags and the BINHOST setting.
The trick is that is doen't work as no BINHOST mirrors exists currently.

My proposal just seeks to clarify some of the technical issues there might be before it could actually work.

Of course if you do not call emerge with -k you do not "pollute" your system with prebuild binaries. So it is totally optional.

Regarding the enormous variation of USE and CFLAGs. I don't belive that we see all permutations like you claim. I'll be willing to bet on that 80%+ users running gentoo stable would fit into maybe 10 different profiles. I don't claim that you did. But I and a lot of other users roaming the forums just copied some sensible looking USE and CFLAGs and went on with that. Like I said, the settings used for building the GRP could be a good starting point to identifying the most common profiles.

If there isn't plans to improve this situation as kurt says. I think we as a community should think hard about how it could be improved.

So basically I belive it is a 80/20 thing. If you are part of the 20% that won't benefit, then be content that you are already content with the system at it works now. If you are part of the 80% that might benefit. Then look forward to upgrading KDE in 20 minutes instead of 20 hours.

I think the advantage to that is pretty obvious.

/Lasse

EvilTwinSkippy · Posted: Sat Jan 03, 2004 1:30 pm Post subject: Cheaters Answer

Well, my answer it to cheat and create a new distro, based on Gentoo. Thus eliminating the flamewar issues, and leaving one free to experiment and modify as needed.

Gentoo is really intended to be a meta-distribution anyway. Why pollute the purity that is Gentoo with the troubles of maintaining a binary system. The developers don't want to hear about it. Most hard-core users don't want to hear about it.

The folks who are interested in it fall into 2 camps: those who need to maintain a butload of machines and don't want to have to reinvent the wheel, and people with crufty old computer that lack the cahoonas to perform a full gentoo build. Many just want something to unstuff onto a box that has half a chance of being kept up to date.

Neither of these camps are as interested in performance as they are in compadibility and relative stability. Rapid deployment and a simplified install process would be a plus.

My misguided effort is being documented at http://www.etoyoc.com/tao-linux
_________________
I've found that people will take what you say more seriously if you tell them Ben Franklin said it first.

klieber · Posted: Sat Jan 03, 2004 1:31 pm Post subject:

klieber · Posted: Sat Jan 03, 2004 1:36 pm Post subject: Re: Cheaters Answer

Lazlo · Posted: Sat Jan 03, 2004 2:20 pm Post subject:

You see debian does a pretty bad job of optimizing their binaries for my (sub)architecture. I just tried the latest and greatest, and I was not impressed. ArchLinux offers some of what I want, but has a very little community. I simply don't see a reason that gentoo couldn't have a binary repository too. And I certainly don't see a reason to create another distribution to add such a relativly small feature.

You might be right that we'd be hard pressed to fit everyone into 10 categories. That was a random number anyway. I wish we could do something more scientific about it that just throw mud though. Take the most common archtectures, use a proven vanilla CFLAGs. A little too heavy USE flags isn't going to kill anyone.

Is there a place where stats are kept on gentoo users in general. Does emerge send any info back to the portage servers ? Isn't AthlonXP and P4 way the most popular ?

How much space does a compiled portage tree take up ?
You could go the other way and see how many binary repositories would fit on a $100 disc. P2P could take care of most of the bandwidth issue.

If gentoo had a binary edge I am sure more users would be attracted. Debian would have very little to offer against us then it seems to me. And as gentoo growth out of its little niece. More joe-average-user is attracted. People who are happy to stick within a predefined profile.

EvilTwinSkippy · Posted: Sat Jan 03, 2004 2:54 pm Post subject: Binary Edges...

EvilTwinSkippy · Posted: Sat Jan 03, 2004 2:56 pm Post subject: Re: Cheaters Answer

QPegasus · n00b Joined: 27 Sep 2003 Posts: 42

With this debate, along with others, a popular arguement seems to be "well, if you want binary packages, go away and use another distro".

Well, I don't use gentoo because its source based, and I'm pretty sure I'm not alone. Infact, I'd rather use a binary distro, but none exist that match up with portage imo. Few that I've tried have been as easy to use and problem free as portage, and those that are either have a poor selection of available packages, or a heavily outdated selection of packages.

Thats the reason I use gentoo....along with the large and helpful community.

I dont see how it can hurt to have a binary repository with packages for the popular CPU's and sensible/common USE/CFLAGS. Obviously its impossible to have a binary available for every combination under the sun, and many would still compile things themself, but how are they losing out? Nothing will change for them, it'll still be the gentoo they know and love.

On the bandwidth issue - surely, adding binaries wont add as much as some say? I know that syncing would certainly use more, but the actual packages? When someone downloads one of the new binaries, arent they just downloading that instead of the source they would have downloaded anyway? Also (someone correct me if I'm wrong), arent sources usually larger than the binaries?

The way I see it, it would make gentoo appeal to a much larger userbase, aswell as making things easier for many current users. The rest wouldnt be affected and would continue the way they always have.

imo, its not a matter of "would it benefit gentoo?", its "is this possible?". The problems are, disk space, bandwidth and any costs associated with those. With so many potential contributers and packages, security would also be an issue. Probably the biggest one imo.

Just my 2 cents....but what do I know.

klieber · Posted: Sat Jan 03, 2004 3:56 pm Post subject:

DrNick · n00b Joined: 02 Jan 2004 Posts: 15 Location: Sweden

I for one, having been through rpm hell of redhat 6 and mandrake 6/7/8 chose gentoo for being the number one source based distro in the world IMHO. Not because I can chose whatever uber cflags when compiling, hell I'd use gentoo if it was the slowest distro out there, but just because doesn't break (...that easy

) as binary distros tend to do.

But maybe there's a middle road to this discussion, there're probably relative few packages that are a real pain to compile and maybe they could be stored somewhere or why not a comunity effort among those interested. Having a common naming scheme for the packages including the USE-flags and p2p sharing.

# emerge_p2p package_useflag1_useflag2 || emerge package

klieber · Posted: Sat Jan 03, 2004 4:21 pm Post subject:

Lazlo · Posted: Sat Jan 03, 2004 10:00 pm Post subject:

It seems like Daniel Robbins is on my side here, if that counts for anything.
http://www.gentoo.org/main/en/philosophy.xml

I can't seem to find the any plans on what portage-ng holds for the future. I'd be happy for a url, if anyone has it. But reading the philosophy page, I find it hard to belive that better support for binary packages is not planned. It is only THE single largest complaint from people who has looked at gentoo.

As for security and binary packages. In the end it is all about trust. You can assume that the binary packages are safe just as safe as you assume that nobody has injected backdoors into the code you are manually building. Like the recent case, that was fortunatly caught in the kernel development.
http://slashdot.org/article.pl?sid=03/11/06/058249&mode=thread&tid=106&tid=185

It is a matter of procedures and engineering to ensure that the system stays as safe as possible from intruders. I am confident it can be done so it is both safe and a small hassle to add new binary packages and an even smaller hassle for maintainers to test and approve these packages.

With the amount of data P2P already serves today, I can't see how a 3-400Gb binary repository would be impossible to support at all. Hard drives are damn cheap. P2P allows for massive downloads without killing the server.
So it would require P2P rather than ftp and rsync and automation rather than manual labour, but impossible? I don't think so!

/Lasse

Lazlo · Posted: Sat Jan 03, 2004 10:20 pm Post subject:

Woops, I was a bit too sloppy seaching there. I found the portage-ng plans myself.
http://www.gentoo.org/proj/en/portage/portage-ng/systemspec.xml

They are very highlevel and seems to mention that it is a rewrite of the current version. Nothing about why the current is so bad, but we'll just have to assume that it doesn't adhere to the architechtural requirements outlined in the document.

The only thing it really say about features is this:

EvilTwinSkippy · Posted: Sat Jan 03, 2004 10:42 pm Post subject: Binaries over P2P

Lazlo · Posted: Sat Jan 03, 2004 11:09 pm Post subject:

Well the difference between the legal and illegal use of P2P is that in the legal version which i belive BitTorrent is a proponent of you have a backbone server that holds the actual file. So if you are alone in the world you'd still be able to pull the file off that one server albeit slower. If more than one server participates in the BT network you'd be able to download a bit from each server at the same time.
So using a P2P technology where there is an actual server holding the entire file the whole time is essential.

As to disc space. Ok, maybe running the operation off IDE discs are not a good idea. So maybe it is a bit more expensive that I would have it sound like. Still servers like ibiblio must have enormous capacities available.

If pressed hard the mainstray of packages would have to be just i686 optimized and more CPU intensive programs like most of the desktop stuff would be more fine tuned for your sub-architechture. I am not sure anyway that I can defend maintaining 15 super optimized packages of the vi editor anyway :P

I'll still see before I belive that the space we need can't be found though.

klieber · Posted: Sun Jan 04, 2004 1:52 am Post subject:

Lazlo · Posted: Sun Jan 04, 2004 2:00 am Post subject:

Perhaps you didn't read the philosophy page I linked to.
Here's an interesting quote:

Kihaji · Apprentice Joined: 12 Sep 2002 Posts: 230

Cerement · Guru Joined: 14 Jun 2003 Posts: 404

Several others have already mentioned that portage can already use a binary package, but this item popped out:

EvilTwinSkippy

Let me try to assemble what we all have said here so far:

First the issue is not about binary package support, portage has that, it's about creating a binary repository. Gentoo Alumni (myself included) generally cast a disparaging eye on the whole concept. We have a hard enough time keeping the sources straight, let alone figure out what canonical package was linked with what version of what library.

In order to successfully create and maintain a binary repository would require such a generalization of features that the result would be almost, but not entirely, unlike Gentoo. The maintenance of the binary repository would require the same time and resources required to maintain a standard binary Linux distro. You are just using Gentoo for a build environment.

Now, I do see merits to the "sub-distro" approach. I maintain my own for my server cluster. With a little communication, and some forethought a small community could come up with a microdistro that serves their specific needs.

Which brings us back to the original intent of Gentoo. It's not a distribution. It's a way to come up with your own. Now, if someone is particularly queer for a distribution that knits in KDE, Cups, Alsa, MySQL, and TCL with the default editor being EMACS, my little 'tao-linux' distro is perfect for you. Some other VI
_________________
I've found that people will take what you say more seriously if you tell them Ben Franklin said it first.

Lazlo · Posted: Sun Jan 04, 2004 7:35 pm Post subject:

I certainly don't see why how this discussion can be summed up to that we should all create our own specialized binary repositories if that is needed.

I would sum it up as follows.

A binary repository would be a great addition to the gentoo world. Users would likely flock to join the gentoo banners, when given the prospect that an updated system could be installed in a matter of hours, rather than days.
The situation as it is now is absurd. Granted, you have the LiveCD that can provide you with a working system in fast. But since the first thing everybody does is to run emerge sync && emerge update. They soon discover that getting an updated system from a stage3 install is slower than it would have been from a stage2 install.
Also you see people like this guy:
https://forums.gentoo.org/viewtopic.php?t=120853
try to do twists in order not having his main CPU occupied with building all the time.
Or - I suddenly find my self urging to watch a divX movie. How long did you say I have to wait before totem and dependencies is ready ???

The reason the binary distos isn't good enough is that they are not compiled for my machine architechture, but rather one 5 or 10 years old. And if a binary install fails on one of the usual distros they can't handle a natural fallback to a build from source.

I outlined a few ideas off the top of my head on how some of the technical difficulties could be handled. Good response back on why it wasn't all as easy as I originally imagined it. So experiments will have to show how many binary repositories it would be feasable to handle and how many % of packages they should hold. My take is that when a package is marked as stable, a binary build should be provided.

Some claim that if I want binary packages I have chosen the wrong distro or that no support for binary packages is planned for the future.
I belive that drobbins proves the first statement wrong if the philosophy page is to be trusted.
http://www.gentoo.org/main/en/philosophy.xml

If this:
http://www.gentoo.org/proj/en/portage/portage-ng/systemspec.xml
is truly all the plans that currently exists of plans for portage-ng, then it is far to early to conclude anything on what amounts of support is or is not planned for improvements on binary packages.

But the plan does state that it is open for suggestions. So I hereby formally suggest that portage-ng should have excellent support for building and maintaining binary repositories.

Now I think I will go and see what thoughts the portage-ng developer group has on the concept.

Has anyone found links to further plans for portage-ng development or even the mailing list archives for the portage-dev mailing lists ?

Thanks for the feedback and the civilized tone so far.