Portage with p2p

[pelux]

I have just submitted this enhancement in bugzilla , see:
https://bugs.gentoo.org/show_bug.cgi?id=23747
Here is the text, let me know what you think about it:

----------------------------------------------------------------------------------------------
I know that suggestions similar to this one had alrady been discussed a lot of time, but before you start complaining please read all i have to say.
The portage system is really powerfull. I'm not using Gentoo becouse i think that optimized packages for my system will be a lot faster than precompiled ones (well, maybe ), but becouse:
1) the stable branch is really stable but not too outdated (see Debian)
2) i can configure my gentoo installation as i really want (only the packages i want, the way i want)
3) if i have to install packages like mplayer , or the latest loster, i can do this with no trouble at all
The only drawback is the compiling time. Yes, i can compile in the night, but sometimes i really need to have a package installed immedialy, of simply i don't want to wait ages to have kde compiled on my not too powerfull laptop.
Finally here is the suggestion:
The idea is to create a p2p network where gentoo users can transparently share the already compiled packages.
Considering one package "name_version" (let's say kdebase_3.1.2) this can be different from the others already compiled one (samepackage_sameversion) only in three ways:
I) used optimizations
II) used USE flags
III) linked library's versions used (let's say it may be compiled using pam_0.73 or pam_0.74 as the dependency rule only states => pam_0.73)
IV) gcc compiler used
On point I i think that a lot of gentoo users will be happy to renunce using exotic CFLAGS settings, it they can get the packages already compiled THE way thay want with "standard" optimizations like CFLAGS="-march=athlon-xp -O3 -pipe -fomit-frame-pointer". Also the gentoo developers team is moving in the direction of defining some "stable&standard" predefined cflags settings for various cpu, as can be inferred from the "CFLAGS/cpuinfo collection project".
Point II should not be that great problem, as the possible combinations of optional USE flags in packages is not that high number. Also it usally happend that there are some "most used" USE flags combinations.
Point III is probably the major one. We have to distinguish between kdebase_3.1.2 compiled upon pam_0.73 and kdebase_3.1.2 compiled upon pam_0.74 (and also between different releases).
Point IV is not a problem as the gcc version used in gentoo doesn't change frequently.
All this system should be based on an existing p2p sharing protocol (maybe adapted) that has to be efficient, fast, scalable, and working also with firewalled users (there are a lot....) (am i asking too much? ).
The key point is the number of users adopting this feature (it would be nice to have an entry make.conf under features , like "binary_sharing"=yes), that should be enaugh large to cover all the needed combinations. The good news is that gentoo's user base is currently growing, with the popluraty of the distro, so i think that using this system a user shouldn't have any problem in findind the packages it need with the right characteristic for popoular (and not only) programs. Kde hell could end!
All the sharing have to be completely transparent to the user! So, if a user enables this feature in make.conf , the following two things will happend:
A) for every ebuild compile, two files will be generated under an arbitrary directory:
1) the binary package (ebuild with -k option)
2) the description (.desc maybe?) file, with the same name of the binary package, cointaing the three infos needed (optimizations, USE flags, version/revision of the packages the package depend upon).
This way the user actively contribute to the gentoo sharing comunity in a transparent way [an algorythm that deletes obsoletes file shold be added later......]
B) for every ebuild compile, portage automatically search with a query on the p2p gentoo comunity if the requested package is already present, with the required characteristic. If so it download it and install it.
A md5 check should be added later for security reason.
I know that a lot more should be added and discussed, but i need to know what you think of this idea. I also have to add that i don't have the programming experience needed to be a "first line developer" that directly modify portage and the sharing protocol, but i want to contribute for what i'm able to.
I really think that ,as the user base is already very large, this project would be succesfull.
This enhancement could improve a lot the gentoo experience, given the reduced package's installation time, while preserving all the portage excellent features.
For the brave who dared to read me till this point, i thank them! (also sorry for my bad english....)
Hope that this contribution will help the gentoo comunity.
Bye

Stefano Peluchetti

[Carlo]

The answer is simple: NO!

I wouldn't trust another gentoo user just because he uses gentoo.


Carlo
_________________
Please make sure that you have searched for an answer to a question after reading all the relevant docs.

[pelux]

But you don't have to trust him becouse you "feel" to do so.
A md5 sum check will be implemented, so you you will be sure that that you are not getting a trojaned version of the program. Anyway the security aspects of this system will be discussed in details if the idea get enough positive feedback. Obviously this point will be taken in grat consideration, as it's clear that you have to be asolutely sure that wahat you are going to install is not compromised.
Thanks for your reply.

[ciguanabo]

I have to agree with Carlo on this. Source code can (in principle) be checked easily, but binaries can't (because of all the different flags, options, etc). A solution could be to have official binaries which could be MD5 checked, of course this would put some restrictions as to how many variants of a binary package are available because not everybody will be able to compile official packages.
This reminds me... what happened to the GRP packages in 1.4rc2? They were really usefull for intalling a desktop system rapidly (and you can always compile the programs later with your own flag and optimalization settings), but I couldn't find them in the latest live-cd versions. Have the packages been deprecated?

[patan]

[Pythonhead]

[pygoscelis]

Here's a rudimentary security system.

We don't share linked binaries, we share object files. Typically there are many object files per shared package (or else it doesn't make sense to share it). Now you select few random files from the bunch and recompile them. If your objects and downloaded objects are identical, consider the entire package good.

This is not 100% bulletproof of course but it's something. Now, bizarrely, TCPA would make it easy to implement a 100% bulletproof check...

[pelux]

Sorry for the late reply, but i was very busy with study.
Anyway.....an idea should be to have users register to a gentoo page to be able to use this p2p system.
This way we could have differen type of "accounts",so you can see who contibuted that package, and check if for example it is a gentoo developer (a D flag for developer and U flag for users should be ok).
Also you can have a "trusted circle" of friends (that you know personally) and that you trust, so you can choose to download by them anyway.
Good night

Pelux

[dasalvagg]

This has probably been proposed before. I dont for a second think this is a new idea....

A vast majority of Gentoo users have broadband internet connections, and a vast majority use portage daily. The nature of Gentoo's source based distribution causes a drain on servers hosting the distfiles. This has been an issue in Gentoo land before, and will continue to be as the Gentoo userbase grows. The bandwidth problem can be eliminated with existing technologies applied to the Gentoo environment. My idea stems from the fact that I, and many other Gentoo users are willing to share a few kb/s, but do not want to dedicate a server to be a mirror. A couple reasons why I don't personally run a mirror.
1. I don't leave my computer on all the time.
2. Bandwidth intensive applications need to utilize full connection.
3. Can't handle a thousand requests a day.
4. Would only like to be able to server 2-3 files at a time.

I, like most other Gentoo users, accumulate a large number of source files in my /usr/portage/distfiles directory. Also, like most Gentoo users my computer has a broadband connection and utilizes very little of its full potential on average. It is a logical choice that ap2p network could alleviate the bandwidth problems. Any potential p2p network must be a voluntary network, to use on either end. By this I mean that if I am going to download a source file, then I must have the option of downloading from a mirror, such as ibiblio, or from the p2p netowrk. And likewise, once I have that file I must have the ability to chose to share the file, or not share the file. The protocol is largely irrelative, as long as it is efficient. Bittorrent, and Gnutella come to mind. In these circumstances it would be to the benifit of Gentoo users to have a seperate p2p network...sidestepping any legal action taken against these.
Key points
1. Use an existing protocol.
2. Voluntary to use the p2p network ie. /etc/init.d/gentooP2P (start | stop | status)
3. Adopt this into portage ie. emerge --p2p gaim Will download off p2p
4. Allow users to stop service if already involved in an intensive operation.
5. By sharing 50kb/s and using something like bittorent we can make a substantial dent on our use of distfile servers.
6. Have backup mirrors, if a file cannot be found on our p2p network, then use one of the main mirrors.

What do people think?

[broschi]

Don't forget that we also need a way to check the authenticity of the file. I guess we might as well simply check the md5sum from the current mirrors and check it against the ones that come from a p2p net. This way we prevent trojan horses in the packages among other things.
_________________
"Is this type of thing going to happen every time we switch to improbability drive?" "Very probably I'm afraid."

[delta407]

[Qball]

The idea is good..

I think for us poor adsl upload people it would be nice to have bandwith throttling..
and partial download (download it from 10 host at the same time) would be needed. (because not everybody has a big 10/100mbit upload. )
btw.. as protocoll we could look into gift.

[dasalvagg]

[pytigger]

This idea looks good. But there are drawbacks, other than authenticity:

1. unofficial mirror have to invest traffic / time for keeping up to date, or they are worthless
2. A P2P network is unreliable. I would not want to rely on it, I'd just use the working mirrors
3. Gentoo should make up it's own network. Existing networks, especially younger P2Ps would be severely polluted by the portage namespace!
_________________
And by the way - Gentoo needs an official WIKI!!

[syko]

[Genone]

Someone is working on it:

http://article.gmane.org/gmane.linux.gentoo.devel/10376

[dasalvagg]

[Ari Rahikkala]

[MooktaKiNG]

I believe bittorrent is the best option.

Personally i have been thinking of modifying the current bittorrent ncurses cleint to allow me to mirror ISO's more efficiently. Its not a good idea to run a new instance of the client for each .torrent file.

So i was thinking there would be like a small server application. That every couple of minutes checks the tracker to see if there are anyone trying to download the files that the server can serve. Then if there is it will start bittorrent client, to allow that person to download the file from the server.

This is just a basic idea, but i'm thinking of making it so that only one instance of bittorrent is running for multiple torrents.

This would work great on mirror servers.

Anyway, its just an idea.
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router

[pytigger]

If you have a dozen or more systems in a network that all use, say, AMD Duron processors, the other components differing more less, and you want to install Gentoo on all of them, if you are not very very clever, a source distribution would take hours, significant updates too.
If the filesharing thingy could be configured to make a local network mirror binary packages, that would be very cool, because every new computer introduced to the lan can install gentoo without the need to compile everything anew. In the internet this might be possible too: Since a P2P network multiplies the gentoo network strength, it can begin to store binary packages for those who want them.
_________________
And by the way - Gentoo needs an official WIKI!!

[ebrostig]

Currently no P2P application has a good solution for people behind firewalls, esp restrictive ones like the one I'm behind.

If this was implemented I would have to find another distro to use since I would never be able to update Portage.

Please, Gentoo is not only used in homes and the environment in the corporate world is very different from your home universe.

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

[Sieg]

It's a good suggestion, but indeed, the option of being able to turn it off would need to be there. There are a lot of reasons why some people would not like to share their bandwith, and the use of Gentoo could not affect that decision. If people want to do it, it would be a great addition, but the choice needs to remain there.

As ebrostig pointed out, this could not be activated in probably 99% of the corporate installations of Gentoo, as this would not only be bad ethics, but important costs on system usage. Home users would be affected in a couple of ways. First of all, not all ISPs offer unlimited bandwith to broadband users. I for one am stuck with a limit of 15GB of download and 15GB of upload from 7am to midnight. I know some ISPs that have limits even lower than that, with as little as 1GB of upload per month. They could just not sacrifice their small ratio for that. The other way that we could be affected is latency. Maybe there are not that many Gentoo users that are heavy gamers, but anyone that plays over the Internet knows that uploading kills your pings for most connections. Therefore, all these people would not want this service to be available during those times.

My 2 cents.

[dasalvagg]

I've never heard of a 1gig upload limit, but maybe i'm just sheltered. Of course....this was always intended to be an addition to the mirrors that are already in place, and not necessarily a replacement.

Gaming, low upload limits, corporate people should all be able to turn it off with a simply command like

[ebrostig]

[dasalvagg]

something of interest to this discussion...
http://news.gmane.org/onethread.php?group=gmane.linux.gentoo.devel&root=%3C20030723194823.GJ9959%40mail.lieber.org%3E