"Ultrapeer" for Gentoo?

Soul-Crusher · Posted: Fri Sep 27, 2002 7:26 am Post subject: "Ultrapeer" for Gentoo?

When using a Gnutella client, particularly LimeWire, I'm presented with the option of using my ultra high-speed connection to help send files to slower people. I think something like this would be a good idea for Gentoo to help take some of the load off their mirrors, like if there was some sort of deamon which used FTP to transfer files that are temporarily stored on a user's computer from previous installs. So for example, if I recently built KDE, then this deamon would send information to some database that would store my IP and use it just like any currently existing Gentoo mirrors for someone else who wants to build KDE.

Just an idea to toy with. I know bandwidth isn't free, maybe Gentoo can use some p2p technology to it's advantage.

rac · Posted: Fri Sep 27, 2002 7:36 am Post subject:

Not a support request, moving from OTG to Gentoo Chat where hopefully people can help flesh this idea out.
_________________
For every higher wall, there is a taller ladder

Soul-Crusher · Posted: Fri Sep 27, 2002 7:59 am Post subject:

D'oh! Sorry!

Pitr · Tux's lil' helper Joined: 23 Jun 2002 Posts: 90

Are you talking about distributing ebuilds, precompiled packages or both... either way I like it.

Soul-Crusher · Posted: Fri Sep 27, 2002 9:04 pm Post subject:

Both, I guess. Sorry my idea isn't very well developed, I'm kind of a newbie to Gentoo.

I noticed there are a bunch of files in my /var/tmp/portage directory, roughly 120MB in my case. I figure that it's an awful waste for me to just wipe them off my harddrive.

I haven't the foggiest where the source packages go, those are what I had in mind. Users should have the option to share source pakages among peers (Provided the source file remains somewhere at least for a little while) rather than having everyone download from one of the official mirrors.

I suppose that could cause security problems as well.

pjp · Administrator Joined: 16 Apr 2002 Posts: 20485

Source ends up in /usr/portage/distfiles. If you emerge using the -b option, the binary packages end up in /usr/portage/packages.
_________________
Quis separabit? Quo animo?

Soul-Crusher · Posted: Sat Sep 28, 2002 3:50 am Post subject:

Neat! I have about 820MB of source packages

that I wouldn't mind sharing with other people in efforts to alleviate stress on the official Gentoo mirrors if only I knew how. I would imagine these are fairly common packages too--Things like KDE + QT, XFree86, Gnome, kernel source, Mozilla, a couple versions of GCC, etc.

phong · Posted: Sat Sep 28, 2002 5:15 pm Post subject:

If you have a reeeally fast connection, you can probably become a Gentoo mirror. But short of that - I think it would be interesting if the portage tree and distfiles were distributed in a p2p fashion. The only problem is the necessary trust that must be put in whomever you get the packages and portage tree from. I don't know md5 signatures are an adequate level of security for this. There was another thread discussing more robust package signing mechanisms.

It might be cool if people could act as low-bandwidth or part-time mirrors. Maybe they have a good interenet connection but not so good that they could become a full-fledged mirror. Maybe they could get a small number of requests (i.e. less likely than normal to show up in the round-robin DNS), or only operate at certain times of the day, or only distribute certain packages.
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer

rac · Posted: Sat Sep 28, 2002 6:16 pm Post subject:

soulwarrior · Guru Joined: 21 Oct 2002 Posts: 331

I have thought about the same idea and I have now seen that "Soul-Crusher" has already brought up this idea.

For overnet there does exist serveral catalogs on the net like sharereactor and the like. They contain a short description of the "packages" (mostly illegal) with a hash to identify the file on the p2p-net.
So, maybe one could put a sort of hash in every ebuild to identify the files one could donwload on the official-gentoo-p2p-net.

Emerge could than try to download first the file on the p2p-net and if the download is to slow, or the file is not already on the p2p-net it downloads the source the old-fashioned way from a known-source with the ftp-protocol.

Maybe one could adapt an opensource p2p system like gnutella for this purpose and create an official gentoo p2p network.

So people could not only help the opensource-idea by donating, posting in the forum, ... but also by giving a small amount of their bandwith to the community.

I hope some people with the necessary skills in the programming area will take up this idea and develop it.
I will be among the first people to test such a system :wink:

yokem55 · Guru Joined: 18 Apr 2002 Posts: 360 Location: Oregon

Would the fact that bittorrent is written in python make it a good candidate for this type of p2p file distribution? They use some pretty strong hash key checking to make sure the file remains consistent.

djco · Posted: Tue May 06, 2003 8:59 pm Post subject:

peepsalot · n00b Joined: 05 Apr 2003 Posts: 52 Location: Texas

I got this idea afeter reading https://forums.gentoo.org/viewtopic.php?t=52348 (now locked), as I was going to suggest a free distributed emerge solution instead of a pay for compilation solution(as suggested in that post). Then, just before starting a new thread, I saw this thread, which my idea also seems to very relevant to... Here is the idea I have in my head:

I think a cooler thing to do would be to have a distributed Emerge@home client similar to seti@home, genome@home, etc.

Anyone that wanted to participate could contribute spare bandwidth and CPU cycles to the community. Then, whenever someone wants to emerge something, you send a request out to the network, and each computer in the network could separately download and compile part of the code. As soon as one part of the code is compiled, it is sent to the requester where all the seperate pieces are put together. I think that distcc is pretty close to this idea already, so maybe it wouldn't even be that hard to put it all together. It seems the only extra thing needed is to turn it into a world-wide community, by having a way to discover hosts that are part of Emerge@home, and to add your computer to the list of hosts. Although I must mention that I know very little about distcc: don't know exactly how it works, so maybe it ain't that simple.

One other problem that might be difficult to overcome would be: in order to compile even one small part of a package, each participating computer would need to download the entire tarball for that package. That would be very wasteful, and probably even impractical for modem users. Here is one solution: each host in the network would only respond to a request to compile a package if it already had that package on it's hard drive. Also, hosts could specify an amount of disk space that they are willing to contribute to the network. So, when a host gets a request to compile a package that is not already installed on their computer and if their specified amount of shared disk space is not already used, they would download the source. They would then also be able reply to all subsequent request for that package, and keep that package in their shared space until it is deemed obsolete.

Gentoo users helping other gentoo users emerge things at blazing speed. Now that would be cool! Kinda seems to fit with the whole Gentoo sense of community too.
_________________
Call me butter, 'cause I'm on a roll!

Naughtyus · Posted: Wed May 07, 2003 12:15 am Post subject:

Another way of doing the above (emerge@home style) would be for packages to automatically build with '-b', and keep a record of the compiler options used, so if someone requests 'Openoffice for Athlon-xp -03 .. etc.', anyone who has already compiled with those options will have the binaries on their computer.

tsawyer · n00b Joined: 06 Jun 2002 Posts: 22

As Gentoo becomes more popular, especially with the introduction Gentoo Games, I would guess that serving ISO’s generates much of the load on the mirrors. Seems to me that if the Gentoo community would adopt an open p2p solution, such as Gnutella, we could make the ISO’s more readily available. This would hopefully reduce load on the mirrors for rsync purposes. Gnutella has the additional benefit of being widely used by Windorks who are likely to download the game CD’s.

We could finally use p2p for legitimate purposes.

Thoughts? Ideas?
_________________
There are 10 types of people in this world, those who understand binary and those who do not.

dweigert · Posted: Mon Jun 02, 2003 6:59 pm Post subject:

Hmmmm... I wonder if the new WASTE p2p thing would work well for this.....
Private nets, encrypted links, hmmmmm....

Dan
_________________
"Always remember to mount a scratch monkey..."

TazG · Guru Joined: 22 May 2004 Posts: 320 Location: Canada

Did this idea ever get anywhere?
I really like the emerge@home idea...

Antimatter · Guru Joined: 11 Aug 2003 Posts: 463

LucaSpiller · Posted: Mon Feb 07, 2005 6:52 pm Post subject:

If you want to help us poor old folks with 56k modems (since 1998

, by the end of this week I should be free though

), you may want to try a Deltup mirror. Basically it is a service that creates something simular to a compressed diff file between package versions and sends this out to the user.

As far as I can tell there are already quite a few servers sending out these packages but only one creating them. Setting up another server to create them would be good because sometimes there are queues or problems with the server. You should probably contact the author for more details on this.

Links:
https://forums.gentoo.org/viewtopic.php?t=215262
http://linux01.gwdg.de/~nlissne/

EDIT: Only just realised the first post is two years old. :roll:

_________________
:: Luca :: Mac Fag :: Original Macbook, 2g RAM :: Closet Linux user (seasoned with salt and pepper) :: C2D E4400 @ 2ghz, 4g RAM (only 3.2g detected under 64bit...), Nvidia 9600GSO ::

electrofreak · l33t Joined: 30 Jun 2004 Posts: 713 Location: Ohio, USA

I like the sound of this, however... it would have to be p2p because many connections for residents don't have much upload bw. However, I do not believe wget can do any sort of p2p downloading. There would have to be a completely different system used, one that when it can find "peers" with the file will then turn to wget and download from the gentoo mirrors.

Now for security... We would know that the files will be correct once downloaded because of the md5 checking. No problem there.

However, I think the deamon would have to have some sort of option for allowed bandwitdh because uploading all the time at full speed would definately slow my connection way down.

We can't share binary packages because they are probably different for each person. We would only be able to use the /usr/portage/distfiles/ . This would lessen the load on gentoo's servers and the mirrors.

truekaiser · l33t Joined: 05 Mar 2004 Posts: 804

interesting. so would this be like B.T. or like emule?

electrofreak · l33t Joined: 30 Jun 2004 Posts: 713 Location: Ohio, USA

truekaiser · l33t Joined: 05 Mar 2004 Posts: 804

yea that would be a great way to releive the stress on the mirrors for the larger files. you know open office(both source and bins of xaiman and normal), kde, gnome, and kernels.

Lews_Therin · l33t Joined: 03 Oct 2003 Posts: 657 Location: Banned

truekaiser · l33t Joined: 05 Mar 2004 Posts: 804