Portage with p2p

panserg · Apprentice Joined: 16 Apr 2003 Posts: 188

is there any p2p protocol that is transparent for a typical http-proxy? The best woul be a p2p based on a wedav server.
_________________
Less is more!

Sieg · Tux's lil' helper Joined: 07 Oct 2002 Posts: 87

delta407 · Posted: Fri Aug 29, 2003 12:44 pm Post subject:

dasalvagg · Apprentice Joined: 26 Jun 2002 Posts: 183 Location: NY

In an e-mail to a Gentoo dev I received this response about protocols.

fava · Retired Dev Joined: 27 Sep 2002 Posts: 8

Regarding Bittorrent:
Bitorrent requires a separate instance and a separate port for every file that you are sharing. When some people will be sharing hundreds of files this is simply not a workable approach.

Bittorent is optimized towards dealing with a small number of large files under very high (initial) demand situations. Our needs are a little different, we are dealing with large number of much smaller files under moderate demand and without the initial spike in demand.

In General:
The content of the network needs to be tightly controlled. We are dealing with source code here and the consequences of someone being able to inject malicious code or malicious files into the network would be unthinkable. Therefore ALL files on the network MUST be vetted by a central server, and ONLY files allowed by the central server would be allowed on the network. All files transfered over the network MUST be checked to make sure that they match the checksums of the authorized files.

I also want to create a network that is 100% legal content so that when the RIAA or MPAA goes on a rampage we will be unaffected.

As far as I know there are no p2p networks that allow this much central control, the trend in p2p networks is away from central controll because of liability issues.

mmealman · Guru Joined: 02 Nov 2002 Posts: 348 Location: Florida

The p2p protocol should only share files that match a master list kept at an official Gentoo mirror with the proper checksums.

Also, the protocol should be limitable in various ways even beyond simple bandwidth throttling.

For example, if my server is on a 100Mbs line with a 150GB monthly limit and a 95% rate at 256k, I'd have no problems at all letting the p2p burst at high rates so long as I could control the frequency and monthly upload GBs.

The upload bandwidth usage need to be very customizable, controllable and monitorable if admins are going to use it.

syko · Posted: Sun Aug 31, 2003 1:22 am Post subject:

It definitely could not be anything but an option, and it would have to be disabled by default. I like the idea, and the plan, but with my 2GB upload cap (not quite the 1GB of others, but still pretty low) I wouldn't be able to participate 24/7.

I think it all depends on continuing to use the standard gentoo mirrors, in addition to a network like this. If a p2p system would offer more benefits, then more and more people would use it, reducing the load on the primary servers, and allowing corporations, and others who cannot use a p2p network for whatever reason to use a less bogged-down standard http or ftp server.

dasalvagg · Apprentice Joined: 26 Jun 2002 Posts: 183 Location: NY

so...looks like everyone is in agreement that this would be great for gentoo. Anyone willing to help out fava? Everytime i think about it, it seems like a larger project.....

ebrostig · Posted: Mon Sep 01, 2003 12:15 am Post subject:

I don't think it necessarily is good unless this is a second option and it can work with any kind of firewall or proxy and not require specific ports to be opened.

I also feel it is not a good thing if it would require people to host parts of portage nor do I see how syncing of Portage is going to work nor a lot of other things.

Please work out a detailed description prioor to starting and coordinate it with the developers prior to start any coding.

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'

dasalvagg · Apprentice Joined: 26 Jun 2002 Posts: 183 Location: NY

Have you read the previous posts?

Firewalls...you're not the largest part of the population. Purpose is to get a good number on it, not necessarlly 100%. Do what is easy first, then add help for others.

Syncing.....was never brought up. This is for distfiles.

fava · Retired Dev Joined: 27 Sep 2002 Posts: 8

progster · Apprentice Joined: 16 Jul 2002 Posts: 271

If it wasn't optional I think gentoo wouldn't run on any servers anymore, as traffic costs money. Also p2p networks are not that great for cable users (fast download but slow upload), for some reason the slow upload speed actually slows down downloads on p2p networks (especially bittorrent)

~progster

pytigger · Posted: Mon Sep 01, 2003 6:43 pm Post subject:

There has been a discussion about improving portage in general, and one of these things is making the download / build process parallel. That would smooth out a few speed issues.

The packages that are downloaded must be verified. One solution might be to check a checksum from portage or any trustable mirror (or even more than one). Gnutella would thus be quite usable (but keeping clear of any traditional Gnutella networks of course).
The problem is, that some lamers could probably modify gnutella or any other protocol to name the correct checksum before downloading, instead of the real checksum from the modified packages. Both with multiple and single source downloading portage would have to wait until the download is complete to see there was a corrupted file download.

And, of Course, the P2P thing has to be kept secure. IP revealance is not really relevant, since we all spread our IP around the net anyway, except maybe some who notoriously use proxys / routes to remain anonymous.
It must be made sure that users don't have security holes activated. There should be documentation about various aspects of possible misconfigurations etc.
_________________
And by the way - Gentoo needs an official WIKI!!

dasalvagg · Apprentice Joined: 26 Jun 2002 Posts: 183 Location: NY

Checksums and the like could be downloaded from a central server everytime a file is downloaded from the p2p system. Or it could be included from an rsync, which would be out of the scope of the p2p system. Either method makes certain that an outside source is available to verify the integrity of the file.

srob99 · n00b Joined: 14 Oct 2003 Posts: 48 Location: Perth WA

People have mentioned that bittorrent requires a separate port and instance for each file - as the main argument against bittorrent.
Sure.. that is for the official bittorrent client. Keep in mind that bittorrent is also a defined open protocol. We are completely allowed to create our own client (which is still bittorrent compliant), but which uses a single port and a single process. For example, see mldonkey's bittorrent implementation. It works reasonably flawlessly, i'm down/uploading about 4 completely different bittorrent streams (each one using a completely different tracker too) to various people/IPs, and only need one process of mldonkey. I also only have one port open on the firewall for bittorrent.

For a Gentoo implementation, the "bittorrent distfiles" service would need to be optionally activated from /etc/init.d in a runlevel, and emerge (with appropriate command-line options to enable this mode) would somehow need to tell the bittorrent client to add a particular package to its "list" of files to download/upload. emerge would then need to have a way of knowing when the download is complete, so it can continue on it's merry way. The bittorrent service, in the mean time, is sitting there day in, day out helping everybody else with all the files in /usr/portage/distfiles that it can find a valid tracker at gentoo.org for.

Probably another fairly important required feature would be bandwidth control, in particular for upstream. eg a modem user would probably want to limit the upstream to 1k/s if he/she wants to anything else useful with the link. still, each 1k/s that people offer is 1k/s less that a mirror has to provide!

Security concerns are of course a non-issue with the bittorrent protocol, as each 256k block is hash protected, the hashes are stored on the central tracker, and the official trackers would be run by gentoo.org (a custom tracker server would need to be created that could track multiple bittorrent streams at the same time, but this is probably already done. [haven't looked at bittorrent tracker for a while])

Indeed, a "bittorrent gentoo distfiles client" project would be a great idea to reduce official mirror bandwidth requirements around the world, but still does not tackle the issue of actual storage volume required on the official mirrors! (which of course still need to be maintained, as they are the backbone of gentoo!! - a P2P distfiles system is only useful as an add-on to those who actually want to "rc-update add " [and start] the service.)

pookey · Posted: Sat Mar 06, 2004 2:30 pm Post subject: Portage with p2p

I had an idea a few days ago, and thought I should voice it here, perhaps someone might have an opinion about it.

When someone compiles a program, it is packageed up, and some kinda hash is makes of the compile and use flags for that package, and this file is stored somewhere on the file system. This file could then be made availible on the peer to peer network for others with the same use/compile flags to download, which would be a lot quicker then compiling it, but would achive the same goal.

This idea might see a little insecure, as someone could build a nasty package. I've not really come up with a conclusion on how to get around this. Perhaps a package wouldn't be considered valid untill there are 10+ sources for it, but even then that's no real security.

I don't have the time or the skills needed to set the ball rolling on this unfortunatly. but If anyone has any ideas on this, I would be interested to read any feedback.

Earthwings · Posted: Sat Mar 06, 2004 4:49 pm Post subject: Re: Portage with p2p

pookey · Posted: Sat Mar 06, 2004 5:15 pm Post subject: Re: Portage with p2p

Earthwings · Posted: Sat Mar 06, 2004 5:46 pm Post subject: Re: Portage with p2p

pookey · Posted: Sat Mar 06, 2004 6:35 pm Post subject: Re: Portage with p2p

sanity · n00b Joined: 19 May 2002 Posts: 39 Location: Iowa

One nice thing about Gnutella, last I checked, is that they actually use HTTP as the transfer protocol, to the extent that if your Gnutella client cooperates, you can actually browse it from a Web browser.

However, Bittorrent provides a single place to get both the file verification and a way to get mirrors -- the .torrent. I have no idea how it's implemented.

Freenet might actually solve both problems -- I'm not sure if it's transparent to proxies, but it certainly allows for entirely p2p operation (no censorship possible, so no worries from RIAA) while assuring data integrity.

Unfortunately, Freenet is written in Java and is very slow and buggy. We'd have to write our own client.

Clients to anything except Konspire2b can download from behind a firewall, but not necessarily upload. It's not too hard to imagine a client which allows no uploads at all -- in fact, giFT allows one to set the maximum number of simultaneous uploads to 0.

If we were to implement our own client, we'd want to make sure it's generic. And if we do that, there's no point without first considering all alternatives -- even Kazaa (FastTrack) via giFT!

Finally, remember that data integrity means we have to sign packages. Downloading an md5 over rsync is not secure -- "some lamer" can intercept your rsync connection just as easily as your p2p connection. There are plans to implment such signing, although I have no idea what progress has been made.

And if we are going to do this for distfiles, it might be nice to also do it for sync, somehow. We'd want a p2p network that scales well to small files, of course, but rsync really doesn't have much point, as we usually update whole files anyway. The advantage of rsync is it's efficient at updating lots of small files, and only the new ones. But everything we do over rsync, if we are signing the portage tree, makes more sense over p2p, if it could be made fast enough.

epall · Posted: Sun May 16, 2004 4:33 am Post subject: anything new?

I thought up pretty much the same idea as has been outlined here a few days ago and came to investigate. As far as I can tell, a bunch of work was done in the Summer of 2003, and then everybody walked away. Is this true? I'm very interested in supporting this in any way I can, but not alone. Is there still interest in this?

One thought I had was that bittorrent may not be the right protocol to use for this. I don't think we should use something like gnutella, in which the goal is one giant swarm, but bittorrent isn't that good for the thousands of tiny distfiles that gentoo has. I propose we consider OpenNap, because it provides the isolated(ish) network that we get with bittorrent, but is far better suited to exchanging many small files. I admit, there still is the issue of large files, because downloading from only one peer is not very good. Maybe we should create our own protocol that's a hybrid?

Malakai · Apprentice Joined: 24 Dec 2002 Posts: 299

[quote="dasalvagg"]

TheChuckster · Tux's lil' helper Joined: 18 Aug 2003 Posts: 91

Couldn't syncing also be done via p2p by simply hosting a tar ball of the current portage tree on the network and extracting it on the user's system like you do during a GRP install?

sanity · n00b Joined: 19 May 2002 Posts: 39 Location: Iowa

No thanks. du -ch /usr/portage (after nuking distfiles) on my reiser4 box shows 92M. That's not really fair -- making a tarball shows 15M -- but it's still absolutely out of the question for a dialup user, and a ginormous waste of bandwidth for everyone else.

Now, let's suppose we have a p2p network where as long as 'emerge sync' is running (unless the person decides to become a permanent mirror), all files in their /usr/portage are available for download. Suppose we also make a global Manifest file (for all the sub-Manifest files, and for files which don't yet have a manifest), which includes the date and time. Now all the user has to download from an official mirror is the Manifest file, probably via http, even checking the timestamp with a HEAD request. Then they can check all their files against that Manifest, then connect to the p2p network, making available only the files that are up-to-date against that latest Manifest. Then, requests are made across the network for a file with a particular name and checksum -- starting by replacing old Manifests with new ones. Each new Manifest gives them more checksums, and so it goes.

I suspect that a system like this will be a lot less efficient per literal byte sent than just a Torrent of a tarball, but it will also ultimately save tons of bandwidth for daily updates of 5-10 files.

It would help if the same system could apply to distfiles -- as long as 'emerge sync' or a particular distfile download is running, all of /usr/portage and /usr/portage/distfiles are available. This is nice because while not everyone will always be downloading the same tarball, if people don't immediately clean them all out (after every emerge), there will be enough left over to share, and if I'm downloading foo.tbz2, someone else can download bar.tbz2 from me.