View previous topic :: View next topic |
Author |
Message |
epall Tux's lil' helper
Joined: 16 May 2004 Posts: 77 Location: Silicon Valley, CA, USA
|
Posted: Sun May 16, 2004 5:24 pm Post subject: |
|
|
It seems to me that trying to distribute the portage tree over p2p is fraught with perils. Trying to distribute compiles has even more problems. I think the problem that we should focus on most is distributing source .bz2/.gz files that live in /usr/portage/distfiles. |
|
Back to top |
|
|
sanity n00b
Joined: 19 May 2002 Posts: 39 Location: Iowa
|
Posted: Sun May 16, 2004 5:59 pm Post subject: |
|
|
the .bz2/etc files in distfiles have the same problems as the portage tree, only less so. What scales best to bittorrent (and not what I just described) is a huge tarball of everything. _________________ Every time you read a signature, God kills a kitten. |
|
Back to top |
|
|
epall Tux's lil' helper
Joined: 16 May 2004 Posts: 77 Location: Silicon Valley, CA, USA
|
Posted: Sun May 16, 2004 6:12 pm Post subject: |
|
|
exactly! That's why I don't think we should use bittorrent. A huge tarball would be a massive waste of bandwidth, because people would only need a few megs of it. BitTorrent would be well suited to a huge tarball, but that's not what we want. I think another protocol would be a better choice. |
|
Back to top |
|
|
sanity n00b
Joined: 19 May 2002 Posts: 39 Location: Iowa
|
Posted: Sun May 16, 2004 7:50 pm Post subject: |
|
|
I wasn't suggesting bittorrent directly. I don't know if there's another protocol that does what... well, what I was suggesting about three posts ago. Scroll up |
|
Back to top |
|
|
epall Tux's lil' helper
Joined: 16 May 2004 Posts: 77 Location: Silicon Valley, CA, USA
|
Posted: Sun May 23, 2004 5:46 pm Post subject: |
|
|
You're right, but like I said, trying to distribute the portage tree over p2p is not as easy as distfiles. I think that a good system would be using your original idea for just distfiles over a napster network. |
|
Back to top |
|
|
mlsfit138 Guru
Joined: 20 Sep 2003 Posts: 406 Location: Washington
|
Posted: Thu May 27, 2004 7:32 am Post subject: |
|
|
Gnunet would probably be more appropriate than freenet. It is written in c++, is searchable, and offers similar data integrity assurance.
Many of you think that it would be a mistake to tap into an existing p2p network, but if gentoo were to use something like gnunet, or freenet, it would lend that network a great deal of credibility, and support those project's goals of promoting free speech.
besides, if this service isn't enabled by default, so what if the network is taken down for legal reasons? (we would still have our mirrors) _________________ "Everytime you justify
another good in you dies"
-Converge, The Saddest Day, Petitioning the Empty Sky |
|
Back to top |
|
|
Squinky86 Retired Dev
Joined: 25 Mar 2003 Posts: 309 Location: Alabama, USA
|
Posted: Sat Sep 04, 2004 9:15 pm Post subject: |
|
|
fava, I've been trying to get in touch with you, but haven't been able to.
For anyone interested, I have some ideas. Please see bug #62331 and happy hacking all .
21:17 <@squinky86> !seen fava
21:17 <+GenBot> fava was last seen 13 weeks, 1 day, 18 hours, 26 minutes and 34 seconds ago, saying 'seemant: thanks' in #gentoo-dev. _________________ Me |
|
Back to top |
|
|
Lokheed Veteran
Joined: 12 Jul 2004 Posts: 1295 Location: /usr/src/linux
|
Posted: Sat Sep 04, 2004 10:03 pm Post subject: Re: Network requirements |
|
|
fava wrote: | As far as I know there are no p2p networks that allow this much central control, the trend in p2p networks is away from central controll because of liability issues. |
Appearently you havent heard the news. P2P was deemed legal in the North America. While sharing copyrighted material still remains illegal, using P2P software itself is completely legitimate.
In Canada file sharing itself using P2P was never illegal. I cant comment on the EUs stance, but from a North American stand point, P2P is not illegal in anyway. |
|
Back to top |
|
|
Squinky86 Retired Dev
Joined: 25 Mar 2003 Posts: 309 Location: Alabama, USA
|
Posted: Sat Sep 04, 2004 10:52 pm Post subject: |
|
|
Useage of this feature can only be used for gentoo distfiles in the form I am working on- you will be unable to use p2p-portage for anything illegal. It will be a separate package, so if it's illegal to use a p2p program in your country, just don't emerge it. _________________ Me |
|
Back to top |
|
|
nmcsween Guru
Joined: 12 Nov 2003 Posts: 381
|
Posted: Sun Sep 05, 2004 12:27 am Post subject: |
|
|
why not use http://konspire.sourceforge.net/ as a backend and modify it a bit to work with portage? _________________ Great Resources |
|
Back to top |
|
|
Squinky86 Retired Dev
Joined: 25 Mar 2003 Posts: 309 Location: Alabama, USA
|
Posted: Sun Sep 05, 2004 12:32 am Post subject: |
|
|
Because I'm unfamiliar with it . Actually, it looks like an interesting project, so if you could develop some code that would integrate it with portage, feel free to. Basically this is a proof-of-concept idea right now. _________________ Me |
|
Back to top |
|
|
trexmaster n00b
Joined: 13 Sep 2003 Posts: 17 Location: Paris, France
|
Posted: Wed Nov 24, 2004 9:58 am Post subject: Re: Network requirements |
|
|
[quote="Lokheed"] fava wrote: | I cant comment on the EUs stance, but from a North American stand point, P2P is not illegal in anyway. |
It's just the same over here in EU too. P2P in itself isn't illegal, but sharing of copyrighted material is. _________________ "Somewhere outside of and beyond our universe is an operating system, coded up over incalculable spans of time by some kind of hacker-demiurge." |
|
Back to top |
|
|
gentoo_lan l33t
Joined: 08 Sep 2004 Posts: 891 Location: Charles Town, WV
|
Posted: Wed Nov 24, 2004 10:35 am Post subject: |
|
|
progster wrote: | If it wasn't optional I think gentoo wouldn't run on any servers anymore, as traffic costs money. Also p2p networks are not that great for cable users (fast download but slow upload), for some reason the slow upload speed actually slows down downloads on p2p networks (especially bittorrent)
~progster |
I would agree this is definitely a problem. |
|
Back to top |
|
|
SPY_jmr1 n00b
Joined: 02 Jan 2004 Posts: 62
|
Posted: Wed Nov 24, 2004 8:57 pm Post subject: |
|
|
but, isn't the GPL a copyright?... and if... we can't share copyrighted things, ... then... we can't... but its... but they... but. . .
*KERBOOM!*
|
|
Back to top |
|
|
Squinky86 Retired Dev
Joined: 25 Mar 2003 Posts: 309 Location: Alabama, USA
|
Posted: Thu Nov 25, 2004 12:11 am Post subject: |
|
|
No files are being mirrored on the gentoo mirrors that would violate an unredistributable copyright. The same will apply to p2p distfile mirrorage. And it will be OPTIONAL: the way I see implementing it is by emerging the right package and setting the FETCH_COMMAND in /etc/make.conf. _________________ Me |
|
Back to top |
|
|
dasalvagg Apprentice
Joined: 26 Jun 2002 Posts: 183 Location: NY
|
Posted: Thu Feb 03, 2005 6:05 pm Post subject: |
|
|
i'm again interested in this area as I have newly found free time. Is anyone working on this currently? Is there proof of concept code? |
|
Back to top |
|
|
RedDawn Guru
Joined: 22 Sep 2003 Posts: 368 Location: Los Angeles, California
|
Posted: Thu Feb 03, 2005 10:10 pm Post subject: |
|
|
Any updates i would be happy to test this out. |
|
Back to top |
|
|
davidsb Tux's lil' helper
Joined: 08 Dec 2002 Posts: 146 Location: Lisbon, Portugal
|
Posted: Tue Feb 15, 2005 2:47 am Post subject: |
|
|
I don't have the skills nor the time to put a good effort on this project, but I've been following the interesting discussion and I'm for sure interested on testing or whatever else comes up that I migth help out. _________________ http://recycler.homelinux.org/~wolfshade/bootlegs/ |
|
Back to top |
|
|
travail101 n00b
Joined: 30 Apr 2004 Posts: 12 Location: United States of America
|
Posted: Sat Apr 02, 2005 10:26 pm Post subject: swarming in any shape or form |
|
|
after throwing the apparently old tired p2p idea into #gentoo on Freenode, seeing interested users, and then promptly being directed to the forums to look at many failed attempts at finding a worthy solution... I thought of a few more ideas.
My ideas may be a bit presumptuos, or forward, or half-baked, but try and here the concept more than the implementation as I lay it out
DISCLAIMER: sorry i didn't read every post in every thread on sharing bandwidth be it BT or some other implementation of p2ping. secondly this idea is being posted here first instead of in a new thread to avoid being flamed for saying the same thing over and over... if it's new enough I may make a new thread for it... once again with the not reading everything in the entire Forum to know if this is new or not. thirdly... this is not an idea to add existing technologies directly to Portage/Gentoo it's an idea to expand existing or create new technologies that would benefit the entire server world, and would then once created be implemented into gentoo if enough people can be convinced.
ok so now that that's out of the way here's the idea. (the word SwarmHash will be used as the concept of a file pointing to a hash and a tracker, similar to torrent files for BT)
the protocol can be new or adapted from existing technologies, it would need at least three things, a stand alone implementation, an FTP extension, and an HTTP extension... (perhaps plugins for existing FTP/HTTP server software like Apache) the last two needs are so that userless "SwarmHash" files can see FTP and HTTP mirrors as other users, this has more than one advantage, some of which I may not even have thought of, but a few I can see are A) no need to fallback on FTP or HTTP if whatever P2P protocol used in Portage has no users... instead the Gentoo Mirrors would be seen as seeds or other users. B) FTP and HTTP mirrors can all keep each other up-to-date via the same swarm, this is a protocol that if implemented could reach Epic proportions, with massive global swarming... and C) well there is no C yet... you tell me if you see any more advantages to this thought. the non HTTP/FTP implementation is of course just the p2p protocol itself and should exist standalone for the sake of keeping FTP/HTTP server plugin sizes down, and for use by people not trying to use it on a grand scale... it may be modified BT, or if may be the next generation of BT and replace it alltogether. the client software needs to be able to run 24/7 and needs to be able to be set to only use idle time, it needs bandwidth throttling, it certainly needs resume, it needs MD5 or other security Hashing built-in, and so on... if possible it should have a dynamice SwarmHash file that can change what files are part of the download.
the FTP/HTTP server integration is key, and is necessary... if this part is NOT possible the whole idea is almost useless...
how Gentoo would fit in if said software and implementation existed?
yes, you can put something like P2P_ENABLE="yes" in make.conf. this would then use the new protocol for downloading files...
the SwarmHash file should be able to be created dynamically with information from portage, even the portage DataBase, a emerge p2psync could exist if needed to update your Portage DB with the info needed by the new p2p client to make it's own SwarmHash file. there's a few ways this could go, and this is probably going to be the biggest issue to decide on if such an solution could be fabricated, how many files the SwarmHash file should manage: my idea is it should manage a complete Portage database, this is why it must be dynamic, if it's one file that lists all files available in portage then it not only needs to be a dynamic implementation but should be created to take advantage of existing databases, like the one portage uses. this i think is the best way to go, nobody wants 400 processes sharing 400 files from there /usr/portage/distfiles dir, one big file that hardly anyone would ever have a 100% of except servers would be ideal if it can be made to take advantage of external databases, like the users portage database. from the clients view it would look like a ton of users with 5-10% of the total SwarmHash file... the client would accept commands to only get a specific file from a simple command, emerge would create this command dynamically, when you type emerge foo with P2P_ENABLE="yes" in mak.conf emerge will send a command like "swarmhash_get -f foo-0.6.5.tar.gz --SwarmHash=portage_distfiles.s2h" this would tell the client to get the file foo-0.6.5.tar.gz from the SwarmHash file portage_distfiles.s2h (which is our Torrent type file with the information on all file in portage including the one we want to merge right now) the SwarmHash file could even be made to check files and version numbers from the server instead of the local database so emerge could simply send the command "swarmhash_get -f foo -v x86 --SwarmHash=portage_distfiles.s2h" to the client which would ask the trackers server what the latest available version is that matches the query, the -v command would be like --variable which could be compared to a server side list of variables, in this case it's the Gentoo ARCH types... so emerge could send ~x86 for the latest unstable version in portage. if this second scenario were implemented, then there would no longer be a need to resync your local portage database to get the latest version... this second implementation may require a lot more work than I think at this point, either way, the first implementation is just fine too.
so that's all I have right now... p2p with HTTP/FTP extensions so mirrors can swarm and users can swarm, cuts the need for volunteer seeders as the current mirrors would simply need to be convinced to use the new protocol extension for their server software... this should be desireable as the more servers using it, the less time it takes to keep themselves updated, certainly a lot of thought needs to go into this, but if it hasn't already been proposed elsewhere, I think it's a good idea... once again, if FTP and HTTP extensions are not possible for some reason, this whole idea is gonna take a big hit, but it would still be possible... just harder to get servers onboard to seed... anyway, what do you people think?
this is brainstorming time btw, not flaming time, constructive critisism only please. and once more... try to see past my implementation scenarios to the concept of the software as a whole before saying it won't work. |
|
Back to top |
|
|
walrus_55 Tux's lil' helper
Joined: 19 Jul 2003 Posts: 122 Location: Melbourne, Australia
|
Posted: Sat Apr 02, 2005 11:42 pm Post subject: |
|
|
What are these bandwith problems you talk about? i download from portage perfect at 160-170 kB/s every time i can never get that on any p2p software even bittorrent the highest ive ever
got is 80kb/s. I think as long as we have servers who are happy to be mirrors then its a much better option. The only thing that could be handy as far as i can see is the ability to stop a download
and resume it later, but hey my longest download is 20 mins anyway. |
|
Back to top |
|
|
LucaSpiller Apprentice
Joined: 10 Sep 2004 Posts: 188 Location: Censorship Land (aka England)
|
Posted: Tue Apr 05, 2005 6:36 pm Post subject: |
|
|
There aren't any major problems at the moment, but if Gentoo continues to grow then the demand on the servers will grow and there will be problems. Bandwidth isn't necessarily the problem, rsync causes a fairly high load on the server. Also the main problem with bandwidth is that it isn't free, somebody has to pay for it so wouldn't you rather put a hand in to lower their costs?
OT: The reason why you are probably getting slow speeds with BT is because you are a) downloading unpopular torrents or b) haven't set up port forwarding correctly. I have a 120kb/s connection and get that nearly everytime. _________________ :: Luca :: Mac Fag :: Original Macbook, 2g RAM :: Closet Linux user (seasoned with salt and pepper) :: C2D E4400 @ 2ghz, 4g RAM (only 3.2g detected under 64bit...), Nvidia 9600GSO :: |
|
Back to top |
|
|
varpath n00b
Joined: 02 Feb 2005 Posts: 18
|
Posted: Sun Aug 28, 2005 8:35 am Post subject: Spread portage distfiles using p2p?? |
|
|
Hi all,
I tried looking around, but it would appear that no one has suggested this idea before.
Instead of grabbing distfiles and emerge --update world from a central server, why cant we decentralise this task to all gentoo users.
I have an unlimited 256k dsl account, I definitely dont mind using my bandwidth to upload my distfiles to other gentoo users, especially when I'm away from the computer. (Another idea, how about using my idle CPU time to help other gentoo users reduce their compile time? )
I suppose that this would be a good way to show the world that p2p is not just good at sharing porn, warez and (for riaa) illegitimate mp3 and movies. (I'm grabbing smashing pumpkins mp3s from http://rspaa.niluje.net/data , btw)
Just a though. |
|
Back to top |
|
|
d4rkwingduck Apprentice
Joined: 11 Aug 2005 Posts: 220 Location: somwhere on this big blue earth
|
Posted: Sun Aug 28, 2005 9:05 am Post subject: |
|
|
Quote: | Another idea, how about using my idle CPU time to help other gentoo users reduce their compile time? | i did get an idea like that a while ago, on the downside tho dont u think that the latency would be trmendous? also security wise what if someone added some niffty extra code to one of the distfiles and distribute that via p2p wouldnt that create a bot net thus loosing the security aspects of linux ..... well those are just some of my thoughts i may be totally wrong _________________ http://www.di.fm/trance |
|
Back to top |
|
|
Shadow Skill Veteran
Joined: 04 Dec 2004 Posts: 1023
|
Posted: Sun Aug 28, 2005 9:29 am Post subject: |
|
|
None of these are bad ideas but security issues aside one has to wonder if concepts like folding@home could really be applied to compiling software since these are not jus mathematical computations but build operations that have various features enabled for a given cpu. It would be nice if it could be done however. _________________ Ware wa mutekinari.
Wa ga kage waza ni kanau mono nashi.
Wa ga ichigeki wa mutekinari.
"First there was nothing, so the lord gave us light. There was still nothing, but at least you could see it." |
|
Back to top |
|
|
aidanjt Veteran
Joined: 20 Feb 2005 Posts: 1118 Location: Rep. of Ireland
|
Posted: Sun Aug 28, 2005 9:57 am Post subject: |
|
|
Thats a pretty good idea actually, and we don't really need to worry about 'security', since we fetch the md5 file for packages from the offical rsync mirror. bittorrent transfers are pretty efficent when everyone chips in, think of how fast grabbing Gentoo LiveCD's with bittorrent is.. its a pretty good system, and everyone win wins all around.
not sure about internet distributed compilations thou :/ |
|
Back to top |
|
|
|