Distcc over the net - Ditributed Computing

bleakcabal

I had this idea ( which I doubt I am the first to have ) during a class, what if there was an application that used Distcc in the manner of Seti At Home. Using idle cpu clock cycles which would be donated to those conected to a large P2P like network of other Gentoo users.

To insure that poeple are not always taking but also giving a kind of quota system could be used so that you cannot abuse this program too much.

Another option would allow poeple to send bandwidth limit so as not to create a large internet bill because of the constant download/upload.

Other than that, this could be a great idea to speed up compilation times if this could implemented ( I don't know if it could ).

What do you all think ?

ckdake · l33t Joined: 10 Apr 2003 Posts: 889 Location: Atlanta, GA

That would be pretty awesome to pull off, but probably not worth it. The problem is that compilation isn't massively parallel so it wouldn't scale that well. Many stages in the compilation depend on the successful completion of previous stages, so not much would be gained by massively distributing the process. Now for making stage tarballs, or binary builds with optimations for amd/p4/ppc/etc, it would help a bit more.

I have 2 machines running gentoo, and distcc on both of them. When I am emerging on the faster one, it ends up waiting on the slower one, not making effective use of its own cpu.

I think there was an idea earlier on the forums to make a distributed network of binary builds avaliable. So you would register with this service, and whenever you compiled something the service would remember the use variables and optimization settings that affected the build, and the next time someone else tried to emerge with the same applicable settings, the would just download the object file from you.

distcc is great and it gets a lot of use in my house, but i dont think that 50 machines would be any faster than 3 or 4 for most compilation jobs.
_________________
http://ckdake.com/

Beekster · Apprentice Joined: 26 Nov 2002 Posts: 268 Location: Sydney

Another problem is the typical imbalance between upload speeds and download speeds with cable or DSL.

From using distcc at home between two roughly equivalent machines, I can see that the transmission of source code is much larger than the returned compiled code.

The code distributed for compiling on the other machine typically ranges from a few hundred k to a meg at a time (per job). On 100 Mbit local connection, no problem. If upload speed reduced the speed of sending code to say 25 to 50 kB/sec, then just distributing one job to the other machine could take more than 20 secs.

The other issue is trust. Would you want portions of, for example, ssh compiled on a machine that you don't trust?

krusty_ar · Posted: Thu Nov 27, 2003 12:47 am Post subject:

This has been discused several times in these forums, the idea has several technical problems (some mentioned here) and ALOT of security problems ATM.
_________________
I am Beta, don't expect correct behaviour from me.
Take part of the adopt an unaswered post initiative

bleakcabal · Posted: Thu Nov 27, 2003 1:48 am Post subject:

Well it seems it's just another stupid idea from a stupid person(tm)

TheCoop · Posted: Thu Nov 27, 2003 7:55 am Post subject:

its not a stupid idea. Its a good idea in theory, its just implementing it will be a lot of hard work thats just not worth it
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

https://forums.gentoo.org/viewtopic.php?t=145494
_________________
Quis separabit? Quo animo?