distcc - it works (quick but dirty...)

[de4d]

if anybody wondered howto use distcc during an emerge process:
(warning: this method might be ridiculous - better ideas appreciated :)

[phong]

Ok, I've been doing a lot of playing with distcc over the past week or two, and I think I can give some tips that will be slightly quicker and a bit less dirty. I planed on writing up a whole HOWTO when I determine if/how distcc can be used for the bootstrap (I've already gotten it to work with emerge system, the kernel compile, most of kde and several misc. things). In the mean time, here's a quickie.

First off, distcc has no built in security mechanisms. Before running the distcc daemon on any volunteer machines, it's critical that you block off tcp port 4200 incoming at your firewall otherwise anybody can run whatever they want with the permissions of the user running distccd (which need not and should not be root BTW).

Second step, take a quick look at the distcc web page and emerge distcc on all the involved machines.

Add the following lines to your /etc/make.conf

[de4d]

tanx a lot!
ive been trying similar stuff before i ended up replacing make by a script - just didnt get it to work ...
should have read some more documentation about make ;))
_________________
void main(){fork();main();}

[zen_guerrilla]

[phong]

Hmmm, does distccd --help give any output? It should give some output to stderr if run with --verbose though... Can you try running it with --log-file=test.log or something to see if anything appears in the log file?
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer

[zen_guerrilla]

The log simply told me not to run distcc as root, oops Now everything works gr8. I' m finally emerging gnome2
phong : Thanx for the great tip on distcc. Now I'm gonna finish my gentoo installation at work.

.:: zen ::.

[verwilst]

I have the following in /etc/make.conf:


CC="distcc"
CXX="distcc g++"
MAKEOPTS="-j2"
DISTCC_HOSTS="gateway localhost"
DISTCC_VERBOSE=1


and i get this when trying to compile an app with emerge:

checking whether make sets ${MAKE}... yes
checking for i586-pc-linux-gnu-gcc... distcc
checking for C compiler default output... configure: error: C compiler cannot create executables

!!! ERROR: The ebuild did not complete successfully.
!!! Function src_compile, Line 3, Exitcode 77
!!! (no error message)

Any suggestions?
_________________
Bart Verwilst
Gentoo Linux Developer
Gent, Belgium

[rac]

Anything in config.log that might give us more detail?
_________________
For every higher wall, there is a taller ladder

[phong]

I assume you built distcc before putting that stuff in your make.conf... Can you try just compiling a single .c file with distcc, i.e. "distcc hello_world.c"? Before you do that, export DISTCC_VERBOSE="1" so it will give you more info.
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer

[gentuse]

I would recommend setting CC="distcc gcc" and CXX="distcc g++" to overcome the bug described here: http://lists.samba.org/pipermail/distcc/2002q4/000308.html

[474]

Thanks so much for this thread, in particular the recommendations for make.conf. It works like a charm, although some builds fail or don't take advantage of distcc in the desired manner (I'm keeping a log of these because I'm setting up a new system).

However, I think I just discovered something rather worrying. I was proceeding to emerge app-misc/endeavour on a pretty clean system. I noticed this message when compilation of the tiff library ebuild dependency began:

[rac]

pahud's question has gone to distcc problem. Please don't post support questions to threads in Documentation, Tips & Tricks.
_________________
For every higher wall, there is a taller ladder

[yokem55]

Just how scaleable is distcc? Could you say have a 100 machines on a network all helping make a single build that could then be applied to all the systems? I mean, if this could cut a KDE compile time down to minutes when distributed so broadly, I wouldn't be suprised to see corporate sysadmins embracing Gentoo since it would be able to produce full binary packages just as fast as they could download them from the net.

[puddpunk]

yep, sure!

just on your emerger be sure to set the config up correctly with your 100 slaves, then emerge with the --buildpkg command, and it will build a package which can be distributed over all 100 machines (you'll have to do that yourself though)

[zojas]

I use distcc in combination with ccache. that way if the ebuild fails, once you fix it there's not as much work to do.

make sure you have distcc and ccache installed. I put /usr/bin/ccache as the first directory in root's path.

then in your make.conf file: (almost the same as the code above):

[mglauche]

KDE does not seem to like distcc, so i doubt the 100 machines will help compiling it ..

I think for compiling on such many machines distcc would have to have finer controll over the make process, in order to be still efficient.

I usually get only 30-40% speed improvement by compiling on 3-4 machines, with the distant once being quite a bit faster than the local machine ...

[zojas]

I posted some timings using ccache+distcc on emerging mozilla. There are three cases: with distcc+cold ccache, distcc+warm ccache, and with no distcc/no ccache. the times come out as you expect.

http://desertsol.com/~kevin/ccache_distcc.html

[fyerk]

[proxy]

I have noticed an issue which was preventing distcc from working at all:

in the newer ebuilds for gcc, there are two new lines in /etc/env.d/05gcc

they are:

[nitro322]

[zojas]

the client tells the distccd what flags to use. I use distcc routinely with a pentium II as a client and an athlon as a server and it works fine. I've also used a 90MHz pentium as a client and that worked too.

[mbp]

I doubt if distcc would scale to 100 machines, for a few reasons:

- Many Makefiles are written in such a way that they can't schedule 100 jobs at once. For example, they may not parallelize across directories, and each directory may have <100 files.

- The client may run out of steam trying to run the preprocessor and distcc client for 100 machines. The client-side overhead is a few percent of the total work, which suggests an upper limit of a few tens of machines.

- The network may saturate trying to pass source and objects around.

Aside from that there is no in-principle reason why distcc couldn't scale up to that number. It would however be interesting to try. Unfortunately I don't have 100 Linux machines to try on.

[Decibels]

I am having a bit of confusion on the CC & CXX flages. I see most people include them in their make.conf.

Then you also see stuff like this:

[nephros]

I have four machines at home, all running distccd, but not all are up all the time.

I use the following script therefore, like this:

[nacs]

After following the instructions in the posts above, I found that distcc wasn't being used. However adding the following line to make.conf fixed it immediately: