distcc: webserver to private hosts?

[Jiokah]

Hey,

I have a web server somewhere on the internet, a home server (acting as a router among other things), and 2 hosts under the home server (with private IPs). They're all running Gentoo. I would like the web server to be able to send compiling jobs (distcc) to the two private hosts. How?

Right now I use distcc to share compiling jobs between my web and home server over SSH with ssh-agent and keychain, so I've managed to get a pretty good idea of what I'm doing (with the above problem aside, heh).

Thanks!

-Matthew

[merlijn]

First off, setup distccd on both the home servers behind the router. Now u should edit the port it listens on and set it to something high for example in /etc/conf.d/distccd. Now forward ports from your router to the different ports on different hosts.

Following the steps provided at http://www.gentoo.org/doc/en/distcc.xml, u can now do this on the webserver: /usr/bin/distcc-config --set-hosts "localhost x.x.x.x:AAAAA x.x.x.x:BBBBB". Where x.x.x.x is your home ip and AAAAA/BBBBB are the ports linked to the different machines for distcc.

Please bear in mind that distcc takes a fairly large amount of datatraffic, not only will it cost you money for your webserver, you will also need to have sufficient speed on your homelink to even benefit from sharing the compiles.

Hope this helps,

[Jiokah]

Hey, thanks for the reply. You have a good idea there, but I'd prefer to use SSH as my webserver is a production server. And I'd prefer not to open more ports than nessesary. Is there a way to tunnel SSH through my router to multiple hosts while still being able to SSH to the router itself?

As per your concern about data traffic, SSH has even more overhead (25% for the processor!). But I'm not doing this for speed. My server doesn't have sufficient memory to compile larger programs like MySQL. My homelink is very fast, and I have way more than enough bandwidth to cover this. Plus I have quite a few projects in the near future and they'll all be hosted on my server, so I will be upgrading soon (which will give me more ram). You have a notable concern none-the-less.

Thanks for the help so far!

[merlijn]

This wouldn't be my way to go for this problem, as you said b/c of the ssh overhead (which is definately less than 25%, but still reasonable). But ofcourse it can be done. I'm assuming u have ssh access to the router only, and u have a setup like this:

router - 192.168.1.1 (ssh open on WAN ip)
distcc pc1 - 192.168.1.2 (distcc running at any port, default will do just fine)
distcc pc2 - 192.168.1.3 (distcc running at any port, default will do just fine)

Assuming both distccd's are running on port 3632 you can do this on your webserver: `ssh -L 44444:192.168.1.2:3632 -L 55555:192.168.1.3:3632 you@your.home.ip` followed by: `/usr/bin/distcc-config --set-hosts "localhost:44444 localhost:55555"`

I'm not 100% sure u can use -L twice in one ssh command, otherwise just open 2 ssh sessions like that

Cheers,

[merlijn]

Come to think of it, if you want to compile the stuff on 2 remote hosts you will need to set something like MAKEOPTS="-j3" in /etc/make.conf assuming the cpu's at your home are single cores. This will take MORE ram than compiling locally with just -j1 or -j2. You might want to look at crossdev (http://gentoo-wiki.com/HOWTO_Cross_Compile) to compile such programs without involving the webserver.

But then again, if u have lots of swap to make up for the limited ram, it might still be faster due to the heads not constantly having to jump between swap and mounted partition.

[Jiokah]

You have an interesting idea there, I just glanced at it so I'm not quite sure what it's supposed to do, but when I get home later tonight I'll toy with it and see what it does. SSH isn't nessesary behind the router as the router can protect the network, I just didn't know you can use distcc half with SSH and half with TCP.

Now regarding the -j# option, I have this set to 4 (two compiling jobs for each host at a time), but you mentioned something about this taking more ram on the SERVER even though it's not compiling at all? Even more ram than giving the server one or two compiling jobs at a time? I'm wondering, why is that?? If this is true, I'm in trouble because my web server does not have any swap (if you're wondering how powerful it is, it's a link-2 VPS from VPSlink.com). If you're referring to more ram per compile job on a host than on the server, then this is fine as my two private hosts have a LOT of ram and my VPS is just plain incapable of compiling large applications (though, I managed to sqeeze PHP out of it using a couple of tricks, heh).

Thanks again

[desultory]

[Jiokah]

[merlijn]

I'm not too sure about why it takes the amount of ram, but if i recall correctly the -j# tells the machine to always have #-1 compiles ready to be executed, which will be stored in ram.

With the SSH option, only traffic over the internet will be encrypted using SSH, the traffic on your home network will just be plain.

I don't know the proper way of building it on your homeboxes and then moving it over, but crossdev might be the answer even though u are not compiling for a different arch (or maybe u are).

http://www.gentoo.org/proj/en/base/embedded/cross-development.xml

Cheers,

[desultory]

[Jiokah]

Hey guys,

Thanks for all the help, but...

I gave up and just paid the extra money to upgrade the server (had to do that in a couple weeks any ways). But I did learn a lot here, thanks for all the explanations and wisdom