Gentoo Distcc Live-Cluster-CD?

bastibasti · Guru Joined: 27 Nov 2006 Posts: 581

Iam thinking about building a Gentoo-based Distcc LiveCD for my own use.

While building I thought about building a DistCC distri, that would use VPN to connect to a server, and build a distcc cluster (like boinc but distcc)
If this is used, we would all gain compilation speed. But does distcc check connection speed? Has someone tried it over internet?

drutten · Posted: Mon Oct 22, 2007 7:50 am Post subject:

I haven't tried it, but i heard that distcc is really slow over internet and generally not worth the effort.

ToeiRei · Veteran Joined: 03 Jan 2005 Posts: 1191 Location: Austria

distcc over inet isn't a good idea, I think. But I could think of using your idea in a LAN environment. The bad thing about it is, that the compile cluster has to be static

_________________
Please stand by - The mailer daemon is busy burning your messages in hell...

Darth Marley · Tux's lil' helper Joined: 25 Jan 2007 Posts: 107

I would love to see a good article about how to roll your own version of this.

Both as a livecd and a bootable USB drive, such a tool wouldbe very helpful when compiling for legacy boxes.

It seems that some of the issues for this would be using the same gcc and libc versions across all the compilers. Those running from the livecd would of course be the same.

Elshar · n00b Joined: 10 Sep 2004 Posts: 15

I was just thinking about this. Was actually hoping someone had already made a livecd with distcc already on it. I suppose you could get around the problem of static distcc hosts by just using a script to add all hosts within your netblock to the list. Really wouldn't be too hard, and distcc ignores down hosts anyways (Although it is spammy during compiles).

The hardest part would be overcoming a difference in gcc versions between the 'master' host and what's on the livecd. Although that too could be overcome with a well-written script or two. You could just have multiple versions loaded, but that wouldn't solve the problem of newer releases (And you'd still need a script to path to the correct gcc bins).

Maybe a better solution would be to net boot the machines instead of using a livecd. Has a couple advantages, mostly the ability to update/add gcc versions to the remote clients on the fly, and saves on media costs. Not that CDRs cost much these days, but if you gotta burn 10 CD's every time you change something it'd add up quick.

I've gotten FreeBSD to net boot fine, has anyone else had experience doing that with Gentoo? Admittedly I haven't even searched yet, so it's likely already in the docs section. If not, I guess that's my next project.

jdepaola · Posted: Tue Jan 08, 2008 4:37 pm Post subject:

I really like the netboot idea, and I've definitely been a fan of PXE vs. Most other methods to manage a clustered environment.

With PXE you could just change the initrd per MAC address allowing you to give static'd IPs to the boxes. In the DISTCC configs, they all could be the same, allowing so many connections / same subnet and actually is the least of my concerns.

I did get gentoo to boot over PXE, which entailed some hurdles in itself, however the only way to truly do it is to mount it over a network file system. Which lead me to even more problems, as with NFS it took about 20-30 minutes to boot over Gigabit Ethernet, on Dual Xeon Nodes. Needless to say this was extremely frustrating and the only way I've been able to think about over coming this hurdle is tarball the install and stream the untar'ing into RAM onto each box. But that method is way way less preferred as it complicates management unnecessarily.

This idea, is actually what lead me to start working on a higher performance network file system.

DISTCC, over the Inet is possible and it doesn't require a VPN, however I did use one for obvious security implications. It really isn't worth your time in doing unless the box is pre-pentiumIII on most inet connections, the latency is extremely poor under these conditions. However if you have two Comast Highspeed connections like we had, we were able to make use of DISTCC up through Dual Xeon equipment. Our ping times over the VPN were less than 20ms. We also have enough boxes to to do a MAKEOPTS="-j 192".

I will also vouch for USB booting as it boots soooo much faster than CDs, and technically easier to manage, as you can script the creation of one and sort of assembly line the in house manufacturing of gentoo'ifying them. Their are 2 draw backs.
1) support of usb booting on legacy boxes (kinda the whole point here)
2) cost, (however I found 1gb flash drives w/ instant rebate promotions for $6/each at OfficeMax, only cool part with them was they let me back-order as many as I wanted because they didn't have the quantity I wanted in-store. No surprise it was a good deal.) I think I picked up 10 of them to try this little idear out.

I'm thinking that we will have to accumulate some community support to collaborate on this concept. Hopefully it won't require a new distro...! (which I also thought of starting for this very reason)

Let me know if there are any interested parties. Maybe we can make this happen together.
_________________
Regards,

J. De Paola

~ Evil is the root of all that which is human, and discipline is what truly differentiates from that which is Good...

~ The only difference between evil men and holy men is that evil men do it for their own glory...

Elshar · n00b Joined: 10 Sep 2004 Posts: 15

I wonder why it was taking so long to netboot? I was doing it with PXE with FreeBSD, using NFS and it booted at normal speed. Everything had gigabit cards in a gigabit switch, and I was only doing 7 boxes off the main one. But I don't see why it would take Gentoo any longer.

Must be nice to do a -j 192. I wonder how long it'd take to go through the whole series of deps to install say KDE or Gnome with that setup? mmm.

jdepaola · Posted: Wed Jan 09, 2008 9:52 pm Post subject:

The main reason is that it is that NFS is extremely inefficient with small files and even worse with 0 byte files.

If you try mounting the /dev/ or /proc/ over NFS it just doesn't seem to like it.

Did you do this? How is your setup? I'm curious to know how you've managed to get it to boot quickly.
_________________
Regards,

J. De Paola

~ Evil is the root of all that which is human, and discipline is what truly differentiates from that which is Good...

~ The only difference between evil men and holy men is that evil men do it for their own glory...

bastibasti · Guru Joined: 27 Nov 2006 Posts: 581

I've just finished my CD setup. Its a complete Gentoo running from disc, hdd, usb or RAM (need at least 1gb)
It contains:
2.6.24-gentoo-r8
gcc-4.1.2
gcc-3.3.6-r1
latest distcc,
openrc

I use linux-live scripts (slax) they are very neat and dead easy to use (even for me) :wink:

i also installed some more apps that I found quite useful:
mc, cryptsetup, X, twm, lynx, firefox.

Unfortunately I dont have any webspace but if anyone is interested... let me know

jdepaola · Posted: Fri May 30, 2008 4:35 pm Post subject:

Did you also enable cross compiling? That way it is a universal distcc liveCD.

I'm interested in hosting it also.
_________________
Regards,

J. De Paola

~ Evil is the root of all that which is human, and discipline is what truly differentiates from that which is Good...

~ The only difference between evil men and holy men is that evil men do it for their own glory...

bastibasti · Guru Joined: 27 Nov 2006 Posts: 581

If you tell me how to do that, I can do it almost instantaneously.

richard.scott · Posted: Tue Jun 03, 2008 10:29 am Post subject:

nativemad · Posted: Tue Jun 03, 2008 4:51 pm Post subject:

I have gentoo over pxe with nfs... It's faster over 100Mbit than my laptop to boot! :wink:

Ok, i must admit that i use a squashfs and no tiny files... But its actually the same squash as i use with usb and cd!
I just mount the nfs-share (or cd/usb), mount the squash (and unionfs over it with a ramdrive) and then pivot root! (everything is directly from the squash over nfs, changes are going to ram)

So if you have a working livecd/usb, its just a matter of pxe-bootloader-config and linuxrc to have it also on pxe...

Cheers!

_________________
Power to the people!