Distcc How-To

Vorlon · Posted: Sat Oct 31, 2020 11:19 pm Post subject: Distcc How-To

Distcc is a fantastic idea: Use the power of multiple computers to speed up compiling programs on a target computer. Gentoo makes this process very easy. Simply install distcc on the "target" computer and the "helper" computer(s), make a few changes to "/etc/portage/make.conf", and go.

There is one HUGE caveat, however. The software environment (processor type and testing-level) must be the same on both the target and the helper, or this system gets very complicated very fast. But there is an easy way to make it work: use a virtual machine for the helper. Here's how.

Definitions - To Keep us Straight

There seems to be a lack of standard naming conventions within the cross-compiling documentation. To keep things straight, here are the definitions used in this How-To, which I think generally follow the conventions used in other Gentoo documents.

Build / emerge / compile - In this How-To, all of these are considered to be synonymous (I know, they're really not, but that's being too pedantic for the purposes of this How-To.)

distcc - "Distributed Cross Compiling" A method of using multiple computers to build programs for another computer, which may have a different architecture. For example, using a multi-core AMD Ryzen system to build programs for a Raspberry Pi. (Please note that bridging different CPUs with fundamentally different architectures is beyond the scope of this How-To. This document assumes that both the Target and Helper are the same basic x86 type of CPU.)

Host - Any computer that compiles programs for another. This is often a synonym for "Helper". Please note that a computer can be a host for itself and other computers at the same time.

Target - The computer that needs the help. The Target is the computer that all the other computers are building software for.

Helper - The computer which compiles programs for another computer. There can be multiple helpers working at the same time to distribute the compiling load. In this How-To, all the Helpers will be VirtualBox virtual machines so that they can EXACTLY match the Target systems in terms of CPU, Tool Chain, and testing/stability (i.e.: "~AMD64" or "AMD64").

Tool Chain - The programs used to compile source code. Typically, this will consist of gcc, libc, binutils, and the kernel. If the required Tool Chain on the Target computer does not match the active Tool Chain on the Helper, the compiled programs may not function, or may function erratically. This incompatibility is a major stumbling block for distributed cross compilation.

Step 1 - Build a Helper virtual machine in VirtualBox

a) Build the Helper virtual machine normally as you would any typical Gentoo system. You do not need to install X or any GUI systems on the Helper. You will need to either set a static IP address for the Helper, or record the assigned IP address for use later.
b) Be sure to set CFLAGS to match the Target system. Set "--march=" to the correct CPU architecture as the Target. Do NOT use "--march=native" on the Helper because the Helper would then build the same CPU architecture as the VirtualBox host instead of the Target system.

Step 2 - Install distcc on all the computers (the Target and the VirtualBox Helpers)

a) Run "emerge distcc"

Step 3 - Enable distccd service on the Helper(s)

a) Edit /etc/conf.d/distccd to enable the subnet where the Target resides so that the Helper will compile for it. The "allow" variable needs to be set. For example, DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24" will enable distccd to run for any Target in the 192.168.1.x subnet.
b) Run "rc-update add distccd default" to enable the service to start automatically at boot.
c) Run "/etc/init.d/distccd start" to start the service now.

Step 4 - Update make.conf on the Target

a) Update the MAKEOPTS variable to specify the number of helper cores and the number of local cores.
a1) -j now signifies the total number of threads to use. It is set to the total number of threads for all the helpers (including the target) + 1. For example, if the local computer has 4 cores and there are 2 helpers with 4 cores each, then the parameter would be "-j13"
a2) -l (lowercase "L") is added to indicate the number of threads on the local machine + 1. For our previous example, the parameter is "-l5".
a3) Combine the two parameters in the MAKEOPTS variable. In our example, the line should be MAKEOPTS="j13 -l5"
b) Add the "FEATURES" option by adding a line that says FEATURES="distcc".

Step 5 - Set distcc variables on the Target

a) Run the program distcc-config to set the Helper system(s) using the "--set-hosts" command. Be sure to append "cpp, lzo" to each Helper host. You can add multiple URLs. For example, if the Helpers are at 192.168.1.23 & 192.168.1.54, the command will be distcc-config --set-hosts "192.168.1.23,cpp, lzo 192.168.1.54,cpp, lzo". This creates the file "/etc/distcc/hosts", which lists all the Helper hosts. You can also create this file manually.

Step 6 - Emerge on the Target

a) Emerge packages on the Target as you normally would. You will see distcc messages on the Target as the packages emerge.

Miscellaneous Notes:

• Sometimes distcc induces errors in the build process and the emerge will fail. I have not been able to determine why some packages build and some don't. Instead, I simply emerged as many as possible using distcc, then commented out the "FEATURES" variable and emerged the rest.
• Based on my somewhat limited reading of anecdotal evidence, distributed computing is still something of a black art. It should work, but often doesn't. At least one person has told me they eventually found distcc more trouble than it is worth. Depending on the Target machine, the help may or may not be worth the extra effort. Using distcc, I was able to install Gentoo on an old Pentium III with only 512K RAM. Distcc was critical to that since the Pentium III is actually waaaaay too old and feeble, and 512K RAM is waaaaay too tiny to build Gentoo.
• I recommend making the Helper a VirtualBox virtual machine so you can ensure the Target and Helper have identical settings and Tool Chains. But the Helper can be anything as long as these items match the Target.
• There are ways of building a unique Tool Chain on the Helper without going through all the VirtualBox stuff, but I found creating a new VirtualBox machine a lot easier than trying to figure out the confusing Wiki info.
• There are 2 different monitoring programs mentioned in the Gentoo wikis, but I have never been able to get them to work. Instead, I simply ran htop on the Helpers and watched as the distcc program executed on them.
• Distcc is potentially a security risk because it allows remote machines to execute programs on the Helper. You need to be careful to restrict which computers can use the Helper in the configuration file /etc/conf.d/distccd.
_________________
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation

NeddySeagoon · Posted: Sun Nov 01, 2020 10:03 am Post subject:

Moved from Installing Gentoo to Documentation, Tips & Tricks.

Its one of these.

distcc just works, ever across architectures. Its key to have identical versions of gcc everywhere.
When jobs are distributed, distcc tells exactly what must be done but not with what compiler version.

The monitoring programs are run on the Target.
DISTCC_DIR must be defined, as that's where they look to see what disstcc is doing.

Not all phases of a build can be distributed, so lots of nothing going on is expected too.
e.g. the target has to do its own preprocessing and linking.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Hu · Administrator Joined: 06 Mar 2007 Posts: 22593

Lemon-Lime · n00b Joined: 27 Apr 2023 Posts: 58

pingtoo · Posted: Fri Sep 01, 2023 2:20 pm Post subject:

Hu · Administrator Joined: 06 Mar 2007 Posts: 22593

Lemon-Lime · n00b Joined: 27 Apr 2023 Posts: 58

Noted Hu! Will give it a try and I'll update this post with my findings.
Thank you so much for your help!
_________________
Crazy frog is the artist, not the song

juliedeville · n00b Joined: 14 Oct 2024 Posts: 27

I followed the instructions, but can't get it working. I think I had it earlier, because I remember seeing distcc messages when merging a package; idk where I went wrong.

Hu · Administrator Joined: 06 Mar 2007 Posts: 22593

juliedeville · n00b Joined: 14 Oct 2024 Posts: 27

Nvm, I got it. Good guide! In /etc/distcc/hosts make sure to only enable "lzo" because "cpp" activates pump mode, which is deprecated.

Ralphred · l33t Joined: 31 Dec 2013 Posts: 635

Bob P · Advocate Joined: 20 Oct 2004 Posts: 3374 Location: USA

It's been a long time since I ran a distcc compiling farm (is anyone here old enough to remember Stage 1/3 Installs or the Jackass! Project?) It had to be 18-20 years ago...

One of the things that I really liked about the GUI version of the distcc monitor was how much real time status information it gave me regarding the progress of a full system recompile that was distributed across a dozen machines. It was a royal PITA to get distcc up and running but once I finally got it working I found the bird's eye view provided by the GUI monitor to be particularly fun to watch during a long system emerge.