Trying to get distcc to work for cross-compiling

juliedeville · n00b Joined: 14 Oct 2024 Posts: 39

Hello,

I followed these instructions https://forums.gentoo.org/viewtopic-p-8844162.html#8844162 to setup distcc on two machines. I am not getting any useful errors or logs, and it does not appear to be compiling on both. Networking is setup, and I am able to ssh into the helper machine. I believe I had it working earlier, because I saw messages when merging a package, but it might have been when I was merging distcc itself.

I am not sure where I went wrong, any assistance is appreciated.

pingtoo · Posted: Fri Oct 25, 2024 4:14 pm Post subject:

If you wish help from other, don't let other guessing

You have yet sharing anything for us to start understand what is the problem. All we know is from your statement distcc is not working.

Please use the terminology in the linked post, show us what does it mean

juliedeville · n00b Joined: 14 Oct 2024 Posts: 39

Sorry, haha

One is a thinkpad t450s with an Intel i7-5600u, and one is an asus prebuilt desktop with an intel i7-7700

The target is the thinkpad

The profile on both machines is default/linux/amd64/23.0/hardened

I am still learning about tool chains

juliedeville · n00b Joined: 14 Oct 2024 Posts: 39

Okay, I added the line CC="distcc" to make.conf on the target, and before that, I tested to make sure distcc itself was working by writing a c program and compiling using distcc, and it worked, so it seems the issue is was just getting it to work with portage. Now, I get this error when running emerge

juliedeville · n00b Joined: 14 Oct 2024 Posts: 39

I think I have got it now. I had the hosts file on the target set to "192.166.7.79,cpp, lzo." I removed "cpp," and I am no longer getting the error. I was under the impression that portage would return messages specifically for distcc at the beginning of an emerge process, and when I run "sudo distccmon-text 1" on the helper, it wasn't displaying anything in the output.

I think I have got it working now, because I can hear it compiling!

NeddySeagoon · Posted: Fri Oct 25, 2024 6:17 pm Post subject:

distccmon is run on the system needing help, not the helper.

distcc-pump has been broken for years. It needs the same (identical) include files on the helpers as the system needing help.
That allows the helpers local files to be used which saves network bandwidth.

Portage no longer supports pump mode.

Your topic title is misleading as you are not cross compiling.
Cross compiling would be using your i7 to build arm code for say, a raspberry pi.
That works too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

juliedeville · n00b Joined: 14 Oct 2024 Posts: 39

Oh I see. I misunderstood the term. Also sorry, I meant to type 'target' and not 'helper' regarding distccmon. Currently I am having an issue where the helper machine will randomly stop compiling..

NeddySeagoon · Posted: Sat Oct 26, 2024 4:06 pm Post subject:

You need distcc installed on both systems as it's in two part.
distccd, which runs on the helpers and listens for cries for help
By default it will start one distccd process for each logical core on the helper.

distccd need not be run on the system needing help.
local compilation will be used anyway when there are problems.
Hence, localhost need not be included in the list of helpers. If it's there at all, it should be last.

When you do not give a maximum nuber of concurrent jobs to distribute to helpers, the default is 4, as if you had used /4
If a helper can do more than that, by all means increase it.
But! MAKEOPS on the weaker system may need to pe increased too.
This is a double edged sword as not everything can be distributed and you don't want to swamp the weaker sysem when it has to do all the work.

You say that the helper randomly stops helping.
Does that mean what it says or does distribution stop?
There should be error messages.

Only C and C++ compile jobs can be distributed.
Preprocessing and linking will still be carried out locally.
All other code will still be built locallaly to ... e.g. rust.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

juliedeville · n00b Joined: 14 Oct 2024 Posts: 39

eccerr0r · Posted: Sun Oct 27, 2024 4:59 pm Post subject:

It is difficult to tell what the optimal number of jobs is -- it depends on the package and also within the package itself (which is difficult to change, so you have to carefully choose one that's not optimal for half of the run.)

Distcc is used only under certain circumstances and is the reason why it's not used often. The things to check:
- distcc is installed and FEATURES=distcc (which you got, but it needs to be checked!)
- networking is working (which you got, but needs to be checked!). ipv6 setup is annoying.
- Only works for gcc and clang, and cross compile works. However, things like xgcc, rust, haskell, perl, python, llvm-tblgen, ld-bfd, etc. don't get distributed.
- target machine must have same architecture cross or native gcc (arm vs i686 vs amd64 vs... mingw!). Clang is a bit different as it seems it has cross compile support built-in.
- target machine must have the same version of gcc enabled (make sure with gcc-config!), or same version of clang installed
- target machine with similar version (like gcc-13 vs gcc-11) may or may not be used depending on how different the preprocessed code is. I found that gcc about 50% of the time will still use mismatched gcc version. This percentage drops to near 0% for mismatched clang.

I hacked up a patch for distcc (distccmon-text, distccmon-gui) so that it works a bit better despite having pid namespaces enabled for portage (default). It's a dirty hack but "works for me" ... might have to post it somewhere if I haven't already, else you can disable pid namespaces (FEATURES=-pid-sandbox I think, and imho my dirty hack is cleaner than using -pid-sandbox but ymmv).
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Fri Nov 01, 2024 10:02 am Post subject:

There is no single optimal MAKEOPTS.

e.g. gcc is written in C++ but it mostly cannot use distcc due to the way it builds
Pass 1 builds C and C++ with any random compiler on the system.
Pass 2 does the same thing but uses the compiler built in pass 1. That only exists on the local host.
Pass 3 builds C and C++ using the compiler from pass 2, then compares the outputs of pass 2 and pass 3.
If all is well, the other languages are built and its all installed.

It is possible to set MAKEOPTS on a per package basis, or you could run your own binhost by building in a chroot on one of your helpers.
That would not use distcc and the more powerful box would lend it's power to everything.
There is a wiki page. I would normally post a link but I'm on my phone right now.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

eccerr0r · Posted: Sun Dec 01, 2024 5:41 pm Post subject:

Rust is also one of those annoying packages like gcc

Phase 0: Build local copy of LLVM using any C compiler. This can be distributed.
Phase 1: build Rust with (originally portage-downloaded) downloaded copy of rust binary, or (currently) rust-bin. This technically can be distributed alas there is no distrust today, so it's run local.
Phase 2: Build rust with the newly compiled rust. Just like gcc, this necessarily is local and cannot be distributed.

There is no MAKEOPTS that will optimize this.

One of these days perhaps cluster computing would be part of portage and if the machine needing compilation help's filesystems are NFS mounted, then it would be possible for the helper to run whatever command is needed...alas it would open another can of worms.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?