emerge -1 clang takes 14 hours? [SOLVED!!]

jesnow · l33t Joined: 26 Apr 2006 Posts: 888

So I don't think 14 hours is normal to emerge clang. What happens is it emerges fine for, it distributes its jobs all just fine, then about 20 minutes in it settles down to a single thread on localhost that takes way longer than normal to compile the rest. I tried a bunch of combinations of ccache, distcc, MAKEOPTS and EMERGE_DEFAULT_OPTS, and those had an effect, but it always settled down to hammering a single processor.

emerge --info says:

alamahant · Advocate Joined: 23 Mar 2019 Posts: 3929

Plz see
https://wiki.gentoo.org/wiki/Clang#Bootstrapping_the_Clang_toolchain
Initially you build them with gcc then with clang via /etc/portage/env/ and /etc/portage/package.env
In my case

Hu · Administrator Joined: 06 Mar 2007 Posts: 22877

I don't think OP is trying to bootstrap clang with itself. In practice, if you only have clang installed to deal with those packages that are not properly tested with gcc, then the steps in that Wiki are unnecessary. Let Portage install clang when it requests to, and trust Portage to use clang when, and only when, necessary.

I routinely see times of less than 1 hour for clang. As best I recall, it does not have an extended period of single threaded operation.

sam_ · Developer Joined: 14 Aug 2020 Posts: 2039

I'd suggest trying to reproduce this with vanilla CFLAGS for a start, as notably something like -Ofast may substantially increase compile times.

mike155 · Posted: Mon Jan 10, 2022 8:41 am Post subject:

Please post the full output of

eccerr0r · Posted: Mon Jan 10, 2022 7:13 pm Post subject:

Yay! -funroll-loops is back!

I haven't seen 14 hour clang build times and a lot of my machines clang is a non-issue after distcc is greased. Rust is still #1 on my slowest machine and it's just about one day to complete.

Not sure about clang, but most of these language compilers have two or more stages where the first part builds with the existing tools on the machine (or in the case of rust, a downloaded copy). Then the second stage it builds itself with what it built in the first stage. This second stage cannot be distcc'ed. Getting ninja/make to work optimally is the key here, as long as you're not using high cost compile options.

Indeed full emerge --info data would be interesting to see what we're dealing with.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

jesnow · l33t Joined: 26 Apr 2006 Posts: 888

Thanks everyone for your thoughtful input! Especially Hu, you have hit the nail on the head. Yes clang is probably compiling itself, that's normal, but it should be using more than one processor and ideally it should be distcc'ing. It does very nicely when it compiles other things. I think I do have some jacked up package settings (thanks to this install being originally cloveros, a whole nother story) If I can find out why these ones are messed up and *where* in the package configuration scheme they got messed up, that would help me understand both the package and how to control portage better.

And save me from waiting while clang compiles on a single thread.

Ok Here we go:

Perfect Gentleman · Veteran Joined: 18 May 2014 Posts: 1256

jesnow · l33t Joined: 26 Apr 2006 Posts: 888

Similar results -- the behavior seems to be newish with clang 13.

mike155 · Posted: Sat Jan 15, 2022 3:58 pm Post subject:

eccerr0r · Posted: Sat Jan 15, 2022 9:59 pm Post subject:

you did something different at clang-11.. once it took 30 minutes, next it took almost 7 hours.. what happened?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

jesnow · l33t Joined: 26 Apr 2006 Posts: 888

eccerr0r · Posted: Sun Jan 16, 2022 12:13 am Post subject:

Thermal throttling perhaps is another possibility?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Hu · Administrator Joined: 06 Mar 2007 Posts: 22877

Thermal throttling could slow execution of running processes, but OP noted at the outset that the clang build drops into a strange single-threaded behavior. I wonder if that single threading alone is a major factor in the absurd compile times.

OP: when clang settles into this mode, what is the system load average, and what does the process tree look like? In particular, is make starting only one child at a time, or is make starting multiple jobs and yet the kernel schedules only one at a time? You have -l12 in your MAKEOPTS, likely to control run-away behavior in case of a distcc problem. Perhaps there is some path where the system load average goes so high that make is constrained by that -l12. Make will always keep at least one child running until it completes, regardless of how high the load average goes. However, if your load average exceeds 12, then Make will only run one child at a time.

Is there any indication in the distcc logs that distcc has locked out use of volunteers, and is trying to run only locally? How is DISTCC_HOSTS set? That is, assuming an unlimited number of jobs to run and system resources to run them, how many jobs would be processed on (1) the build machine and (2) all the other volunteers?

eccerr0r · Posted: Sun Jan 16, 2022 3:10 am Post subject:

Actually I sort of saw this behavior during some thermal throttling problems I had with one machine. I can't explain why this happened however, but for some reason my machine was throttled down to one job when it was overheating, for whatever reason. Must be some OS-HW interaction but it's not exactly clear to me how.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

HealerLFG · n00b Joined: 08 Feb 2022 Posts: 15

Good evening. I'm a bit late to this party, but I was having this exact same problem and found this thread today, as well as the solution... an emerge of CLANG that began with make opts and jobs properly, but then settled down to a single thread maxing out 1 logical core on my cpu, with no other threads being made. I have a similar configuration: broadwell-e, liquid cooled, and 32GB of ram backed by 32GB more of a swap partition, with rather conservative jobs/opts, so it didn't seem like a thrashing or thermal issue. I also used a tad less aggressive optimizations and flags, including PGO + LTO. (no Ofast or anything known to cause UB or break strict-standards though...)

*my* issue, was an invalid /etc/portage/make.conf.lto file. I LTO'd my system with the LTO-Overlay here: https://github.com/InBetweenNames/gentooLTO . I had glossed over a step or two while configuring that /etc/portage/make.conf.lto symlink that the overlay script creates. If you used that overlay and instructions as well, I urge you to double check that .conf. After adjusting mine, it seems to properly utilize all my jobs/opts during the link phase.

cheers

jesnow · l33t Joined: 26 Apr 2006 Posts: 888

Should clang be built without LTO?

Hu: Many thanks for your input. Who knows what makeopts and JOBS= does under the hood. I mean, you do. Presumably something that low level would be consistent between big builds. But my machine right now is happiliy compiling chromium and libreoffice at the same time, and dutifully hammering the distcc volunteers. So that suggests the problem is something unique to clang that I can fix.

Cheers,
Jon.

eccerr0r · Posted: Sat Feb 12, 2022 4:54 pm Post subject:

Pretty sure my machine was running one compile job at a time then when it was exhibiting weird behavior. I'll need to see if it happens again and investigate further, after fixing up my machine I haven't seen this behavior again.

But if you're focused on completion time versus optimized binaries, yes you should disable LTO/PGO/... - will get a bit of reprieve now but will add a few seconds to all the other programs you compile with it... Tradeoffs...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

jesnow · l33t Joined: 26 Apr 2006 Posts: 888

Still a problem. Clang compiles with one local thread, and takes nearly a day. It is a beast of a package to begin with, but compiling on a single core you're waiting a while. chromium by contrast on the same system compiles in 2 hours with 30 or so threads on five machines running distcc.

I've tried turning off distcc and ccache, no change in behavior. LTO, which does cause a lot of my compile-time issues, was off anyway.

Here it is doing its single thread thing and taking up 44% of memory (I have 16G), there are currently 4G occupied in the swapfile.

eccerr0r · Posted: Thu Mar 10, 2022 3:04 pm Post subject:

Since you're using distcc, are you sure your other distcc boxes aren't being hammered? How long does it take without distcc?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Hu · Administrator Joined: 06 Mar 2007 Posts: 22877

If clang is compiling single-threaded, adding more memory might help that thread swap less, but will not make better use of your other CPU cores.

Your load average looks reasonable, and would not cause -l12 to matter. I also asked about the process tree during the time that this is unexpectedly single threaded. I would like to see all the Portage-related processes, tree-threaded, with arguments. ps -Hfwwww -u portage should suffice.

How exactly did you disable distcc?

eccerr0r · Posted: Thu Mar 10, 2022 6:38 pm Post subject:

Should not be a RAM issue, I've built clang-13 on a old 4 core, 8GB machine. This is also with it running at least 2GB worth of virtual machines at the same time. I use a much lower -l and -j option for MAKEOPTS as historically I don't have that many cores though now I may need to tweak things with more distcc cores available.

Somehow you have a setting to get it to take 7GB RAM for one *gcc* job that has taken 100 minutes to run... I haven't seen something more than 3GB yet, this takes the cake...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

jesnow · l33t Joined: 26 Apr 2006 Posts: 888

Dear Hu:

Thanks, you are always on the case. This is kind of a murder mystery to me too. It just doesn't make sense that a single compile job can create such havoc. My new memory is arriving tomorrow, so maybe brute force will do the job. My current gentoo install started as a cloveros, and I still don't understand all the settings he tweaked. I noticed after my last post that -Ofast is set, so maybe that's doing extreme precompiling of things and loop-unrolling. But I tried to turn all the funky stuff off for clang.

Before I install the memory I will "emerge -1 clang" and get the more granular picture of whats running you ask for once it hits single core mode. Maybe as eccerr0r says there's just one weird setting that's blowing it all up.

I got home from work tonight and it did compile finally! Maybe single core mode is better than compiling away for 10 hours and then running out of memory and crashing.

eccerr0r · Posted: Sat Mar 12, 2022 4:09 pm Post subject:

Ah... did not catch this was formerly a cloveros install, not installed from Gentoo, so technically this thread belongs in the Unsupported forum, no wonder why none of us can reproduce the issue. Also the fact it looks like cloveros went belly up 2 years ago or so, so you're stuck with the parent distribution.

It looks like cloveros has a bunch of optimizations that are not default which also includes LTO, but this would be a linker thing not compiler...

So is this the first time you tried updating with solely the Gentoo repositories? And was the plan to switch to a more base Gentoo-like install that doesn't take 14 hours to complete clang and only need < 8GB RAM? Problem is if the distribution of jobs is bad due to the package constraints, yes it's possible to end up running single threaded, which was the whole problem of going to multi core/thread software programming in the first place...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

jesnow · l33t Joined: 26 Apr 2006 Posts: 888