emerge -1 clang takes 14 hours? [SOLVED!!]

[jesnow]

Currently these are the compiler flags in use for clang:

[pjp]

[Hu]

The advice to add more memory is for people who are plagued by swapping due to insufficient memory relative to the number of programs they have open. They run 8+ jobs in parallel, run out of memory, start swapping, and all 8+ jobs run slowly as the kernel swaps in and out pieces of each of them. As I understand your problem report, you already knew that was not your failure mode. Your failure mode is that the system stops trying to run multiple jobs in parallel. We still don't have any indication why it decides to go to running one process at a time with no parallelism, though I have asked a few times for information that might lead us in that direction.

[jesnow]

I noticed the following: (Sorry for the multiple posting, this is a new subtopic. )

Registry.cpp is where the compilations of the clang compiler stall out single threaded. And what stalls out? cc1plus, the preprocessor. So lets just go look at the code, maybe we'll get a clue:

https://clang.llvm.org/doxygen/Registry_8cpp_source.html

So this is very interesting: It defines a bunch of preprocessor macros, but the most important one is this one:

[jesnow]

Hi Hu, you're quite right, I had indeed more or less eliminated insufficient memory as the problem, so I kind of knew the pile of extra memory wasn't likely to solve this particular issue. I had noticed that the machine swapped a little bit before settling down to single thread mode, and so it's not awful to have way too much memory in a gentoo machine. At least I can eliminate that bottleneck. I *like* running gentoo even though it's super-unproductive for the rest of my life. My son the developer tells me its a kind of pointless auto-gratification to spend so much time on compiling my operating system every weekend.

[jesnow]

Ok found it. Marking this [solved], thanks to all that helped. People shouldn't hate on cloveros so much. It was always gentoo through and through, just kind of a really over optimized one. Unlike lots of other gentoo derivatives that put a lot of their own code in, This was just gentoo with a complicated set of /etc/portage files, a binhost and a nicer install script than the standard one.

Anyway:

Cloveros had compiled clang with his "no-nosinter" compile flags:

[eccerr0r]

Well the point is, cloveros added something you didn't add explicitly, else you would have remembered, I would hope... or else you'd basically have Gentoo with another set of instructions, which you also should have remembered. So that environment file must have came from cloveros however it was distributed - which is not supported by Gentoo proper. There is no way I could have acquired a copy of those files from the Gentoo repositories or the installation wiki - hence my concern this not being a supported configuration, even if the base system is still Gentoo.

In any case glad you found your problem. Most likely it's solely due to turning on LTO whether inadvertently or not, I don't think graphite or other options kills performance that much but worth testing anyway if you're still trying to squeeze performance out. But now your ryzen beats my 10+ year old core2 quad as it should :)
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[pjp]

I'm glad you figured it out.

[jesnow]

OK, I didn't really mean "hate", but in fact everything cloveros did was "in bounds" for a gentoo distribution. And If I can just learn how to control it all, which I am, it will improve my gentoo skills. Which have improved as a result.

For example, with distcc turned on (but mt /etc/distcc/hosts needs adjusting):
All of my distcc running on various boxes around the house are Calculate linux, which is much more vanilla. Only my server and my workstation are still compiled gentoo.

[lefsha]

I don't know how much the issue has been solved, but I do have exact the same issue.
I do see no solution for that. And I was not able to find any solution in that thread.
Anyone who does really believe there was a solution I have missed, please point me to that text.
Thx.

The issue so far is only existing at GCC 12 and 13. I didn't check all 13.X versions, but only
one for each major version.

The issue doesn't exist for GCC-14 as well it doesn't exist for ANY version of CLANG.

Obviously, not settings, no complier flags no nothing can be a reason. Something is wrong
inside of GCC or the way it is compiling for Gentoo. I didn't check other versions of GCC from Debian
neither I have compiled them manually. I did so with CLANG. Both ebuild version of CLANG and
manually made one are free of that issue.

I recall the issue is, that after starting the 16 jobs in my case with -j16, after certain period
of time by reaching 98-99% of compilation the cc1plus process became single threaded
and steadily growing in memory consumption, sometimes reaching 40-60%.

It can take 1-2 days to compile clang by gcc. It can take 1-2 days to compile gcc by gcc.
Some other bulky software (vtk etc) has the same issue. In all those cases the compilation
is successful! It just takes terribly long and uses only 1 core of 16 cores.
It's not a linking phase, so I cannot understand what compiler can do that long with text files.

Switching to another gcc version like gcc-14 or moving to clang solves the issues.

I cannot claim that this issues is really solved like OP reported.
I have seen similar reports on reddit.

I will maybe try to compile gcc-12 manually and see the difference.

So far it is very annoying. Never before I have experienced something like that.
Unfortunately I cannot move to gcc-14 yet a lot of software still depends in gcc-12.
_________________
Lefsha

[stefan11111]

[Zucca]

[stefan11111]

[Zucca]

[sam_]

As "cool" as this derivative might be, they're using several flags known to both cause substantial slowdowns in compile-time and also have dubious benefits on runtime performance. -fipa-pta being an example of that where the GCC developers specifically say not to bother using it unless a measured significant improvement in runtime is measured.

[jesnow]

Thanks, yes: My trying out (now-defunct) Clover on my (then) new machine was really the gift that kept on giving. Since starting this thread I got an AMD 5950X to go with all that memory, toned down the "optimizations" and all compiles smoothly. But in the meantime, we have slotted clang, and I have to compile it three times. And it's gotten bigger. I will start another thread about that. I have three clangs and four pythons, and it's supposed to be a "stable" system, with only a small number of active keywords.

Ironic.

[Zucca]