emerge uses more CPU cores (threads) than permitted

[pietinger]

... and only at the start of the compilation phase.

I am posting here because I am not sure if this is a bug and needs a bug report. But I suspect a bug.

I usually do my word update at night (unattended) so I don't know when this behavior started. Today I could observe (with btop) that during today's word-update almost all packages after the configuration phase started the compilation phase immediately with 32 threads and after these were “processed”, portage “only” started the allowed 24 threads. My setting in make.conf is very simple:

[Hu]

I have seen reports like this before. It is unfortunate, but it is not a Portage bug. When you set MAKEOPTS, you tell GNU Make (and certain other build systems that, for compatibility, try to understand at least some GNU Make options) how many jobs it can start. Some build systems just guess on their own how many parallel jobs they should start. For example, dev-build/ninja states in its documentation:

[pietinger]

Hu, thank you very much for your (detailed) answer. If I understand correctly, I only have two options: Either there is the possibility to give NINJA this parameter (for each run), or it is only possible via the cgroups? (otherwise I have to live with the behavior)
_________________
https://wiki.gentoo.org/wiki/User:Pietinger

[Hu]

I thought most ebuilds that used non-GNU Make systems already handled this for you, when they can. Some build systems might not allow handling this (whereas Ninja allows you to set this right, but is greedy by default) and I don't know if Ninja is even involved in the packages you reported. I expect cgroups could enforce the limits you want, but I am not aware of any native Portage support for applying the requisite limits.

Can you capture a process tree of a misbehaving build, so we can determine what build system it uses and what, if any, options were passed that should have constrained it?

[Zucca]

As far as I know if the ebuild uses correct eclasses (and their functions) for building with ninja, the switches specified in MAKEOPTS should be automatically passed to ninja. I've observed this by looking at the command line arguments ninja gets.
But does ninja respect those in all cases...

I'll spin up a qtwebengine build...
_________________
..: Zucca :..

[sam_]

[Zucca]

Ok. First observations:

[keekkenen]

[logrusx]

[Zucca]

keekkenen,

Unless I'm totally lost, -j is shorthand for --jobs.

Anyway. My second test showed some interesting signs... Now with ninja -v -j72 -l68, the load has been staying at 68, except for the start (obviously). But at some points the load spiked up to 120. I compile everything in RAM, and it's far from exhausted. Not a single page of swap has been allocated. So it cannot be disk load.
_________________
..: Zucca :..

[logrusx]

[Zucca]

Yes.
After the load spiked at 120, ninja stopped creating new threads until the load was again at 68.
So far no other load spike has happened.
I believe qtwebengine ebuild does respect MAKEOPTS quite well. At least ninja quickly reacts to change in load.

pietinger, did you observe load going up to 32 for how long?
Or was it only spiking sometimes for short durations, like what I've observed?
_________________
..: Zucca :..

[keekkenen]

[pietinger]

[pietinger]

[pietinger]

[logrusx]

[sam_]

[pietinger]

[keekkenen]

[sam_]

I'm not sure what you mean. Portage defaults to that in GNUMAKEFLAGS (IIRC) if MAKEOPTS, GNUMAKEFLAGS, and MAKEOPTS aren't set (it's just a fail-safe default if the user specifies nothing at all in make.conf).

[logrusx]

[quote="pietinger"]