emerge --jobs=n but exclude packages by name?

[mcnutty]

I have the following options in my /etc/make.conf:

[Ralphred]

Have you had any luck with /etc/portage/{env/,package.env/}, or is the die "already rolling" at that point?

[bstaletic]

[mcnutty]

The problem arises because when chromium, or some other package, starts emerging the load factor is low and below my threshold, so emerge decides to start merging another package. But, when that next package is another large one, it will cause the failure because now I've got two huge packages building in parallel. (edit: for a variety of reasons this tends to happen on almost all large world updates)

I can reduce the MAKEOPTS jobs in an env file, but then that package will take forever if that package is being merged by itself. I'm not sure if you had something else in mind though.

[Ralphred]

[pingtoo]

If load average 24.0 is something you machine can handle. Then you can try setup the MAKEOPTS=-j8 that way each job can spawn at most 8 compilations so assume all three emerge job kick in, you will get 3x8 compilations.

You can also add MAKEOPTS="-j8 -l8" so the build server (make, ninja and mason) will control depend on load.

Or make a very very large swap

[mcnutty]

I appreciate the responses, but they don't really solve the problem in a satisfying way. It is easy to prevent the system from crashing by setting 'emerge --jobs=1', but this significantly increases the time to update when there are lot of small packages (the normal case).

At least on my system, a large majority of installed packages are small and many don't even need compilation (e.g., perl, python, etc). This is what makes having 'emerge --jobs>1' really useful and speeds up the updating process significantly. But 'emerge --jobs>1' becomes useless in my case because '--load-average' is not an effective way to prevent large builds from happening concurrently. There are too many ways for emerge to get fooled.

So, I was hoping I missed something and there was some better way to exclude large packages from building concurrently with other ones.

[Slippery Jim]

Some of my builds used to get assassinated by the OOM for similar reasons. A bunch of parallel configuration phases would all be started in the first few seconds, and then they would all start compiling and blow up the load average to like a hundred or something. What pingtoo said is what fixed it for me. I set my max load average to 90% of my jobs value, which is set to my number of physical cores. That way the system still has 10% free for responsiveness. my settings on a ryzen 4750G are:

[logrusx]

[mcnutty]

[pingtoo]

[mcnutty]

[logrusx]

[pingtoo]

You are more likely run into problem when updating many very large package with emerge's parallel control than make.

GNU make does know and respect load average reported by system. However emerge failed in this respect.

Emerge tend to start many jobs (not always respect the -j and/or -l) all at once but will not start more when job count is lower than specified --job <count> and load average lower than specified --load-average control so you tend to wait at that large job finish then emerge will continue to move on to remaining jobs. (I test this many time, it is not always because dependency)

I did not suggest use env file.

I suggest use command line with environment variable to override the default.

[mcnutty]

[Genone]

[mcnutty]

I've been evaluating how the system responds after modifying my '/etc/portage/make.conf' file by adding '--load-average' to both 'EMERGE_DEFAULT_OPTS' and 'MAKEOPTS'.

It does seem to prevent the system from overloading when multiple large packages are scheduled for a world update. However, it more or less works the way I expected/worried it would. What tends to happen is the load average will be high merging the small packages and then chromium (or qtwebengine, etc) will start. Since the load average is high they will build using only 1 or 2 cores/threads and take forever to complete (24+ hours).

It's not the end of the world, but I was hoping to have my cake and eat it too.