Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
emerge --jobs=n but exclude packages by name?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
mcnutty
Tux's lil' helper
Tux's lil' helper


Joined: 29 Dec 2009
Posts: 127

PostPosted: Mon Oct 07, 2024 8:34 pm    Post subject: emerge --jobs=n but exclude packages by name? Reply with quote

I have the following options in my /etc/make.conf:

Code:
EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=24.0"
MAKEPOPTS="-j24"


In general this significantly reduces the amount of time an update takes. However, there are a couple packages that screw everything up (i.e., chromium and qtwebengine). Chromium (or some other big package) will start to emerge, it will then take a a few minutes before it starts compiling heavily. During that time one or more other large packages will start emerging. Once they all get going, the load average will be well over 24 (possibly in the 50-60s) and I will run out of memory.

Is there a way to tell emerge to never merge a specific package (or packages) in parallel with others?
Back to top
View user's profile Send private message
Ralphred
Guru
Guru


Joined: 31 Dec 2013
Posts: 597

PostPosted: Mon Oct 07, 2024 8:38 pm    Post subject: Reply with quote

Have you had any luck with /etc/portage/{env/,package.env/}, or is the die "already rolling" at that point?
Back to top
View user's profile Send private message
bstaletic
Guru
Guru


Joined: 05 Apr 2014
Posts: 333

PostPosted: Mon Oct 07, 2024 8:46 pm    Post subject: Re: emerge --jobs=n but exclude packages by name? Reply with quote

mcnutty wrote:
Is there a way to tell emerge to never merge a specific package (or packages) in parallel with others?


Not really, but you can get creative with /etc/portage/env and /etc/portage/package.env
https://wiki.gentoo.org/wiki/Package.env
Back to top
View user's profile Send private message
mcnutty
Tux's lil' helper
Tux's lil' helper


Joined: 29 Dec 2009
Posts: 127

PostPosted: Mon Oct 07, 2024 8:52 pm    Post subject: Reply with quote

The problem arises because when chromium, or some other package, starts emerging the load factor is low and below my threshold, so emerge decides to start merging another package. But, when that next package is another large one, it will cause the failure because now I've got two huge packages building in parallel. (edit: for a variety of reasons this tends to happen on almost all large world updates)

I can reduce the MAKEOPTS jobs in an env file, but then that package will take forever if that package is being merged by itself. I'm not sure if you had something else in mind though.
Back to top
View user's profile Send private message
Ralphred
Guru
Guru


Joined: 31 Dec 2013
Posts: 597

PostPosted: Mon Oct 07, 2024 9:24 pm    Post subject: Reply with quote

mcnutty wrote:
edit: for a variety of reasons this tends to happen on almost all large world updates

You are beyond the native capabilities of portage. Start by making a "blacklist" of updates that break your "usual model", then push them into a temporary "package.mask", then run them as one package at a time after the fact as "one_at_a_time" packages.

Things like "qtwebengine", "firefox" and "chromium" aren't hard to identify before the fact, and /etc/portage/bashrc offers more than you need to sidestep "paradigm breaking" builds.
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1154
Location: Richmond Hill, Canada

PostPosted: Mon Oct 07, 2024 10:29 pm    Post subject: Reply with quote

If load average 24.0 is something you machine can handle. Then you can try setup the MAKEOPTS=-j8 that way each job can spawn at most 8 compilations so assume all three emerge job kick in, you will get 3x8 compilations.

You can also add MAKEOPTS="-j8 -l8" so the build server (make, ninja and mason) will control depend on load.

Or make a very very large swap 8O
Back to top
View user's profile Send private message
mcnutty
Tux's lil' helper
Tux's lil' helper


Joined: 29 Dec 2009
Posts: 127

PostPosted: Mon Oct 07, 2024 10:51 pm    Post subject: Reply with quote

I appreciate the responses, but they don't really solve the problem in a satisfying way. It is easy to prevent the system from crashing by setting 'emerge --jobs=1', but this significantly increases the time to update when there are lot of small packages (the normal case).

At least on my system, a large majority of installed packages are small and many don't even need compilation (e.g., perl, python, etc). This is what makes having 'emerge --jobs>1' really useful and speeds up the updating process significantly. But 'emerge --jobs>1' becomes useless in my case because '--load-average' is not an effective way to prevent large builds from happening concurrently. There are too many ways for emerge to get fooled.

So, I was hoping I missed something and there was some better way to exclude large packages from building concurrently with other ones.
Back to top
View user's profile Send private message
Slippery Jim
Apprentice
Apprentice


Joined: 08 Jan 2005
Posts: 281

PostPosted: Tue Oct 08, 2024 12:31 am    Post subject: Reply with quote

Some of my builds used to get assassinated by the OOM for similar reasons. A bunch of parallel configuration phases would all be started in the first few seconds, and then they would all start compiling and blow up the load average to like a hundred or something. What pingtoo said is what fixed it for me. I set my max load average to 90% of my jobs value, which is set to my number of physical cores. That way the system still has 10% free for responsiveness. my settings on a ryzen 4750G are:

Code:

MAKEOPTS="-j16 -l14.4"
EMERGE_DEFAULT_OPTS="-j16 -l14.4"


I needed both lines. MAKEOPTS takes care of individual ebuilds starting too many threads, and EMERGE_DEFAULT_OPTS ensures that portage doesn't start too many emerges. Even if portage starts a lot of emerges, make should notice when the load average spikes, and hold off on its stuff until the load average drops, IIUC.
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2279

PostPosted: Tue Oct 08, 2024 4:44 am    Post subject: Reply with quote

mcnutty wrote:
I appreciate the responses, but they don't really solve the problem in a satisfying way.


I'm sorry if that bothers you but that's the current reality. I guess using the binhost will also be not satisfying, so here's another reason to be unhappy :D

Best Regards,
Georgi
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum