View previous topic :: View next topic |
Author |
Message |
mcnutty Tux's lil' helper
Joined: 29 Dec 2009 Posts: 127
|
Posted: Mon Oct 07, 2024 8:34 pm Post subject: emerge --jobs=n but exclude packages by name? |
|
|
I have the following options in my /etc/make.conf:
Code: | EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=24.0"
MAKEPOPTS="-j24" |
In general this significantly reduces the amount of time an update takes. However, there are a couple packages that screw everything up (i.e., chromium and qtwebengine). Chromium (or some other big package) will start to emerge, it will then take a a few minutes before it starts compiling heavily. During that time one or more other large packages will start emerging. Once they all get going, the load average will be well over 24 (possibly in the 50-60s) and I will run out of memory.
Is there a way to tell emerge to never merge a specific package (or packages) in parallel with others? |
|
Back to top |
|
|
Ralphred Guru
Joined: 31 Dec 2013 Posts: 596
|
Posted: Mon Oct 07, 2024 8:38 pm Post subject: |
|
|
Have you had any luck with /etc/portage/{env/,package.env/}, or is the die "already rolling" at that point? |
|
Back to top |
|
|
bstaletic Guru
Joined: 05 Apr 2014 Posts: 333
|
Posted: Mon Oct 07, 2024 8:46 pm Post subject: Re: emerge --jobs=n but exclude packages by name? |
|
|
mcnutty wrote: | Is there a way to tell emerge to never merge a specific package (or packages) in parallel with others? |
Not really, but you can get creative with /etc/portage/env and /etc/portage/package.env
https://wiki.gentoo.org/wiki/Package.env |
|
Back to top |
|
|
mcnutty Tux's lil' helper
Joined: 29 Dec 2009 Posts: 127
|
Posted: Mon Oct 07, 2024 8:52 pm Post subject: |
|
|
The problem arises because when chromium, or some other package, starts emerging the load factor is low and below my threshold, so emerge decides to start merging another package. But, when that next package is another large one, it will cause the failure because now I've got two huge packages building in parallel. (edit: for a variety of reasons this tends to happen on almost all large world updates)
I can reduce the MAKEOPTS jobs in an env file, but then that package will take forever if that package is being merged by itself. I'm not sure if you had something else in mind though. |
|
Back to top |
|
|
Ralphred Guru
Joined: 31 Dec 2013 Posts: 596
|
Posted: Mon Oct 07, 2024 9:24 pm Post subject: |
|
|
mcnutty wrote: | edit: for a variety of reasons this tends to happen on almost all large world updates |
You are beyond the native capabilities of portage. Start by making a "blacklist" of updates that break your "usual model", then push them into a temporary "package.mask", then run them as one package at a time after the fact as "one_at_a_time" packages.
Things like "qtwebengine", "firefox" and "chromium" aren't hard to identify before the fact, and /etc/portage/bashrc offers more than you need to sidestep "paradigm breaking" builds. |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1153 Location: Richmond Hill, Canada
|
Posted: Mon Oct 07, 2024 10:29 pm Post subject: |
|
|
If load average 24.0 is something you machine can handle. Then you can try setup the MAKEOPTS=-j8 that way each job can spawn at most 8 compilations so assume all three emerge job kick in, you will get 3x8 compilations.
You can also add MAKEOPTS="-j8 -l8" so the build server (make, ninja and mason) will control depend on load.
Or make a very very large swap |
|
Back to top |
|
|
mcnutty Tux's lil' helper
Joined: 29 Dec 2009 Posts: 127
|
Posted: Mon Oct 07, 2024 10:51 pm Post subject: |
|
|
I appreciate the responses, but they don't really solve the problem in a satisfying way. It is easy to prevent the system from crashing by setting 'emerge --jobs=1', but this significantly increases the time to update when there are lot of small packages (the normal case).
At least on my system, a large majority of installed packages are small and many don't even need compilation (e.g., perl, python, etc). This is what makes having 'emerge --jobs>1' really useful and speeds up the updating process significantly. But 'emerge --jobs>1' becomes useless in my case because '--load-average' is not an effective way to prevent large builds from happening concurrently. There are too many ways for emerge to get fooled.
So, I was hoping I missed something and there was some better way to exclude large packages from building concurrently with other ones. |
|
Back to top |
|
|
Slippery Jim Apprentice
Joined: 08 Jan 2005 Posts: 281
|
Posted: Tue Oct 08, 2024 12:31 am Post subject: |
|
|
Some of my builds used to get assassinated by the OOM for similar reasons. A bunch of parallel configuration phases would all be started in the first few seconds, and then they would all start compiling and blow up the load average to like a hundred or something. What pingtoo said is what fixed it for me. I set my max load average to 90% of my jobs value, which is set to my number of physical cores. That way the system still has 10% free for responsiveness. my settings on a ryzen 4750G are:
Code: |
MAKEOPTS="-j16 -l14.4"
EMERGE_DEFAULT_OPTS="-j16 -l14.4"
|
I needed both lines. MAKEOPTS takes care of individual ebuilds starting too many threads, and EMERGE_DEFAULT_OPTS ensures that portage doesn't start too many emerges. Even if portage starts a lot of emerges, make should notice when the load average spikes, and hold off on its stuff until the load average drops, IIUC. |
|
Back to top |
|
|
|