View previous topic :: View next topic |
Author |
Message |
mcnutty Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 29 Dec 2009 Posts: 130
|
Posted: Mon Oct 07, 2024 8:34 pm Post subject: emerge --jobs=n but exclude packages by name? |
|
|
I have the following options in my /etc/make.conf:
Code: | EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=24.0"
MAKEPOPTS="-j24" |
In general this significantly reduces the amount of time an update takes. However, there are a couple packages that screw everything up (i.e., chromium and qtwebengine). Chromium (or some other big package) will start to emerge, it will then take a a few minutes before it starts compiling heavily. During that time one or more other large packages will start emerging. Once they all get going, the load average will be well over 24 (possibly in the 50-60s) and I will run out of memory.
Is there a way to tell emerge to never merge a specific package (or packages) in parallel with others? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Ralphred Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 31 Dec 2013 Posts: 598
|
Posted: Mon Oct 07, 2024 8:38 pm Post subject: |
|
|
Have you had any luck with /etc/portage/{env/,package.env/}, or is the die "already rolling" at that point? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
bstaletic Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 05 Apr 2014 Posts: 333
|
Posted: Mon Oct 07, 2024 8:46 pm Post subject: Re: emerge --jobs=n but exclude packages by name? |
|
|
mcnutty wrote: | Is there a way to tell emerge to never merge a specific package (or packages) in parallel with others? |
Not really, but you can get creative with /etc/portage/env and /etc/portage/package.env
https://wiki.gentoo.org/wiki/Package.env |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mcnutty Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 29 Dec 2009 Posts: 130
|
Posted: Mon Oct 07, 2024 8:52 pm Post subject: |
|
|
The problem arises because when chromium, or some other package, starts emerging the load factor is low and below my threshold, so emerge decides to start merging another package. But, when that next package is another large one, it will cause the failure because now I've got two huge packages building in parallel. (edit: for a variety of reasons this tends to happen on almost all large world updates)
I can reduce the MAKEOPTS jobs in an env file, but then that package will take forever if that package is being merged by itself. I'm not sure if you had something else in mind though. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Ralphred Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 31 Dec 2013 Posts: 598
|
Posted: Mon Oct 07, 2024 9:24 pm Post subject: |
|
|
mcnutty wrote: | edit: for a variety of reasons this tends to happen on almost all large world updates |
You are beyond the native capabilities of portage. Start by making a "blacklist" of updates that break your "usual model", then push them into a temporary "package.mask", then run them as one package at a time after the fact as "one_at_a_time" packages.
Things like "qtwebengine", "firefox" and "chromium" aren't hard to identify before the fact, and /etc/portage/bashrc offers more than you need to sidestep "paradigm breaking" builds. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pingtoo Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/gallery/Star Wars/movie_star_wars_storm_trooper.gif)
Joined: 10 Sep 2021 Posts: 1167 Location: Richmond Hill, Canada
|
Posted: Mon Oct 07, 2024 10:29 pm Post subject: |
|
|
If load average 24.0 is something you machine can handle. Then you can try setup the MAKEOPTS=-j8 that way each job can spawn at most 8 compilations so assume all three emerge job kick in, you will get 3x8 compilations.
You can also add MAKEOPTS="-j8 -l8" so the build server (make, ninja and mason) will control depend on load.
Or make a very very large swap ![Shocked 8O](images/smiles/icon_eek.gif) |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mcnutty Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 29 Dec 2009 Posts: 130
|
Posted: Mon Oct 07, 2024 10:51 pm Post subject: |
|
|
I appreciate the responses, but they don't really solve the problem in a satisfying way. It is easy to prevent the system from crashing by setting 'emerge --jobs=1', but this significantly increases the time to update when there are lot of small packages (the normal case).
At least on my system, a large majority of installed packages are small and many don't even need compilation (e.g., perl, python, etc). This is what makes having 'emerge --jobs>1' really useful and speeds up the updating process significantly. But 'emerge --jobs>1' becomes useless in my case because '--load-average' is not an effective way to prevent large builds from happening concurrently. There are too many ways for emerge to get fooled.
So, I was hoping I missed something and there was some better way to exclude large packages from building concurrently with other ones. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Slippery Jim Apprentice
![Apprentice Apprentice](/images/ranks/rank_rect_2.gif)
![](images/avatars/123412392546489e1012e20.jpg)
Joined: 08 Jan 2005 Posts: 281
|
Posted: Tue Oct 08, 2024 12:31 am Post subject: |
|
|
Some of my builds used to get assassinated by the OOM for similar reasons. A bunch of parallel configuration phases would all be started in the first few seconds, and then they would all start compiling and blow up the load average to like a hundred or something. What pingtoo said is what fixed it for me. I set my max load average to 90% of my jobs value, which is set to my number of physical cores. That way the system still has 10% free for responsiveness. my settings on a ryzen 4750G are:
Code: |
MAKEOPTS="-j16 -l14.4"
EMERGE_DEFAULT_OPTS="-j16 -l14.4"
|
I needed both lines. MAKEOPTS takes care of individual ebuilds starting too many threads, and EMERGE_DEFAULT_OPTS ensures that portage doesn't start too many emerges. Even if portage starts a lot of emerges, make should notice when the load average spikes, and hold off on its stuff until the load average drops, IIUC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
logrusx Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/176594205966916859b5145.jpg)
Joined: 22 Feb 2018 Posts: 2287
|
Posted: Tue Oct 08, 2024 4:44 am Post subject: |
|
|
mcnutty wrote: | I appreciate the responses, but they don't really solve the problem in a satisfying way. |
I'm sorry if that bothers you but that's the current reality. I guess using the binhost will also be not satisfying, so here's another reason to be unhappy :D
Best Regards,
Georgi |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mcnutty Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 29 Dec 2009 Posts: 130
|
Posted: Tue Oct 08, 2024 4:21 pm Post subject: |
|
|
Slippery Jim wrote: | Code: |
MAKEOPTS="-j16 -l14.4"
EMERGE_DEFAULT_OPTS="-j16 -l14.4"
|
I needed both lines. |
I think this might be the key. I didn't realize 'make' also had the '--load-average' option. My only concern is that a large package like chromium might get started with a small number of jobs and then take forever. Definitely worth a try though, thanks ![Very Happy :D](images/smiles/icon_biggrin.gif) |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pingtoo Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/gallery/Star Wars/movie_star_wars_storm_trooper.gif)
Joined: 10 Sep 2021 Posts: 1167 Location: Richmond Hill, Canada
|
Posted: Tue Oct 08, 2024 5:47 pm Post subject: |
|
|
mcnutty wrote: | Slippery Jim wrote: | Code: |
MAKEOPTS="-j16 -l14.4"
EMERGE_DEFAULT_OPTS="-j16 -l14.4"
|
I needed both lines. |
I think this might be the key. I didn't realize 'make' also had the '--load-average' option. My only concern is that a large package like chromium might get started with a small number of jobs and then take forever. Definitely worth a try though, thanks ![Very Happy :D](images/smiles/icon_biggrin.gif) |
May be you fall in to a wrong area of thinking. Is one package finish time important OR all packages finish time important?
Adjust one package parallelism not necessary equate to complete update. so should you wish to only compile 1 package then use command line environment override may be is a way to go. For example Code: | MAKEOPTS="-j24" emerge -a chromium |
|
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mcnutty Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 29 Dec 2009 Posts: 130
|
Posted: Tue Oct 08, 2024 6:16 pm Post subject: |
|
|
pingtoo wrote: |
May be you fall in to a wrong area of thinking. Is one package finish time important OR all packages finish time important?
Adjust one package parallelism not necessary equate to complete update. so should you wish to only compile 1 package then use command line environment override may be is a way to go. For example Code: | MAKEOPTS="-j24" emerge -a chromium |
|
I want world updates to finish as quickly as possible. It is not uncommon, for example when there are KDE/Plasma updates, for there to be 50-100 packages to merge. Most of these packages are small and merge relatively quickly. Using 'emerge --jobs>1' helps a lot to speed these small packages up because compilation is generally not the bottleneck.
However, what tends to happen is the big packages are all that's left toward the end of the update after the small packages are done. Due to quirks in how the packages are merged those big packages all wind up being emerged in parallel and eventually overload the system. Yes, reducing the parallelism using an env file will prevent the system from overloading, but will then cause the world update to take forever.
I think adding '--load-average' to the 'MAKEOPTS' is exactly what I want, but it will depend on how smart 'make' is. I have a feeling that it is "dumb" in that it will check the load average when 'make' is first executed and limit the number of jobs at that point. But, if it is able to dynamically adjust the number of jobs after the compilation has started when the load average changes, then it be exactly what I am hoping for. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
logrusx Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/176594205966916859b5145.jpg)
Joined: 22 Feb 2018 Posts: 2287
|
Posted: Tue Oct 08, 2024 6:30 pm Post subject: |
|
|
mcnutty wrote: | I have a feeling that it is "dumb" in that it will check the load average when 'make' is first executed and limit the number of jobs at that point. |
I think you're wrong here. To my knowledge, load-average will cause make to only spawn a new job if load average is below the threshold. It doesn't make sense to do what you suspect.
However that can't save you from low memory conditions which packages like qtwebengine and webkit-gtk cause so often. Anyways, I hope that solution works for you.
Best Regards,
Georgi |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pingtoo Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/gallery/Star Wars/movie_star_wars_storm_trooper.gif)
Joined: 10 Sep 2021 Posts: 1167 Location: Richmond Hill, Canada
|
Posted: Tue Oct 08, 2024 6:49 pm Post subject: |
|
|
You are more likely run into problem when updating many very large package with emerge's parallel control than make.
GNU make does know and respect load average reported by system. However emerge failed in this respect.
Emerge tend to start many jobs (not always respect the -j and/or -l) all at once but will not start more when job count is lower than specified --job <count> and load average lower than specified --load-average control so you tend to wait at that large job finish then emerge will continue to move on to remaining jobs. (I test this many time, it is not always because dependency)
I did not suggest use env file.
I suggest use command line with environment variable to override the default. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mcnutty Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 29 Dec 2009 Posts: 130
|
Posted: Tue Oct 08, 2024 7:27 pm Post subject: |
|
|
logrusx wrote: |
I think you're wrong here. To my knowledge, load-average will cause make to only spawn a new job if load average is below the threshold. It doesn't make sense to do what you suspect. |
I don't use 'make' very often so it's good to hear that it continuously monitors the build process and will adapt the number of jobs as conditions change during compilation.
logrusx wrote: |
However that can't save you from low memory conditions which packages like qtwebengine and webkit-gtk cause so often. Anyways, I hope that solution works for you. |
I don't have low memory conditions unless multiple large packages start merging with high '-j' values for 'make', so in theory this (adding '--load-average' to the MAKEOPTS) should solve my problem. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Genone Retired Dev
![Retired Dev Retired Dev](/images/ranks/rank-retired.gif)
![](images/avatars/21004157514287ab2b29021.jpg)
Joined: 14 Mar 2003 Posts: 9598 Location: beyond the rim
|
Posted: Wed Oct 09, 2024 9:33 pm Post subject: |
|
|
mcnutty wrote: | logrusx wrote: |
I think you're wrong here. To my knowledge, load-average will cause make to only spawn a new job if load average is below the threshold. It doesn't make sense to do what you suspect. |
I don't use 'make' very often so it's good to hear that it continuously monitors the build process and will adapt the number of jobs as conditions change during compilation. |
Well, make works just like portage in that regard: Only create a new process if the load limit allows it. It doesn't impose a limitation on the number of jobs directly, that is what -j is for (it sets an upper limit if you specify a number). The difference between make and emerge is that processes launched by make usually only last a few seconds while processes launched by emerge can last hours, so naturally the limit is checked much more often.
One problem you may run into though is that not all build systems will evaluate MAKEOPTS and/or support load-limitations in the same way. Also some processes launched by make in turn may create their own subprocesses and therefore bypass the load limit. Therefore you may want to set the load limit lower than you'd normally do. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
|