| View previous topic :: View next topic |
| Author |
Message |
mudspitease
n00b
Joined: 09 Mar 2025
Posts: 4
|
 Posted: Sun Mar 09, 2025 3:43 pm Post subject: Optimization of MAKE.CONF due to slow emerge/compile times |
 |
|
Hello,
I want to make sure that my make.conf file is optimized properly...
13th Gen Intel(R) Core(TM) i5-1334U, 10 cores / 12 threads + 16gb ram on nvme
I currently have a few programs that use | Code: | | jumbo-build, -03, and LTO, |
like qtwebengine, firefox, etc, in their specific package.use files... but I'm finding that I need take off those settings, since it seems to stop them from compiling...
I have noticed that my emerge/compile times a lot slower recently, and I want to make sure I'm not tripping over myself with all these settings. Is my CPU supposed to shoot up to almost 100% while compiling?
For context, the packages that are currently emerging are:
| Code: |
app-office/libreoffice-24.2.7.2-r1::gentoo
x11-libs/wxGTK-3.2.5::gentoo
|
Any tips on how to improve my MAKE.CONF? Anything "bad practices" that I've inadvertently falling into?
Thanks!
| Code: |
# These settings were set by the catalyst build script that automatically
# built this stage. Please consult /usr/share/portage/config/make.conf.example
# for a more detailed example.
# Common compilation flags
COMMON_FLAGS="-march=native -O2 -pipe -fdiagnostics-color=always"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
RUSTFLAGS="${RUSTFLAGS} -C target-cpu=native"
# Locale settings
LC_MESSAGES=C.utf8
# Make options
MAKEOPTS="-j7 -l14"
EMERGE_DEFAULT_OPTS="--jobs=12 --load-average=15 --keep-going --verbose --with-bdeps=y"
# Portage scheduling policy
PORTAGE_SCHEDULING_POLICY="idle"
# Features
FEATURES="parallel-fetch parallel-install ccache"
CCACHE_SIZE="50G"
CCACHE_DIR="/var/cache/ccache"
# License and keywords
ACCEPT_LICENSE="*"
ACCEPT_KEYWORDS="amd64"
# USE flags
USE="systemd -elogind \
kde qt6 \
-gtk -gnome \
-telemetry bash-completion \
apparmor \
alsa pulseaudio sound"
# GRUB settings
GRUB_PLATFORMS="efi-64"
# Video card settings
VIDEO_CARDS="intel"
# Gentoo mirrors
GENTOO_MIRRORS="https://mirror.clarkson.edu/gentoo/ \
http://mirror.clarkson.edu/gentoo/ \
rsync://mirror.clarkson.edu/gentoo/ \
http://www.gtlib.gatech.edu/pub/gentoo \
rsync://rsync.gtlib.gatech.edu/gentoo \
https://mirrors.mit.edu/gentoo-distfiles/ \
http://mirrors.mit.edu/gentoo-distfiles/ \
rsync://mirrors.mit.edu/gentoo-distfiles/ \
https://gentoo.osuosl.org/ \
http://gentoo.osuosl.org/ \
https://mirrors.rit.edu/gentoo/ \
http://mirrors.rit.edu/gentoo/ \
ftp://mirrors.rit.edu/gentoo/ \
rsync://mirrors.rit.edu/gentoo/ \
https://mirror.servaxnet.com/gentoo/ \
http://mirror.servaxnet.com/gentoo/ \
http://gentoo-mirror.flux.utah.edu/" |
|
|
| Back to top |
|
 |
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5479
Location: Bavaria
|
|
| Back to top |
|
|
mudspitease
n00b
Joined: 09 Mar 2025
Posts: 4
|
| Posted: Sun Mar 09, 2025 5:08 pm Post subject: Optimization of MAKE.CONF due to slow emerge/compile times |
|
|
| pietinger wrote: | mudspitease,
Welcome to Gentoo Forums! 
|
Thanks! It's been a long ride to get here - but VERY glad I made it!
| Quote: | Never start more threads than CPU cores ... if you do it will be slower and not faster.
Never start more threads than your amount of RAM is able to hold ... if you do and you will have then swapping, it will be slower.
So for your CPU - having 12 logical cores (with "hyperthreading") - BUT only 16 GB of RAM you could use a maximum that is between MAKEOPTS="-j8" and MAKEOPTS="-j12" (depending of the package you compile). Try -j12 and proof if there is swapping. If your machine swaps go down. |
Just made the update in my MAKE.CONF... I just commented out my old MAKEOPTS, and put in -j12 in MAKEOPTS.
I commented out my EMERGE_DEFAULT_OPTS in my MAKE.CONF.... but should I put it in a package.use file for libreoffice, for instance? Or, if I were to use /etc/portage/env like you mention, would I put it as -j4, per my 16 gb of RAM? |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2816
|
| Posted: Sun Mar 09, 2025 6:03 pm Post subject: |
|
|
| mudspitease wrote: | Hello,
I want to make sure that my make.conf file is optimized properly...
13th Gen Intel(R) Core(TM) i5-1334U, 10 cores / 12 threads + 16gb ram on nvme
I currently have a few programs that use | Code: | | jumbo-build, -03, and LTO, |
|
As a start, remove LTO and change O3 to O2. It's not worth neither the trouble, nor the performance gains if at all noticeable. Both of them actually.
| mudspitease wrote: |
I commented out my EMERGE_DEFAULT_OPTS in my MAKE.CONF.... but should I put it in a package.use file for libreoffice, for instance? Or, if I were to use /etc/portage/env like you mention, would I put it as -j4, per my 16 gb of RAM? |
No. It's not package specific, it determines how many packages are compiled concurrently, hence "emerge" jobs. Multiply that by the number of make jobs and you get maximum total number of threads that can be spawned on your CPU. That's way too many.
Last but not least, have you considered using the binhost?
Best Regards,
Georgi |
|
| Back to top |
|
|
mudspitease
n00b
Joined: 09 Mar 2025
Posts: 4
|
| Posted: Sun Mar 09, 2025 6:28 pm Post subject: |
|
|
| logrusx wrote: | | mudspitease wrote: | Hello,
I want to make sure that my make.conf file is optimized properly...
13th Gen Intel(R) Core(TM) i5-1334U, 10 cores / 12 threads + 16gb ram on nvme
I currently have a few programs that use | Code: | | jumbo-build, -03, and LTO, |
|
As a start, remove LTO and change O3 to O2. It's not worth neither the trouble, nor the performance gains if at all noticeable. Both of them actually.
| mudspitease wrote: |
I commented out my EMERGE_DEFAULT_OPTS in my MAKE.CONF.... but should I put it in a package.use file for libreoffice, for instance? Or, if I were to use /etc/portage/env like you mention, would I put it as -j4, per my 16 gb of RAM? |
No. It's not package specific, it determines how many packages are compiled concurrently, hence "emerge" jobs. Multiply that by the number of make jobs and you get maximum total number of threads that can be spawned on your CPU. That's way too many.
Last but not least, have you considered using the binhost?
Best Regards,
Georgi |
So, if I have 12 threads, if I put it in, I would divide that by number of MAKE jobs to get the EMERGE_DEFAULT_OPTS?
I would have to do half my RAM, since that's less than my cores... so, 12/8=....1.5 for my EMERGE_DEFAULT_OPTS? |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2816
|
| Posted: Sun Mar 09, 2025 7:12 pm Post subject: |
|
|
| mudspitease wrote: |
So, if I have 12 threads, I would divide that by number of MAKE jobs to get the EMERGE_DEFAULT_OPTS?
|
No. EMERGE_DEFAULT_OPTS contains options that will be passed to your every emerge command, one of them can be --jobs which instructs portage how many parallel emerges it should run. Make jobs are how many thread will be run by each emerge. Generally, because it's a concept from GNU make, which sometimes has no equivalent. Some build systems will never run more than one thread, some will never respect such hints. But generally, almost all ebuilds strive to transform that into whatever option the build system of the package respects if at all.
Yes, it's a good idea to keep the product of the two below the number of hardware threads (logical cores) but better leave emerge jobs alone, unless you're monitoring the process all the time and you can intervene wherever only a single package is available to increase the number of threads and decrease the number of emerge jobs and vise versa. It's not worth it, been there, done that. Better leave emerge jobs alone and all cores available to a single job. This is the most straightforward way of handling it.
Also EMERGE_DEFAULT_OPTS can serve surprises. You can think of one thing, forget about one option there and run a command that does not act as expected. Example - I had added -av there and one night leaving for the bedroom I issued emerge <a very large package here>. Dependency resolution was a long process back then, so I figure I'll issue the command without -a/--ask and it'll continue automatically. Surprise in the morning - I see the confirmation prompt. A whole night wasted.
Also, this is a mobile CPU, I'm asking you once again, have you considered using the binhost? Do you know what it is?
Best Regards,
Georgi |
|
| Back to top |
|
|
mudspitease
n00b
Joined: 09 Mar 2025
Posts: 4
|
| Posted: Sun Mar 09, 2025 7:18 pm Post subject: |
|
|
| logrusx wrote: | | mudspitease wrote: |
So, if I have 12 threads, I would divide that by number of MAKE jobs to get the EMERGE_DEFAULT_OPTS?
|
No. EMERGE_DEFAULT_OPTS contains options that will be passed to your every emerge command, one of them can be --jobs which instructs portage how many parallel emerges it should run. Make jobs are how many thread will be run by each emerge. Generally, because it's a concept from GNU make, which sometimes has no equivalent. Some build systems will never run more than one thread, some will never respect such hints. But generally, almost all ebuilds strive to transform that into whatever option the build system of the package respects if at all.
Yes, it's a good idea to keep the product of the two below the number of hardware threads (logical cores) but better leave emerge jobs alone, unless you're monitoring the process all the time and you can intervene wherever only a single package is available to increase the number of threads and decrease the number of emerge jobs and vise versa. It's not worth it, been there, done that. Better leave emerge jobs alone and all cores available to a single job. This is the most straightforward way of handling it.
Also EMERGE_DEFAULT_OPTS can serve surprises. You can think of one thing, forget about one option there and run a command that does not act as expected. Example - I had added -av there and one night leaving for the bedroom I issued emerge <a very large package here>. Dependency resolution was a long process back then, so I figure I'll issue the command without -a/--ask and it'll continue automatically. Surprise in the morning - I see the confirmation prompt. A whole night wasted.
Also, this is a mobile CPU, I'm asking you once again, have you considered using the binhost? Do you know what it is?
Best Regards,
Georgi |
Thanks for the info! Seems like it's best to just leave it out entirely, then! Seems to be more trouble than it's worth....especially due to what pietinger said...
I have... but I want to try the "full Gentoo experience" just compiling, first. Do precompiled binaries have USE flags, still? I believe Vivaldi does, and seemed to respect them...? But that's the only one I've tried.
Last edited by mudspitease on Sun Mar 09, 2025 7:23 pm; edited 1 time in total |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23171
|
| Posted: Sun Mar 09, 2025 7:23 pm Post subject: |
|
|
| Binaries from the binhost have USE flags, and Portage will only pick the binary from the binhost if (a) your requested USE flags match one of the variants from the binhost or (b) you tell Portage to ignore your USE flags and take a binary package that does not match. Regarding variants, for some packages, the Gentoo binhost builds the same package version multiple times with differing USE flags, in an attempt to cater to users with different preferences. The binhost does not build all possible combinations, so unusual flag choices may leave you without a matching binhost package. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54988
Location: 56N 3W
|
| Posted: Sun Mar 09, 2025 7:31 pm Post subject: |
|
|
mudspitease,
A good limit to the number oy concurrent make threads is the lower of RAM GB/2 on threads.
For you, that's 8.
| Code: | # Make options
MAKEOPTS="-j7 -l14"
EMERGE_DEFAULT_OPTS="--jobs=12 --load-average=15 --keep-going --verbose --with-bdeps=y" |
Here you allow 7 concurrent make threads and at the same time you allow portage to run 12 packages at the same time.
That's 84 concurrent make threads.
Not all build systems honour MAKEOPTS. so it could be worse than that.
You have neither the RAM nor the threads. That will force you into swapping and CPU cache thrashing, both of which contribute nothing to the task at and and only slow things down.
jumbo-build and LTO are both memory hogs.
jumbo-build gets a package that supports it to build faster if you have the real RAM. If it pushes you into swapping, it will be much slower.
LTO uses lots of memory to not link things that will not be used. Binaries load faster, as they will be smaller. Sometimes it gets it wrong and the resulting binaries don't work.
Neither are worth the trouble.
-O3 is interesting. It extends the build time by adding optimisations that make the code bigger in an attempt to make it faster.
It's a double edged sword. The bigger code may not fit into the CPU cache, so the resulting code is slower as the CPU cannot cache everything it need to at the same time.You cache misses go up. Further -O3 breaks things. It should only be used where the upstream have determined that is both safe and effective.
ccache is useful for developers building the same C or C++ code over and over with small changes.
The first build is slowed down by the cache creation process. Subsequent builds are faster as the cache is used.
If that,s not you ccache is an overhead that you don't need.
There there are the problems of ccache returning the wrong thing.
If you want to push the boundaries. use | Code: | # Make options
MAKEOPTS="-j12"
EMERGE_DEFAULT_OPTS="--jobs=1 --keep-going --verbose --with-bdeps=y" |
and throttle back MAKEOPTS= to 8 or less for big packages that push you into swapping or complain about lack of RAM.
You appear to know how to do that.
Drop, -O3 back to -O2. Do not use LTO, jumbo build or ccache.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
