| View previous topic :: View next topic |
| Author |
Message |
Spanik
Veteran
Joined: 12 Dec 2003
Posts: 1059
Location: Belgium
|
 Posted: Sat Dec 03, 2022 1:44 pm Post subject: building new kernel is very slow |
 |
|
I read another thread where the update and build process was slaow and that was a reason for the TS to put Gentoo aside. Now I could understand that as I have the same feelings, update and building a kernel has become very slow over the years. I stated that in the thread and was told I probably did something wrong. Now that is very possible. So I ran a few things now and timed them. I'm now very confused from what I found.
Let's start with the system. It is a older Lenovo E530 laptop with an i7-3632QM 2.2GHz (4 cores/8 threads), 8GB ram and a 120GB Plextor PX-128M5M sata SSD. So nothing special, rather older run of the mill system.
It hadn't been updated for a couple of weeks due to the job and a holiday. It was running a self configured kernel 5.15.74. So I started as I always do:
| Code: | emerge --sync
emerge -puvUD @world (nothing blocking showed)
emerge -uvUD @world |
289 packages needed an update. This update ran for 1 day and 7 hours. Which I find very, very long. Ok, it included Libreoffice and Firefox but rust is the binary. Those who are interested can find the emerge.log of this here https://quirinus.one/gentoo/emerge_log.txt.
As there was a new kernel I decided to update that one and time it. My make.conf:
| Code: | # These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
COMMON_FLAGS="-march=native -O2 -pipe -fomit-frame-pointer"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
# NOTE: This stage was built with the bindist Use flag enabled
PORTDIR="/var/db/repos/gentoo"
DISTDIR="/var/cache/distfiles"
PKGDIR="/var/cache/binpkgs"
# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C
# toegevoegd
CHOST="x86_64-pc-linux-gnu"
MAKEOPTS="-j5"
INPUT_DEVICES="libinput synaptics virtualbox"
VIDEO_CARDS="intel nouveau virtualbox"
USE="X -systemd -pulseaudio -mono -oss fftw jack pdf exif jpeg bindist -bluetooth" |
I ran as usual:
| Code: | eselect kernel list
Available kernel symlink targets:
[1] linux-5.15.11-gentoo
[2] linux-5.15.23-gentoo
[3] linux-5.15.26-gentoo
[4] linux-5.15.32-gentoo-r1
[5] linux-5.15.41-gentoo
[6] linux-5.15.59-gentoo
[7] linux-5.15.69-gentoo
[8] linux-5.15.74-gentoo *
[9] linux-5.15.80-gentoo
eselect kernel set 9
cd /usr/src/linux
cp /usr/src/linux-5.15.74-gentoo/.config /usr/src/linux/.config
make oldconfig |
For those interested, the .config is here: https://quirinus.one/gentoo/.config
Then as usual "make && make modules_install", noted the time when I pressed Enter. I also opened another terminal with "top".
Building this took 2 hours and a couple minutes. But what I found strange is that in "top" there always was a single line at the top of the list of processes with "cc1" at 100%. Total cpu was maximal 12% which fits with a 4 core/8 thread cpu running a single thread at max cpu.
So it seems that "make && make modules_install" does not care about the "MAKEOPTS="-j5".
To make sure I did the following:
| Code: |
make mrproper
cp /usr/src/linux-5.15.74-gentoo/.config /usr/src/linux/.config
make oldconfig
time make -j5
....
real 31m19.879s
user 145m21.906s
sys 9m28.036s |
So about 4 times faster and indeed in "top" I could now see 3-5 threads of cc1 running at 100% and total cpu usage at 50-60%.
Now, then what is wrong with my make.conf or .config or what else am I doing wrong to get make ignoring the make.conf? I guess that also means that the update @world could also be a lot faster.
_________________
Expert in non-working solutions |
|
| Back to top |
|
 |
asturm
Developer
Joined: 05 Apr 2007
Posts: 9373
|
| Posted: Sat Dec 03, 2022 1:47 pm Post subject: |
|
|
| pietinger wrote: | | (when compiling the kernel it does not take MAKEOPTS=... from make.conf) |
| NeddySeagoon wrote: | | Its worse than that ... make.conf is not consulted at all. |
Only emerge cares about your make.conf. You build your kernel manually, evidently, by not using emerge. |
|
| Back to top |
|
|
Spanik
Veteran
Joined: 12 Dec 2003
Posts: 1059
Location: Belgium
|
| Posted: Sat Dec 03, 2022 2:07 pm Post subject: |
|
|
| asturm wrote: | | pietinger wrote: | | (when compiling the kernel it does not take MAKEOPTS=... from make.conf) |
| NeddySeagoon wrote: | | Its worse than that ... make.conf is not consulted at all. |
Only emerge cares about your make.conf. You build your kernel manually, evidently, by not using emerge. |
So you always need to add that manually? And make.conf is only for emerge which I find strange, I would think that it is a configuration file for "make" as a utility. Or is that another make.conf that hides somewhere else?
While I was doing this on the laptop, I also had the desktop doing its "emerge -uvUD @world" and a terminal with "top". Indeed in that case I could see often multiple instances (>10) of "cc1" and "cc1plus". But when it was python then at most 2-3 instances. Rust was mixed, going from 2-3 to a lot of (couldn't count them fast enough). Anyway, the desktop did its update in 3 hours 45 minutes. Better than the laptop but with 24 cores/48 threads and 64GB ram and a more performant SSD I would hope so.
_________________
Expert in non-working solutions |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5593
Location: Bavaria
|
| Posted: Sat Dec 03, 2022 3:35 pm Post subject: |
|
|
| Spanik wrote: | | So you always need to add that manually? [...] |
Yes. The kernel is an execption, because it brings its own build-system. For your desktop you can try from "make -j32" until "make -j48" (dont use "make -j49") ... why ?
... this paramter has one maximum with (logical) cores of CPU, AND
It also depends how many RAM is consumed (you have "only" 64 GB) ... if you can use "make -j48" for your kernel, this must not be true for MAKEOPTS in make.conf because there exists some packages (e.g. rust) wich needs upt to 4 GB / job ... |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5593
Location: Bavaria
|
| Posted: Sat Dec 03, 2022 3:53 pm Post subject: |
|
|
I also took a look into your .config. Almost very fine, but two options are really bad:
| Code: | CONFIG_X86_5LEVEL=y
CONFIG_DEBUG_FS=y |
I would disable both. Maybe you want take a look into another thread from today: https://forums.gentoo.org/viewtopic-t-1159535-highlight-.html |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23305
|
| Posted: Sat Dec 03, 2022 4:51 pm Post subject: |
|
|
| Spanik wrote: | | So you always need to add that manually? And make.conf is only for emerge which I find strange, I would think that it is a configuration file for "make" as a utility. Or is that another make.conf that hides somewhere else? |
The name is a bit unfortunate, but make.conf has no effect on GNU make, just as a hypothetical /etc/portage/cmake.conf would not automatically be used by CMake, nor /etc/portage/scons.conf automatically be used by SCons. As a convenience to the user, Portage's helper program emake passes the value of MAKEOPTS as a command line argument to make. Ebuild best practices encourage using emake over make. Similarly, as a courtesy to the user, many maintainers write their ebuilds to try to extract the most interesting options from MAKEOPTS and pass equivalent directives to other non-Make systems, to give you as much value as possible even when upstream is not using GNU make. | Spanik wrote: | | But when it was python then at most 2-3 instances. |
Parallelism is always limited by what the upstream build can achieve. Some build systems are not well prepared for high levels of parallelism, and simply cannot find enough work to do in parallel to reach the limit you set. When that happens, you may see it running far fewer processes than you allowed, because your setting is a maximum that you will permit, not a minimum that the build system is guaranteed to reach. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Dec 03, 2022 5:11 pm Post subject: |
|
|
The problem is the 8GB ram; using more than -j3 threads is a bad idea with that amount of ram, in general (e.g. for huge c++ projects). For the kernel, a higher number might be used. For this reason, it might not even be a good idea to re-use the value from /etc/portage/make.conf for the kernel.
That being said, you can of course use a script which compiles the kernel and reads the arguments from /etc/portage/make.conf. sys-kernel/kernel from the mv overlay does this, though this (as most kernel build scripts) is somewhat specialized to particular needs and might not necessarily fit your use case. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55142
Location: 56N 3W
|
| Posted: Sat Dec 03, 2022 7:41 pm Post subject: |
|
|
Spanik,
Portage, via ebuilds and emerge is a wrapper around the upstream build systems that the individual packages use.
When you compile the kernel, that wrapper is not used. It's just you and the kernel build system.
Very few packages can get more than 30 make threads in flight at one time.
I did try on a 96 core box. That was a few years ago now, where I used to use -j100 :)
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Sat Dec 03, 2022 9:34 pm Post subject: |
|
|
| NeddySeagoon wrote: | Spanik,I did try on a 96 core box. That was a few years ago now, where I used to use -j100  |
Q: How long would it take to compile the Linux kernel with all modules enabled on one of the TOP 500 computers (with 8,730,112 cores)?
A: 5 seconds:
- 2.9 seconds for make to figure out what to do
- 0.1 seconds for GCC to compile the kernel and all modules
- 2 seconds for the linker to bind everything together into a single binary image
 |
|
| Back to top |
|
|
Spanik
Veteran
Joined: 12 Dec 2003
Posts: 1059
Location: Belgium
|
| Posted: Sat Dec 03, 2022 10:08 pm Post subject: |
|
|
Thanks for all the replies. I have been using Gentoo now for years but still learn things almost every day. Let's take your points.
| pietinger wrote: | I also took a look into your .config. Almost very fine, but two options are really bad:
| Code: | CONFIG_X86_5LEVEL=y
CONFIG_DEBUG_FS=y |
|
Great to get some feedback on this, I'm very happy that you think it is "almost very fine". I would like to take your advise, but the issue is that I have no idea how such options come into the .config. You see, I do a manual config. This involves going through the menuconfig and removing everything I'm sure of I don't need. Like when I use an AMD cpu, I remove everything that relates to an Intel cpu and vice-versa. Likewise I remove everything of drivers for chips or cards I'm sure that are not in the system. No need to build for a Renesas chip if you are sure you have an Intel phy. Or "Amateur radio support".
But this stops when it comes to things like "SLUB allocator". Or "deadline sheduler". There I'm lost. If the help says "Y if you're insure" then "Y" it is. And the options are just ballooning! I mean what is "Eckelman SIOX support" or "DAX: direct access to differentiated memory"? If you are a user then you just can't keep up with all this.
So you are probably right that those 2 are bad options. But I have no idea at all where they come from. And even less idea what they do. So how would I be able to know they are bad?
Don't get me wrong, I really appreciate it all. But it is getting far to complicated to follow. And I really can believe that 99% of the users are giving up and just going with a default genkernel.
| Hu wrote: | | Parallelism is always limited by what the upstream build can achieve. Some build systems are not well prepared for high levels of parallelism, and simply cannot find enough work to do in parallel to reach the limit you set. When that happens, you may see it running far fewer processes than you allowed, because your setting is a maximum that you will permit, not a minimum that the build system is guaranteed to reach. |
I do understand that what I set is a maximum. But I was always convinced that portage is the build system. It really is becoming an issue that parallelism is not becoming a goal in itself for developers. Speed of cpu's seems to have hit a limit and since then we are offered more and more cores. But using them seems to become the bottleneck. When I see that LibreOffice Calc is only using a single core to calculate large spreadsheets then there is a problem. Graphical applications like Gimp also have this issue. I understand that not all problems allow themselves to be parallelised but surely there is a lot of progress still possible.
A bit of a hobbyhorse of me. I got interested in this when Transputers came out. Still have the Inmos "instruction set" and "applications handbook" in my bookshelf. (yes, that dates me as well) That was one of the reasons I went to AMD when they brought out the Athlons and Opterons, the interconnections between cpu's was right out of the Transputer ideology.
| mv wrote: | | The problem is the 8GB ram; using more than -j3 threads is a bad idea with that amount of ram, in general (e.g. for huge c++ projects). For the kernel, a higher number might be used. For this reason, it might not even be a good idea to re-use the value from /etc/portage/make.conf for the kernel. |
| pietinger wrote: | | It also depends how many RAM is consumed (you have "only" 64 GB) ... if you can use "make -j48" for your kernel, this must not be true for MAKEOPTS in make.conf because there exists some packages (e.g. rust) wich needs upt to 4 GB / job ... |
The 8GB ram is indeed a limit for the laptop. And yes, the desktop could benefit from another 64GB added as that would double the memory bandwidth by using another 4 ram channels extra. Only thing is when I check memory use it never gets that far. The kernel only went up to a couple of GB and even the complete @world on the desktop never broke more than 10GB. So yes, memory may be the limit but it rarely is.
| mike155 wrote: | | NeddySeagoon wrote: | | Spanik,I did try on a 96 core box. That was a few years ago now, where I used to use -j100 |
Q: How long would it take to compile the Linux kernel with all modules enabled on one of the TOP 500 computers (with 8,730,112 cores)?
A: 5 seconds:
- 2.9 seconds for make to figure out what to do
- 0.1 seconds for GCC to compile the kernel and all modules
- 2 seconds for the linker to bind everything together into a single binary image
|
I spend some time looking at the "emerge -uvUD @world" output going by (yes, it was a very productive afternoon) and it seems to me that the things taking relatively most time where
- configuring
- checking if gcc supported this or that option
All these are sh scripts or python and they are single threaded. Next were unpacking files and copying them to replace older ones.
But again, thank you all for the instructive feedback.
_________________
Expert in non-working solutions |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55142
Location: 56N 3W
|
| Posted: Sat Dec 03, 2022 10:34 pm Post subject: |
|
|
Spanik,
| Fred Brooks wrote: | | The Bearing of a Child Takes 9 Months No Matter How Many Women Are Assigned |
There are some things that cannot be parallelised.
Calculating a spreadsheet is one of them. All the cell references have to be followed in the right order.
You need to dip into The Mythical Man Month. Which is where that quote comes from but you may already know it.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23305
|
| Posted: Sun Dec 04, 2022 12:01 am Post subject: |
|
|
| Spanik wrote: | | But I was always convinced that portage is the build system. |
Portage is a wrapper that provides a standardized interface to invoke a variety of different build systems, of varying quality and capability. In very simple cases, the ebuild may be the build system, but usually Portage's ebuild delegates all the work to upstream's build system. | Spanik wrote: | | It really is becoming an issue that parallelism is not becoming a goal in itself for developers. |
I agree. Part of the problem is that many people like to think in procedural terms, rather than in terms of dependency graphs. Tooling then follows that mental model. However, for best parallelism, you want the build to be expressed in terms of a dependency graph, with as fine a granularity as people can reasonably handle. | Spanik wrote: | I spend some time looking at the "emerge -uvUD @world" output going by (yes, it was a very productive afternoon) and it seems to me that the things taking relatively most time where
- configuring
- checking if gcc supported this or that option
All these are sh scripts or python and they are single threaded. |
Yes, and in principle those are great places to do better. Many configure checks do not depend on other configure checks, and could be run in parallel. However, the preferred way of writing configure scripts does not allow the interpreter (usually, a Bourne shell) to see that, so it runs them serially. Rewriting the configure script as a list of goals ("Does gcc support -fFOO?", "Does gcc support -fBAR?"), each of which sets a result variable that can be retrieved later, could let configure go much faster. Unfortunately, that is a massive amount of work, and most developers will take the view that doing it might save them an hour of time over their lifetime as the maintainer, then decide that is insufficient savings to justify the effort. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55142
Location: 56N 3W
|
| Posted: Sun Dec 04, 2022 9:50 am Post subject: |
|
|
Hu,
Once upon a time there was confcache (sp)?
However, it was unreliable and went away.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Spanik
Veteran
Joined: 12 Dec 2003
Posts: 1059
Location: Belgium
|
| Posted: Sun Dec 04, 2022 1:21 pm Post subject: |
|
|
| NeddySeagoon wrote: | Spanik,
| Fred Brooks wrote: | | The Bearing of a Child Takes 9 Months No Matter How Many Women Are Assigned |
There are some things that cannot be parallelised.
Calculating a spreadsheet is one of them. All the cell references have to be followed in the right order.
You need to dip into The Mythical Man Month. Which is where that quote comes from but you may already know it. |
Have read it quite a while ago.
Not all problems can be parallelised as you say. But when I see (and make) a typical spreadsheet, the dependence on cell references is almost always very limited. Like you have a row where each cell is dependent on the previous cells, but never on those above or below. Likewise, I always have vlookups where you look up the value of a cell in a range that does not change while doing the lookup. And looking up a value of a cell does not depends on the cell above or below. It could be that the code to take those cases into account might be worse than just keeping it sequential. But there are a lot of cases that could benefit I think.
_________________
Expert in non-working solutions |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5593
Location: Bavaria
|
| Posted: Sun Dec 04, 2022 5:46 pm Post subject: |
|
|
| Spanik wrote: | | Thanks for all the replies. I have been using Gentoo now for years but still learn things almost every day. [...] |
You are very welcome !
To your 2nd sentence I can only say: Me too !
| Spanik wrote: | | [...] And I really can believe that 99% of the users are giving up and just going with a default genkernel. |
Yes ! Fully agree ! I understand it also.
The problem is ... there is no book you can learn ... there are hundreds of sources of information ... in every source you have just some information ... and some other you have only heard by other IT people, or accidently read in an article from some Linux sites, or even in a forum post ... short: Experience is if you have to collect yourself all information ... sadly but true
| Spanik wrote: | | [...] But I have no idea at all where they come from. And even less idea what they do. So how would I be able to know they are bad? |
More sadly: All configuration options are not statically; they can change from version to next version. Of course it is easier if you have read all information beginning with kernel version 0.1 ... but there are not many people which have done this ... please dont believe I know all options ... I hope I know all relevant options ... until the next version  |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55142
Location: 56N 3W
|
| Posted: Mon Dec 05, 2022 10:42 am Post subject: |
|
|
This post from 2005 remains true.
The more detail you put into a document, the faster it gets out of date.
With regards | Quote: | | large pot of strong coffee |
you may need several now. :)
-- edit --
Hmmm .... | Quote: | | and this too shall pass away |
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
