(CLOSED) Parallel make of kernel?

Helena · Last edited by Helena on Sat Sep 12, 2020 7:26 pm; edited 1 time in total

Just recently I found out that I could dramatically improve the time needed to build a new kernel by

RayDude · Posted: Fri Sep 11, 2020 10:33 pm Post subject:

I've done make -j=n (where n>1) on the kernel for at least 10 years. I'm currently using -j16 for my Ryzen 7 4800H work laptop.

I've never had any problems with make with jobs set to large numbers on the kernel.

I suspect your problem is related to something else and somehow rebooting fixed the issue, not rebuilding the kernel.

Have you looked at dmesg and /var/log/messages? after starting X windows to see if there are any error messages?

You could also look at /var/log/Xorg.0.log
_________________
Some day there will only be free software.

CaptainBlood · Advocate Joined: 24 Jan 2010 Posts: 3999

Here's

mike155 · Posted: Fri Sep 11, 2020 11:43 pm Post subject:

The number 13 is considered an unlucky number in some countries.

You may choose any number for 'make -j N'. But not 13! Sometimes that leads to broken kernels.

Helena · Posted: Sat Sep 12, 2020 1:24 pm Post subject:

NeddySeagoon · Posted: Sat Sep 12, 2020 2:10 pm Post subject:

Helena,

The 'optimum' value for X in MAKEOPTS="-jX" is no longer tied to just CPU cores.
Large values of X can lead to swapping because there is not enough real RAM to support X parallel make threads.

A few C++ packages need 2G RAM per thread, so gcc may want 26G RAM with -j13.
If you build in tmpfs too, you need a lot of RAM.

Swapping does not mean writing pages of dynamically allocated RAM to swap space. The kernel has other ways of swapping too.

I've built one or two kernels on a 96 core box using -j100. They seem to work.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Helena · Posted: Sat Sep 12, 2020 5:59 pm Post subject:

NeddySeagoon,

Thanks for clarifying this. I've noticed with my 16 GB RAM that make -j13 for the kernel never used more than 1.5 GB so that would be plenty. Moreover, swapping because of RAM shortage should not lead to errors, just take longer, right?

On the contrary, this would indicate that I could increase n in "make -j<n>" for kernel builds to the point where swapping starts to take place...

NeddySeagoon · Posted: Sat Sep 12, 2020 6:10 pm Post subject:

Helena,

Thats right, unless the kernel invokes the Out of Memory Manager, which may kill your compile job. You notice that though.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Helena · Posted: Sat Sep 12, 2020 6:13 pm Post subject:

To further increase my curiosity, I should perhaps repeat and elaborate on what exactly happened the past few days/weeks with "make -j13":

When upgrading gentoo-sources (and therefore for the first time building this particular kernel version) I have no problems.
When I have changed my .config afterwards (especially, but perhaps not only, in the Drivers section) the build only recompiles kernel parts affected by my .config changes (presumably). This sometimes (more often than not) leads to a problematic reboot (I suspect the kernel is faulty but have not as yet definite proof)
When I precede the next kernel build by "make clean" the reboot with the new kernel is just fine.

So my theory is that the second step, an "incremental" kernel build, is not bugfree with "make -j<n>" when n>1. It may sound weird and I hope I am wrong.

toralf · Posted: Sat Sep 12, 2020 6:32 pm Post subject:

Helena · Posted: Sat Sep 12, 2020 7:25 pm Post subject: