| View previous topic :: View next topic |
| Author |
Message |
Kream
n00b
Joined: 19 Apr 2002
Posts: 30
Location: New Delhi, India
|
 Posted: Wed Dec 11, 2002 11:13 pm Post subject: Compiling kernels with distcc |
 |
|
Compiling kernels with distcc
I test a number of kernels including vanillla, mandrake, lolo-sources, linux-beta and, of course, gentoo-sources so I find myself compiling the kernel lots of times. Even if you use distcc to emerge packages like I do, kernel compilations aren't distributed because the kernel make doesn't use /etc/make.conf. This is how I use distcc to radically speed up my compile times:
Overview
- Set up distcc
- Emerge distcc on the computers
- Start distccd on the slave systems
- Configure the kernel
- Edit Makefile
- make the kernel
Set up distcc
Set up distcc on all computers that you want to use, including the host computer. To accomplish this, a simple emerge command suffices:
Once done, make sure you start the distccd daemon on all the computers that you want compilation to be distributed to. This is done by running this command on all of them:
| Code: | | /etc/init.d/distccd start |
Configure the kernel
Change to /usr/src/linux and configure the kernel using make menuconfig or make xconfig (does anyone still use make config ????  ). Save your config and and do a make dep.
Edit the Makefile
Fire up your favourite editor and load /usr/src/linux/Makefile. Find this section: | Code: | #
# Include the make variables (CC, etc...)
#
|
And change "gcc" in the CC line to
| Code: |
CC = $(CROSS_COMPILE)distcc
|
Append this line to the section:
| Code: |
DISTCC_HOSTS=slave1 slave2 slave3 ... slaveN
|
You can enter the hostnames of the computers if you're sure that they resolve. If they don't, you can use their IP addresses. It's best to enter the computers in descending order of CPU power availability.
If you want the local machine to participate in the compile, remember to start distccd on it as well and add localhost to the list of DISTCC_HOSTS. I don't recommend this, though, as the local machine will already be doing the preprocessing and co-ordinating.
Add DISTCC_HOSTS to the line which says | Code: | | export VERSION PATCHLEVEL SUBLEVEL EXTRAVERSION KERNELRELEASE ARCH \ |
and save and exit.
Make the Kernel
When making the kernel, do a
where Q is the number of parallel makes that you want to run. I recommend that Q be the number of slaves you have + 2. Examine the terminal's output to make sure that distcc is being called instead of gcc. If you're a gkrellm2 user like I am, you'll notice lots of data traffic outbound from your machine (preprocessed code) and a small amount of inbound traffic (compiled code).
I found my kernel being built *much* faster with distcc.
Aniruddha Shankar
Bangalore, India.
Vipassana Works. |
|
| Back to top |
|
 |
kyron
Apprentice
Joined: 26 Aug 2002
Posts: 198
Location: Montreal, Qc.
|
| Posted: Fri Apr 11, 2003 1:45 am Post subject: |
|
|
To add to this wonderful document of yours, here is a script which I use each time I compile a kernel....just to make sue I don't forget anything 
| Code: | make dep && make clean
make -j6 bzImage
make -j6 modules
make modules_install
mount /boot/
#keep a date stamped copy ;-P
cp /boot/bzImage /boot/bzImage.`date +%F_%k-%M`
cp arch/i386/boot/bzImage /boot/bzImage
umount /boot/
#Making sure alsa-driver
export ALSA_CARDS=ens1370
emerge alsa-driver nvidia-kernel nvidia-glx
depmod -a
update-modules |
_________________
M$ Windows: When in doubt, REBOOT
Linux: When in doubt, RTFM  |
|
| Back to top |
|
|
kyron
Apprentice
Joined: 26 Aug 2002
Posts: 198
Location: Montreal, Qc.
|
| Posted: Fri Apr 11, 2003 1:49 am Post subject: |
|
|
One other thing I forgot, I also added ccache to the following line:
| Code: | | CC = $(CROSS_COMPILE)ccache distcc |
_________________
M$ Windows: When in doubt, REBOOT
Linux: When in doubt, RTFM |
|
| Back to top |
|
|
matroskin
Apprentice
Joined: 21 Jan 2003
Posts: 214
|
| Posted: Wed Jul 16, 2003 1:31 pm Post subject: Re: Compiling kernels with distcc |
|
|
| Kream wrote: | does anyone still use make config ????
|
yes, i do. make config and make oldconfig are pretty good for configuring kernels throu scripts. |
|
| Back to top |
|
|
kurifu
Tux's lil' helper
Joined: 28 Jul 2003
Posts: 77
|
| Posted: Sun Mar 28, 2004 10:00 pm Post subject: distcc while using genkernel. |
|
|
For anyone who wishes to use distcc and genkernel to make their kernel, all you have to do is follow your normal distcc setup (emerge -v distcc, visit the distcc howto link provided at the end of the emerge), follow the instruction at the top of this thread to prepare your kernel make file and use the following command with genkernel:
| Code: | | genkernel all --menuconfig --makeopts="-j5" |
Obviously you can make a few changes to the genkernel command line, depending on how you wish to build the kernel. The important command is --makeopts="-j5" which actually sets up the number of parallel makes at once, needed to push some of the processes to your host. |
|
| Back to top |
|
|
LordVan
Developer
Joined: 28 Nov 2002
Posts: 67
Location: Austria
|
| Posted: Thu Nov 22, 2012 1:14 pm Post subject: |
|
|
Thanks for the well written instructions  I was looking for just this (my laptop is too slow to compile kernel in reasonable time -- and also tends to overheat  )
_________________
I don't suffer from insanity. I enjoy every minute of it. |
|
| Back to top |
|
|
Yamakuzure
Advocate
Joined: 21 Jun 2006
Posts: 2297
Location: Adendorf, Germany
|
| Posted: Tue Jul 12, 2016 7:32 am Post subject: Re: Compiling kernels with distcc |
|
|
| Kream wrote: | And change "gcc" in the CC line to
| Code: |
CC = $(CROSS_COMPILE)distcc
|
|
Hi everybody!
If you came here, because you wanted to have distcc to help to compile a kernel on your Raspberry Pi 3, like me, then please notice, that distcc will fail when used like this, but works exceptionally well with: | Code: | # Make variables (CC, etc...)
CC = $(CROSS_COMPILE)armv7a-hardfloat-linux-gnueabi-gcc |
You should have set up your system like described in the Raspberry Pi/Cross building wiki entry.
_________________
Edited 220,176 times by Yamakuzure |
|
| Back to top |
|
|
Yord
n00b
Joined: 21 Aug 2016
Posts: 3
|
| Posted: Sun Aug 21, 2016 2:11 am Post subject: Re: Compiling kernels with distcc |
|
|
| Yamakuzure wrote: | | Kream wrote: | And change "gcc" in the CC line to
| Code: |
CC = $(CROSS_COMPILE)distcc
|
|
Hi everybody!
If you came here, because you wanted to have distcc to help to compile a kernel on your Raspberry Pi 3, like me, then please notice, that distcc will fail when used like this, but works exceptionally well with: | Code: | # Make variables (CC, etc...)
CC = $(CROSS_COMPILE)armv7a-hardfloat-linux-gnueabi-gcc |
You should have set up your system like described in the Raspberry Pi/Cross building wiki entry. |
Hi,
I came here for the RPI 3, but after much searching (albeit above didn't work for me, I think my distcc must be messed up somehow) I've found a simpler way!
e.g. make CC="distcc" bzImage
I don't know if you need to add the makeopts -- isn't that sourced through /etc/portage/make.conf? |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54585
Location: 56N 3W
|
| Posted: Sun Aug 21, 2016 9:39 am Post subject: |
|
|
Yord,
Kernel builds do not use /etc/portage/make.conf
You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.
For readers wanting kernels for arm and the like, cross compiling the kernel is a good introduction co the minefield that can be cross compiling.
The kernel is well behaved under cross compiling.
Do take care with | Code: | | make modules install |
though. Its a really bad idea to install arm kernel modules on your x86_64 build host.
It might be harmless, it might not.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Yord
n00b
Joined: 21 Aug 2016
Posts: 3
|
| Posted: Sun Aug 21, 2016 12:17 pm Post subject: |
|
|
| NeddySeagoon wrote: | Yord,
Kernel builds do not use /etc/portage/make.conf
You need to add -jN to your command, at least, or make will only run one concurrent job, even though it will use distcc.
For readers wanting kernels for arm and the like, cross compiling the kernel is a good introduction co the minefield that can be cross compiling.
The kernel is well behaved under cross compiling.
Do take care with | Code: | | make modules install |
though. Its a really bad idea to install arm kernel modules on your x86_64 build host.
It might be harmless, it might not. |
Ah, thanks. I actually have problems with compiling the way I did - distcc won't recognise the slave and it distributes locally instead.
Just to clarify, I'm trying to compile a kernel from the rpi, using distcc. There wouldn't be a problem with cluttering the build host in this way would there? |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54585
Location: 56N 3W
|
| Posted: Sun Aug 21, 2016 12:52 pm Post subject: |
|
|
Yord,
That's fine. You need to ensure that the Pi and helpers all have the same versions of gcc.
distcc will still be slow. The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.
The Pi3 is slightly different as its WiFi is on a 50MHz 4 bit wide SDIO. That gives a 25MB/sec transfer rate, if it can run the WiFi crypto fast enough.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Yord
n00b
Joined: 21 Aug 2016
Posts: 3
|
| Posted: Mon Aug 22, 2016 8:39 pm Post subject: |
|
|
| NeddySeagoon wrote: | Yord,
That's fine. You need to ensure that the Pi and helpers all have the same versions of gcc.
distcc will still be slow. The Pi has all of its IO on a single 400Mbit/sey USB2 root hub.
The Pi3 is slightly different as its WiFi is on a 50MHz 4 bit wide SDIO. That gives a 25MB/sec transfer rate, if it can run the WiFi crypto fast enough. |
Ah, Thanks! 
I'm still having problems, my rpi conf looks like this:
| Code: | # --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
192.168.1.90/8 |
| Code: | # /etc/conf.d/distccd: config file for /etc/init.d/distccd
DISTCCD_OPTS=""
# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"
# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"
# set this option to run distccd with extra parameters
# Default port is 3632. For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"
# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
# SECURITY NOTICE:
# It is HIGHLY recommended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example: --allow 192.168.0.0/24
# example: --allow 192.168.0.5 --allow 192.168.0.150
# example: --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"
# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15" |
/usr/lib/distcc/bin looks like this:
| Code: | lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 20 18:31 armv7a-hardfloat-linux-gnueabi-gcc-4.9.3 -> /usr/bin/distcc
-rwxr-xr-x 1 root root 85 Aug 20 18:37 armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 c++ -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 cc -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 g++ -> armv7a-hardfloat-linux-gnueabi-wrapper
lrwxrwxrwx 1 root root 38 Aug 20 18:37 gcc -> armv7a-hardfloat-linux-gnueabi-wrapper |
and on the PC doing the loading:
| Code: | # --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
# 127.0.0.1 |
| Code: | # /etc/conf.d/distccd: config file for /etc/init.d/distccd
DISTCCD_OPTS=""
# this is the distccd executable
DISTCCD_EXEC="/usr/bin/distccd"
# this is where distccd will store its pid file
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"
# set this option to run distccd with extra parameters
# Default port is 3632. For most people the default is okay.
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"
# Logging
# You can change some logging options here:
# --log-file FILE
# --log-level LEVEL [critical,error,warning, notice, info, debug]
#
# Leaving --log-file blank will log to syslog
# example: --log-file /dev/null --log-level warning
# example: --log-level critical
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
# SECURITY NOTICE:
# It is HIGHLY recommended that you use the --listen option
# for increased security. You can specify an IP to permit connections
# from or a CIDR mask
# --listen accepts only a single IP
# --allow is now mandatory as of distcc-2.18.
# example: --allow 192.168.0.0/24
# example: --allow 192.168.0.5 --allow 192.168.0.150
# example: --listen 192.168.0.2
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.1.0/24"
#DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.2"
# set this for niceness
# Default is 15
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15" |
and again, /usr/lib/distcc/bin looks like this (shouldn't matter?):
| Code: | lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 armv7a-hardfloat-linux-gnueabi-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-c++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-g++-4.9.3 -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Aug 21 14:34 x86_64-pc-linux-gnu-gcc-4.9.3 -> /usr/bin/distcc
|
I've also rebuild the armv7a-hardfloat toolchain on my pc as a counter-measure.
So, for some reason distcc works perfectly when I'm using emerge, but it doesn't when I'm trying to compile a kernel (on the rpi). Is this the correct way to make a kernel?
| Code: |
make ARCH=arm CC=distcc bcm2709_defconfig
make ARCH=arm CC=distcc oldconfig
make ARCH=arm CC=distcc menuconfig
make ARCH=arm CC=distcc -j8 bzImage modules
make ARCH=arm CC=distcc -j8 modules_install
|
... and obviously copy the kernel img over.
EDIT: So, when I try compiling the kernel, I still get these errors:
| Code: |
wonghau linux # make ARCH=arm CC=distcc -j8 bzImage modules
scripts/kconfig/conf --silentoldconfig Kconfig
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
make[1]: 'include/generated/mach-types.h' is up to date.
CHK include/generated/timeconst.h
CHK include/generated/bounds.h
CHK include/generated/asm-offsets.h
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
CC arch/arm/kernel/smp.o
CC [M] drivers/bcma/main.o
distcc[6332] ERROR: compile arch/arm/kernel/smp.c on 192.168.1.90/8 failed
distcc[6332] (dcc_build_somewhere) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retrying locally
distcc[6332] Warning: failed to distribute arch/arm/kernel/smp.c to 192.168.1.90/8, running locally instead
CC block/cfq-iosched.o
distcc[6395] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] drivers/base/regmap/regmap-i2c.o
distcc[6401] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] mm/zsmalloc.o
distcc[6411] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] crypto/seqiv.o
distcc[6412] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] fs/9p/vfs_super.o
distcc[6414] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6368] ERROR: compile drivers/bcma/main.c on 192.168.1.90/8 failed
distcc[6368] (dcc_build_somewhere) Warning: remote compilation of 'drivers/bcma/main.c' failed, retrying locally
distcc[6368] Warning: failed to distribute drivers/bcma/main.c to 192.168.1.90/8, running locally instead
CC [M] drivers/base/regmap/regmap-spi.o
distcc[6453] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6332] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'arch/arm/kernel/smp.c' failed, retried locally and got a different result.
LD arch/arm/kernel/built-in.o
LD kernel/trace/libftrace.o
CC kernel/trace/trace_events.o
LD drivers/block/built-in.o
distcc[6506] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] drivers/block/pktcdvd.o
distcc[6514] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] drivers/bcma/scan.o
distcc[6536] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
distcc[6368] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'drivers/bcma/main.c' failed, retried locally and got a different result.
CC [M] drivers/bcma/core.o
distcc[6559] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC kernel/trace/trace_export.o
distcc[6569] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] fs/9p/vfs_inode.o
distcc[6583] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] crypto/echainiv.o
distcc[6596] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] crypto/crypto_user.o
distcc[6608] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
CC [M] crypto/cmac.o
distcc[6617] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
|
|
|
| Back to top |
|
|
laineg
n00b
Joined: 13 Nov 2016
Posts: 3
|
| Posted: Thu Nov 17, 2016 1:07 am Post subject: Re: Compiling kernels with distcc |
|
|
| Kream wrote: |
| Code: |
CC = $(CROSS_COMPILE)distcc
|
|
That won't work for the for the cross-compiling scenario, because distcc isn't CHOST-prefixed
| Code: |
CC = distcc $(CROSS_COMPILE)gcc
|
Tells distcc to execute gcc for the target, assuming the cross compiler is on your compile servers
And for ccache:
| Code: |
CC = ccache distcc $(CROSS_COMPILE)gcc
|
_________________
/sbin/fsck: error while loading shared libraries: /lib/libc.so.6: invalid ELF header |
|
| Back to top |
|
|
ville.aakko
Tux's lil' helper
Joined: 06 Aug 2006
Posts: 113
Location: Oulu, Finland
|
| Posted: Fri Mar 08, 2019 3:26 pm Post subject: |
|
|
Hi,
Thanks for this guide.
I've just noticed that for some reason distccd parallelizes really poorly (despite using -j5) - only one thread is compiling on the slave. Perhaps there's just too much overhead on the master to distribute compilation effectively / at all? (localhost is not in /etc/distcc/hosts)
I'm compiling a kernel for amd k6-3 box, and obviously it would be quite slow on the box itself.
Also, indeed the original instructions doesn't work if the slave computer is not the same arch. I needed to use:
| Code: | | CC = distcc $(CROSS_COMPILE)i586-pc-linux-gnu-gcc |
It's a bit confusing, as one can not (or should not) set CROSS_COMPILE in the Makefile, as he/she is not cross-compiling in the Make process, i.e. on the target computer (amd k6-3 in my case) ?
Maybe this should be added to the distcc cross compilation wiki, as Kernels are quite a large compile. AFAIK it is otherwise outdated, too (the wrapper script is not needed anymore?).
EDIT: The wiki is missing instructions to use "--jobs" on the slave configuration. Now that I added it there, it's starting to give more jobs to the slave Still, quite some overhead and the master is not distributing processes as fast as it could - maybe 2-3 threads running on the slave (which could handle more, max jobs set to 6)
_________________
- Ville |
|
| Back to top |
|
|
NightMonkey
Guru
Joined: 21 Mar 2003
Posts: 357
Location: Philadelphia, PA
|
| Posted: Wed Jul 14, 2021 1:22 am Post subject: Kernel Makefile changes? |
|
|
Howdy. It seems that 'CC' has gone away. I see there's "HOSTCC' and 'HOSTCCX' in the Makefile. Anyone in the know on how best to enable Distcc kernel compilation these days? Thanks in advance.
_________________
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
