The Compile 'Cluster'-Raspberry pi

jacealvejetti · n00b Joined: 29 Mar 2019 Posts: 13 Location: Michigan, USA

So in short I have an abundance of pi's, and nothing todo, figured i would recreate a project i did a few years back on the original pis(I used three instead of nine) and see how beneficial it would be to whip them all up in the following configuration:

Currently they are all running the stock raspberry pi 4.19 kernel, though I am hoping to change that soon.

pi3- stock-builder: Handles most of the base system compiling, build package for the rest to update from.
pi3- Nas: Has two 128GB sticks using mdadm to set them in raid 1, NFS exporting packages for binhost and distfile
I believe I read that it may be better to do a full portage export, if anyone can chime in I would be interested, can this be shared by 9 systems at the same time?
pi3- bramble3: nothing special just a builder
pi3- bramble4: nothing special just a builder
pi3+- bramble5: nothing special just a builder
pi3+- bramble6: nothing special just a builder
pi3+- bramble7: nothing special just a builder
pi2- g-pig: test bed
pi2- murmur server

Each is setup as a distcc node and set to use the cluster when compiling.
Is there a way to set distcc to use the hosts in a random order rather then as listed in hosts every time?

As the pi2 are in there you probably guessed I am running this in 32bit mode and compiling for armv7.

I plan to add more duties to the others as i see fit.

I want to move them from the stock kernel to one that is compiled, but wouldn't know where to start or how to automate the roll out process, I figure it could be packaged but again no idea.

I do have a pi0, I may look into seeing if I can get crossdev on each of these to compile for it, mainly to see if I can rather then practicality.

This was my project over the last few days, looking for any suggestions or answers aside from "just use a crossdev/x86" I have had one setup before this was merely using unused hardware.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs

NeddySeagoon · Posted: Sat Mar 30, 2019 10:42 am Post subject:

jacealvejetti,

Welcome to Gentoo

jacealvejetti · n00b Joined: 29 Mar 2019 Posts: 13 Location: Michigan, USA

Thanks for the welcome Neddy!

I should have read the manual... I wonder if that was there a few years back when i was doing this on the original pi 1bs, I appreciate the --randomize insight implemented that about 5 minutes ago based on your suggestion and its working just as i wanted it to.

I am moving on to the kernel, I think I will make one of the spare "bramble" boxes my kernel builder, installing git now then I will attemp to follow the link you sent.

Thanks for the insights, they are much appreciated!
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs

NeddySeagoon · Posted: Sat Mar 30, 2019 4:39 pm Post subject:

jacealvejetti,

The kernel is one of the easiest packages to cross compile.
You will save a lot of time building Pi kernels on, say, an amd64 system.

You can use distcc to build the kernel too. The Pi is not best suited to distcc, as its network port is on USB2.
Its 100Mbit on Pi3 and less and a bit more on Pi3B+ as it has a 1GB network chip. Even there, its on USB2 so 400MBit/sec is doing really well.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

jacealvejetti · n00b Joined: 29 Mar 2019 Posts: 13 Location: Michigan, USA

NeddySeagoon,

I do have the ability to setup a crossdev environment, I have reservations as the last time I did this project on the original pis setting up crossdev was a nightmare, currently its not a question of efficiency, more of killing time while learning.

I 'beat' you to the distcc for the kernel though I appreciate it, after googling and a bit of rtfn I found how to invoke distcc along side of make, I am currently building the rpi-5.0.y across the cluster, after about two hours, give or take me not watching it like a hawk, its done. Now to look into installing it not only on the one that compiled it but rolling it to the others.

I want them to be 'self reliant' while this may not be fast eventually I just plan to use them for micro services, ntp, murmur, maybe a backup dns server, I may even make one sit there a "WHOPPER" by playing freeciv against its self.

Again, I appreciate the insights!
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs

jacealvejetti · n00b Joined: 29 Mar 2019 Posts: 13 Location: Michigan, USA

Follow up:

So I went with nfs mounting the linux src directory from the build pi to the others, from there I just followed the post compile/"install" commands on each pi in the cluster, this did indeed work, my only question is in this configuration am I looking at potential problems in the future?

Currently they all mount the nfs as read only as to not allow the rest to tamper with the source.

Now rocking 5.0.3 over the cluster.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs

NeddySeagoon · Posted: Sun Mar 31, 2019 12:09 pm Post subject:

jacealvejetti,

With distcc in use /usr/src is only used by by the system doing the build. It pushes out preproceesed sources to helpers.
Making /usr/src read only is fine as long as you set to output directory to some read/write space.

You can also share /usr/portage and /usr/portage/distfiles this way but maybe not /usr/portage/packages unless you maintain /etc/portage in sync across all the Pis.
An nfs mount would fix that.

I'm not a big fan of nfs on Raspberry Pi, it works but it soaks up a lot of the IO bandwidth.
The SDcard interface can run at 50MHz, which is 25MB/sec and its not on the USB2 port. That's a large chunk of the Pis IO bandwidth going to waste.
On the Pi3, the WiFi is on the second mmc card interface, so there is scope for networking there, the down side is that any WiFI crypto is done in the CPU, so its CPU load.

I've not tried a second SD card on the Pi, in place of the Wifi. That would involve the use of a soldering iron but the thought had crossed my mind.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

jacealvejetti · n00b Joined: 29 Mar 2019 Posts: 13 Location: Michigan, USA

Just following up a few months later, the little cluster is chugging along, it auto updates weekly, both emerge and kernel.

NeddySeagoon, I followed your advice, I made one of the cluster a "nas" and it stores portage, including distfiles, for the cluster.

Another is dedicated to kernel work and hosts it's own builds for the rest to install.

Saddly outside of the learning aspect the one used the most is the murmur pi, though I'm shocked at it's stability.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

jacealvejetti · n00b Joined: 29 Mar 2019 Posts: 13 Location: Michigan, USA

Honestly dont know if it keeps track which server is doing the work, I know if a server takes to long to return it will pass the work to another and just disregaurd the work when the first catches up.

I also though about going the DNS route specialy as the "random" server use in distcc leaves something to be desired, as far as i can tell you can have it randomize its hosts, but it only does so on service launch or call, I would love to have it shotgun the cluster everytime a compile piece is pulled, I have noticed during long builds that 2/6 of the ompile cluster remain unused generally.

Though this dosent happen for the linux kernel, probably the biggest things this cluster builds at the moment.

I have come up with a spare pi-3b+ I may make into a desktop and let it rely on the cluster for building, but I also want to move the cluster to aarch64 even it it would leave the pi2s out of operation.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs

NeddySeagoon · Posted: Sat Jun 15, 2019 8:08 pm Post subject:

jacealvejetti,

arm64 on a Pi3 is fun. I started that within a month or two of the Pi3 appearing.
Its got a lot easier since then.

I have a public binhost for arm64 Pi3 too. Help yourself if its useful.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

I think I said this before in another thread. I'd like to test a cluster of PIs (i'm thinking 4 or

, to see how they are doing with general C code. Like the kernel. I know it's not a tell_you_all test, but it's a good place to start.

But with a few amendments. First of all, the current "distcc server" I use, is a VM that has crossdev, and toolchains for every arch I own. I think I made a thread about it. Its something of the sort:

NeddySeagoon · Posted: Sat Jun 15, 2019 8:35 pm Post subject:

axl,

The 1G RAM limit will mean you can only run -j1 for lots of things.
You need a good thermal solution to avoid thermal throttling too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

axl · Veteran Joined: 11 Oct 2002 Posts: 1144 Location: Romania

jacealvejetti · n00b Joined: 29 Mar 2019 Posts: 13 Location: Michigan, USA

NeddySeagoon,

One of these days I am going to have to hop in the cluster and turn them all down to -j1, distcc down to 6 and see if theres a big difference.

I have each of them set to -j4, mind they are all only cli so nothing truly big crosses them.

I have spun up a single aarch64 node and likely over the coming weeks I will slowly migrate the cluster, as I ended up with an extra node, again, It will likely replace the murmur pi.
_________________
Current Project: 9 Node Distcc compile cluster composed of four Raspberry Pi 3b, three 3b+ and two 2bs

Gavinmc42 · n00b Joined: 23 Sep 2019 Posts: 21 Location: Brisbane

Can I recycle this now that Pi4s are out?

I am now using a Pi4 for a Desktop replacement PC at home.
Sakaki Gentoo64 and repo has lots but not everything has been built for A72 Aarch64.

Unlike Debian Gentoo64 build for source that takes time.
Just got my Pi4B4 today, it is replacing the 4B2 which replaced the 4B1.

Have a 5 Port switch I can stack Pi's on.
The question I have is is the 4B4 the better Pi4 to put on the cluster for compiling compilers?
Or will the 2GB one be enough?

As a Desktop Chromium takes heaps of memory and 1GB swap file was needed on the 4B1.
Medium compiles ran out of ram on 4B2 which could do bigger but still cannot have many tabs open.
So the idea is to have a cluster of Pi's, headless doing all the emerging and other grunt work.

Has anyone done this yet with Pi4's?
What is the optimum setup?
_________________
Don't get Pi's if you are scared of learning.

NeddySeagoon · Posted: Mon Sep 23, 2019 1:55 pm Post subject:

Gavinmc42,

I cheat, Packet.net has made a 96 core, 128G RAM, Cavium Thunder II available for Gentoo to use.
I build there and push the binaries to my [url=http://bloodnoc.org/~roy/BINHOSTS/]BINHOST[/ur/] for anyone to use.
I use these on my Pi3, Pi4 and Acer R13. Its all A53 rather than A72 at the moment.

Not everything will distribute. gcc can't by the nature of its build system.
glibc builds but is broken every now and again.
rust, needed by firefox and a few other things is written in rust.

Most things that will use distcc will also use cross distcc. The latter tends to be faster as an amd64 build system generating arm64 code has more RAM, more cores and has a higher clock rate.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Gavinmc42 · n00b Joined: 23 Sep 2019 Posts: 21 Location: Brisbane

Yep 96 cores etc are serious tools.
Sakaki told me what she uses back when all I had was a 3B+, heavy metal CPU's.
I did manage to compile the complete GCC toolset for RISC-V, it took 13hours.
That did convince me Gentoo and Pi's can do serious stuff.

Now some guys are doing parts of RISC-V GCC on Pi4's in one hour.
That's good enough for home schooled coding.
As a dedicated compiler box the Pi4 can use "make -j4 or j5" for more speed.
But I noticed last night using the Pi4B4 emerging a cross compiler, I still had plenty of ram left.
It was borderline with the 2GB.

DistCC is unknown territory for me, it can wait until I have another 3 Pi4's for my 5 port switch.
I now also have a 16port 1Gbs switch waiting in case I win lotto and Pi4B4's suddenly become easier to get.
15 Pi4's = 60 cores, 60GB memory

Not as cheap as Packet monthly but not a bad home setup

In the meantime an old Pi3B+ on the cluster can be a local Binhost for me.
I have Sakaki's binhost bookmarked but cannot see yours.
_________________
Don't get Pi's if you are scared of learning.