| View previous topic :: View next topic |
| Author |
Message |
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
 Posted: Sun Feb 24, 2019 1:11 am Post subject: [TIP] Build everything for Pi devices in a virtual machine |
 |
|
I was looking for a simple way to build packages for Raspberry Pi and Orange Pi. Today, I figured out a good simple solution.- Make a btrfs partition in an ARM gentoo virtual machine
- Turn /etc/portage into a btrfs subvolume so that you can send the content of /etc/portage to Pi devices with `btrfs send` and `btrfs receive`. If /etc/portage differs between the virtual machine and Pi devices, binary packages may not work.
- In the virtual machine, set EMERGE_DEFAULT_OPTS to
| Code: | | "--quiet --buildpkg-exclude='virtual/* sys-kernel/*-sources'" |
in /etc/portage/make.conf. You can also add www-client/firefox-bin and www-client/google-chrome to --buildpkg-exclude.
Build and install every binary package needed by Pi devices in the virtual machine.
Send the content of /etc/portage to Pi devices with `btrfs send`. You can later send the content of /etc/portage from Pi devices to the virtual machine.
Fetch and install binary packages from the virtual machine on Pi devices.
You can build kernel in an ARM virtual machine, too. There will be no more cross-compilation. |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Sun Feb 24, 2019 10:35 am Post subject: |
|
|
crocket,
That looks like the ARM chroot solution adapted to btrfs.
QEMU will still be used to emulate the ARM CPU so its still suffers the software emulation of hardware slowdown.
You could add in cross distcc between the VM and the host. For arm64 on amd64 that appears to give a worthwhile performance improvement.
The kernel is one of the easiest packages to cross compile. The cross compile will be faster.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
| Posted: Sun Feb 24, 2019 10:56 am Post subject: |
|
|
| NeddySeagoon wrote: | crocket,
That looks like the ARM chroot solution adapted to btrfs.
QEMU will still be used to emulate the ARM CPU so its still suffers the software emulation of hardware slowdown.
You could add in cross distcc between the VM and the host. For arm64 on amd64 that appears to give a worthwhile performance improvement.
The kernel is one of the easiest packages to cross compile. The cross compile will be faster. |
1. What is ARM chroot solution?
2. What is cross distcc between the VM and the host? |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Sun Feb 24, 2019 11:12 am Post subject: |
|
|
crocket,
The ARM chroot solution is on the wiki.
Essentially, you do an arm (or any other arch) install in a chroot on your amd64 system.
You add in a statically build QEMU to emulate the foreign arch, then you can chroot into the foreign install.
The static QEMU emulates the CPU to run the foreign arch install.
Considering an amd64 system with the above install, its possible to set up distcc between the foreign arch, in your case the guest, and the host. The host then runs a cross compiler when the guest sends it something to compile, just like two physically separate systems.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
| Posted: Sun Feb 24, 2019 2:19 pm Post subject: |
|
|
| NeddySeagoon wrote: | | The cross compile will be faster. |
Do you suggest that the best way to build Raspberry Pi packages on my desktop is to use the desktop as a distcc server and Pi as the client? Not sure if that's going to be the best way because this method seems to leave a mess on my desktop. It can also incur a huge amount of downtime on Pi when an upgrade fails in the middle. To minimize downtime on my desktop during upgrades, I build binary packages in a ZFS clone on my desktop and take a ZFS snapshot before installing binary packages.
Do you know a simple good way to build Raspberry Pi packages and kernel on my desktop? |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22819
|
| Posted: Sun Feb 24, 2019 4:30 pm Post subject: |
|
|
| No, he is telling you to have the emulated ARM64 VM run a distcc master and delegate all the work to natively-executed ARM64 cross-compiler instances on your AMD64 host. This minimizes the amount of work done by the virtual ARM64 chip, and avoids using the real ARM64 system at all. Running an arm64 native compiler (runs on arm64, generates code for arm64) emulated is more expensive than running an amd64-hosted arm64-targeted cross-compiler (runs on amd64, generates code for arm64). |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Sun Feb 24, 2019 7:01 pm Post subject: |
|
|
crocket,
What Hu said and why do you need a snapshot before you install binary packages?
A build arm64 binary packages natively on a arm64 box. (Thank you Packet.net)
Push them to my binhost then install the binaries on my Pi3 and Acer R13 Chromebook.
Help yourself if its useful.
My arm64 chroot has two different make.conf files.
One to build things while chrooted. The other to build and install using a cross compiler.
Both ways build binary packages.
For the few things that won't cross compile nor build in the chroot (QEMU is not perfect) I build using the Pi and distcc, where possible.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
| Posted: Sun Feb 24, 2019 10:06 pm Post subject: |
|
|
| NeddySeagoon wrote: | | why do you need a snapshot before you install binary packages? |
I don't need one. But, it helps to roll back my system to the point before an upgrade when I find that an upgrade broke. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Sun Feb 24, 2019 10:11 pm Post subject: |
|
|
crocket,
Ahh OK. I keep all my distfiles and binary packages as an aid to rollback, if I need to.
Its only been the occasional package, ever.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
| Posted: Sun Feb 24, 2019 10:25 pm Post subject: |
|
|
| NeddySeagoon wrote: | | For the few things that won't cross compile nor build in the chroot (QEMU is not perfect) I build using the Pi and distcc, where possible. |
1. How can Pi use a distcc server on the desktop for packages that don't support cross-compilation? The desktop needs cross-compilation.
2. Does a proper QEMU virtual machine support compilation of packages that QEMU user-mode emulation can't compile? A QEMU virtual machine is going to still be a lot faster than Raspberry Pi 3 B+.
3. Do you have an armor case for Pi? I'm going to buy Pi with an armor case because it effectively keeps Pi cool at constant full load. The case itself is a huge heatsink. It's an extreme passive cooler.
Last edited by crocket on Sun Feb 24, 2019 10:34 pm; edited 2 times in total |
|
| Back to top |
|
|
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
| Posted: Sun Feb 24, 2019 10:29 pm Post subject: |
|
|
| NeddySeagoon wrote: | Ahh OK. I keep all my distfiles and binary packages as an aid to rollback, if I need to.
Its only been the occasional package, ever. |
When you discover that Anki doesn't support the latest stable versions of QT packages, it's going to take a lot of work to downgrade all the QT packages and rebuild packages that depend on QT.
It's simpler to just roll back when you have anki decks to review and don't want to spend the next 2 hours on manually downgrading QT. It minimizes downtime for such critical packages as Anki.
Anki is critical to me.
Without binary packages and ZFS snapshots, gentoo is not suitable for critical production machines where hours of downtime are not acceptable.
Last edited by crocket on Sun Feb 24, 2019 10:48 pm; edited 1 time in total |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Sun Feb 24, 2019 10:47 pm Post subject: |
|
|
crocket,
A QEMU chroot and a QEMU VM will suffer in the same way from QEMU.
A Phenom II 1090T, 6 cores at 3.2GHz is about twice the speed of a Raspberry Pi 3. The 3B will be faster, if you can keep it cool.
Cross distcc and cross compiling are not the same thing, so cross distcc can work where cross compiling fails.
For cross compiling, the build system needs to be cross compile aware.
For cross distcc, to the build system, its a native build. distcc passes preprocessed sources to the helpers, which run the cross compiler and pass the output back to the Pi.
The build system runs on the arm64 system.
Generating the correct code from the cross compiler is rarely an issue. That would be a gcc bug.
cross distcc has a go faster mode, called pump mode. There be dragons. When it works, preprocessing can be offloaded too.
I have a sawn off 486 or Pentium heatsink on my Pi.
As its only held on with an elastic band, there is no heatsink compound. That would be too messy.
Take care with custom heatsinks like you linked to. The component layout differs between the Pi 3 and Pi 3B+ It matters for cases like that.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
| Posted: Sun Feb 24, 2019 10:53 pm Post subject: |
|
|
| NeddySeagoon wrote: | | Take care with custom heatsinks like you linked to. The component layout differs between the Pi 3 and Pi 3B+ It matters for cases like that. |
It supports 3B and 3B+. Plus, the armor case is sturdy and makes a Pi look like a tank.
| NeddySeagoon wrote: | | A Phenom II 1090T, 6 cores at 3.2GHz is about twice the speed of a Raspberry Pi 3. The 3B will be faster, if you can keep it cool. |
I never knew Raspberry Pi 3 was significantly faster than Intel Pentium E5500 which used to my desktop CPU. Maybe, Pi 3 should be an additional distcc server.
I could also build binary packages in a btrfs/ZFS clone on Pi and use my desktop as a distcc server.
| NeddySeagoon wrote: | | One to build things while chrooted. The other to build and install using a cross compiler. |
How does an ARM chroot have access to cross compilation toolchain? Does the chroot just use distcc server on localhost?
Last edited by crocket on Sun Feb 24, 2019 11:19 pm; edited 1 time in total |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Sun Feb 24, 2019 11:15 pm Post subject: |
|
|
crocket,
Gentoo does have some large production deployments.
All the production users I know of build binary packages and test the proposed update well away from production.
That includes a snapshot of ::gentoo with their own overlays, so that they don't have to deploy ::gentoo as it was at any particular point.
When testing is complete they update production from their own repo and binary packages.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Sun Feb 24, 2019 11:20 pm Post subject: |
|
|
crocket,
| Quote: | | How does an ARM chroot have access to cross compilation toolchain? |
I linked to that already. It uses the localhost IP address to get to the host system outside the chroot.
My example used 127.0.0.2, so that distcc to the host outside the chroot could be monitored. localhost is 127.0.0.0/8
Much as a chroot can use the real network.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
crocket
Guru
Joined: 29 Apr 2017
Posts: 558
|
| Posted: Mon Feb 25, 2019 12:28 am Post subject: |
|
|
| NeddySeagoon wrote: | I linked to that already. It uses the localhost IP address to get to the host system outside the chroot.
My example used 127.0.0.2, so that distcc to the host outside the chroot could be monitored. localhost is 127.0.0.0/8
Much as a chroot can use the real network. |
So, you are saying there is no local cross-compilation. Only cross distcc.
Do you compile kernel with local cross-compilation toolchain? Or, do you utilize cross distcc to compile kernel in an ARM chroot or an ARM virtual machine? |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54627
Location: 56N 3W
|
| Posted: Mon Feb 25, 2019 2:12 pm Post subject: |
|
|
crocket,
I cross compile the Raspberry Pi arm64 kernel.
The kernel is the first thing ported to a new arch. That has to be done with a cross compiler because there is as yet, no native code for the new arch.
It follows that kernel cross compiling is very well tested.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
