How to manage multiple gentoo installations

luckylinux · n00b Joined: 17 Mar 2012 Posts: 48

Hello,

Maybe the question is quite common (probably it is) but I wonder how do "experienced" users manage multiple Gentoo installations ?
I have about 15 Gentoo installations, which I group in "profiles" for which I build binary packages:
- Minimal: DNSServer, Distcc host, Crossdev, ...
- Server: NAS, Other kind of servers requiring Apache/PHP/...
- Desktop

At the moment I'm using FEATURES="buildpkg distcc distcc-pump" whenever I run an update on a host. Therefore the next time other hosts having the same profile get updated, they can simply install the binary packages. USE flags is common amongst packages and CFLAGS/CXXFLAGS is set to lowest common setting (currently Intel Sandy Bridge but I set a generic x86-64 profile, CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe")

Packages are shared using NFS from the NAS and mounted in each client to /usr/portage/distfiles and /usr/portage/packages.

Next steps I plan on implementing:
- Implement "chattr +i" on /usr/portage/distfiles and /usr/portage/packages so that if NFS cannot be mounted I won't have each host compiling packages in his corner and not being able to tell. I should have done this already, so much time and disk space wasted :oops:

- Use Salt/Saltstack to manage and distribute configuration files. Due to the common settings amongst the above mentioned "profiles", all files in /etc/portage and other files such as /etc/genkernel.conf should be in sync. At the moment I'm doing this by hand

- Distribute binary kernel to each host. Again this makes sense since all hosts have basically the same configuration and no specific driver requirement. All Gentoo currently (except Desktop) reside on ESXi. I should probably just tar /boot/{linux,initrd,System.map} as well as /lib/modules/linux{version}, .config and /etc/genkernel.conf and append some requirements on the ebuild (such as gentoo-sources, linux-firmware, ...)

However I wonder how do you guys manage multiple Gentoo installations concerning the updates. Mainly the security updates. I currently use Royal TS on Windows and after compiling on one machine, I just send the same command to all others automatically. Probably the same can be done from a Bash shell, I'm not sure. This works very well when things go OK. But when things go south, I quickly lose a few hours trying to debug what package is creating issues and why.

I read on the wiki there are some scripts for "automatic" updates which are *highly discouraged*. If using these is highly discouraged, then what do you use and how do you implement it, at least for security updates ?
- You run glsa-check and automatically install updates
- You subscribe on gentoo-announce mailing list for security updates then perform the updates automatically

Furthermore, can you keep the list of installed packages in sync between the different machines ? I would like to have the machines installed with identical packages. This includes the troublesome emerge --depclean at the end of the update. How do you perform that ? My idea was to create a custom ebuild like desktop-package-list where I add lots of DEPEND and RDEPEND items for each package I want installed. This way I could also "enforce" the removal of a package I don't want anymore. I just wanted to know if there was a more elegant way to do it

I hope somebody can shed a bit more light on this

mike155 · Posted: Mon Mar 26, 2018 10:16 pm Post subject:

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

One note on modifing the world file, it is NOT designed for you to edit directly. You should use portage to add/remove packages to/from the world file.

3 main options to keep in mind, is --oneshot installs a package but does NOT add it to the world file. Without that, it would add the package to the world file.
To remove a package entry from the world file, you can use --deselect would remove that package from the world file. (this simply removes the entry in the world file, does not uninstall the package.
Lastly, you have --noreplace, which does the opposite of deselect, adds a package to the world file (does not recompile the the already installed package).

mike155 · Posted: Tue Mar 27, 2018 12:18 am Post subject:

krinn · Watchman Joined: 02 May 2003 Posts: 7470

portage doesn't have any real trouble to handle world trouble

mike155 · Posted: Tue Mar 27, 2018 1:57 am Post subject:

I browsed through the man pages and I'm now convinced that it's safe to edit or copy '/var/lib/portage/world':

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

A few things I do:

- PORTDIR on a ro NFS mount
- DISTDIR on a rw NFS mount (DISTDIR should never be inside PORTDIR, the default is terrible)
- PORTAGE_RO_DISTDIRS="/mnt/nfs/http-replicator" (or whatever fetch proxy you're using) is a good idea too
- /etc/portage on one machine is a ro NFS mount that the others can symlink config files from

My hardware is too varied to try this but mounting root via NFS and booting a kernel from PXE may also be options.

musv · Advocate Joined: 01 Dec 2002 Posts: 3337 Location: de

I'm using some kind of squash_portage, but have written the scripts on my own, when I switched to Sytemd a few years ago.

I have a directory /var/portage, which is mounted rw via NFS. This directory includes a distfiles directory (DISTDIR) and a portage-current.sqfs file (SquashFS image of the Portage). This image is mounted to /usr/portage (ro). Furthermore I use a OverlayFS on /usr/portage to be able to update and modify the portage. When I unmount /usr/portage, the changed portage tree will be written to my SquashFS image in the NFS folder.

Advantages

The portage image has a size of 146 mb including several overlays.
You don't need the portage tree physically stored on any machine (unmounted /usr/portage is empty).
It can be updated on a machine of you choice and is up to date on any other machine in the network (after remounting the image).
Distfiles are downloaded only once

Disadvantages

Portage becomes incredibly slow. The get a list of packages to be updated can take up to 10 minutes.
If you update the portage tree, you have to restart/remount the portage tree on every machine to have the updated tree available.
Doesn't work together with Networkmanager, which always stopps NFS before unmounting the SquashFS image and NFS portage directory. Solved by throwing NetworkManager away.

luckylinux · n00b Joined: 17 Mar 2012 Posts: 48

Wow so many answers so quickly I'm amazed

So you basically all suggest to do this with integrated tools rather than relying on an external configuration system such as Puppet/Cfengine/Saltstack/...

@Ant P.: what is exactly the difference between DISTDIR and PORTAGE_RO_DISTDIRS ? I'm a bit confused since both concer distfiles. Furthermore you say "DISTDIR should never be inside PORTDIR, the default is terrible". I would say that also applies to packages /usr/portage/packages. Why not putting it a bit more structured say /usr/portage/ports, /usr/portage/distfiles and /usr/portage/packages ?

Furthermore mounting root via NFS, although attractive (space savings, ...) may defy one purpose: availability. If for whatever reason the network goes down, I lose all access to all machines. Not a good solution. Would any of you reccomend running an rsync (or say zfs send -R // replication, although at the moment I'm using ext4 but could move to ZoL I guess) to the "slave" machines ? At least this could be done for the 6-7 distcc hosts I have in my lab

@mike155: I'm not exactly fan of editing the world file directly. However, even if I do, how can I enforce that the changes will be applied ? emerge -auDn @world ? Not sure if it's a good idea to let this run unattended (say via cron)

@All: nobody install updates automatically ?

musv · Advocate Joined: 01 Dec 2002 Posts: 3337 Location: de

luckylinux · n00b Joined: 17 Mar 2012 Posts: 48

I'm wondering then what to do with /etc/portage. To my the "/etc" folder has to be always available, even in the event of a network failure.
Furthermore mounting using NFS to /etc/portage presents an additionnal problem: NFS is prone to give "stale file handle" errors (already had it happen on my media server).

I wonder therefore what the best approach would be:
- Using rsync ?
- Using zfs send -R (assuming I would convert all Gentoo installations to ZFS) ?
- Using saltstack ?

soparla · n00b Joined: 21 Aug 2018 Posts: 72 Location: Florida

Why should we mess around with portage setup and etc... next to the usual support for source-based ebuilds, Portage also supports building and installing binary packages.

Below guide explains how to install them, create them, and how to setup a binary package (binhost) server.
https://wiki.gentoo.org/wiki/Binary_package_guide

To make emerge happy and sync repos between the nodes, simply host your own local rsync mirror on a single dedicated machine (perhaps the same binhost):
https://wiki.gentoo.org/wiki/Local_Mirror

You should also cache the dist files (again, on binhost):
https://wiki.gentoo.org/wiki/Local_distfiles_cache

Beside all of the above, my farm is also using distcc so compilation is processed in the cluster but the binaries are stored on a single node (binary package server: binhost) and then distributed to the rest:
https://wiki.gentoo.org/wiki/Distcc

For me... the annoying part was setting up the SSH user config to work correct. The trick is to have the user created on the server, generate its keys and export both private/public to all of the machines. Then you might find it easier to alias the binhost via the ~/.ssh/config on all the clients: