Installing Gentoo on 250 Servers

[rollexus]

Hello Everyone,

We currently have 10 clusters with 25 nodes per cluster with gentoo installed on them. We have learned that not updating them regularly could cause major problems which leads to emrge -e system and --emerge world which could go through compiling up to almost 200 packages... That is if we are successful in doing it the first time, and if not, fix the block and rerun the commands again. This is obviously not feasible 250 times... We have decided to upgrade all machines from the ground up. That being said, what we are looking for is installing gentoo server or even hardened profile on 250 machines. What we are looking at is:

Having one machine as a gold client (or server mind you): This machine is responsible for the initial install of Gentoo, stage 3, and the compiled kernel
Having one machine as a build machine: This machine is in charge of making packages, which the other computers use to pull the built binary packages
250 Hardened Gentoo machines humming gracefully.

The scares we have expereined in the past have to do with the dreaded kernel updates, gcc updates etc... If you gents and ladies can share your expertise on how to lay this down with the least bit of pain in the near and far future. These machines are all the same at the moment, but future machine that get added to the center may differ, this is something to also consider... Though a "Gentoo Chat" like discussion in the moment, mods please leave this thread here because it will eventually transition to a "Gentoo Install" in the near future.

Thanks in Advance!

Nick

[dirkfanick]

Maybe I got your question wrong and you already know all about it, but BINHOST-usage is what I would suggest:

http://en.gentoo-wiki.com/wiki/Using_Portage_BINHOST

http://www.gentoo-wiki.info/HOWTO_setup_a_PORTAGE_BINHOST_server

[dwbowyer]

Two thoughts the second of which I think you will like better.

Could also make sense to use distcc? Depending how critical your cluster is, this might not be an option, and if you don't want to have full build environment on the nodes, distcc obviously not an option. I have a only 2 desktop machines, but can build some 500 packages without failures using distcc. Between an AMD64x2 and Pentium3, it gave the Pentium3 a 500% boost in building it's packages. With 250 servers, I imagine you could build a set of 200 server packages in a tenth the time while nicing the compiles to only hit about 5% cpu load on each node....

EDIT: Note I mean to use distcc to build binary package set on one machine and then pull those to the nodes, after testing the set on the "gold" system. Since production servers don't want/shouldn't be updated weekly, except for security fixes, I recommend the option below.


I switched to setting setting up a build-host in a chrooted environment on the AMD64, and found that to be even faster in my use case, due to IO bottleneck on the P3.

With a decent script, building bin-packages for other systems can be fully automatic, after the initial set-up of the chroot environment. With custom chroot environments, even building/maintaining diverse hardware from a single buildhost is possible. I even build the Pentium3 kernel on my AMD64, and an "emerge -uDN world" on the P3 does everything else in a few minutes. Even "emerge -e world" on the Pentium3 only takes about 90 minutes, rather that 36 hours alone or 8 hours with distcc.

https://forums.gentoo.org/viewtopic-t-451680-highlight-buildhoster.html was meant for large server clusters. Unmaintained scripts since 2007, but needs only minors tweaks to work. Also easily extensible. It's just BASH scripts. The list of packages to compile in bootstrap.sh had to be modified to reflect minor changes since 2007. I highly recommend it.

Take a few days to familiarize yourself and set up the chroots, tweak the existing scripts to suit your needs and this would just about automate all upgrades, even allowing you to test MOST packages within the chroot (not kernel and udev etc). Like any large roll-out you'd want to pull the packages to a few systems and test them before deploying to all your clusters. Having built binary packages would however also allow easy reversion to working set on the tested systems if problems were found. Buildhoster makes that easy.

IF you look at buldhoster and you like what you see, you can PM me if you have any questions, since the original author no longer supports/maintains it. I'll help with what I can.

[jormartr]

If I understood the problem, I suggest also the binary packages way. Compile on one machine and install on every other host with that binaries.

Chroot is comfortable, but a physical host, or better, a virtual machine (with snapshots feature) lets you test things like startup/shutdown scripts, kernels, extra modules... these which a chroot does not let you do.