Gentoo in production

[kittykat990]

Hi Guys

New to Gentoo having used Debian and Centos. I am keen to find out about the challenges of running Gentoo in production. One of the concerns I read is of the time it takes to update. Are there any other challenges?

Thanks
Kitty

[eccerr0r]

TBH the main problem is having to roll with the punches as it's a rolling release.
I suspect a lot of production servers rely on stability and if one has to keep on upgrading a piece of software, the maintenance issue becomes a problem.
Completely up to you to do. I run it on my home servers but I don't have mission critical apps, things can go down for a few hours/days while trying to come up with a different solution because the old solution is no longer supported, and it's fine for me.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[pietinger]

Some thoughts (also some security thoughts):

- Use only a stable system
- Set up a server for rsync: https://wiki.gentoo.org/wiki/Home_router#Rsync_server
- Work with binpackages https://wiki.gentoo.org/wiki/Binary_package_guide
- Do a kernel hardening for all servers (sorry only german: https://forums.gentoo.org/viewtopic-t-1112848.html) - Dont believe this is done in our hardening sources ...
- Set up a validating, recursive, caching DNS resolver (I use "unbound", also only german: https://forums.gentoo.org/viewtopic-t-1125184.html)
- Install a personal firewall on every server (... additional to your main firewall(-s)
- Do all "emerge -uUDv @world" first on your own Admin-Station (and check out for problems; do updates to your servers after that)

- Maybe ... only maybe ... use for your clients Windows ...

[pjp]

I'll second the rolling release issue. In the case of Gentoo, that results in a lot more updates that need to be addressed. A binary distro generally has a fixed point from which updates occur. Commonly major version numbers. That allows for a smaller footprint of code that needs to change.

That brings up what I think is a bigger issue. Binary distros provide a lot of upfront work for you that becomes "your job" when running Gentoo. In addition to any dev/test environment upgrade evaluation that you would perform for a binary distro, you have to build the binaries and make sure they work (or be comfortable with the possibility that something might not). Well known binary distros that are commonly used or associated with use in enterprise environments also add a possible benefit of how many people use them and the number of installed system on which they are running. There are likely to be fewer differences between systems as well.

Regarding the time it takes to perform updates, I think that is somewhat exaggerated. If you're thinking of configuring each system uniquely and updating each one individually, then I have to believe you aren't an experienced administrator. Deciding how consistently things are built is now your decision. Are you going to modify compiler settings, rely on hardware uniqueness? That increases the variations you'll have to deal with when upgrading.

Gentoo does provide a feature to deploy binaries using a "binhost" (https://wiki.gentoo.org/wiki/Binary_package_guide). I remember a different document, but this may be the "current" format. Setting that up is an additional "one time" burden.

Another somewhat more complicated matter is in deciding if you don't want certain binaries deployed on certain systems. Portage the tool requires python. Compiling requires a compiler. Do you want a compiler or any dynamic languages on your web facing systems? Or any systems for that matter? If not, that's an additional complication in deploying and maintaining systems.

In my experience, I would not recommend Gentoo for the environments in which I have worked. That doesn't mean it isn't suitable for production / enterprise use, only that it wouldn't make sense for those environments. Two, but not the only factors to consider would be availability of Gentoo knowledge in the available candidate pool, and whether or not the additional burdens offer enough advantage. How difficult will it be for the organization to expand or replace its employees with Gentoo knowledge? Does the organization gain enough value in choosing Gentoo to outweigh the additional challenges that come with Gentoo?

Before I'd consider such a role, I'd want at least 2 fairly capable build hosts for redundancy (could have other roles without uptime requirements) and multiple hardware incompatible systems for use in a test lab. The more "weird" hardware the organization used, the more of that I'd want in the test lab. In my experience, organizations tend to put a low priority on systems administration testing environments.
_________________
Quis separabit? Quo animo?

[NeddySeagoon]

kittykat990,

For production use you separate building and deploying.

When you deploy, you use your snapshot of the portage tree that you used for building and the binary packages that you have built and tested.
It much like deploying any other binary distro.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[kittykat990]

Some really wonderful comments here, I am appreciating them.

[kittykat990]

[figueroa]

I run a remote Gentoo stable x86 server and maintain it with updates live. It's updated frequently, every 20-30 days or so. I haven't had any issues in at least five or six years. I reboot about once every couple of months for kernel updates. Once upon a time (8-9 years ago) I froze the kernel at a pretty good good version and had it running for something around 600 days, after which I became a bit paranoid over rebooting, so I don't do THAT anymore.

I get selected logs a couple of times every day by email for what amounts to a health check. I'm more confident in maintaining this box than any of the equivalent Debian stable machines that I've run. I like Debian stable, and that would be my second choice.

The OS is frequently backed up, automatically, in what amounts to stage4 equivalent tarballs. I anticipate problems. I'm careful. I have a couple of similar (one is identical) servers locally, one is usually updated daily, the other at the same frequent intervals as the remote server. These updates help me debug update problems in advance.

There is a helper at the remote location who can do hardware maintenance with my advice, and if necessary boot it from a flash drive for OS maintenance. Over the last 15 years we've had to replace the main hard drive, a power supply x2, a network card. Downtime has only occurred for broken hardware. I installed this machine around 2008-2009 to replace a much older machine that was also running Gentoo, doing mail, web, and file service.

Time to maintain isn't an issue. I don't watch packages compile.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

[kittykat990]

[figueroa]

[NeddySeagoon]

kittykat990,

I don't know of all the information collected in one place but here is a high level overview.
To keep it simple, lets assume your production fleet are all identical. If they are not, some fine tuning will be required but the concept is the same.

You build and test your potential deployment in a build/test system that is representative of production.
Its the normal gentoo install and update process with two exceptions.
1) You keep a snapshot of the the ::gentoo (and all other) repos that you use for building and testing.
2) As you go, you build binary packages of everything. That's FEATURES=buildpkg.

If you use rsync to update your repos, make a tarball when you ace happy its all built and tested.
If you use git, export a tarball.
You now have BINHOST that looks something like my arm64 binhost and the matching ebuilds, so that other systems can use them.

Distribute the repo tarball to the production systems to be updated.
Make the BINHOST available to production systems. That's a make.conf entry.
emerge -K on the production systems will install from the binhost or fail.

Play with it in a VM or two.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[kittykat990]

[quote="figueroa"]

[kittykat990]

[marduk]

[sam_]