Gentoo as a Production OS

crackytron · n00b Joined: 16 Nov 2007 Posts: 24

Hey there

I was wondering whether many of you think Gentoo would be stable and secure and reliable enough for being a production OS - this means it must be low maintenance, system must work when things are updated, uptime must be 100%...etc. The stuff I've read has recommended Debian over Gentoo because it is easier to keep stable, however I'm still considering Gentoo as I just like the way its laid out, where the config files are - generally feels better.

I thought I'd get it straight from the horses mouth and see if you lot think it'd be wise to go with Gentoo.

As for my skill level, I'm fairly confident doing a fair few things, but I'm far from being an experienced sysadmin.

Thanks!

PS - what is this "religious" thing I hear about Debian :S
PPS - and fyi, the server will be handling things like ftp, http, mysql, mail. Nothing overly mad but it will be taxed by a fair bit of traffic.
_________________
welp

SwissBushIndian · n00b Joined: 13 May 2007 Posts: 27

The "religious" bit about Debian is that they are, some may say overly, devoted to open source, which effectively means that they do not support non-free software in their repositories.

Kasumi_Ninja · Posted: Sat Nov 17, 2007 10:26 pm Post subject:

It depends on your preferences. Personally I find Gentoo the most stable and consistent OS thusfar (I have supported and tested lots of distributions) . Setting up may take some time, but it is imo by far the easiest OS to maintain and more important the most consistent OS I know. If you need a server OS then you can also try Centos.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered

Drysh · Posted: Sun Nov 18, 2007 7:39 am Post subject:

Gentoo isn't that hard to maintain. The problem is that you have to learn a new way to handle updates.

While using other distros you will plan for a big update once in a while, with Gentoo you will have updates on a daily basis. That means: (1) You need to learn that you don't have to update just because it is available. There are tools to find what you need to update (like glsa-check), and you can check the forums, IRC, mailing lists, etc.. This is valid for any Distro, but much more for Gentoo.

Just like any other update with any other distro, (2) always keep a backup. Nobody is insane to update to a new release of Debian without a backup. It's no different with Gentoo: but the "new release" is any time you want. I think this is the key point with Gentoo: you have to keep in mind this is a meta-distribution with continuous releases (instead of a periodic release). Any time you run "emerge -DNu world" you are creating a new release.

Since Gentoo is source based (and not binary based) you will need time (a processing cycles) to update. (3) Never emerge if you are in a hurry or if the machine is running something else that needs to be finished soon. Before emerging, make sure you will have some time to fix a eventual error. Also: (4) Test first. Don't start your update with the mission critical server, start with your own box.

Most important: (5) Don't ever use exotic CFLAGs, or ~arch in a production environment. Don't even think about that. Using ~arch is like using a beta OS (it may be fun at home, but it can't be trusted).

Following these 5 tips, Gentoo is almost as stable as Debian. I won't say it's as stable because Gentoo usually have newer packages that aren't as tested as Debian's are. If you need to have the latest version of some packages, Gentoo may be even more stable than Debian (it's much easier to use one ~arch package in Gentoo that the same version in Debian).

Gentoo is different than most distros. It can be as stable as others if you learn how to work with Gentoo (but it takes some time to learn the Gentoo right way). I recommend that you start using Gentoo at home or with a non-critical machine. Since Gentoo gives you much more freedom than other distros, it takes discipline to use Gentoo. It's not for everyone, but if you have the wisdom, Gentoo may be a wonderful tool.

BTW: I don't have the wisdom to use Gentoo in a production environment, at least not if I I'm alone. I always think that new package is too cool, and it won't take more than an hour for a full update. Bah, "emerge -DNu world" and go to lunch when I'm back everything will be running fine, just a "etc-update" and I'm ready. (Sometimes it works that way! Sometimes...)

Kasumi_Ninja · Posted: Sun Nov 18, 2007 9:43 am Post subject:

alistair · Retired Dev Joined: 15 Jul 2005 Posts: 869

I would run gentoo in production.

But I would make sure I knew exactly when downtime/slowtime was acceptable.
I would update frequently the "core packages". (Monthly at most)
I would watch for gcc, glib updates and plan updates for those (I'm taking major version changes, not -r* ).
I would have/do have standard CFLAGS.
I would have a development/test box where an update occurs and is shakedown tested first. I would attempt to mirror the production server as much as possible.

note: I am reasonably paranoid

_________________
______________
Help the gentoo-java project. Visit Gentoo Java Project

what good are admin powers if you don't abuse them for personal gain - mark_alec

i92guboj · Posted: Mon Nov 19, 2007 12:44 am Post subject:

I don't think Gentoo is any more or any less stable. My vision is a bit different.

Those numbers that I will say are just illustrative, and they do not mean a concrete thing, so just bear with me.

In a regular binary distro, 75% of the stability (or instability) is given by the OS itself, and 25% by the administrator. In Gentoo those numbers are the contrary. That means that you have more responsibility. You have the power to screw up yourself, and no one is going to stop you. That can be good if you know how to do the things, or bad if you don't and don't have the time or will to learn a bit.

Again, I know that that picture is way too abstract. Gentoo also offers the glsa system, and while you stick to only the updates that are strictly needed to cut any security risk or support new hardware, you should be on the safe side. You have also an excellent documentation on how to harden your system, starting from the basics like iptables and up to more advanced topics like selinux and the like. Few distros out there have such amount of documents and such a useful forums like Gentoo.

For those brave enough, you even have the choice to use portage over FreeBSD, though I must admit that I am not sure that that would be any good at the current stage (I don't even know if there is a current stage, indeed).

About religious wars, I always avoid them. Everyone has the right to use whatever s/he thinks that fits better for him or her. Those kind of discussions are not only useless (no one is gonna change his mind just because you don't like the way he thinks... it'd be stupid to even think about that), but also counterproductive.

Linux is just linux, after all. You can do the same things on any distro. In ultimate instance, the only thing that matters is the support you are going to find on the concrete community, and regarding that, Gentoo is superb.

Kasumi_Ninja · Posted: Tue Nov 20, 2007 8:16 pm Post subject:

eccerr0r · Posted: Tue Nov 20, 2007 9:08 pm Post subject:

I think I've already put my spiel about this in the thread titled "Gentoo for Enterprise" -

For home/hobbyist/tinkerer/educational/entertainment use, this is fine, but

chance(breakage from emerge -u package) >> chance(breakage rpm -U file)

At least it seemed that way - I never had a broken system when I used Redhat after an update, and if there was breakage, it was for that package only, not anything else.

Gentoo? There's a nonzero chance of widespread breakage. Once, without non-portage intervention, almost required a boot disk to fix because portage broke after an upgrade.

I don't know way too much about how SuSE works, but I also felt that Debian was similar to Redhat, there were a lot of packages that apt would simply not upgrade in fear of ending up with an unusable system. Or at least a system that needs serious work to get it usable again (such as upgrading apache)... It's sometimes hard to tell when to just start fresh with Gentoo or not, and also minimize the number of fresh installs...

"Moving Target Syndrome"
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Kasumi_Ninja · Posted: Tue Nov 20, 2007 11:20 pm Post subject:

ats2 · Apprentice Joined: 22 Apr 2005 Posts: 297

To get the best out of the two worlds : make a gentoo host server with vserver-sources (Linux-Vserver) , then set up a debian vserver with all your services (or one for each service depending on what you want to do) . Thus you wont need to update your gentoo box so often (as it only hosts your virtual servers) and you'll run rock stable services. Another good thing with this setup is you can actually test new configurations in new vservers before running them for real.
A little overhead while learning vservers but great benefit in the long run.

PS : vservers use a *very small* amount of the host machine, so you're not limited by their numbers.

crackytron · n00b Joined: 16 Nov 2007 Posts: 24

So I can gather from this topic that Gentoo is good in production if you have the knowledge and experience to keep it running well. Now, I don't want to cop-out entirely, but if there are production system (gentoo specific) checklists for things like security (IE - you must reset these passwords and get rid of these accounts) it would be incredibly useful. At the moment I'm more of a "programmer guy who knows the most about system administration" than a "system administrator". I'll admit that I'm inexperienced, but I'm being paid a low wage and I've set up numerous linux systems but never in a production environment. I think if I work hard I could do it, and I'll be covering my ass in writing, but if anyone has any good, up-to-date resources on production server administration, dos/donts, security checklists, it would be exceptionally helpful.

Ats2's suggestion is quite an interesting idea. I've never worked with virtual hosts but it does seem like an interesting idea.

FYI the reason I like Gentoo is that because I inherited a network that has a load of issues, been around for 10+ years, etc etc so I don't have all of it inside my brain. I like starting from the beginning and understanding exactly what is going on. This is the great thing about Gentoo - if something is there it is because you have put it there.

Also, what are your opinions on leaving things like packaging tools on live systems - my friend says that if you leave emerge/apt etc it could allow the system to be compromised more easily - is this true? Can emerge be locked to certain groups?

Thanks
_________________
welp

Kasumi_Ninja · Posted: Sat Nov 24, 2007 3:18 pm Post subject:

JC Denton · Apprentice Joined: 16 Apr 2003 Posts: 151 Location: USA

rgk · Posted: Sun Nov 25, 2007 3:49 am Post subject:

I use Gentoo as a production server and it works great, and I constantly update it. I don't unmask anything and I normally only do an full update if I am sure it will work. But I am also not that worried about 100% uptime, any problems that I have fallen into are fixable within a few minutes. I would highly recommend testing out Gentoo on an other server and seeing how everything works before putting it on a production server. Maybe also learn your way around before putting it to a production server too. I used Gentoo for a few months before I thought putting it on a server that I would need, had a few problems at first but now everything is smooth.
_________________
MadGizmo.com is awesome.

crackytron · n00b Joined: 16 Nov 2007 Posts: 24

I'll definately give that handbook a read.

The reason I'm considering Gentoo so heavily is that I've found emerge far more reliable than apt-get, rpms, whatever in the while I've been using linux. I've used Fedora, Debian, Ubuntu, Mandrake and SuSE and I've found that Gentoo is the most logical, most enlightened (compare the default Gentoo apache config to anything else, it is beautiful).

I just need to cover my ass in writing so I can say - I did this research which showed that Gentoo was a suitable OS, I did these measures to protect security etc etc - if everything went pear shaped.

I'm pretty much settled as other distros seem to annoy me every time I have to use them.
_________________
welp

i92guboj · Posted: Wed Nov 28, 2007 11:05 am Post subject:

gringo · Advocate Joined: 27 Apr 2003 Posts: 3793

ChojinDSL · l33t Joined: 07 Jul 2003 Posts: 784

Just thought I'd give my 2cents here.

At my Job I'm the Lead System Administrator. I manage and deploy our servers as well as train new sysadmins.

Gentoo is the sole reason why I can work as a System Administrator. I had tried linux in the past, but gentoo was the first one where I actually learned about linux (because your forced to), and as a result of that I also managed to get everything working under Linux that I wanted to.

Because of all this, my personal preference is always Gentoo, mostly because thats the distro Im familiar with the most. Sometimes out of boredom, I will try another distro on my laptop, but in the end I always come back to gentoo.

Considering this, my view is probably tainted, but here are some points about my experience with gentoo.

First of all, one of the things I love about gentoo is its flexibility of installation. Since you install everything by hand anyways, you can easily perform a "Parasite" install. e.g. Boot a livecd of any distro, and then just proceed to do a chroot installation, and download all the packages you need of the net.
I'm sure that this is possible with other distros as well, but the process just doesnt seem as straightforward and requires a lot of tutorial hunting.

Compilation Time:
One of the most cried about (dis)advantages of gentoo.
I have to say, that while I will think twice about installing gentoo on someone else's Desktop or Laptop, I have no problems with installing it on a server.
If you are only installing a Server environment withouth X, then the biggest packages you have to worry about are maybe Apache, Mysql, glibc and gcc. But even with that in mind, on a modern server compile time is not that big of an issue.

If you are renting a root server from a provider, you can get a DualCore or X2 system rather cheaply. Even if you build a server, you can get a cost-effective system and still have a Dual Core or X2 cpu.

Maintenance:
While the initial configuration will take some time. After all you have to do it all by yourself. This results in a "less bloat" situation. Since your are more likely to activate only what you need if starting more or less from scratch, rather than if you had a "activate-everything-just-to-be-on-the-safe-side" default config file, where you had to specifically disable stuff you didnt want.

Once the initial configuration is done, updates are usually painless. Portage will inform you if any config files need updating. With etc-update or dispatch-conf you can easily select which configs can be blindly updated and which need to be edited by hand.
It is rather rare that I had to manually update configs by hand. Its only when a new package version changes the way it does things or changes the syntax of its config file. I dont know how much of headache this is on other distros, but obviously BIG upgrades should never be taken lightly. e.g. PHP4 to PHP5.

Thinks to be wary of, or simply be careful about are major updates of things like: Apache, PHP, MySQL, Postfix.
There was one instance where Apache especially caused a major headache for me, since it shifted around some config file locations and some of our vhosts entries did not work properly anymore.
The problem was easily fixed once I dug a little deeper into the apache projects Website. It was mostly my fault anyways, since I didnt inform myself ahead of time of potential issues with updates.

Golden Rule of Thumb: If you have something mission critical like postfix or apache, make sure you read the changelogs and stuff, BEFORE you update it.
Security updates should usually not pose a problem. Especially if applied via glsa-check.

The only other big problem I ran into with gentoo, or rather a colleague of mine, was a update of amavis. The syntax changed slightly and this resulted in our mails being stuck in an infinite loop. Easily overlooked but also easily solved.

All that being said, if you want to make automatic system updates for everything except your mission critical software, then make extensive use of the package.mask config.
You can define there if a certain package should only be installed with a specific version number. To be on the safe side, you should also never delete that package source file from /usr/portage/distfiles, just in case the file is no longer available from the mirrors.

Conclusion:

From my personal experience I can say, that Gentoo can be just as stable or unstable as any other distro. I have only had two major issues with it on a production system, which was mostly our own fault. That being said, you should always choose the distro you are most familiar with for a production system. Regardless of what the distro might be.
And with "most familiar" I do not only mean your level expertise with that distro, but also the availability of support. Corporate or community wise.

Supportwise I prefer gentoo, since pretty much ANY question I ever had about gentoo, was either solved through forums.gentoo.org or www.gentoo-wiki.com.
I also find that the gentoo community in general seems more knowledgeable about linux than for example the ubuntu community. (only my personal opinion.)

My recommendation would be for you, choose what your familiar with AND what you can get support for. Preferrably community support, unless your willing and able to spend the cash required for commercial support.

crackytron · n00b Joined: 16 Nov 2007 Posts: 24

Thanks for all the advice. I'm not sure if rackspace actually allow you to reinstall your OS (after all this :/), but if anyone knows anything about linux in general it'd be nice to know the answer to this question:

On our current rackspace/tdmgroup servers which I figure are CentOS or RHEL (any help in how to identify which/version etc would be brilliant), what is the best way to make sure they are up to date? I'm not sure the person previous to me was particularly hot on updating things, so I have a bad feeling that there our servers are full of holes. It appears to have RPM but not yum or anything.

I also have a feeling that upgrading lots of things will break lots of things. So its a tradeoff between risking lots of downtime (argh), or knowing that our server is most likely horrendously insecure. Fuck, for all I know, rackspace are doing it for us. The previous sysadmin left me with NOTHING to work with. There's like a black hole in the history of *what the fuck happened with our servers* for about 2 years.

The servers do appear to be extremely stable, however.
_________________
welp

Urban Cowboy · n00b Joined: 09 Oct 2007 Posts: 64

The problem is.. if it went to production, you'd have to have tech support - which would be a nightmare. There are way too many variables and configurations with gentoo.

You'd have to pigeonhole gentoo to specific cflags, use flags, packages, drivers - which would essentially ruin exactly what makes gentoo so awesome - configure-ability.

Imagine someone explaining how they deleted python and can't re-emerge it and need help.
_________________
Anything worth doing is worth over-doing. Moderation is for cowards.

Suicidal · l33t Joined: 30 Jul 2003 Posts: 959 Location: /dev/null

Gentoo is fine as a production server, based on you build everything on a image host.

I had at one time ~20 gentoo servers.

I had one that I did all of the building on, I ran it like so:

Kasumi_Ninja · Posted: Mon Dec 03, 2007 4:11 pm Post subject:

gringo · Advocate Joined: 27 Apr 2003 Posts: 3793

red-wolf76 · Posted: Tue Dec 04, 2007 11:18 pm Post subject:

The two posts above me seem like a sane way of doing things. You'd lose a bit of the rice if you had one testbed and one powerbox for compiling stuff and only used generic 686 code that will run on all of them. But in essence you'd not be doing anything different from what binary distros do. You'd be compiling binary packages that your production box(en) will use. The test server will give you some amount of warning if you're about to bork things up big time.

Gentoo is a long-winded install and if you're not familiar with Linux you have to step up to the plate and be willing to learn, but after I've tried some of the more "mainstream" distros out there (suse, debian, [K-X]Ubuntu), I'm still in love with it...
_________________
0mFg, G3nt00 r0X0r$ T3h B1g!1111

Use sane CFLAGS! If for no other reason, do it for the lulz!