Goodbye my Gentoo servers, I hardly knew you

kashani

Intro:
It all started in Dec '05. I arrived at my new job at a new startup as their jack of all trades IT guy for http://jumpcut.com/ They were running on three dev boxes with CentOS and having a number of samba issues, cvs, and apache issues. I'd been running Gentoo for about three and half years at this point and felt it could do everything we wanted and make my life easier. As the only sys admin, network engineer, desktop support, etc guy I took a vote and Gentoo won.

I: Fix the office stuff
Consolidated dev into one CentOS box. Installed Gentoo onto sys01 which was now free to be CVS with cvsd, which wasn't in the Centos package, Bugzilla, Cacti, net-snmp, RT, Nagios, Samba 3.0.22c or whatever fixed what Centos' Samba 3.0.14 couldn't do with the latests XP patches. Additionally followed the nice how-to the Gentoo Wiki to integrate Samba into the existing active directory.

Number of packages I didn't have to build RPMs for: 150+ mostly because of RT and Bugzilla deps.
Total time: two weeks for building, configuring packages, testing, and releasing to all employees

II: Moving more to Gentoo
Dev had standardized on PHP and then used a home built version of PHP 5.1.1 and Mysql 5.0.11 since neither existed in our CentOS. Additionally their original consultant did not setup software RAID in any of the dev machines. It was a bit of a mess, but we were early in the dev cycle so weird dependencies on custom builds had not crept in.

Built dev01 with LVM and software RAID, Apache 2.0, PHP 5.1.2 w/hardened patch and limited subset of functions to reduce binary size, current cvs ffmpeg and assorted libs, etc. Moving devs over and set each up with their own Apache instance. No one noticed a difference other than they really liked the default Gentoo colors in their terminal.

Number of packages I didn't have to build RPMs for: 50+ mostly due to ffmpeg and PHP required libs.
Total time: one week to build, configure, test, and release.

III: Dev and staging fully on Gentoo
dev02 and dev03 rebuilt with LVM and software RAID. dev02 becomes dev Mysql 5.0 and dev03 becomes dev NFS storage.

Number of packages I didn't have to build RPMs for: none
Total time: four hours as I used a stage4 I made from dev01

IV : Setting up production on Gentoo
We used what we called minimal stage4s generated from to dev to build the eventual twenty machines that ran the site. Each class of machine eventually got it's own stage4 like db-stage4, www-stage4, which allowed us to install a single machine in one hour or roughly six machines in one and a half hours which is the most we ever received at one time.
We rebuilt the database servers as x86_64 eight months later as well as the dev db server, but otherwise updated machines as packages became available. All new ebuilds were tested on dev, then staging, and then made it into production. Generally updates made it into production within three weeks of the initial package update hitting portage. Some hit portage quicker if they were security issues.

Total number of additional packages I never had to build: 30+
web stats, Postfix, postgrey are some of the highlights.
Total number of Gentoo caused problems in production: 0
Uptime: 99.99%, stupid power breaker in our colo.

V: How are we doing?
I've worked in a few large shops of 100+ machines and some small shops. As a one man shop I do far less work and update faster with Gentoo than I ever did with Redhat, Fedora, or Solaris in other < 50 machine shops. Remember as a startup being able to get new versions, especially versions that fix issues quickly and easily is key. Additionally using Gentoo ebuilds was vastly superior to any package I could build on my own. I don't have the time to QA in house packages and even if I did Gentoo builds are used by thousands of people rather than just one.
I don't come out so well against a large shop. However without the infrastructure or the budget to pay for large shop tools we did quite well. Additionally I am most comfortable in Gentoo and there is something to be said for using the tools you're most familiar with.
Where Gentoo has really shines is in projects that fail. No really. We've all done that "hey lets try upgrading to Apache 2.2 and see how well it works." In Gentoo you change a few lines, emerge apache, run some tests, realize it's not quite there, change a few lines, emerge apache again, and you're back to where you started. Total time about two hours.
Or even projects that go somewhere. "Hey I need X packages for testing." Gentoo installs, some minor tweaks, and hand off to the dev. When we go to production I know I can get the same package because I let Gentoo do the work rather than half ass a build because I didn't have time for non production issues when the project had no priority. Naturally there some changes in config in production, but we can go to production faster without having to repackage, re QA, and then release.

VI: Goodbye Gentoo servers
As of last week we are now owned by Yahoo! which uses Redhat. We're starting the process of moving to their infrastructure and it looks to be straightforward. I'm sad to see our Gentoo servers go, but trying to convince Yahoo! to run a distro they have no experience with just for us isn't going to happen.
I did however get geek points from a number of Yahoo engineers who thought it was very cool that we used Gentoo to build our system. Of course I never said things like, "Gentoo rulez" when I describe our infrastructure and I did say "We were relying on a number of bleeding edge packages so we picked a distro that catered to that. There are some quirks, but if you're prepared to deal with that it can be great."

kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape.

andrewd18 · Posted: Thu Oct 05, 2006 3:39 am Post subject:

I was thinkin' this was gonna be another "Wah, I hate Gentoo" thread. Glad to see you enjoyed your experience.

_________________
Keep Your Toolchain Stable! - emwrap.sh

There's no place like ::1

padoor · Advocate Joined: 30 Dec 2005 Posts: 4185 Location: india

it is a pity yahoo does not want part of gentoo.
_________________
reach out a little bit more to catch it (DON'T BELIEVE the advocate part under my user name)

anello · Posted: Thu Oct 05, 2006 5:32 am Post subject:

Thank you kashani!

That's how a "goodbye/I'm back" post gotta look like.

Finally something that's worth the time reading.

It's sad to hear that yahoo is insisting on RedHat. I would not expect a conservative attitude from such a big company. On the other side, it makes the work of the IT Dep. much easier.
_________________
Antonino Catinello | http://catinello.eu

vibrokatana · Guru Joined: 09 Feb 2006 Posts: 328 Location: o0o0oo

The problem with large companies is that most decisions are made on the top and is unchangable. When you move to a structure like that it hurts performance on the lower levels, because the leaves hate to be dragged in the wind. The larger the structure, the more chance for resistance, expecially when leaves are transplanted.

Gentoo Servers!!!! I love gentoo; The server I set up at my high school (im now in college) is nearing its one year mark, as in uptime and age

Red hat is nice when you have the tech support, let them fix it when you dont know something. Of course this leads to lazy admins, but then again, its better then lazy windows admins.
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

mark_alec · Posted: Thu Oct 05, 2006 8:00 am Post subject:

Sad to see that your Gentoo boxen have been migrated. Glad you have had a good experience using Gentoo all these years.
_________________
www.gentoo.org.au || #gentoo-au

uxbod · Guru Joined: 30 Dec 2002 Posts: 474 Location: UK

I hate to say but personally I feel that Gentoo will not easily make it into the Blue Chip arena until such time one of the large vendors ie. HP, Dell provide commercial support for it. RedHat and Suse have the commercial ties, and upper management look far more favourable on these distros because of that. Commercial acceptance and support does not equal a better product though

BastianBalthazarBux · Retired Dev Joined: 10 Dec 2004 Posts: 78

Happy to see you had good time with gentoo.
I've also apreciated your frequent answers on the server mailing list, hope that you decide to keep at least some gentoo box and still lurk on our mail list.
Having prepared ppl like you help us is a great bonus to the community.

krgds, Francesco R.

kierans · n00b Joined: 07 Apr 2005 Posts: 51

I'm in my final year of university (down in the land of Australia), and I've been using Gentoo as my distro for a few years. I even got a guy that I did group projects with into it. In our final year this year, myself, my Gentoo friend and some other guys had to do a full year project for a client external to university. Turns out our project wasn't going to be the standard software development stuff that they usually throw at you, to see if you've being paying attention the last few years. The project was more a research project, which means we needed a server box that was highly configurable and easy to rollback in case something went wrong. So my friend and I turned out to be the main code monkeys. Where did we turn - Gentoo of course

Turns out my old man was upgrading, so I got his old machine, installed Gentoo within a couple of days, whacked in on the back of the uni network, and we were off. Subversion, apache2, and all the Java tools (being the language of the project) were all installed quickly and efficiently. When we wanted to try something, we let portage do the work, had a play and if it didn't work, a couple of hours to undo the damage. If it did work, it stayed. In terms of software configuration the biggest hurdle was documentation of some of the software, but that's not Gentoo's fault.

All in all the group agreed that if it wasn't for Gentoo, we would have never gotten the project done in the allocated time span. Thanks to portage, we didn't have to spend huge amounts of time dealing with dependencies, or getting the damn thing compiled. No we sat back while Gentoo did it's thing, and got the important stuff done.

vibrokatana · Guru Joined: 09 Feb 2006 Posts: 328 Location: o0o0oo

Doesnt portage have an option to grab precompiled packages? It would be interesting if someone were to start a commercialized branch of gentoo that had the option to use bin packages for most of the packages (or atleast a system to drop request for precompiled).

I dont mind the other distros, but I love gentoo, it doesnt complain, it works great, and I think businesses would love it IF they were to know about it.

I only figured out about gentoo because I heard about it from the 3rd person from one of my friends. Maybe we should start a gentoo awarness initiative?
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

amne · Posted: Thu Oct 05, 2006 5:33 pm Post subject:

Nice to hear Gentoo doesn't suck.

_________________
Dinosaur week! (Ok, this thread is so last week)

kashani · Posted: Thu Oct 05, 2006 6:10 pm Post subject:

I figured it was about time Gentoo Chat had a good sucess story rather than some of those *other* posts.

In regards to Yahoo I still remain the Jumpcut Tech Guy or whatever my title ends up being, but Servers, Storage, Networking, etc are all individual groups outside of us at Y! They install and manage the servers that us, Flickr, de.lic.io.us, Yahoo Mail, etc all use. Trying to add Gentoo to that infrastructure would be impossible since they're already managing thousands of RedHat servers. However a few Y! engineers expressed intrest in setting up a few boxes to play with.

Of course some of the local boxes in the office won't be changing like sys01 and my workstation. I'll still be on the mailing lists and hanging around in the forums. I also keep meaning to sign up for x86 and amd64 arch testing, but have never had the time as you might imagine. My personal webserver will also stay Gentoo so I'm not going anywhere, just biding me time until I can Gentoo-ize another shop.

kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape.

Zepp · Posted: Thu Oct 05, 2006 7:14 pm Post subject:

Finally a nice goodbye (sorta) story

. Sorry to hear, but glad it worked out so well while it lasted.

and well at least they didn't make ya switch to windows

steveL · Posted: Sun Oct 08, 2006 1:16 pm Post subject:

Yeah good to hear a success story, even if that company's changed it's infrastructure.

dundas · Guru Joined: 16 Dec 2004 Posts: 317 Location: China, Earth

The hardest thing for me is also to convince other engineers to realize the beauty of Gentoo, to overcome the compiling fear and linux fear in general.

But damn it, I'm trying and keeping myself updated wiz the latest Gentoo stuff.
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer

mudrii · l33t Joined: 26 Jun 2003 Posts: 789 Location: Singapore

Yup really nice story
_________________
www.gentoo.ro

vibrokatana · Guru Joined: 09 Feb 2006 Posts: 328 Location: o0o0oo

most people dont even know what compiling is... people fear the unknown...
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

renrutal · Posted: Thu Oct 12, 2006 9:50 pm Post subject:

kashani, sir, you really should set a page, a blog, or a very complete guide about how to manage 100+ Gentoo machines.

Many people here, including me, know Gentoo is awesome, but they don't have a clue about how to do that miraculous feat.

deanpence · Apprentice Joined: 08 Nov 2004 Posts: 158 Location: Earth

vibrokatana · Guru Joined: 09 Feb 2006 Posts: 328 Location: o0o0oo

Whenever someone asks me how to make an init script I always ask which distro they are running, if its gentoo ill gladly help them, but red hat needs to despirately do something with their startup, its a royal pain without a gui
_________________
My Systems - "I suggest the whole thing be coded in whitespace. Henceforth the code will be obscure and functional at the same time."

karsten_s · Posted: Sat Oct 14, 2006 12:01 pm Post subject:

y'know, it's been just great reading this. Sometimes I was close to tears from sensing the passion you guys have for gentoo. And do you know what's the greatest thing about that is? I can relate! I've got only one gentoo box (started Jan 02, 06) and I love it! It may seem somewhat geekish to love a (meta-) distribution the way we love gentoo, but, hey, we do love it. Both portage and the community have become the main reasons for me turning into a linux missionary more than ever.

I'm losing my point. What did I want to say? Basically thanks, kashani, for sharing, and also thanks whoever got this thread mentioned in the gwn.
_________________

nixnut · Posted: Sat Oct 14, 2006 4:23 pm Post subject:

ErZo · n00b Joined: 23 Jul 2005 Posts: 6 Location: Sweden.

Thanks for sharing this! Awesome, really really awesome. I'm glad you're in our community :-)

Neat

What more can I say?

symbiat · n00b Joined: 20 Aug 2003 Posts: 36 Location: New York

symbiat · n00b Joined: 20 Aug 2003 Posts: 36 Location: New York