Long-Term Stability == Gentoo Is Broken

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

This is now the third time I'm having to rebuilt a server that I'd managed to get stable and leave alone for a while.

When I went to update some packages I found that many of the installed packages had been subjected to a "Courtesy Mask" because they were "too old". I'm pretty much wedged at this point because I can't upgrade to anything that is current enough in a single go.

Is there any expectation that this sort of behavior is going to stop? I've been using Gentoo since forever and once or twice a year I wind up with a system that gets into this state -- I get it where it works perfectly, no bugs, no nothing. And it even stays up and running for MONTHS on end. The currently broken server has 268 days of uptime on it, and I need to keep servers up for as much time in a row as possible.

This problem has become so severe that I've not installed Gentoo on a new server in over a year now. I no longer use Gentoo for my company's embedded products. But until I can migrate the last of my servers off Gentoo, could y'all please stop being a bunch of Net Nannies and trying to force people who are running production to update their entire system every other week or so?

GES · Posted: Tue Apr 26, 2011 7:19 pm Post subject:

I daily update my hosted gentoo server and workstation.
There are no such problems with them.
This is a rolling release distribution not a debian that enough to be updated annually.

avx · Advocate Joined: 21 Jun 2004 Posts: 2152

Little harsh for a first post, don't you think?

If you're running this for production, ie. making profit with it, why not pay for one of the devs to look over your servers? Or get a SUSE/Redhat-contract?

Or maybe keep a clone of the production system and try updates first on that one, as one should? For me, servers are running fine, but maybe that's just me...
_________________
++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.

aCOSwt · Last edited by aCOSwt on Tue Apr 26, 2011 7:52 pm; edited 2 times in total

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

I have 18 years of experience with Linux, which puts me up there with everyone else who's used Linux since Linux was distributed on floppies. I picked Gentoo years ago because those other distros are giant bloated piles of crud.

There's no reason for "Courtesy masks", which is what broke me this time. I'm not going to upgrade X -- the thing that seems to be in the way -- until I take the server down. The server is on a UPS, so it isn't going down any time soon. There's also no need for someone with 30 years of UNIX kernel experience to "hire a dev", unless the goal is to force people into hiring developers or buying service contracts. I fix my own kernels, thank-you.

As for updating your "server" every day, I run production 24/7. That means I can't afford to have some package come out with an "Oops!" and then emerge for the next few hours or waste time chasing every single last update every single day.

Just tell people to quit adding "courtesy masks". I'm not a pimple-faced teenager or college geek who needs to have her hand held.

aCOSwt · Posted: Tue Apr 26, 2011 7:55 pm Post subject:

John R. Graham

SlashBeast · Retired Dev Joined: 23 May 2006 Posts: 2922

@aCOSwt I found you annoying like frack, behave yourself and maybe limit your posts to something worth writting, about topic.

@tallgirl: Gentoo need much more love on production, there is no releases of it with frozen packages like ubuntu or redhat tho.

As I use gentoo on production I have some things deployed to make it painless.

- Before upgrade any package, my portage's bashrc is doing binary package including all config files.
- I have binhosts to build packages.
- I also have ccache, when i really have to compile something.
- Daily incramental backup, lvm snapshot where it is possible
- I never upgrade system withour reviewing what is to update, you never know what just Vapier pushed broken to tree and was rsynced to mirrors before he fixed it in gentoo's cvs. True story.

Simba7 · Posted: Tue Apr 26, 2011 8:16 pm Post subject:

Oh, everyone chill. I've been running Gentoo solid for a few years now. Sure, you need an update now and then (updates, fixed exploits, etc), but it's sure as hell better than a bloated pile of crap. Gentoo is the *ONLY* reason I moved to Linux from FreeBSD.

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

The issue was really simple and I stated it from the outset -- I keep seeing these "Courtesy Masks". They aren't a "courtesy". If someone has major control issues and they need to break people's systems, tell them to go work for Microsoft.

And no, the servers DO NOT GO DOWN. There is no "monthly maintenance schedule" that allows them to be shut down.

And yes, I have a VM for making sure whatever I decide has to be upgraded can be made to work.

I think y'all live in a different world than commercial production. My rackspace provider has not had a second of power or network down time in more than 2 years. The server in question has had 15 minutes of unscheduled downtime in 268 days. That's 99.995% uptime, which is okay by me. Tomorrow it will be 15 minutes in 269 days. Half and hour or an hour of "scheduled server downtime" a month is 99.9% uptime, which sucks.

I have managed to make the upgrade I needed to make -- I unmasked a boat load of "courtesy masks". I'm also building an Ubuntu server (LTS) that will replace the Gentoo system whenever I get a chance, which could still be months away.

Gentoo is much better than dealing with Red Hat, which is what I used from the early 90's until switching to Gentoo, but y'all have to figure out how to keep people from breaking stuff for no good reason.

Raptor85 · Apprentice Joined: 29 Dec 2007 Posts: 208

My servers are updated every 6 months or whenever there's a critical update that applies to me, so long as you watch what you're updating, and of course always build and test in the test environment first, it's fine. The masks are easy enough to remove, and it's easy enough to cache ebuilds and distfiles locally (you should be doing this anyways on a server environment). Binhost is definitely the way to go for production servers. Other than the bandwidth bill i forgot to pay a few months back resulting in a temporarily downed server (whoops) most of my machines have > 400 days uptime. (well, had, i just did kernel updates across all my computers, except my netbook, still can't get a 2.6 kernel small enough to fit in the 1 meg SOC :/ )

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Long-term ignorance of how a distro works is not long-term stability. If you're installing Gentoo in production then letting it go 290 days without even so much as a glsa-check — security updates — and then coming here and whining that we don't have a magic update fairy for people living under a rock, then sorry but in all honesty you shouldn't be administrating Linux systems.

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

Simba7 · Posted: Wed Apr 27, 2011 12:14 am Post subject:

gentoo-dev · Apprentice Joined: 24 Jan 2006 Posts: 172

phajdan.jr · Posted: Wed Apr 27, 2011 10:52 am Post subject:

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

phajdan.jr · Posted: Thu Apr 28, 2011 6:09 am Post subject:

So you unmasked the packages and the update apparently succeeded. What's the problem then? Maintaining older packages is a maintenance burden, and we don't always have manpower for that. If you have or want to maintain them locally, and want Gentoo to change, consider becoming a developer and helping maintaining those packages in Gentoo. Or become a proxy maintainer. This has a chance to work, but I doubt whether complaints like this thread can change something.

Please be constructive. If you want, you can publish an overlay with those older packages unmasked (in fact, people have been doing just that with for example KDE 3, which is no longer in portage). If it's useful for other people, they will take that.
_________________
http://phajdan-jr.blogspot.com/

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

krinn · Watchman Joined: 02 May 2003 Posts: 7470

you seems to love your uptime status more than anything, well, i'm pretty sure your boss could live with a 2 minutes offline status to boot a new kernel to avoid your server been compromise and taken down (by taken down i don't mean offline, sadly still online, but no more doing its tasks now that the server is doing the hacker's tasks) for X times and than another X times to investigate the hole. And worst, than again X times (in the rush) to fill the holes to get your server back to online with a safe status.
Because you can take 1 year to build a kernel (lol) but your server won't take much time to boot it (if you're doing a good job, i mean no mistake when doing it, an easy task, that we can all sometimes fail at first try)

for your programs, portage never remove your ebuild (except if your a --depclean freak, no comment on that), you can just move them in an overlay to keep using them, masking ones you don't like and unmasking ones you wish to keep...

And maintaining an old package isn't just "if it build, it work". Securities issue apply to a newer package might not get easy to apply to the same package with an old version.
So even your "old" version still build, it's not a proof of sanity, just a proof it still can be built.
Why would you finally expect gentoo devs doing what the author isn't doing ? Pickup any program you wish, and tells its author "hey, i'm still using v0.3 and i see you've update v1.3 to 1.4 to correct this bug, but version 0.3 also have the bug, why aren't you updating the 0.3 so ?"

devs, distro... make choices, if you don't agree with them, use another one. The good part is that gentoo is flexible enough that you still have many ways to not agree with the changes and still don't apply them.
I wonder what you will wrote when you get your ubuntu up and running, because i doubt you will have that many exits gentoo gave you to avoid anything they will do that you don't agree with them. What about for example if you don't like their new unity desktop? Does they gave their users the choice to not using it, does they gave you choice of what desktop you wish use? (this are real questions, answer might be yes, i don't know anything about ubuntu). And to my knowledge, i don't know any ubuntu v8 user saying they keep updating programs to correct bugs in it. It's just v8 is bug, use ubuntu v9, v9 is bug? ok go with v10...

You're using a distro that is so easy that you can just don't use the ebuild and use the source file yourself directly, maintaining it and using it as you wish. It's also doable with a binary distro, it just take more time because the further you will take that path the higher the number of sources you will need to keep because of the dependencies list getting bigger (the common bloat people rant against).

And remember, like we said here (france), so it might be a bad translation : "grass looks greener elsewhere"

tallgirl · n00b Joined: 26 Apr 2011 Posts: 14

Krinn,

I'm the owner of the company. I'm the one who makes the decisions.

For some applications two minutes of downtime once a month for a reboot is entirely too much. By the time applications have been shutdown, the box rebooted, and the applications up and running again, 2 minutes can turn into an eternity. There is no such thing as a "weekend" in a 24/7 production environment. Software is written to run 24/7 with all the attention to detail that's required to do that.

Anyway, thanks for the feedback. It's convinced me that Gentoo is completely inappropriate for a high-availability environment.
_________________
Big: AMD Phenom quad core, 3.2GHz, 8GB DDR3, 3TB Disk, Dual 1000BaseT NICs.
Little: AMD Geode, 400MHz, 256MB DDR, 16GB CF, 100BaseT NIC.

Seek balance in life.

John R. Graham · Posted: Fri Apr 29, 2011 9:05 pm Post subject:

tallgirl,

I think enough has been said and I think you may have legitimate reasons for the decision you've come to.

However, I'd like to give you one little piece of justification for why the Gentoo Portage tree is managed the way it is. Gentoo's main servers' Internet bandwidth is donated (by OSUOSL, I believe) to Gentoo, not paid for. If all extremely old (but not known to be harmful) ebuilds were left in the tree then "emerge --sync" time for everyone and outbound bandwidth for our benefactors, would go up. We're incented to keep the tree lean for both of those reasons. The "Courtesy Mask" you're railing about really is a courtesy; it's much more friendly than just deleting the ebuild without warning. Maintenance effort is also an issue, possibly equally as important as the bandwidth issue; Gentoo seems to be perennially understaffed and reducing the number of supported versions of a package definitely has a positive impact on maintenance effort.

Now, that all said, I'm not completely sure that Gentoo has perfectly balanced all of these factors today, but there are legitimate reasons for doing things the way that we do.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.