Raid5 isn't so redundant after all? - SW Raid5 failure

Gr|ffous · n00b Joined: 27 Apr 2004 Posts: 35

Hi Guys,

I recently went through the highly traumatic experience of loosing a disk in my file/backend server. No worries I thought, "I raided it good!"

Specifically, I'm running raid1 for /boot, and 3 raid-5 arrays across 4 Pata IDE disks.
At the time I was copying a movie over nfs to the largest of the stripes when it just went splat. The entire system hung which was definitely not cool, but I put that down to that fact I was using IDE rather then Scsi, old Pata IDE, and using onboard and highpoint controllers - you can't expect too much out of that lot can you? Or should I be expecting more?

I had to hard reset it, and it came up missing a disk in the bios, although oddly gentoo found it again albeit at the wrong ultra/dma level. I rebooted again and the bios found it this time - I guess it just needed a rest

I'll point out at this point that I realise the disk is probably poked and won't last for much longer - but still, a tired disk in a raid-5 is still better then a degraded raid5 right?

The mirror obviously came up fine, the other 2 raid fives started ok (degraded, but operational), but my main media raid was gone!!

No matter how much I yelled at mdadm, it just wasn't coming back again, and I'd get these messages in my /var/log/messages

glory raid5: cannot start dirty degraded array for md3

The first 2 degraded raid 5s were degraded, but not dirty, so I was able to hotadd the 'spare' (original) disk back in, which reconstructed and ran fine.
md3 however was "degraded and dirty" - which is something I didn't really understand at first.

Degraded simply means it's missing a disk but the data is still ok.
Dirty means that the consistancy of the raid isn't known, so I needs to rebuild itself.. which isn a little confusing, and a new one to me.

However when you have both at once you are seemingly screwed as the md doesn't trust itself across the 3 remaining disks (it's dirty), and without a 4th for parity checksums it can't fix itself, and you can't add the 4th until it's not dirty

Around in hopeless circles we go....

I was able to finally bring the raid online again with a mdadm -f which caused the raid to suddenly not be dirty, and only be degraded - allowing me to hotadd my 4th disk back in, rebuild, and fsck.

So I guess I"m just trying to figure out what I can do about this for future.

According to the md manpage located here [url] http://dc.qut.edu.au/cgi-bin/man/man2html?4+md[/url] This will happen any time I'm writing to an array and the power goes out, or simply the system doesn't shutdown properly marking the raid as clean.
Is this just a fundemental flaw in raid5 that no one ever told me about? Or is this only a problem for S/W raid5?

To me the chances of my fileserver loosing a disk and hanging again seem quite high, and from what just happened it appears to me, that contry to what I've been taught so far - raid5 CAN'T survive loosing a disk in these circumstances. Must have been in the fineprint, cause that's news to me.

Am I missing something here?
Helpful comments appreciated.

Thanks,
Gr|ffous

qmp · Posted: Fri Mar 11, 2005 2:40 pm Post subject:

well I cant offer help but:

Dont ever use raid5 for enterprise applications. Get Raid 1, 10 and a streamer. Its worth it.

(6 hds in 6 months)

Gr|ffous · n00b Joined: 27 Apr 2004 Posts: 35

Thanks for the reply all the same.
In this case, it's just a home raid, but tradionally raid5 was domain of the enterprise. I'm surprised to hear people suggesting against it.

In my case, I guess things like batteries on the raid cards mean that whatever stripe it may have been in the middle of, it can safely resume after a power failure - thus solving the problem of a dirty array.
If the array isn't dirty, then it can loose a disk just like you would expect.

Gr|ffous

Kurse · Posted: Fri Mar 11, 2005 4:09 pm Post subject:

Dunno if he was refering to software raid5 only, but hardware raid5 is definately common on enterprise class servers. Works very well.

mpsii · l33t Joined: 11 Mar 2003 Posts: 658 Location: Jackson, TN

Kurse · Posted: Fri Mar 11, 2005 4:17 pm Post subject:

qmp · Posted: Fri Mar 11, 2005 6:04 pm Post subject:

Raid 5:

Disadvantages:
- Software:uses a lot of cpu Hardware: Promise crap controllers (e.g. SX6000) dont have the cpu power to support 5 fast ide drives in raid 5.
- Hardware Controllers are ridiculously expensive (get twice the disks for the money instead)
- You cannot swap disks for backup purposes (e.g. remove one of the disks in Raid 1 on Friday and put another one).

Advantages (over Raid 1/10):
- Cheaper.

I use a Raid 1 at home + Streamer (daily) + CD-R (weekly).

You ask me why? I had 6 hds crashing on me in 6 months in 2002 (IBM-Deathstar+Maxtor). Dont ask me what I would use for business ;-)

Either you have important data or you dont. I think I do (and I learned from the crashes that I really really had a hard time without that data). Its worth couple hundred bucks to me to know I am safe.

yottabit · Posted: Fri Mar 11, 2005 6:29 pm Post subject:

I agree with everything said, but I do want to say something in IBM/Hitachi's defense... IBM did produce that lemon drive model and there was quite a stink about it, but all of their other models have been and are rock-solid. I will only sell IBM/Hitachi disks for ATA/SATA systems. I have eight of them myself, of various capacities and interfaces.

(Ever since WDC went to liquid bearings right around the end of the 1990s their drives have been total crap. From King to dirt, flat. Now they're existing on their Cadillac name without the proof to back up claims. For ATA/SATA drives I will only use IBM/Hitachi. Don't even get me started on Seagate or Maxtor.)

/tangent-off
_________________
Play The Hitchhiker's Guide to the Galaxy!

ledskof · Apprentice Joined: 22 Oct 2002 Posts: 209

Over the years I've had a lot of enterprise (both servers and workstations) experience with IBM/Hitachi, Maxtor, Seagate, WesternDigital, Fujitsu, and Quantum. (and some others). The only hard drives I've had a significantly larger amount of trouble with were fujitsu and quantum. Of course I had the problems with the deathstars but at least that was a known model to avoid.

Speaking of the deathstars. I've got a couple of those that I bought back right before the problems started cropping up... They are still in RAID0 on a gaming workstation

yottabit · Posted: Fri Mar 11, 2005 7:20 pm Post subject:

Yes, I agree. I have 15 years of experience with all sorts of computing systems. For consumer disks, my order of preference would be the following:

Hitachi/IBM
WDC
Seagate
Maxtor
Quantum (they don't exist now)
Fujitsu (they don't exist now, either)
Seagate (based on their drives 6 years ago)

It's worth noting that I place lightyears of difference between Hitachi and WDC, and then group WDC, Seagate, and Maxtor very close together, and then a couple more lightyears between Maxtor and Quantum.

It was a happy day for me when Fujitsu Storage was bought out since many many notebooks used Fujitsu's 2.5" disk drives (because they were so much cheaper than Hitachi/IBM).

All of my product judgments are based on my own empirical experience over quite a long time.

J
_________________
Play The Hitchhiker's Guide to the Galaxy!

futile1 · n00b Joined: 23 Feb 2004 Posts: 20

I think your problem with your software raid 5 was the fact that the system froze. Usually when a drive is lost it is marked bad and no more attempts to access the device are made. The raid should still be in working order and the system will continue to function as normal. Then after the degraded raid is unmounted (marked clean) and a new drive is inserted it will rebuild to an undegraded array. I wouldn't blame the concept of a software raid 5 for your problems, but some unstable code somewhere in the kernel.
_________________
AMD64 3000+, MSI K8N neo2-platinum, 1gb dual-channel DDR, Antec P160 case