raid5 failed... suggestions?

SeeksTheMoon · Apprentice Joined: 24 Sep 2003 Posts: 163

I am using hardened gentoo with a softraid level 5 with dmcrypt and XFS upon three identical disks sd(abc), which are about 4 months old. And today this raid suddenly failed. Well not that suddenly, because mdadm sent me an email that sda failed two days ago but I kind of overlooked this mail and now as I copied data to that raid, sdc failed too, leaving me with [_U_], which technically means that the raid should be destroyed

The interesting part is, that all the smart data from all three drives is ok, but the kernel.log is a huge sh*tstorm of I/O errors, corrected read errors and uncorrectable read errors on several blocks. I made sure to run badblocks and filesystem checks after I created that raid. So what exactly is happening (and broken) here? I cannot believe that all those sectors suddenly break and cause the raid to fail.

The next question is: What can I do now? The raid wiki says to shutdown the machine, replace the drives, power up and wait until the raid has been recreated. But I have this feeling that this may be a bad idea and might do something while the system is still running, but I don't know what exactly...

Any suggestions?

SeeksTheMoon · Apprentice Joined: 24 Sep 2003 Posts: 163

I was able to make a complete recursive directory listing and saved it and I was able to copy about 5 unimportant textfiles from 6.6k files. yeah, this rocks...

eccerr0r · Posted: Thu Sep 30, 2010 6:17 am Post subject:

usually unless something really catastrophic happened, having two disks fail at the same time is unlikely, and more like perhaps controller or power failure. See if it's possible to reassemble manually, might have to force it since it may be out of date, but yes there's likely data loss here.

You're probably screwed either way. See if the disks can stlil be read, if not, shutdown and restart and see if the disks can be read. If they can, see if you can force reassemble the array -- this is dangerous but since you probably lost everything, it can't hurt...

and yes please heed... RAID is not backup. RAID is for uptime. I still backup my RAID regularly just in case something like this happens...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

cach0rr0 · Posted: Thu Sep 30, 2010 6:22 am Post subject:

regarding the IO errors, had this on a drive specifically when one directory was read - it's like it caused the drive to go into some hideous loop or some such that i guess caused it to heat up?

I shut the machine down, completely, leave it off for a half hour or thereabouts...Fire it back up, and I can keep using it, until i hit that sector.

So shut it down and let it cool off?

My external drive is doing this now too (which, unfortunately, is my backup drive), and the same routine sorts things out at least temporarily.
If you can just get it back up and happy temporarily, you can copy off the data - fingers crossed.

Sorry not very technical, but it's been my course of action.
_________________
Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash

SeeksTheMoon · Apprentice Joined: 24 Sep 2003 Posts: 163

the disk temperatures are between 36-39C, which is fine.