EVMS RAID5 failed--need plenty of help [SOLVED]

[jjlawren]

Apparently last night my server had some errors with one of the disks in my 4 disk RAID5 array. The following kinds of errors were repeated for several minutes (all on sdb):

Jun 28 05:01:45 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:01:45 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:01:45 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:01:49 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:01:49 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:01:49 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:01:53 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:01:53 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:01:53 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:01:56 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:01:56 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:01:56 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:02:00 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:02:00 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:02:00 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:02:03 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:02:03 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:02:03 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:02:03 backend sd 1:0:0:0: SCSI error: return code = 0x8000002
Jun 28 05:02:03 backend sdb: Current: sense key=0x3
Jun 28 05:02:03 backend ASC=0x11 ASCQ=0x4
Jun 28 05:02:03 backend end_request: I/O error, dev sdb, sector 541945335
Jun 28 05:02:49 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:02:49 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:02:49 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:02:53 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:02:53 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:02:53 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:02:56 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:02:56 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:02:56 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:03:00 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:03:00 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:03:00 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:03:04 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:03:04 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:03:04 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:03:07 backend ata2: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
Jun 28 05:03:07 backend ata2: status=0x51 { DriveReady SeekComplete Error }
Jun 28 05:03:07 backend ata2: error=0x40 { UncorrectableError }
Jun 28 05:03:07 backend sd 1:0:0:0: SCSI error: return code = 0x8000002
Jun 28 05:03:07 backend sdb: Current: sense key=0x3
Jun 28 05:03:07 backend ASC=0x11 ASCQ=0x4
Jun 28 05:03:07 backend end_request: I/O error, dev sdb, sector 544130991
Jun 28 05:03:08 backend raid5: read error corrected!!

From the system log, it looks like the machine rebooted after these errors. Now EVMS is reporting that the MD superblocks on the first 2 drives (of 4) are not valid, and should be removed from the region. Is there any chance of this being recovered?

[jjlawren]

Can I use mdadm in some way? I assume EVMS is just a wrapper for the kernel support.

Also, LVM2 is being used as well, but I assume that if I can solve the superblock problem, that won't be as big of a problem. EVMS does show the LVM volumes, but cannot activate the array.

[jjlawren]

Fixed with the following command:

mdadm -C /dev/md0 --level=5 --chunk=128 --layout=la --raid-devices=4 /dev/mapper/sda1 missing /dev/mapper/sdc1 /dev/mapper/sdd1

Since one disk was marked as missing, the array would not attempt to resync and effectively made it read-only (unless I made changes to it myself). I ran fsck on each of the LVM2 partitions that existed on the array and everything looked fine. Added sdb1 back into the array as a spare and let it sync.

Unfortunately it looks like sdb is actually bad, and it was removed a few hours later when it failed. But my data is intact and a replacement drive is on the way.