Raid5 unusable after reboot (help)

Muddy

Ok here goes.
I've been messing with this for two days straight now and I'm at the end of the rope.

I have 4x Sata drives on a sil3114 pci add on card that I've setup in a raid5 group on /dev/md2 using mdadm.

I've done this four of five times now and it always ends up the same, I reboot to test and make sure the raid group
is ok and the kernel (dmesg) complains the superblock is bad.

Here is the scan after this last setup.
(my other two raid groups are raid1 and part of a lvm group.
This raid5 is not)

Muddy · Posted: Wed Jan 17, 2007 4:12 pm Post subject:

Tried it again this time with no mdadm.conf file to see what would happen, same thing.

here is the output from dmesg..

user123 · n00b Joined: 28 Jun 2005 Posts: 14

Try to assemble the array manually instead.

Stop any running arrays using mdadm -S /dev/md0 (for example).

Use mdadm -E /dev/hda1 (also an example) to examine the md-superblock on /dev/hda1.
Do this for every partition you know is part of an array and figure which devices belong to which array.

Say you've found out /dev/hda1, /dev/hdc1, /dev/hde1 are part of your raid 5 array.
Then just assemble it using mdadm --assemble /dev/md0 /dev/hda1 /dev/hdc1 /dev/hde1.

Muddy · Posted: Wed Jan 17, 2007 6:53 pm Post subject:

did that all looks good, however here is something odd.. ran monitor for the new raid5 array and check this out.

user123 · n00b Joined: 28 Jun 2005 Posts: 14

It might be something in /etc/mdadm.conf that confuses mdadm --monitor.
Just a wild guess though.

Muddy · Posted: Fri Jan 19, 2007 2:50 pm Post subject:

no, I'm at a loss at this point.
I've even tried to create the raidgroup using the new metadata specs 1.0, 1.1 and 1.2.
For whatever reason I can't create the raidgroup at all unless I use the default 0.90.
Then no matter what I try upon reboot the raidgroup fails to creaate with bad superblock in dmesg.

here is the current config file

Muddy · Posted: Mon Jan 22, 2007 7:34 pm Post subject:

Still can't get this working.

Just for kicks made it raid0 and it worked fine.

Tried Raid6 and it failed on reboot as well.

Any ideas on if it's something in the kernel?

user123 · n00b Joined: 28 Jun 2005 Posts: 14

Try upgrading to 2.6.19.2.
There's a data corruption bug in dm/dm-crypt with canceled BIOs, if i recall correctly, in earlier versions which was fixed in 2.6.19.

The fix was backported to 2.6.18.6 aswell.

Muddy · Posted: Tue Jan 23, 2007 5:10 pm Post subject:

went to 2.6.19-gentoo-r2 and created the raid5 group again, formatted it with reiserfs, updated the mdadm.conf file and rebooted.

with this.. again.

user123 · n00b Joined: 28 Jun 2005 Posts: 14

1) Does it work if you skip the whole mdadm.conf part and assemble it manually on boot?

2) Are you sure you don't have a hardware error?
Faulty disks, faulty memory, a faulty or not fully plugged in PCI IDE controller or something?

You can test memory by emerging memtest86 and add it to grub.conf and then boot it using grub.

Muddy · Posted: Tue Jan 23, 2007 5:51 pm Post subject:

Like I said I have 2x raid1 groups using lvm and it's flawless.
Only thing I can think is maybe the sil3114 pci sata card or one of the drives are goofed... however I've run the dd zero disk on them so many times now without error that I'd guess that it would of show itself by now.

user123 · n00b Joined: 28 Jun 2005 Posts: 14

Weird.. and you created your LVM volume like this (assuming md2 is your raid5 array)?
pvcreate /dev/md2
vgcreate vgraid5 /dev/md2
..

..and not:
pvcreate /dev/hda2 (assuming /dev/hda2 is part of the raid5 array "md2")
pvcreate /dev/hdc2 (assuming /dev/hdc2 is part of the raid5 array "md2")
pvcreate /dev/hde2 (assuming /dev/hde2 is part of the raid5 array "md2")

Because in the later case I could understand your corruption problems..

Muddy · Posted: Tue Jan 23, 2007 6:00 pm Post subject:

well the lvm2 groups are on md0 and m1 putting them together.
Aside from that yea, similar to what you typed in.
md2 the raid5 group that is killing me is/was going to be a stand alone volume from the lvm2 stuff.

The more I think about it the more I'm thinking udev is screwing me.
The UUID for the raid5 group changes each time I make it, should that be happening?

user123 · n00b Joined: 28 Jun 2005 Posts: 14

Hmm.. Very odd.

What if you do something like,

# Zap old superblocks to minimize the risk of md getting confused
for part in sda1 sdb1 sdc1 sdd1; do mdadm --zero-superblock /dev/$part ; done

# Zap partitions and re-read partition tables
for drive in sda sdb sdc sdd; do
dd if=/dev/zero of=/dev/$drive count=1
blockdev --rereadpt /dev/$drive
done

# Try to create array, this time not using partitions, and storing the superblock 4K into the drive instead (-e 1.2) -- just to make sure the problem doesn't have to do with a read/write problem at the end of the disk or somethign
mdadm --create /dev/md2 -l5 -n4 -e 1.2 /dev/sdb /dev/sda /dev/sdd /dev/sdc

# Stop the array
mdadm -S /dev/md2

# Try assembling it again
mdadm -A /dev/md2 /dev/sda /dev/sdb /dev/sdc /dev/sdd

# Create reiserfs on /dev/md2
mkreiserfs -q /dev/md2

# Stop the array
mdadm -S /dev/md2

# Reboot and see if the filesystem is still there after manually reassembling the device
reboot
# ..and then when the system comes up again, mdadm -A /dev/md2 /dev/sda /dev/sdb /dev/sdc /dev/sdd)

Don't forget to remove the corresponding lines from /etc/mdadm.conf

user123 · n00b Joined: 28 Jun 2005 Posts: 14

Also..

I think you said it never worked when you rebooted.
But it worked if you manually stopped the array and re-assembled it again.

Doesn't that indicate that there's something in either the shutdown- or boot-scripts that mess with those devices?

How about trying to flush the buffers with the sync command, wait a few secs and then do a hard poweroff, boot into single-user mode, then try to manually reassemble the array and see if it gives any errors.

At least that would rule out the shutdown- and init-scripts.

Muddy · Posted: Tue Jan 23, 2007 6:48 pm Post subject:

I've not been able to re-assemble after a boot.
Waiting on it to finish the current sync, then trying some of what you said.
Will update when I know more.

Muddy · Posted: Tue Jan 23, 2007 8:13 pm Post subject:

lmao, oh man I've got some screwed up stuff.

finished building the array on /dev/sd[abcd]1 and then ran mdadm -S /dev/md2

then ran the mdadm -A /dev/md2 and it failed, but for kicks I just arrowed up and hit enter a bunch of times, then the freak show started.
every few times it would actually build with only one drive (random), then once it built with two... but no more.

I'm at a total loss.

Muddy · Posted: Wed Jan 24, 2007 12:32 am Post subject:

cleaned all four drives with no partitions at all, ended up having to use the --force option on creation

Muddy · Posted: Wed Jan 24, 2007 2:53 am Post subject:

no go same crap

here is after the sync before I did the mdadm -S on /dev/md2

drescherjm · Posted: Wed Jan 24, 2007 6:15 am Post subject:

Do you have a recent version of mdadm installed?
_________________
John

My gentoo overlay
Instructons for overlay

Muddy · Posted: Wed Jan 24, 2007 3:43 pm Post subject:

drescherjm · Posted: Wed Jan 24, 2007 6:03 pm Post subject:

I thought that might be the problem as a lot of updates/changes have occurred to linux software raid in the last couple of kernels.

At home I am using that same version of mdadm with a 2.6.18 kernel but that it is at raid1 and at work we have many systems using a few different kernels and software raid 1, 5 and 6 but most of the systems still have sys-fs/mdadm-1.12.0 ( I just checked) which is no longer in portage and I assume there is no support for raid5 reshaping.
_________________
John

My gentoo overlay
Instructons for overlay

Muddy · Posted: Wed Jan 24, 2007 6:10 pm Post subject:

yea, I've been pulling what's left of my hair out for a bit now.
My best friend suggested genkernel, so giving that a shot to see if it helps.

Muddy · Posted: Mon Jan 29, 2007 9:35 pm Post subject:

genkernel did not do anything different, will do some other (outside Gentoo) trials and report back.

Laitr Keiows · Posted: Tue Jan 30, 2007 8:13 pm Post subject: