| View previous topic :: View next topic |
| Author |
Message |
redbeardmcg
Apprentice
Joined: 12 Mar 2006
Posts: 171
Location: Suburbia Massachusetts
|
 Posted: Tue Aug 12, 2008 2:16 am Post subject: Raid5 really bad day help |
 |
|
It's definitely been a while since I've been on this forum, but if there is any group of people on the internet capable of making my life better tonight it is you guys. Me and my bottle of scotch certainly hopes so...
So the story... I'm at work and I noticed my imapd was down... I try to ssh into my fileserver through my Linux router and nothing. I called home, had the girlfriend reset the file server, and bang, dead....
Booted up a Gentoo livecd and here is what mdadm seems to think....
| Code: | livecd ~ # mdadm -E /dev/sda3
mdadm: No md superblock detected on /dev/sda3
livecd ~ # mdadm -E /dev/sdb3
/dev/sdb3:
Magic : a92b4efc
Version : 00.90.00
UUID : 8c394ca9:8c59f11a:fd614d7c:a0cc1838
Creation Time : Mon Aug 11 22:04:07 2008
Raid Level : raid5
Used Dev Size : 388644416 (370.64 GiB 397.97 GB)
Array Size : 1165933248 (1111.92 GiB 1193.92 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 2
Update Time : Mon Aug 11 22:04:14 2008
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 858a0af - correct
Events : 0.4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 19 1 active sync /dev/sdb3
0 0 0 0 0 removed
1 1 8 19 1 active sync /dev/sdb3
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
|
Looks OK right, 3 out of 4 drives?
| Code: |
livecd ~ # mdadm --create /dev/md2 --assume-clean --level=5 --verbose --raid-devices=4 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/sdb3 appears to be part of a raid array:
level=raid5 devices=4 ctime=Mon Aug 11 22:04:07 2008
mdadm: /dev/sdc3 appears to contain an ext2fs file system
size=1165933248K mtime=Mon Jul 14 12:26:34 2008
mdadm: /dev/sdc3 appears to be part of a raid array:
level=raid5 devices=4 ctime=Mon Aug 11 22:04:07 2008
mdadm: /dev/sdd3 appears to be part of a raid array:
level=raid5 devices=4 ctime=Mon Aug 11 22:04:07 2008
mdadm: size set to 388644416K
Continue creating array? y
mdadm: array /dev/md2 started.
(from dmesg)
raid5: device sdd3 operational as raid disk 3
raid5: device sdc3 operational as raid disk 2
raid5: device sdb3 operational as raid disk 1
raid5: allocated 4211kB for md2
raid5: raid level 5 set md2 active with 3 out of 4 devices, algorithm 2
RAID5 conf printout:
--- rd:4 wd:3
disk 1, o:1, dev:sdb3
disk 2, o:1, dev:sdc3
disk 3, o:1, dev:sdd3
livecd ~ # mkdir /mnt/recover
livecd ~ # mount /dev/md2 /mnt/recover/
mount: unknown filesystem type 'ext4' |
EXTENSION 4??????? WTF???????? This was formatted EXT3???
| Code: | livecd ~ # fsck.ext3 /dev/md2
e2fsck 1.40.8 (13-Mar-2008)
fsck.ext3: Filesystem revision too high while trying to open /dev/md2
The filesystem revision is apparently too high for this version of e2fsck.
(Or the filesystem superblock is corrupt) |
SO....
Have I lost it all? I'd probably snap if I did, EVERYTHING is on there.
I found a post from someone that never got an answer and seems to have the same problem:
http://lkml.org/lkml/2007/7/12/156
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you. |
|
| Back to top |
|
 |
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23071
|
| Posted: Tue Aug 12, 2008 2:53 am Post subject: |
|
|
On the theory that your superblock is slightly corrupt and managed to somehow obtain a version number that flags it as ext4, try directing fsck to use an alternate superblock. The exact location will vary based on how the filesystem was formatted. Try at 8193 or 32768.
Also, what does file -s /dev/md2 show? Does head -c8192 /dev/md2 | od -tx1z given any indication that some other data has replaced your filesystem? |
|
| Back to top |
|
|
redbeardmcg
Apprentice
Joined: 12 Mar 2006
Posts: 171
Location: Suburbia Massachusetts
|
| Posted: Tue Aug 12, 2008 1:24 pm Post subject: |
|
|
Thanks for the reply!
Unfortunately I don't have a console on the box and I am at work so I will have to try this when I get home unless I can get the girlfriend to boot to the livecd and fire up sshd.
Any further ideas are all welcome!
-Ryan
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you. |
|
| Back to top |
|
|
redbeardmcg
Apprentice
Joined: 12 Mar 2006
Posts: 171
Location: Suburbia Massachusetts
|
| Posted: Wed Aug 13, 2008 6:29 pm Post subject: |
|
|
It looks like file isn't on the livecd
| Code: | livecd ~ # find / -name file -print
livecd ~ # |
Here is the only readable text in the first 8192 bytes of md2 that seems to have anything to do with the filesystem / superblock:
| Code: |
0002140 06 00 00 04 03 00 00 04 31 4a 1a ae 02 59 43 1e >........1J...YC.<
0002160 9a 75 9c 4d 69 e9 5f 50 52 61 69 64 35 20 72 6f >.u.Mi._PRaid5 ro<
0002200 6f 74 20 2c 4b 65 67 2d 00 00 00 04 fe 7d 00 00 >ot ,Keg-.....}..< |
The rest is distorted fragments of txt from what was once a filesystem:
| Code: |
11201040 00 00 00 00 00 00 00 00 2f 68 6f 6d 65 2f 72 65 >......../home/re<
11201060 64 62 65 61 72 64 6d 63 67 2f 2e 6d 6f 74 64 00 >dbeardmcg/.motd.<
...
11201240 00 00 00 00 00 00 00 00 2e 2e 2f 2e 2e 2f 73 62 >........../../sb<
11201260 69 6e 2f 66 75 6e 63 74 69 6f 6e 73 2e 73 68 00 >in/functions.sh.<
...
11204040 00 00 00 00 00 00 00 00 2f 65 74 63 2f 69 6e 69 >......../etc/ini<
11204060 74 2e 64 2f 63 6f 75 72 69 65 72 2d 69 6d 61 70 >t.d/courier-imap<
11204100 64 2d 73 73 6c 00 00 00 00 00 00 00 00 00 00 00 >d-ssl...........< |
Does this seem strange to you?
| Code: |
livecd ~ # dumpe2fs -h /dev/sda3
dumpe2fs 1.40.8 (13-Mar-2008)
dumpe2fs: Filesystem revision too high while trying to open /dev/sda3
Couldn't find valid filesystem superblock.
livecd ~ # dumpe2fs -h /dev/sdb3
dumpe2fs 1.40.8 (13-Mar-2008)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdb3
Couldn't find valid filesystem superblock.
livecd ~ # dumpe2fs -h /dev/sdc3 | head -n 10
dumpe2fs 1.40.8 (13-Mar-2008)
Filesystem volume name: Raid5 root (Keg)
Last mounted on: <not available>
Filesystem UUID: 314a1aaa-fc24-431e-9a75-9c4d69e95f50
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr filetype needs_recovery sparse_super large_file
Default mount options: (none)
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
livecd ~ # dumpe2fs -h /dev/sdd3
dumpe2fs 1.40.8 (13-Mar-2008)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdd3
Couldn't find valid filesystem superblock. |
It looks like SDC is the only disk with valid info on it, is it possible to copy the superblock on sdc3 to sd[abd]3 ?
Many thanks,
-Ryan
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23071
|
| Posted: Thu Aug 14, 2008 2:57 am Post subject: |
|
|
| redbeardmcg wrote: |
Does this seem strange to you?
| Code: |
livecd ~ # dumpe2fs -h /dev/sda3
dumpe2fs 1.40.8 (13-Mar-2008)
dumpe2fs: Filesystem revision too high while trying to open /dev/sda3
Couldn't find valid filesystem superblock.
livecd ~ # dumpe2fs -h /dev/sdb3
dumpe2fs 1.40.8 (13-Mar-2008)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdb3
Couldn't find valid filesystem superblock.
livecd ~ # dumpe2fs -h /dev/sdc3 | head -n 10
dumpe2fs 1.40.8 (13-Mar-2008)
Filesystem volume name: Raid5 root (Keg)
Last mounted on: <not available>
Filesystem UUID: 314a1aaa-fc24-431e-9a75-9c4d69e95f50
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr filetype needs_recovery sparse_super large_file
Default mount options: (none)
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
livecd ~ # dumpe2fs -h /dev/sdd3
dumpe2fs 1.40.8 (13-Mar-2008)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdd3
Couldn't find valid filesystem superblock. |
|
No, it does not seem strange. Since RAID5 is a striping design, it seems perfectly reasonable that a filesystem created on the md container would have its header appear on only one of the underlying disks. I would expect to get more useful output from running dumpe2fs on the md container device, but the output so far proves that at least some data has survived.
| redbeardmcg wrote: | It looks like SDC is the only disk with valid info on it, is it possible to copy the superblock on sdc3 to sd[abd]3 ?
|
It is possible, but my understanding of your disk usage is that such copying would be equivalent to copying the superblock into the middle of some other piece of the disk, likely damaging a file or corrupting metadata. |
|
| Back to top |
|
|
jcat
Veteran
Joined: 26 May 2006
Posts: 1337
|
| Posted: Thu Aug 14, 2008 9:11 pm Post subject: |
|
|
You should probably clone your disks if you have the room, that was you can be sure you aren't going to loose any more during recovery attempts.
Once yo have backups, why not try and mount the container as Ext4 if that's what the FS thinks it is  It's worth a try, I'm no disk or FS guru, but I'm not sure how else you might be able to recover this. Needless to say you've been _very_ unlucky.
Cheers,
jcat |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
