| View previous topic :: View next topic |
| Author |
Message |
yagu
n00b
Joined: 13 Oct 2003
Posts: 18
|
 Posted: Tue Jan 29, 2008 5:17 am Post subject: Catastrophic loss of all filesystems across 2 drives?! |
 |
|
Hello,
Here's the setup I was running with just recently (and for the past year or two):
- Shuttle SD11G5 (http://global.shuttle.com/product_detail.jsp?PI=81)
- Gentoo Kernel 2.6.19
- Two 500GB WD Caviar SE16 SATA Harddrives
- System spread across 5 or 6 reiserfs partitions plus one ext2 boot partition
- Media was spread with LVM2 over a couple more partitions on drive 1 plus another partition that took up the rest of drive 2
- Everything running through a Belkin UPS
I was streaming a movie over SAMBA, when the smb server crashed. I investigated the system and found that no commands were working ("cat: command not found", etc.), except for "ls" and the output didn't look too good (files that weren't symlinks were reported as broken symlinks, etc.).
I initiated an Alt-SysRq emergency restart and the system wouldn't boot.
memtest86 ran fine.
But when running the live CD and Knoppix, it looks like both harddrives are complete toast, all partitions.
For example:
| Code: | root@Knoppix:~# fsck /dev/sda1
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
fsck.ext2: Attempt to read block from filesystem resulted in short read while trying to open /dev/sda1
Could this be a zero-length partition? |
| Code: | # reiserfsck /dev/sda3
...[snip]...
Will read-only check consistency of the filesystem on /dev/sda3
Will put log info to 'stdout'
Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
The problem has occurred looks like a hardware problem. If you have
bad blocks, we advise you to get a new hard drive, because once you
get one bad block that the disk drive internals cannot hide from
your sight,the chances of getting more are generally said to become
much higher (precise statistics are unknown to us), and this disk
drive is probably not expensive enough for you to you to risk your
time and data on it. If you don't want to follow that follow that
advice then if you have just a few bad blocks, try writing to the
bad blocks and see if the drive remaps the bad blocks (that means
it takes a block it has in reserve and allocates it for use for
of that block number). If it cannot remap the block, use badblock
option (-B) with reiserfs utils to handle this block correctly.
bread: Cannot read the block (2): (Input/output error).
Aborted
|
| Code: | root@Knoppix:~# mount /mnt/sda1
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
|
| Code: | root@Knoppix:~# reiserfsck /dev/sdb1
...[snip]...
Will read-only check consistency of the filesystem on /dev/sdb1
Will put log info to 'stdout'
Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
reiserfs_open: the reiserfs superblock cannot be found on /dev/sdb1.
Failed to open the filesystem.
If the partition table has not been changed, and the partition is
valid and it really contains a reiserfs partition, then the
superblock is corrupted and you need to run this utility with
--rebuild-sb.
|
| Code: | root@Knoppix:~# dmesg | tail
Buffer I/O error on device sda4, logical block 0
Buffer I/O error on device sda4, logical block 0
EFS: cannot read volume header
ReiserFS: sda3: warning: sh-2006: read_super_block: bread failed (dev sda3, block 16, size 512)
ReiserFS: sda3: warning: sh-2006: read_super_block: bread failed (dev sda3, block 128, size 512)
ReiserFS: sda3: warning: sh-2021: reiserfs_fill_super: can not find reiserfs on sda3
ReiserFS: sda5: warning: sh-2006: read_super_block: bread failed (dev sda5, block 8, size 1024)
ReiserFS: sda5: warning: sh-2006: read_super_block: bread failed (dev sda5, block 64, size 1024)
ReiserFS: sda5: warning: sh-2021: reiserfs_fill_super: can not find reiserfs on sda5
EXT2-fs: unable to read superblock
|
Etc.
Although the live CD's are at least acknowledging the existence of the partitions:
| Code: | root@Knoppix:~# ls /dev/sd*
/dev/sda /dev/sda2 /dev/sda4 /dev/sda6 /dev/sda8 /dev/sdb
/dev/sda1 /dev/sda3 /dev/sda5 /dev/sda7 /dev/sda9 /dev/sdb1
|
What the heck happened? Did both my drives suddenly implode? It looks like it's even having problems with the DVD drive (although the disk could be scratched):
| Code: |
root@Knoppix:~# dmesg | grep hdc
hdc: media error (bad sector): status=0x51 { DriveReady SeekComplete Error }
hdc: media error (bad sector): error=0x34 { AbortedCommand LastFailedSense=0x03 }
ATAPI device hdc:
end_request: I/O error, dev hdc, sector 109148
Buffer I/O error on device hdc, logical block 27287
Buffer I/O error on device hdc, logical block 27288
Buffer I/O error on device hdc, logical block 27289
Buffer I/O error on device hdc, logical block 27290
Buffer I/O error on device hdc, logical block 27291
Buffer I/O error on device hdc, logical block 27292 |
Please help!! Do I have to RMA both my drives and start from scratch? Say it ain't so! :( |
|
| Back to top |
|
 |
Section_8
l33t
Joined: 22 May 2004
Posts: 627
|
| Posted: Tue Jan 29, 2008 7:08 am Post subject: |
|
|
| My guess would be, if you also get errors from your DVD drive reading known good disks, that it's a fried controller on the motherboard. |
|
| Back to top |
|
|
yagu
n00b
Joined: 13 Oct 2003
Posts: 18
|
| Posted: Tue Jan 29, 2008 9:06 am Post subject: |
|
|
Thanks for the reply; I tried a newly burned and verified LiveCD and there were no problems:
| Code: | livecd ~ # dmesg | grep hdc
hdc: _NEC DVD_RW ND-3550A, ATAPI CD/DVD-ROM drive
hdc: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache
|
Is there a good way to identify if the motherboard controller is at fault?
Thanks!
| Code: | livecd ~ # lspci | grep -i sata
00:1f.2 IDE interface: Intel Corporation 82801FBM (ICH6M) SATA Controller (rev 04)
|
|
|
| Back to top |
|
|
robdd
Tux's lil' helper
Joined: 02 Jan 2005
Posts: 142
Location: Sydney Australia
|
| Posted: Tue Jan 29, 2008 10:18 am Post subject: |
|
|
Hi yagu - that doesn't sound good 
Sweeping generalisation - if you haven't emerged anything recently or done anything to de-stabilise the kernel then it's unlikely that some kind of kernel crash caused the problem - more likely some faulty hardware, which then crashed/hung the kernel.
Do you have an alternate system that you can hook the SATA drives to ? If you have valuable data on the drives it would be worth buying some spare disks (dirt cheap nowadays), and use dd to create an image copy of the disks. So at least you can try to recover any data at your leisure. If on another working system you can copy the disks using dd without error then the disks are probably OK, and it may be the motherboard or disk controller - but I wouldn't experiment until I had copies of the disks. Once you've saved any (possibly) valuable data you could try using the SMART tools to check the disk drives, and I'm sure there's a Linux disk destructive read/write tester that would verify your suspect system (just can't remember what it's called for now).
Not sure how you'll go trying to recover the partitions if you've been using LVM (I stay away from LVM for precisely that reason). Maybe some LVM guru can chime in here ?
Anyway, good luck with getting your system back up and recvoering any data.
Regards - Rob
_________________
Rob Diamond
Gentoo Hack, hack, hacker
Sydney, Australia |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
