BAD HDD!! Please help [solved]

redbeardmcg · Posted: Wed Aug 02, 2006 3:56 pm Post subject: BAD HDD!! Please help [solved]

So the last few days, I kept forgetting to backup my web server / imap server / portage cache / sql server / everything that i dont want to lose server, and while I was working on it yesterday, I started getting the "read only filesystem" messages. I re-booted and tried to fsck the disk, and when it tried to start back up it kernel panicked :evil:

.

NOT GOOD, I've spent the last month working on a web site for my girlfeind who is going in for back surgery and will be out for 2 months and wants something to do (its an online store), so everything else aside (all of my email, my personal blog / web sites, etc...) I NEED to recover the data on this disk.

Is there any way I can access the information and ignore the filesystem errors? I ran fsck -cp and after 2 hours our power went out, so I don't know if it finished. I just need access to this data.

Thanks for any advice on how to recover this information

Ryan
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you.

NeddySeagoon · Posted: Wed Aug 02, 2006 4:31 pm Post subject:

redbeardmcg,

Get dd_rhelp and make a image of the drive in a file.
dd_rhelp was not in portage, last time I looked.
You should not need to babysit it unless the drive goes offline as a result of the error.

Avoid operating the failing drive at all, its only going to get worse.

edit ... its is now emerge sys-fs/dd-rhelp
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

forgotten1 · Posted: Wed Aug 02, 2006 4:51 pm Post subject:

First rule of thumb - Don't do anything else until you have a plan, you don't want to cause any more changes to disk than necessary.

Initial questions:

When exactly did the kernel fail, at boot, or when you attempted to run fsck?
What type of partitions were in use, primary or extended, was logical volume management in use?
What filesystem types were in use on the disk?
Do you know where the data is you are looking for?

Without knowing any of the above, I'd attempt to use a LiveCD (Gentoo, Knoppix, SystemrescueCD, etc.) to [/list]recover the data along with a USB harddrive to copy files over. The process would go something like this:

Insert LiveCD
Plug in USB harddrive
Boot box from LiveCD
Use dd to copy entire disk to USB Harddrive. You will of course need a USB harddrive bigger than that you are trying to clone.
Code:

dd if=/dev/<bad drive device> of=/dev/<USB drive device>
Then attempt to mount the 'faulty' harddrive.
Mount the USB harddrive.
Create a special directory on the USB harddrive for the valuable files.
Locate and copy all the valuable files over the the freshly created directory.

If the harddrive is truly broken, you may not be able to do the above, in which case your only alternative is to send it out to a special data recovery lab. That will be as expensive as it sounds.

If it's just corrupt, then the above will at least allow you to make a clone, and recover your working files.

Good luck.
_________________
Return the Favor | Problem fixed? Add [Solved] to your title | Google the Forums

redbeardmcg · Posted: Wed Aug 02, 2006 5:12 pm Post subject:

Ok, I will pop the drive into a host machine and run dd_rhelp on my lunch break.

This will be the last maxtor drive I run, and the LAST time I have ANYTHING important on a single disk machine.

Thanks for the fast reply!

Ryan
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you.

redbeardmcg · Posted: Wed Aug 02, 2006 5:18 pm Post subject:

NeddySeagoon · Posted: Wed Aug 02, 2006 5:36 pm Post subject:

redbeardmcg,

If your disk is actually damaged, real bad sectors, dd_rhelp cannot read it all but will get bcalk what it can.
Read the link to see how it works.

You can then do data recovery on the image file, which you may want a backup of, so you have an undo function.
If the drive is damaged, you may not be able to read it again.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

redbeardmcg · Posted: Wed Aug 02, 2006 7:42 pm Post subject:

All seemed to be going well...

It found some bad sectors (nothing big, jus a few k here and there) and then this:

NeddySeagoon · Posted: Wed Aug 02, 2006 8:10 pm Post subject:

redbeardmcg,

You can run dd_rhelp again and it will build on what its already done.
Some sanity checking is in order before you try.

How big is the drive ?
The output snippit suggests 41048004.0k or about 40Gb of which its got 27262986.0k or 27Gb
So ... is the drive really 40Gb or is that wrong?
If it were a 30Gb drive, you are not going to read 40Gb from it. I know it sounds silly but the drive size could have been incorrectly established at the start.

Provided its a 40Gb drive try again.
Its also worth allowing the drive to cool down and running it at odd angles - on either edge or even upside down.
The idea being to get gravity to align worn spindle bearings, which is the normal cause of sudden faileres.

If you are really lucky, the data you want may have been read already. To check that out you need to get into the intracases of the mount command with -o loop,offset= options, where offset is in bytes to the start of the partition you wish to mount.

redbeardmcg · Posted: Wed Aug 02, 2006 8:27 pm Post subject:

Yes, the drive is a 40 gig drive. I ran dd_rhelp on the third partition (hdd3). Did this really start it there, or is it dumping the whole drive? How do I determine the start of the 3rd partition?

I am no filesystem expert, but the most important information is the information that I recently wrote to the drive, so I am assuming I am more concerned with what is at the end of the drive? I am running it again, and it got 5 errors right around 27262986. Does this mean that the bad sectors are around the 27 gig point, or do I lack a basic understanding of how information is stored to disk in an ext3 fs?

Thanks again for your help,

I will try putting the disk upside down when I get out of work

Ryan
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you.

NeddySeagoon · Posted: Wed Aug 02, 2006 8:40 pm Post subject:

redbeardmcg,

I misread the dd_rhelp output. You will only have an image of the partition. No need to mess about with -o offset=.

dd_rhelp is not recovering the filesystem for you. Its making an image of the partition in a file. It cares nothing for the filesystem that may be there. It reads raw disk blocks. When it finds an error,, it begins reading again in the middle of the largest unread area and does a binary search to recover all the 'good' sectors. Eventually, only bad sectors are left. Now it tries harder by stepping the head in from above and below, doing retries and so on.

When you were writing data, ext3 find s some space and saves the file, I'm not sure how it determines where on the partition to save a file but it may not be close to the end. Sight of the output bar chart would be useful
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

redbeardmcg · Posted: Wed Aug 02, 2006 9:23 pm Post subject:

Thanks for the clarification. I have used dd in the past (but not dd_rhelp), and I know how it works, I just didnt know if ext3 stores files in sequential order, so if the most recent modifications would be towards the end of the image.

Here is a screen
http://mcguireusa.no-ip.org/dd_rhelp.jpg

It's still running, and my fingers are crossed

Thanks again!
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you.

NeddySeagoon · Posted: Wed Aug 02, 2006 9:32 pm Post subject:

redbeardmcg,

The important line is the

Ast0r · Posted: Thu Aug 03, 2006 4:34 am Post subject:

redbeardmcg · Posted: Thu Aug 03, 2006 5:11 am Post subject:

NeddySeagoon · Posted: Thu Aug 03, 2006 10:51 am Post subject:

redbeardmcg,

You can treat the file image as if it were a real partition, so anything you do to a real partition you can do to the image.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

redbeardmcg · Posted: Thu Aug 03, 2006 1:28 pm Post subject:

I do know that this file can be treated as a filesystem, I was more asking if you knew where mysql stored its database files so that I could copy them over to the server once the new hdd is in it.

Let it run overnight, its got 35/40 gigs (or atleast thats what du says about the file, dd_rhelp still says it only has 20 gigs).

Thanks again for the help, you guys are lifesavers!

Now... linux + raid

_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you.

forgotten1 · Posted: Thu Aug 03, 2006 1:38 pm Post subject:

redbeardmcg · Posted: Thu Aug 03, 2006 2:38 pm Post subject:

Got em! looks like all of my databases are intact! I guess I lucked out, /dev, /etc, /usr seem to be all bad, but everything I needed was good

Thanks again,

Ryan
_________________
Sacrifice:::Your role may be thankless, but if you're willing to give it your all, you just might bring success to those who outlast you.