| View previous topic :: View next topic |
| Author |
Message |
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
 Posted: Sun Jan 13, 2008 10:45 pm Post subject: (SOLVED) Help! Ext3 file system crash: all data gone |
 |
|
I was installing Gentoo for my father in law, when disaster struck. Here's what I did:
1 Fired up openSUSE -his prior OS-, logged in as root and renamed his old home folder (which resides on a different partition, sda4 ) to username_backup.
2 Konqueror crashed, nog big deal I thought I checked if the home folder was properly renamed to username_backup., which was the case.
3 I rebooted and installed Gentoo from a stage 4 on sda3,at the end of the install I mounted sda4 into the home folder to check if everything was alright. And then the shit hit the fan. When trying to enter the username_backup folder I got I/O input output error. Wtf I thought and rebooted in my freshly installed Gentoo OS
4 Upon booting in Gentoo it reported that sda4 was damaged and need to be fixed. I ran ef2chk -vy /dev/sda4 and rebooted
5 Upon reboot I checked the contents of my sda4 partition which was empty  , what was also weird is that I can't create directories (both as user and as root) because i get the same I/O input output error 
Who can tell me:
-If and how I can retrieve the home directory.
Thanks in advance!
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered
Last edited by Kasumi_Ninja on Thu Mar 06, 2008 8:49 am; edited 2 times in total |
|
| Back to top |
|
 |
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Sun Jan 13, 2008 11:04 pm Post subject: |
|
|
What does SMART have to say about your hard disk. Can the device itself be completely read without errors (dd if=/dev/sda4 of=/dev/null bs=1M). If not, then your hard disk is malfunctioning or the partition table is broken (unlikely). If it can be read, make some room somewhere else and pull a copy of the partition (dd if=/dev/sda4 of=/mnt/yourplace/sda4.img bs=1M conv=noerror,sync), make a copy of the copy (cp sda4.img sda4.img.original), and then run your experiments on sda4.img and if it fails copy sda4.img.original back to sda4.img. As a quick check, you could run 'strings sda4.img' to see if there is any data of interest at all in it or not. If there isn't, you'll have to assume that you're on the wrong partition, or your partition table is messed up, and instead of just pulling sda4, you'll have to pull sda as a whole and analyze the whole thing.
Did you rename the home folder of a user, while that user was logged in somewhere, somehow? That will cause serious problems because no application expects its own home to simply vanish like that. So if konqueror was running as username, it crashed because you killed the home. By the way, renaming something to something_backup is always wrong, because you do not create backups by renaming something, but by creating an actual backup copy. If your backup is the only remaining original as seems to be the case here, the backup is obviously not a backup anymore.
That said it's still strange that a simple rename should cause errors in a file system, since no data is actually moved. No data is deleted either, so while the directory structure and everything may very well be lost, the file contents themselves should still be on there, and even if the filesystem can't be repaired anymore (e2fsck -vy and rw-mounting it several times probably repaired it to death already), you should be able to find something using forensic tools or whatever (see gentoo portage app-forensics category). |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Sun Jan 13, 2008 11:11 pm Post subject: |
|
|
Aniruddha,
First the bad news ... ef2chk -vy changed the only copy of the filesystem you had and that may have been damaged before you ran it. Rule 1. make a backup before you start. You may well need an undo function as fsck and make things worse.
Its a little late now but you know for next time.
Work on the partition in read only mode.
Look for a directory called lost&found in the root of sda4, anything that e2fsck found and didn't know what to do with should be in there named by its inode number.
is useful. You can usually identify directories from contents.
can help with files
Its a long job.
Sleuth-Kit, and hexedit can be useful too.
If you are really desperate, there is debugfs but do read its man page.
Do not attempt to recover anything in place, copy recovered information to another filesystem.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Sun Jan 13, 2008 11:15 pm Post subject: |
|
|
Thanks for the quick replies! I am going to catch some sleep and try again tomorrow.
P.S.
Neddy, the profound wisdom of your signature has become apparent to me 
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
Simba7
l33t
Joined: 22 Jan 2007
Posts: 706
Location: Billings, MT, USA
|
| Posted: Sun Jan 13, 2008 11:18 pm Post subject: |
|
|
| Aniruddha wrote: | | Neddy, the profound wisdom of your signature has become apparent to me |
Ya, I seen that. I lost 2GB of WestPac '01 pictures due to my HDD crashing awhile ago. It then happened AGAIN on my parents trip to Minneapolis in '04 when my laptop HDD seized and I lost 2-3GB of pictures. I was PISSED!!
After that, I back EVERYTHING up. Sometimes in multiple places. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Sun Jan 13, 2008 11:50 pm Post subject: |
|
|
Aniruddha,
I've just had a thought ...
| Code: | | ef2chk -vy /dev/sda4 |
on an ext3 filesystem only replays the journal, if needed.
Its not supposed to do the whole 9 yards of reading the entire filesystem, like it has to with no journal.
Make an image of the filesystem, so you can get back to where you are now then try | Code: | | ef2chk -nf /dev/sda4 |
-n don't fix it, just tell me
-f is force ... needed to check an apparently clean fs.
You may want to try with some alternate super block numbers too.
You could also try | Code: | | mount -o ro -b <some-superblock> /dev/sda4 /mout/point |
I'll need to refer you to the man pages to discover how to work out where the superblocks are.
Note also that mount and ef2chk have different ways of expressing this.
Good luck
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
andreas_st
Tux's lil' helper
Joined: 26 Dec 2007
Posts: 112
Location: 48N 16E
|
| Posted: Mon Jan 14, 2008 3:28 am Post subject: |
|
|
| NeddySeagoon wrote: | | I'll need to refer you to the man pages to discover how to work out where the superblocks are. |
Just as a hint for the future and for others: I usually keep the output of mke2fs and store it with the backups to have the block numbers of the copies of the superblocks at hand when needed. It would also be a good idea to save the output of fdisk -ls. Just for the case that the partition table gets corrupted. | Code: | mke2fs -j -L $LABEL /dev/$DEVICE | tee /root/mkfs/mke2fs.$LABEL.$DEVICE
fdisk -ls > /root/mkfs/fdisk-ls |
|
|
| Back to top |
|
|
schachti
Advocate
Joined: 28 Jul 2003
Posts: 3765
Location: Gifhorn, Germany
|
| Posted: Mon Jan 14, 2008 6:28 am Post subject: |
|
|
| andreas_st wrote: | | Just as a hint for the future and for others: I usually keep the output of mke2fs and store it with the backups to have the block numbers of the copies of the superblocks at hand when needed. |
You always can generate this output by using mkfs.ext2 -n (of course you have to use the same options as you did when creating the file system, but if you used the standard options...).
_________________
Never argue with an idiot. He brings you down to his level, then beats you with experience.
How-To: Daten verschlüsselt auf DVD speichern. |
|
| Back to top |
|
|
andreas_st
Tux's lil' helper
Joined: 26 Dec 2007
Posts: 112
Location: 48N 16E
|
| Posted: Mon Jan 14, 2008 7:33 am Post subject: |
|
|
| schachti wrote: | | You always can generate this output by using mkfs.ext2 -n |
Thanks, I didn't remember that but I don't want to rely on it because between the creation of the file system and the time when it gets corrupted may be a long time and I'm almost sure that there will be several new versions of e2fsprogs in the meantime. Probably nobody will guarantee that a newer version will use the same number of superblock copies. |
|
| Back to top |
|
|
Roman_Gruber
Advocate
Joined: 03 Oct 2006
Posts: 3846
Location: Austro Bavaria
|
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Mon Jan 14, 2008 5:54 pm Post subject: |
|
|
I did also run PhotoRec but it didn't find anything
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Mon Jan 14, 2008 8:51 pm Post subject: |
|
|
andreas_st,
The problem is that mount and fsck use different units to find the alternate superbolocks.
One uses filesystem allocation blocks, the other uses disk sectors.
If you get it wrong, the results are just disappointing, theres no damage.
Its debugfs and hexedit if nothing else works but that won't be easy.
What size was the filesystem ?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Fri Jan 18, 2008 10:35 am Post subject: |
|
|
I'll try to fix the damage tomorrow. I'll make a backup of the partition with this command (thanks frostschutz!)
| Code: | | dd if=/dev/sda4 of=/mnt/yourplace/sda4.img bs=1M conv=noerror,sync |
But I guess I the space needed for the image is equivalent the the partition size right? How long will it take to create a 250 GB image (mostly empty)?
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
Habbit
Apprentice
Joined: 01 Sep 2007
Posts: 237
Location: 3.7137 W, 40.3873 N
|
| Posted: Fri Jan 18, 2008 12:49 pm Post subject: |
|
|
| Aniruddha wrote: | | But I guess I the space needed for the image is equivalent the the partition size right? How long will it take to create a 250 GB image (mostly empty)? |
If your partition is really mostly empty and you're short on space, you can save a lot by piping the dd outfile through gzip like this:
| Code: | | # dd if=/dev/sda4 bs=1M conv=noerror,sync | gzip > /mnt/yourplace/sda4.img |
The default compression settings won't spend a lot of time trying to compress normal files, while large runs of zeros will be packed. This will only work, of course, if the free space in your partition actually contains just zeros, which is not usually the case unless you _never_ filled the partition too much or you zeroized it prior to mkfs.
EDIT: about your second question: depends on your HD. You can, however, query dd for progress from another virtual terminal. If no other dd is running on the system at the time, do "killall -USR1 dd" and dd will print stats to stderr (so it will work even in the gzip-redirected mode).
_________________
| Code: | ~ $ objdump -d ./habbit_mind
90 xchg %rax, %rax
EB FD jmp $-3 |
|
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Fri Jan 18, 2008 2:16 pm Post subject: |
|
|
Habbit,
If you need conv=noerror, then the image isn't likely to be very useful as dd will not do retries.
I agree you can't do much else if you don't have space.
The 1M block size is also a bad idea on a damaged filesystem. dd will skip 1M in the event of a single sector error,
Its worth using 4k or 512 as a block size to recover everything dd possibly can.
4k being the fs blocking size and 512B being the disk sector size.
Smaller block sizes take longer but you get more
dd_rhelp or ddrescure (no hyphen) will recover more data than dd from a damaged fs but they both need space for an uncompressed image and a log file
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
frostschutz
Advocate
Joined: 22 Feb 2005
Posts: 2977
Location: Germany
|
| Posted: Fri Jan 18, 2008 4:57 pm Post subject: |
|
|
| NeddySeagoon wrote: | | If you need conv=noerror, then the image isn't likely to be very useful as dd will not do retries. |
I just add conv=noerror,sync for safety. Because if there is an error and it breaks, or if there is an error and it doesn't sync, the result will really be utterly useless and broken in a way that can't be fixed. If the disk is actually damaged of course there are more aggressive solutions like ddrescue that try to scratch as much data as possible off the disk. The bs= don't matter as long as you make it big enough to be performant, as the default block size of dd slows down the copy process considerably. Dunno what'd be the smallest block size that does not hurt performance, I usually use 1M for simplicity's sake.
| Quote: | | ddrescue will recover more data than dd from a damaged fs |
minor nitpick: from a damaged hard disk. dd doesn't care about fs and wether it's damaged or not at all |
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Sun Jan 20, 2008 12:15 am Post subject: |
|
|
Well I tried to copy the partition today but failed. For some reason the bios only recognized 130 GB of the 320GB harddisk. Maybe this has caused the problems in the first place?! Oh well, I'll try a bios reset soon.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Sun Jan 20, 2008 12:45 am Post subject: |
|
|
Aniruddha,
The BIOS may well show about 130G of the drive but Linux will see it all.
Thats what happens when the BIOS does not support 48 bit logical block addressing.
To boot with such a BIOS /boot must be in the part of the drive the BIOS can see.
If you are not booting it, its not important.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Sun Jan 20, 2008 10:16 am Post subject: |
|
|
| NeddySeagoon wrote: | Aniruddha,
The BIOS may well show about 130G of the drive but Linux will see it all.
Thats what happens when the BIOS does not support 48 bit logical block addressing.
To boot with such a BIOS /boot must be in the part of the drive the BIOS can see.
If you are not booting it, its not important. |
Strangely enough Linux did see the same as the bios fdisk -l shows two hard disks of 130 GB.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Sun Jan 20, 2008 11:03 am Post subject: |
|
|
Aniruddha,
What doesshow?
You can make it show partition sizes in disk sectors (512B units) or in cylinders, which have a variable size as shown in the header
| Code: | | Units = cylinders of 16065 * 512 = 8225280 bytes |
Many years ago there was a kernel option to switch to 48 bit LBA after boot but grepping the .config shows its not there any more.
How does a liveCD show the drive size ?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Sun Jan 20, 2008 6:13 pm Post subject: |
|
|
| NeddySeagoon wrote: | Aniruddha,
What doesshow?
You can make it show partition sizes in disk sectors (512B units) or in cylinders, which have a variable size as shown in the header
| Code: | | Units = cylinders of 16065 * 512 = 8225280 bytes |
Many years ago there was a kernel option to switch to 48 bit LBA after boot but grepping the .config shows its not there any more.
How does a liveCD show the drive size ? |
I just fired up the PC after resetting the bios and now it doens't see the sata drives anymore Afaik sata support is enabled in the bios.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Sun Jan 20, 2008 6:27 pm Post subject: |
|
|
Aniruddha,
Does your BIOS have SATA options, like IDE Emulation or ACHI ?
IDE Emulation is a very bad thing, as you will not get DMA if you use that and your SATA drives will appear as /dev/hd...
You will also need to choose LBA mode for the drives, if there is a choice.
What does lspci say about your hardware?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Sat Feb 09, 2008 5:10 pm Post subject: |
|
|
| NeddySeagoon wrote: | Aniruddha,
Does your BIOS have SATA options, like IDE Emulation or ACHI ?
IDE Emulation is a very bad thing, as you will not get DMA if you use that and your SATA drives will appear as /dev/hd...
You will also need to choose LBA mode for the drives, if there is a choice.
What does lspci say about your hardware? |
I finally managed to get the drives working again. Still the size ain't right
| Code: | # fdisk -l
Disk /dev/sda: 137.4 GB, 137438952960 bytes
255 heads, 63 sectors/track, 16709 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 6375 51207156 7 HPFS/NTFS
/dev/sda2 6376 6506 1052257+ 82 Linux swap / Solaris
/dev/sda3 * 6507 13034 52436160 83 Linux
/dev/sda4 13035 38913 207873067+ 83 Linux
Disk /dev/sdb: 137.4 GB, 137437871104 bytes
255 heads, 63 sectors/track, 16709 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table |
| Code: | # lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev a2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev a3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev a3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:0a.0 FireWire (IEEE 1394): Texas Instruments TSB82AA2 IEEE-1394b Link Layer Controller (rev 01)
05:00.0 VGA compatible controller: nVidia Corporation NV43 [GeForce 6600 GT] (rev a2) |
| Code: | lshw
description: Motherboard
product: GA-K8NF-9
vendor: Gigabyte Technology Co., Ltd.
physical id: 0
version: x.x |
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54328
Location: 56N 3W
|
| Posted: Sat Feb 09, 2008 5:57 pm Post subject: |
|
|
Aniruddha,
The partition table says you have 38913 cylinders of 8225280, which is 320,070,320,640B or 320Gb.
The 137Gb reported by fdisk is the old LBA limit.
Linux should not do that (it doesn't for me) even though my BIOS has the 137G limit.
Did you use fdisk running on your own kernel or did you use a liveCD to run fdisk.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Kasumi_Ninja
Veteran
Joined: 18 Feb 2006
Posts: 1825
Location: The Netherlands
|
| Posted: Sat Feb 09, 2008 8:30 pm Post subject: |
|
|
| NeddySeagoon wrote: | Aniruddha,
The partition table says you have 38913 cylinders of 8225280, which is 320,070,320,640B or 320Gb.
The 137Gb reported by fdisk is the old LBA limit.
Linux should not do that (it doesn't for me) even though my BIOS has the 137G limit.
Did you use fdisk running on your own kernel or did you use a liveCD to run fdisk. |
This fdisk is from my own kernel.
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved. Help answer the unanswered |
|
| Back to top |
|
|
|
Other Things Gentoo
