[SOLVED] Booting in single-user mode?

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

Hi,

I am still fighting against my corrupted file system and wondering how I could do a xfs_check / xfs_repair now.
To summarize:

The problems occurr on the root partition so I cannot just unmount the part. and xfs_check it, nor can I just mount it readonly, just does not work.
Since this is a machine I have no physical access to (ssh only) I have to be careful not to boot it in a way preventing me from accessing it any more.

So, as there is no remote console available, I was thinking of booting in single user mode by editing grub...
Could that work?
Could i make ssh available in single user mode?

Any ideas?

platojones · Posted: Sun Apr 24, 2005 10:13 pm Post subject:

Well, you could do a 'touch /forcefsck' and reboot. When the init scripts see that file on bootup, it will force an fsck of all of your partitions. You can also mount a livecd and mount your root partition there, then fsck it. There are a lot of possibilities.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

Will this touch also work for XFS and force a xfs_check since I am using XFS as file system?
The output can then be seen in dmesg?

The option with the CD is not possible as I have no phyiscal access to the machine.

platojones · Posted: Sun Apr 24, 2005 10:21 pm Post subject:

It will work with any filesystem recognized by the system. fsck, when invoked, will use the appropriate version of fsck for your filesystem type. You can verify by looking in /sbin and doing an 'ls -l fsck.*'. If there is a symlink for fsck.xfs (or whatever it is called), then it will work.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

And the results can be found in dmesg then?
If any errors appear, can I also do a xfs_repair in a similiar way?

platojones · Posted: Sun Apr 24, 2005 10:27 pm Post subject:

Wait, I stand corrected. Root is already mounted at this point and it doesn't check that. Single user mode is your best bet.

platojones · Posted: Sun Apr 24, 2005 10:29 pm Post subject:

Nope, wrong there too! Single user will mount root as well. Without physical access this is a tough one! Sorry to have led you down the wrong path here.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

Exactly... the touch has not worked...
Ok, and single-user mode is is no option either?

Any more ideas?

platojones · Posted: Sun Apr 24, 2005 10:36 pm Post subject:

Well, the problem is that to go into single user mode, root has to be mounted. In fact, to do anything root has to be mounted (read only at least). If you want to go into single user mode, you can edit the kernel line in your grub config a boot time (by typing e) and put the word 'single' at the end of the line. Then type 'b' to boot.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

That is what I still see in dmesg:

kjournald starting. Commit interval 5 seconds
EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
EXT3 FS on hda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 506036k swap on /dev/hda2. Priority:-1 extents:1
EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
EXT3 FS on hda3, internal journal

Moreover, when I switch to certain directories I observe weird things.
E.g. if I issue a simple ls in some dir I get:

app-dicts # ls -l
ls: freedict-eng-ita: Input/output error
ls: freedict-ita-eng: Input/output error
total 0

If I leave the system as is it does not take long and the "kernel" decides to mount this drive read-only.
I think thís is done automatically to prevent further errors.

I did some smartctl checks with the smartmontools which told me that at least the drive itself seems to be fine.

What can I do?

platojones · Posted: Sun Apr 24, 2005 10:51 pm Post subject:

Ok, I think you can fsck the root directory if it is in read-only mode. It's only in read-write mode that it can't be fsck'ed.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

boot # mount -o remount,ro /
mount: / is busy

Am I too stupid to do that? :-(

platojones · Posted: Sun Apr 24, 2005 10:59 pm Post subject:

Wait, i just noticed this, but the mount errors you are getting are referring to EXT3, not XFS. Is your root partition actually an XFS partition or an EXT3 partition?

platojones · Posted: Sun Apr 24, 2005 11:03 pm Post subject:

Sorry, you and I just posted at the same time and I didn't see your last question. Well, as for the remount, I was referring to your previous statement that the kenel automatically puts it into read-only mode when after a short period of time. When the kernel does this you should be able to fsck it. But I think we need to figure out what kind of file system this is before we do that.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

I was told it is XFS (I set up everything myself except formating the partitions), but if I look into fstab it seems to be xfs:

/dev/hda3 / XFS noatime 0 2

Is there a common way to verify this?
I was wondering about these EXT3 messages, too...

platojones · Posted: Sun Apr 24, 2005 11:10 pm Post subject:

That may explain how the filesystem got corrupted in the first place. Paste the output from 'df -aT'. It shows the type the system finally mounted it as.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

Argh... now I did a mistake, I thought I issue an

init 1

command and try to mount readonly then.
But just after having issued the init 1 I lost my ssh console.

I think I have to wait for the staff coming back to reboot my machine :-(

I will get back to here tomorrow :-(

platojones · Posted: Sun Apr 24, 2005 11:18 pm Post subject:

Ok, sorry about that.

platojones · Posted: Sun Apr 24, 2005 11:34 pm Post subject:

BTW, since the staff is going to reboot the machine anyway at this point, perhaps you can ask them to verify the parition type of /dev/hda3. The kernel thinks this is an ext3 partition. If you are trying to mount it as an xfs partition, things are going to go wrong. In fact, I'm shocked that it's possible for mount to do that. It is possible with ext2/ext3 since they are so similar, but I would think that the xfs_mount command would not allow it.

platojones · Posted: Mon Apr 25, 2005 1:24 am Post subject:

One final post and I'll leave this alone for tonight. I've given this a lot of thought, and there is probably no way you are going to be able to fsck '/' remotely over a network with ssh. I would advise you talk the staff into doing it for you tomorrow. Please have them also look at your fstab to make sure it matches the actual partition types on the server. Something is wrong there, and very likely the source of your corruption problems. Best of luck and please post here with the results.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

Hi,

df -aT returns the following:

df -aT
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/hda3 XFS 38914860 2763780 34174284 8% /
devfs devfs 0 0 0 - /dev
none proc 0 0 0 - /proc
none sysfs 0 0 0 - /sys
none devpts 0 0 0 - /dev/pts
/dev/hda1 ext2 101105 7979 87905 9% /boot
none tmpfs 119568 0 119568 0% /dev/shm
none usbfs 0 0 0 - /proc/bus/usb

Actually that looks fine and matched my fstab, so I do not know why I see these error messages.

platojones · Posted: Mon Apr 25, 2005 10:41 pm Post subject:

I'm completely mystified by those results. If there are no other entries for '/' in your fstab, I cannot fathom why init would try to mount /dev/hda3 as an ext3 filesystem. Anyway, was the staff able to help with the fsck after they booted the box?

platojones · Posted: Mon Apr 25, 2005 10:54 pm Post subject:

Oh, BTW, I should have thought of this earlier. Is XFS compiled into your kernel or is it a kernel module? Post the results of lsmod please.

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

Hey,

sorry for my late response.
We have not yet rebooted with the LiveCD (staff was too busy, which is annoying in my opionion).
So I became a bit too brave and thought "Ok, if it tells this is a EXT3 fs, let's change the fstab entry from XFS to ext3".
I rebooted some mins ago and now my system is gone.

I have to reboot with a LiveCD tomorrow now... then undo this change and will then issue the lsmod etc. stuff as suggested by you.

I will keep you informed.

Thanks for all your help again!

NautilusIII · Apprentice Joined: 23 Nov 2003 Posts: 162

Question:
Assuming the fs really was ext3 and not XFS (for whatever reason the system could boot with XFS in the fstab), would I have had to do more than just changing the entry from XFS to ext3 in my fstab for /dev/hda3?

Any modules to be emerged and added to "somewhere"?