All my devices are gone! -- SOLVED

Moriah · Last edited by Moriah on Mon Jul 10, 2006 2:40 pm; edited 1 time in total

I recently did an update and noticed in one of the config files that tarring all the /dev files had changes from yes to no. I just had to reboot a system tonight because of a failed drive in a raid5 array, and when it rebooted, it dropped into single user mode saying it could not mount my root file system.

The system boots from a non-raid lvm2 volume via an initrd script, and has been working well for over 6 months now. If this saving all the devices in a tarball is the problem, how do I recover from it? I suspect all my systems will have similar troubles, as I did the update to all of them last week. Do I need to change the no back to a yes? If so, why was it changed in the first place?

I am really concerned about this, as I may have all my systems in a bad state if I have to reboot any of them. They are all on UPS's, but that will only hold them for about 15 minutes, and they are predicting thunderstorms this coming week.

Thanks in advance for any help!
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

Moriah · Posted: Mon Jul 03, 2006 4:22 pm Post subject:

I just rebooted a non-critical box that received the same updates, and it booted fine, so something must have gone haywire on the machine with a virtually empty /dev directory. I may just use this as the opportunity to upgrade its raid5 from 3 * 250 GB drives to 3 * 400 or 3* 500 drives instead. I have complete backups of everything.

_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

chrismortimore · Posted: Mon Jul 03, 2006 4:35 pm Post subject:

The file that holds the setting you want is /etc/conf.d/rc, and is called "RC_DEVICE_TARBALL". Setting that to yes should get you up and running again. Consider installing udev once you're back running though.
_________________
Desktop: AMD Athlon64 3800+ Venice Core, 2GB PC3200, 2x160GB 7200rpm Maxtor DiamondMax 10, 2x320GB WD 7200rpm Caviar RE, Nvidia 6600GT 256MB
Laptop: Intel Pentium M, 512MB PC2700, 60GB 5400rpm IBM TravelStar, Nvidia 5200Go 64MB

Moriah · Posted: Mon Jul 10, 2006 2:35 pm Post subject:

I am not sure what caused the original problem -- a failure of one of the drives in a raid5 -- probably a random glitch or a thunderstorm or whatever.

As a result of this I tried to fdisk the device to see if it was alive, and it was not, so I tried to shut down the machine and reboot it to see if that would make the drive accessible again. The shutdown had some problems. I do not remember the error messages at this late time. I had to punch the reset button. When I tried to reboot, /dev had very little in it; in particular, it had no /dev/hd* entries! Ouch!

After some thought, I remembered that a recent update had changed /etc/conf.d/rc so that RC_DEVICE_TARBALL="no" instead of "yes", as it formerly had contained. Thanks to chrimortimore for pointing me at the correct file for this. :-)

I think this was only a problem because the root filesystem is under LVM2/device-mapper control. When I booted the box, it fell into single user mode, and from there I could see that my root filesystem was indeed mounted, but I still had no /dev/hd* entries. Apparently the initrd file's linuxrc script had no trouble setting things up; it was just the real system that could not fsck my root filesystem.

After I remounted the root filesystem so I could write it, I changed the RC_DEVICE_TARBALL back to "yes" and created an entry for /dev/hda, /dev/hda1, and /dev/hda2. hda1 is my /boot partition, and hda2 is the physical volume that holds my lvm controlled / filesystem. I also had to change the lvm volume names in /etc/fstab from the old format of /dev/volgroup/volume to the newer /dev/mapper/volgroup-volume format.

Then I rebooted again. This time, when it came up, it was able to get to the first IDE disk drive, and udev was able to populate the rest of /dev correctly.

Possible moral of the story: Don't set RC_DEVICE_TARBALL="no" if your root filesystem in under LVM.

_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

chrismortimore · Posted: Mon Jul 10, 2006 4:47 pm Post subject:

Moriah · Posted: Mon Jul 10, 2006 5:28 pm Post subject:

Well, I was using udev, but I do have numerous additional IDE controllers to handle the raid array, and maybe that was confusing things. I don't know; that's why I said a possible moral. What do you think might have happened?
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

chrismortimore · Posted: Tue Jul 11, 2006 10:43 am Post subject:

Moriah · Posted: Tue Jul 11, 2006 12:35 pm Post subject:

The machine is my automatic backup server. I can take it offline for short periods without impact, but it really should be up every night to make its nightly backup run on the other machines on my network. I certainly do not want it offline for more than a day or so.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.