Disk cannot be booted/chrooted - mounts ok [Thread Terminat]

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

Non-stop thrills and chills to do an update in gentoo... I think this might be a udev thing this time but I cannot be certain. I did am emerge update a few days back and pulled in a few system packages including a new udev. When the update finished I reinitialized udev based on the post-install message. I did not reboot however, and continued with the old boot (did do env-update and source /etc/profile). A few days later suddenly started getting messages that /bin/COMMAND was not present even though if I ls the /bin directory I see it there.

I was certain this was trouble; but I knew I had no choice so I rebooted. The reboot lead to a kernel panic.

I loaded a livecd and planned to do a chroot into to try and at least triage if not update my way out of it.... but chroot tells me /bin/bash no such file or directory. So now I cant even get into the environment. Is there anyway to salvage this one? I do not understand what has happened or how I can fix this. Any help is appreciated.
_________________
...and it should be known by now

Logicien · Posted: Tue Dec 22, 2015 7:33 am Post subject:

If you try to chroot this mean I hope that the disk partitions table is there, that the partitions of the disk are there and the filesystems are there too. If I understand, you can mount the Gentoo root filesystem. What does fdisk (or gdisk), blkid, df and du say about those?

What is the state of the filesystem? If the data are there have you try to verify the filesystem with fsck?
_________________
Paul

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

Logicein,
Thanks for the rapid response! Ok, fdisk and df seem to report accurate and OK info for the most part. I did do an fsck on the ext3 partitions and it did fix a number of sectors on the /boot, but didnt appear to fix anything. No other partition reported an error.

Acutally, before the fix of /boot it was giving a kernel panic saying: Not syncing: No working init found try passing init= option to kernel.

Now it is failing at runlevel 2 with a lot of nonesense that i cannot replicate. Still not able to get into the environment either.

My suspicions are leaning towards a bum drive actually. I did have an issue with the drive getting borked about a month ago using a file manager that might have exceeded partition permissions. But I would love a solid confirmation of this somehow.
_________________
...and it should be known by now

Logicien · Posted: Tue Dec 22, 2015 9:01 am Post subject:

Be sure that the bootloader and the Linux kernel are searching everything, kernel, initramfs, root and boot partitions in the right partition paths. If you use Systemd, you may need to pass

krinn · Watchman Joined: 02 May 2003 Posts: 7470

as stupid as it is, did you ever check /bin/bash exists?
you won't be the first one that remove its /bin content after abusing drink or abuse its root power on his fs.

NeddySeagoon · Posted: Tue Dec 22, 2015 11:27 am Post subject:

phobos13013,

Has some other random drive failed, so that your root partition is no longer pointed to by your root= statement in your boot loader setup?
This could change your drive order and some random filesystem is now being mounted as root.

Do you have a removable drive attached during the boot now?
That can cause the same effect.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

Well... the winner is..... krinn!!! I am very very stupid.... I blame the drink and majorly distracted and there are no other excuses but I will make a few up. My only question now is... how do I repopulate my ENTIRE lib directory :/

To the other questions, do not use systemd. And the drive did have an error on boot but that was fixed, I think it was a relic to a problem I had before where superblocks got confused due to a program I was using, but that is fixed now.

No, my main problem now is trying to figure out how to get a new lib directory rebuilt. I would even jus rebuild world if I had to but literally the entire directory is gone so there is nothing I can do. I am doing a major self-facepalm right now.
_________________
...and it should be known by now

NeddySeagoon · Posted: Tue Dec 22, 2015 4:38 pm Post subject:

phobos13013,

You need some binary tarballs of key packages that match your hardware.
Once you have these, you untar them to your root as if each one was a stage3 tarball.

There is another way, which I will outline here and one of us should detail on the wiki :)

Mount your broken install at /mnt/gentoo
Make yourself a directory where you have 20G or so of space. You will get it back later.
Say /mnt/gentoo/home/phobos/rescue
Untar a stage3 to /mnt/gentoo/home/phobos/rescue as if you were doing an install.
Do not get a portage snapshot.
Do all the mounts and other chroot steps but do not chroot yet
Copy over your /mnt/gentoo/etc/portage to /mnt/gentoo/home/phobos/rescue/etc/portage
Bind mount your /mnt/gentoo/usr/portage to /mnt/gentoo/home/phobos/rescue/usr/portage

Now you have an embryonic Gentoo install in /mnt/gentoo/home/phobos/rescue with all your settings in it, which you can chroot into.
It shares /usr/portage with your real install.

Do the chroot to /mnt/gentoo/home/phobos/rescue but don't build anything yet.
Edit make.conf in the chroot to add buildpkg to your FEATURES.
Every package you build will be saved to /usr/portage/packages, which is the same place inside and outside the chroot.

Now build the packages you need.
A good start will be to run /usr/portage/scripts/bootstrap.sh as that will build you a tool chain.
Its actually the stage1 of a stage1 install.
When that finishes, you will have some packages in /mnt/gentoo/usr/portage/packages (outside the chroot)

emerge can use these, with the -K option but you can't use emerge until you can chroot into your real install, so you need to use tar.
From outside the chroot, your packages are in /mnt/gentoo/usr/portage/packages

For glibc you would do

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

Neddy,
Thanks again as always! This rescue process sounds interesting... it seems like there are plenty of opportunities for me to break it along the way since I am sort of like a blind person where I just feel my way through by random chance. I have been very lucky over the years having not fallen into a ravine or being eaten (completely) by tigers. Nonetheless, this sounds like more time and effort than I have at the moment :/. I need this up and running by tomorrow night and luckily this sounds doable. So I will go about the individual package route; is there a list of minimum packages needed to run? Portage (and then run the bootstrap script)? GCC? Note, when i first got this box up and running about a year and a half ago, it had been sitting unused for a few years and I had to rebuild the entire system using this method since it was so far out of date the world had left it in the dust. As a result, I am pretty sure all the pkgs are built with the buildpkg option. Does this mean I can run the emerge -K on packages once portage is working?

One other thing, I was trying some things waiting for reply and one of them was to dl a stage3 tarball. I did extract it into the mounted root folder (not chrooted of course); would this have done any further damage? Oddly enough the system froze right after this finished but I am not certain if this is related.
_________________
...and it should be known by now

NeddySeagoon · Posted: Tue Dec 22, 2015 5:11 pm Post subject:

phobos13013,

If you already have packages, there is no need to rebuild them. They will be in your /usr/portage/packages already
Also, buildpkgs will be in your FEATURES ... Is it?
The hard work is done if that's the case. You can get straight to using tar.

Take great care to get the -C right. tar is going to spray files all over the target.
Oh ... I missed the -p option.
You need tar -pxf as you want to preserve permissions on the untarred files.

My lib contains

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

Thanks Neddy... I indeed wiped out the entire lib directory, its empty; I had a lib32 dir but I am no longer on multi-lib so think it is a relic but lib64, lib and lib.bak are empty. So I do have FEATURES=buildpkg in make.conf so I have the pkgs on root but I have no way to build them in that environment with no chroot... I am at a loss for how I can build them in there! in the past I at least had basic functionality of the root system so was able to do basic things like emerge -K random pkg. But not being able to chroot, I just do not understand where I even begin!

Also want to reposit the question of the repercussions if any of having downloaded a stage3 tarball and extracted that into root (dont need to be in the environment to do that one). It has been ages since I have had to do a fresh install and so cant recall the full effect. My hope was that base libs would be installed but didnt seem to have the intended effect.
_________________
...and it should be known by now

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

OK disregard the previous two posts. I have made sense of it all after actually reading all of Neddy's post about using the tar command with option -C! So I have reinserted a lot of the base components but I am still not fully functioning. I am basically going through and trying to install anything that I screams important to me... and I have a few guides on the interwebs about this kind of thing happening before (and my own experience with it). But if someone can provide a list of the absolute necessaries to start building pkgs, much appreciated!
_________________
...and it should be known by now

NeddySeagoon · Posted: Wed Dec 23, 2015 9:54 am Post subject:

phobos13013,

From the Olde Fashioned Gentoo install guide. Sorry its from the original Guide-XML source.
You will need to decode the xml and fix the source paths.

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

Neddy,
I was so close! I was proceeding with the unpack; I had gotten a chrooted environment. I had untared the list you had provided and many more. But I could not get a working emerge; it would either complain of a bad c compiler, give error with some src_configure error, or multilib error. Also, it was reporting my profile as all the options; actually there were numerous runtime errors I was encountering with python, c, and other core errors.

Then I basically did what I had done to get me into this mess and had to replace the /lib folders by hand. Now some things will not untar there. After going back through the list with re-scrubed lib dirs, i get a complaint that libnurses.so.5 is not there. No chroot allowed.

So, that pretty much killed it right? Is there a way to get the pkgs to unpack to lib again? Does the system have to create the folders?

I guess I am going to try a fresh install unless I get some good news in response. I havent done a fresh install since the commandline days I am pretty certain, is it still this way? If I cannot chroot into the environment, I am not sure how to do the install... Do I just use a dvd now and it will get a base environment to get into? I am going to migrate all the config/files from my existing install to a temp folder and then try to bring them back in, but my system is going to take forever to figure out. Can I just copy the kernel files and then reinstall them once I get the new environment working? Will the dvd method erase the partition or will it only copy in the directory structure and files? Definitely abandoning this hope of having it done tonite; anyone with thoughts or ideas to salvage this current install its appreciated. Otherwise, I will solve the entry when I start the install... feel free to comment until then.
_________________
...and it should be known by now

NeddySeagoon · Posted: Thu Dec 24, 2015 9:36 am Post subject:

phobos13013,

You can do the untar as many times as you like. Its just spraying files over your fileststem.

The -C /mnt/gentoo is for untarring to your root from outside the chroot.
Once inside the chroot it becomes -C /

It sounds like you need to use eselect to reset your profile and lots of other things, e.g python.
You may need that list of symlinks too.
Don't forget gcc-config.

It sounds like you were close.

Don't even think about a binary install from the DVD. Its 18 months out of date and will teach you lessons about updating Gentoo that will take you longer to learn than a new install.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

I was trying to do the eselect and python-updater when I could get in there, but things would break before completing.

It seemed when I did run the eselect it wrote over my make.profile to have selected all the profiles in the list which I had never seen before. Now it thinks I want a multi-lib profile despite what my make.conf says.

I also realize I was also doing tar -xpf instead of tar xpf.... would it still have read the parameters corrently as -xpf? Regardless, I blanked the lib directories again and when I ran the tar again, Its not performing correctly and when I chroot it says that libncurses.so.5 does not exist (which it shouldnt right)? Thats a 32-bit driver isnt it? So am I locked out of my chroot for good now since the make.profile is broken on top of all the pkgs?

I hate it! Still migrating files from home, hoping to sit down with the install sometime by the end of the weekend.
_________________
...and it should be known by now

phobos13013 · Apprentice Joined: 22 Sep 2004 Posts: 277 Location: PNW

Well, this thread is abandoned... lesson learned... do not rm /lib.... kind of a wasted lesson i guess. IF for some reason you do rm /lib... do not install a stage3 tarball on top of it... unless you plan on starting over, which I have... but I am up and running again after full redo of system.

Frankly it was well needed, the box had been resurrected like a Phoenix only at least three occassions. It was magic but not even this powerhouse could survive the chaos candidate: ME!

Heres to new beginnings and happy 2016!
_________________
...and it should be known by now