SOLVED! Kernel Panic on 6.9.x Upgrade

[Vorlon]

I'm running a monolithic kernel on ~amd64, as I have done for about a decade. I routinely upgrade the gentoo-sources kernel as they become available. Recently, I've been getting a kernel panic for 6.9.0 and 6.9.1. Kernel 6.8.9 runs fine.

Reviewing the dmesg output for the 6.9.x kernels vs correctly working 6.8.9, I see that the new kernel doesn't seem to try to access my nvme disk, but looks at the SATA disks instead. It eventually gives me a "/dev/root: Can't open blockdev" error and then panics.

I've booted the Gentoo live disk image and verified that all the kernel modules loaded by the live system are selected in my .config file. I've also used the hwinfo utility to list all the required modules, and have verified that they are selected for inclusion in the kernel.

Something has changed with the 6.9.x kernel, but I can't figure out what it is. I've got all the nvme options selected to include in my monolithic kernel.

Can somebody suggest something else to try?

TIA!
_________________
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation

[NeddySeagoon]

Vorlon,

Post your

[Vorlon]

Thank you for your help!!

lspci -nnk

[pietinger]

Maybe it is not your new kernel ... maybe it is your bootmanager/bootloader ... giving wrong kernel command line parameters to your kernel ...

Do you use grub ? If yes, I would like to see the /boot/grub/grub.cfg AND the output of "blkid" (and the information which of your partitions is your root partition).
_________________
https://wiki.gentoo.org/wiki/User:Pietinger

[Vorlon]

Hmmm... interesting idea. Grub has been kinda goofy lately. (I've had to mask some version for my other stable gentoo systems.)

Grub.cfg is here: https://pastebin.com/973DdzWY

Here is blkid

[pietinger]

Hmmm ... grub.cfg looks good ... yes, I see you have every module which kernel needs to access its root partition in your .config (EFI_PARTITION, BLK_DEV_*, SCCI, NVME, AHCI, FS) ... but you have activated some options that I don't know if they cause problems (they are options that are hardly ever used) ...

Do you really use CONFIG_POWER_RESET and CONFIG_NVMEM_REBOOT_MODE=y ?

And these: CONFIG_NVME_KEYRING=y and CONFIG_NVME_AUTH=y ?

You said you are using a monolithic kernel (like me) ... but do use also an initramfs ?

Because you have enabled it (CONFIG_BLK_DEV_INITRD=y) AND grub gives also an initramfs to your kernel (/boot/amd-uc.img)

When a kernel has an initramfs THEN it is the job of this initramfs to mount the rootpartition ... but you said you had a kernel panic ...

I would need additionally the "dmesg" (after booting one of your old kernels) to see more.


BTW: I have seen this:

[NeddySeagoon]

Vorlon,

You have 20 polaris10 firmware files listed in your kernel and

[Vorlon]

Update 2024-05-20:

Things tried:
1. Ensured I have all the "Must Haves" in pietinger's write up. (Very nice write-up, BTW!)
2. Selected CONFIG_IRQ_REMAP = Y
3. De-Selected CONFIG_BLK_DEV_INITRD. (Kernel notes say "mark Y if unsure")
4. Added missing polaris firmware file to kernel. (Good work spotting a very subtle error, NeddySeagoon!)
5. Ensured all the selected items in NVMe wiki are set (https://wiki.gentoo.org/wiki/NVMe)
6. Verified sufficient space in /boot (/boot is part of the main partition). Also ensured /efi has free space.
7. Rebuilt gentoo-sources 6.8.9, so it's (probably) not a grub issue

Results: Still the same kernel panic.
I can still boot to kernel 6.8.9, but 6.9.0 & 6.9.1 cause panic. <sigh>

So I'm still running, but perplexed.

Much appreciate all the suggestions!
_________________
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation

[NeddySeagoon]

Vorlon,

Please pastebin the changed kernel .config file and the kernel panic reason.
The message will say Kernel Panic ...
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[pietinger]

Vorlon,

the reason why I wanted to check the grub first is as follows:

I have seen that you have all the modules necessary for the kernel to access the root partition. And there are actually only two ways to cause a kernel panic during a CONFIGURATION: One of these modules is missing (or you configure the kernel for the wrong CPU architecture). In addition, it should not be possible to create an incorrect configuration if you already have an (old) working one and then only use this with "make oldconfig". If kernel configuration AND grub are correct, there is of course still the (very small) chance that we have a regression in the kernel. But before we assume this and create a kernel bug report, I wanted to have a look at the "dmesg" of the previous (working) kernel, because I see a lot about the system here:
https://wiki.gentoo.org/wiki/User:Pietinger/Overview_of_System_Information#dmesg

(I am also already on 6.9.1 and have no problems at all. Maybe you want to see what's new:
https://wiki.gentoo.org/wiki/User:Pietinger/Experimental/Manual_Configuring_Kernel_Version_6.9 )
_________________
https://wiki.gentoo.org/wiki/User:Pietinger

[Vorlon]

pietinger,

Here is my dmesg for a good boot using Kernel 6.8.9. https://pastebin.com/iaPjzJx7
A screen shot of the boot process for kernel 6.9.1 is here: https://pasteboard.co/cwW5BQD4aaus.jpg

NeddySeagoon,

Here is the .config file for 6.9.1:https://pastebin.com/YZuKwLHL

Thank you both for your continued help!
_________________
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation

[keltar]

I have similar problem on my laptop but not on desktop system, running nearly identical systems, with the same .config. Not sure if it is the same problem though. I'll look into it when I have time. In my case the error code for root mount is -16, which seems to by EBUSY.

[Goverp]

This is probably irrelevant, but I started using the 6.9 series when 6.9.1 went stable. First boot seemed fine, but the second boot hung. After some investigation, it turned out it wasn't hanging, it was heavily delayed by my initramfs's "fsck" of my rootfs. Under previous kernels that was a short run that took a second or two to be happy. On 6.9.1, after the first boot, it now spots some error (I forget what) in the metadata, which triggers a full "fsck -f" that takes a minute or two.

Note that this was on an f2fs root partition, and is probably specific to that. I've not had time to investigate further - not too keen on bisecting something that's presumably slightly corrupting my rootfs!
_________________
Greybeard

[NeddySeagoon]

Vorlon,

Your image shows the kernel trying to mount 103:3 as root.
That's the major:minor. device number.

/usr/src/linux/Documentation/admin-guide/devices.txt says that is

[grknight]

The photo shows a line Can't open blockdev which comes from file system mounting.
This changed in 6.9 with this commit as its start
For whatever reason, the symbolic /dev/root isn't pointing to the NVMe partition. This should be updated into the DEVTMPFS.

https://github.com/torvalds/linux/blob/master/block/bdev.c#L986 is now the function responsible for opening devices.

Since certain CGroup options are not set, the ret = bdev_permission(dev, mode, holder); always returns true.

I would suggest adding some pr_warn("<message>") macros into the bdev source above before each step to know where it stops. I only say this because there isn't much talking out via printk (what dmesg shows) even with a debug flag that I can tell.

The other alternative is to use an initramfs, like dracut, to control the mounting and would also give a rescue shell to inspect /dev.

[keltar]

Thanks for the hint grknight.
In my case it failed at bdev_open. But interesting thing is it was called twice, and failed on second call. So I digged a bit and managed to boot 6.9.1 after removing resume= and resume_offset= options. I've also stumbled upon https://bugzilla.kernel.org/show_bug.cgi?id=218845 which may be about the same problem I had.

But Vorlon's grub config don't seem to have resume option, so it is probably different problem from mine.
Do you have an error code specified somewhere near error message? Image seems to be clipped so can't tell.

[pietinger]

[Vorlon]

Woo Hoo! I'm booting to 6.9.1.

I tried removing the CONFIG_POWER_RESET, but it didn't make a difference.

The fix was to disable suspend and resume functions in the kernel. (I usually don't use that anyway.)

Much thanks to all who dug into this and offered suggestions!!
_________________
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation

[pietinger]

[keltar]

pietinger, linked patch works fine for me

[pietinger]

[Vorlon]

FYI. Kernel 6.9.3 worked for me, even with the hibernation and sleep enabled in the kernel.
_________________
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation

[grknight]

[Vorlon]

Ooops. You're right. For some reason portage asked for the kernel to be updated. I assumed this was the next version bump.

Sorry.
_________________
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation