Kernel doesn't load on Lenovo X1 Carbon 12th Gen [SOLVED]

[nagmat84]

TL;DR: The crucial and missing kernel options were "CONFIG_X86_X2APIC=y" and "CONFIG_X86_INTEL_LPSS=y". If you are looking for a working configuration, see: https://pastebin.com/QGB8K3J.

-------------------------------------------------

I have a Lenovo X1 Carbon 12th Gen and the Linux kernel directly crashes immediately after loading. I probably missed some important kernel config, but I don't have any clue which one.

Configuration

Boot manager: refind (refind does load)
Kernel: 6.11.8 with EFI stub support
Partition layout: nvme0n1p1 (EFI System), ..., nvme01p4 (Linux Swap), nvme01p5 (Linux Root), nvme01p6 (Linux Home)

Output on console after invoking Gentoo from refind

[pietinger]

I can see from your kernel configuration that you have some experience with it. Everything is configured for the console and also for accessing the hard disk. But it could be that you have a special PCI controller. Boot again with the MinimalCD and do a “lspci -nnk” and check if you need the kernel module “vmd” (which is very often used in Intel machines). If so, this is exactly what is missing (# CONFIG_VMD is not set). (See also: https://wiki.gentoo.org/wiki/User:Pietinger/Experimental/Manual_Configuring_Current_Kernel#Accessing_the_Root_Partition )

Also missing:

[nagmat84]

I assume I do not need the kernel module "vmd". lspci from the installation media does not list it. Here ist the pastebin for lspci: https://pastebin.com/Y0JuUngQ

[pietinger]

Uii ... a brand new MeteorLake ... please check with "dmesg | grep firmware" - after booting with our LIVECD (not the minimalCD; because of graphics driver) - needed firmware files, and put it into CONFIG_EXTRA_FIRMWARE. Does this help (together with my already mentioned changes)?
_________________
https://wiki.gentoo.org/wiki/User:Pietinger

[pietinger]

Maybe you need also: CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON=y
_________________
https://wiki.gentoo.org/wiki/User:Pietinger

[nagmat84]

I enabled CONFIG_VMD and CONFIG_EFI_HANDOVER_PROTOCOL to give it a try. No luck with that. (But I did expect that.)

I will try the Live Image. I am downloading ... However, please note that the Minimal Installation CD is able to boot and uses i915. So I doubt it is a firmware problem, but I will try anyway.

The minimal installation CD loaded the following FW files:

• regulatory.db
  
• regulatory.db.p7s
  
• iwlwifi-ma-b0-gf-a0-83.ucode
  
• iwlwifi-ma-b0-gf-a0.pnvm
  
• iwl-debug-yoyo.bin
  
• intel/ibt-0180-0041.sfi
  
• intel/ibt-0180-0041.ddc
  
• intel/sof-ipc4/mtl/sof-mtl.ri

I already have put those FW into my initramfs except for iwl-debug-yoyo.bin, because I wasn't able to find out which Gentoo Package provides that one. Note that the minimal installation CD is able to use FB on i915 without any graphics FW. Hence, I expect the Live Image to show nothing else.

[pietinger]

[nagmat84]

I tried to other things, both without success:

I compiled my kernel with i915 as a module in case this would have been the culprit and loaded too early. There is no difference, the kernel hangs directly on or after loading the initramfs.

I booted from the Live CD. The module intel_vpu of Live CD tried to load some additional firmware "vpu_37xx.bin, "mtl_vpu.bin" and "intel/vpu/vpu_37xx_v0.0.bin". However this failed as the FW was not available. But this does not seem to be related to graphics.

[pietinger]

[nagmat84]

[pietinger]

[nagmat84]

This is not a direct answer to the previous post, because I was busy trying different things, but I made some progress.

• gentoo-sources-6.11.8 with my custom configuration does not boot at all. This is variant which this post has been about originally and whose config is in the pastebin above.
  
• gentoo-kernel-6.6.58 (stable) with the configuration that is shipped with the ebuild produces the kernel null pointer error and the Oops which I wrote about in my previous posting. I don't get a graphical login, because sddm cannot find a usable display, but at least I get a console.
  
• genoo-kernel-6.11.8 (testing) with the configuration that is shipped with the ebuild simply works. There are no errors in dmesg, no firmware problems and I get a graphical output.

Note: Refind invokes all three kernel in the exact same way and all three kernels use initramfs which include the exact same set of files I assume that proofs that the problem actually lies within the kernel and is not a problem due to refind or some missing firmware files.

I guess one can assume that 6.6.58 might just be too old for my hardware. The null pointer exception within the kernel indicates a kernel bug. So let's forget that kernel version. I only accidentally emerged that, because I forgot to unmask ~amd64 for gentoo-kernel.

So the interesting question is what is the crucial difference between gentoo-sources-6.11.8 with my custom configuration and gentoo-kernel-6.11.8 with the distribution-provided configuration that makes the first one unbootable and the latter one working smoothly?

[pietinger]

[dmpogo]

[nagmat84]

Yes, I am absolutely sure that the initramfs is compressed with LZ4. I manually unpacked all three initramfs with cpio and LZ4 to check if and how they differ. (They don't.) And there should be an early kernel error if the kernel wasn't able to decompress the initramfs. Shouldn't it?

I don't believe the embedded kernel command line is the problem. It's only for localization.

We have to wait until next weekend until I can do more tests. I don't have the laptop with me during the workweek.

One other difference between the distribution kernel and the custom kernel are the IIO-related config options (IIO = industrial input output?). For what are those? Can this be the culprit?

[pietinger]

[nagmat84]

[nagmat84]

I am lost and hit a dead end. I took the distribution-provided kernel configuration for 6.11.8 from /etc/portage/savedconfig, copied that configuration to the kernel sources for 6.12.1, ran "make oldconfig" and compiled gentoo-sources-6.12.1 myself. I ran "make all && make modules_install && make install" and everything was fine. The kernel booted. So rEFInd and my Dracut configiuration are definitely fine, too.

Then, I took my custom kernel config and ran "make oldconfig". Then I incorporated all the tips from here (i.e. enabled CONFIG_I2C_HID_ACPI, disabled CONFIG_SYSFB_SIMPLEFB, etc.). Additionally I inspected lspci, usb-devices and lsmod what else is loaded by the kernel with the distribution-provided configuration. I ran

[pietinger]

These MUST be activated:

[dmpogo]

[pietinger]

[nagmat84]

Good news first. I implemented all recommendations by @pietinger and finally got a booting kernel. Yipeeh! This means I finally have a baseline from where I can start reverting some of the changes (like module signing) until I find the crucial settings (I guess it will be CONFIG_X86_X2APIC and CONFIG_X86_INTEL_LPSS). When I will have found out, I am going to report back.

However, first I need to find some other missing pieces. Unfortunately, the touchpad and touchscreen both stopped working. I assume it is some missing driver which has been statically built into the distribution kernel, because I couldn't find anything suspicious with "lsmod". Once again, I put my hope into this fantastic forum community.

Here are the pastebins for the custom kernel which implements pietinger's recommendations:

• Kernel config: https://pastebin.com/wracLhtd
  
• dmesg: https://pastebin.com/4Bxd37UQ
  
• lspci: https://pastebin.com/63969LZG
  
• lsusb: https://pastebin.com/EUbV61WH

Here are some of the differences which caught my eye. "-" always is the distribution kernel, "+" is the custom kernel.

Some addresses seems to have changes, but I assume that is harmless and expected:

[pietinger]

[nagmat84]

I am going to have a look into that tomorrow.

Is there any way to systematically find out which kernel driver is used for what /sys/devices/**/input? Beides the touchpad and touchscreen the distribution kernel registers a lot more input devices (29 instead of 6). Hence, some other drivers besides those for the touch input seems to be missing, too.

Should I worry about the disabled message-signaled interrupt (MSI) for audio controller (PCI device 00:1f.3)?

[pietinger]

Hmmm ... have you made number 2 of -> ?
https://wiki.gentoo.org/wiki/User:Pietinger/Tutorials/Manual_kernel_configuration#Before_you_start

If you dont find a module, try to search it in "make menuconfig" with a /
(see also: https://wiki.gentoo.org/wiki/User:Pietinger/Tutorials/Manual_kernel_configuration#Basics )

Type there: "IPU6" ... and yes its a module for your video camera ... and then these errors "int3472-discrete INT3472 ..." should be gone also.

Maybe you need # CONFIG_SERIAL_MULTI_INSTANTIATE is not set (you should see in your "lsmod" list) ?

Do you have emerged sys-firmware/sof-firmware for your sound? (If yes, we have a little problem ... maybe your machine is too new). If no, it should be the reason for the different capability of Meteor Lake-P HD Audio Controller.

You have a Synopsys Desingware PCI Controller ... you will need for your touchpad (yes, you have everything for it already configured, except designware):
* I2C_DESIGNWARE_PCI
* I2C_DESIGNWARE_PLATFORM
and maybe also:
* PCIE_INTEL_GW
* SPI_DESIGNWARE (?? here I am very unsure)

The rest of the system log looks really good.
_________________
https://wiki.gentoo.org/wiki/User:Pietinger