gentoo-sources-5.6 fail to boot without kernel panic

[Hupf]

Hi,
I'm trying to debug why after upgrading from gentoo-sources-5.5.x to 5.6.x, my machine fails to boot. Rollback is easy at the moment, but at some point I'd like to be able to make the switch.
On 5.5.x, my working setup is as detailed below. I've verified that my toolchain should be properly configured by recently upgrading to gentoo-sources-5.5.19 without issues.

Boot stack (working with 5.5):

• Kernel is manually configured* with embedded firmware (AMD Picasso GPU + CPU microcode), EFI stub, embedded command line and embedded initramfs
  
• UEFI loads EFI stub from NVMe
  
• CONFIG_CMDLINE omits the root=, init= or initrd= options, neither are they passed via the UEFI/efibootmgr (i.e. defaults are used)
  
• EFI stub loads embedded initramfs
  
• initramfs built using custom initramfs.list including custom /init script, which invokes cryptsetup on the actual root partition (on NVMe)
  
• custom /init hands over to systemd

What I see with 5.5.19:

• printk's relating to various hardware, e.g. USB keyboard, NVMe partions, unused SATA slots etc
  
• "Freeing unused kernel image [...] memory: xxxK"
  
• "Run /init as init process"
  
• <output from initramfs's /init script>

Config-wise,

What I see with 5.6.10:

• printk's relating to various hardware, e.g. USB keyboard, NVMe partions, unused SATA slots etc
  
• no further boot progress, only a blinking cursor after the last printk

So, with 5.6.10, there is no panic/error message (as far as the scrollback goes - right now I don't have hardware for serial console over USB available). The boot process is missing / stopping before the steps "Freeing unused kernel image memory" and "Running /init as init process".
The NVMe device seems to be recognized properly and partitions listed. The embedded initramfs seems to be there just fine - it is built during make and the resulting bzImage is of very similar size to the 5.5.19 one. I'm uncertain if the handover to the initramfs is the problem at all, since then I would expect complaints about the kernel being unable to find/call /init (or the root device). Also I wonder why I'm not seeing the "Freeing unused kernel memory".

I tried scanning the Documentation/, kernel.org and gentoo bugs as well as changelogs/commit histories for anything related to e.g. initramfs, EFI stub and NVMe but couldn't make out changes specific to my problem. I also tried explicitly specifying root=/dev/ram0 init=/init without seeing changes in the output.


* .config for 5.6.10 (not booting)

[dr_wulsen]

Have you tried to set the kernel config CONSOLE_LOGLEVEL to 15 ?
With some good luck, you might be able to circle the issue with more output.

Of course, you could try to enable any other relevant VERBOSE option in your kernel for fun and see if output becomes more meaningful.
_________________
There's no stupid questions, only stupid answers.

[NeddySeagoon]

Hupf,

Wild thought. Its all working except the console.

If you can ssh in and read dmesg, you may find that you need a new (extra) firmware file for your GPU.
I have a Polaris11 card and that's bitten me twice since the 4.16 kernel
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Hupf]

Some updates.
After a session of https://wiki.gentoo.org/wiki/Kernel_git-bisect I identified https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=33960acccfbd7f24d443cb3d0312ac28abe62bae as the offending commit.
So I started playing around (on the most recent gentoo-sources:5.7.0) with the kernel config options related to crypto devices and trusted execution environment.
The culprit seems to be CONFIG_CRYPTO_DEV_CCP_CRYPTO.
Before 33960acccfbd7f24d443cb3d0312ac28abe62bae, my system will boot with CONFIG_CRYPTO_DEV_CCP_CRYPTO=y. After the commit (up until 5.7.0), the system will bot with CONFIG_CRYPTO_DEV_CCP_CRYPTO=n or CONFIG_CRYPTO_DEV_CCP_CRYPTO=m, but not with CONFIG_CRYPTO_DEV_CCP_CRYPTO=y. Also, with CONFIG_CRYPTO_DEV_CCP_CRYPTO=m, the module will load just fine after boot (i.e. without halting/crashing the system for example - I haven't tried actually using the functionality).
Here is the working config-5.7.0-gentoo for reference.
Furthermore, as hinted by the commit's diff, I indeed have a

[Goverp]

Don't assume filing a bug on bugs.kernel.org will achieve anything. You need to look at /usr/src/MAINTAINERS to see where the people who look after the bit of code you have a problem with. When I had a problem with the AMDGPU driver, I needed to send a polite explanation of the bug, a git bisect and context info to the mailing list. A patch would be even more successful! They were very quick to fix it then. Before that, I and others had mistakenly put a bug on bugs.kernel.org and waited, and waited....
_________________
Greybeard

[Ant P.]

On my system the ccp driver outright refuses to load complaining that the BIOS is playing tricks. It's possible yours is similar and the kernel doesn't detect it.

[toralf]

You bisected it - good work!
Now just identify the To: and Cc: by doing something like