[solved] dump-capture kernels and initramfs (kexec -p)

sublogic · Last edited by sublogic on Thu Mar 24, 2022 7:06 pm; edited 1 time in total

I use lvm over LUKS to encrypt root, /var, /home and swap. The initramfs came from genkernel.

I'm trying to capture a crash dump after a kernel panic (see this topic). I read the kdump wiki article. I had a lot of trouble with video modes but I finally got "kexec -l" to work with the following:

Reboot.
Press "e" in grub to edit the chosen entry.
Comment out the set gfxpayload=keep that came from my /etc/default/grub.
In the "linux" line, add options "radeon.modeset=0 init_opts=S"

Taking out "gfxpayload" makes sure the console starts in ugly 80x25 VGA mode.
The "radeon.modeset" thing sabotages my graphic chip's kernel module and keeps the console in ugly VGA. This is of course specific to my machine.
The "init_opts" is genkernel initramfs syntax to stay in single-user mode so we don't try to start X (which would fail because radeon didn't load).
Press F10 to boot the entry.
Enter the root password for single-user mode.
Mount /boot. Optionally remount / read-write.

Run "./kexec.sh -l", this script.

Code:

#! /bin/bash

usage() {
echo Usage: "$0 -l|-p"
}

case $1 in
-l|-p) kopt=$1;;
*) usage; exit 1;;
esac

cmdline="root=/dev/vg0/root debug ro dolvm crypt_root=/dev/sda3"
cmdline="$cmdline hpet=force resume=/dev/vg0/swap init_opts=S"
cmdline="$cmdline irqpoll maxcpus=1 reset-devices"

set -x
kexec $kopt /boot/vmlinuz-5.15.26-gentoo-x86 \
--initrd=/boot/initramfs-5.15 \
--reset-vga --console-vga \
--append="$cmdline"

sync; reboot.

I use a script to be sure that I call kexec consistently. The script is also specific to my machine. The only parameter to the script is whether to run "kexec -l" (warm reboot) or "kexec -p" (capture kernel). I don't know if the "--reset-vga --console-vga" is necessary but it didn't hurt.

Well, the warm reboot works perfectly. No POST, go grub, the kernel restarts immediately, the initramfs loads, I get a rescue shell ("debug" in $cmdline), I exit the rescue and type my LUKS passphrase, I exit the next two rescue shells and I end up in single-user mode with the ugly VGA. Success! Yea. Now for the crash test. The capture kernel will be the same as the crashing kernel.

Reboot.
Press "e" in grub.
Comment out the set gfxpayload=keep .
Add options "radeon.modeset=0 init_opts=S crashkernel=64M nokaslr"
Press F10. Play in the rescue shell, enter the LUKS passphrase, etc.
Mount /boot. Optionally remount / read-write.
Run "./kexec.sh -p" .
At this point I'm in single-user mode with a capture kernel preloaded.
sysrq-c to trigger a panic.

FAIL ! Panic (expected), capture kernel starts (good), but it panics immediately:

sublogic · Posted: Thu Mar 24, 2022 7:06 pm Post subject:

Ah. It's a big initramfs (genkernel) and the capture kernel needs more reserved memory. in my case: "crashkernel=172M nokaslr".

Found by bisection. After the test panic, the capture kernel kicks in and enters early userspace no problem. I can see the /proc/vmcore from the rescue shell.

Alas, there is not enough memory left to open my LUKS2 volume

. (This operation is a memory hog; the cryptsetup FAQ explains why.)

Okay, not quite solved, but I have options.

Shrink the initramfs; I don't need its /lib/modules.
Copy /proc/vmcore to a USB stick from early userspace.

sublogic · Edit 2023-04-03:

Found a better way than the hack to stay on the 80x25 VGA: in my case: compile radeon built-in rather than as a module, and add the required firmware to the kernel.

To know what module takes over your console,