luksOpen fails in initramfs

[Moriah]

I have recently upgraded to the 6.6.13 kernel using gentoo-sources. The kernel worked with my old disk and booted with my old initramfs, so I built a new disk from scratch (after carelessly clobbering my old disk) using a live usb. I prepared the disk with the cryptsetup on the live usb using luksFormat The old disk and its kernel's initramfs were built with tools from 2010. The live usb that was used to setup the new disk uses the 6.1.67-gentoo-x86_64 kernel.

I use a separate usb stick to boot my system, with the kernel on the usb stick and the disk fully encrypted with LUKS, then LVM on that. The disk is not partitioned at all, just LUKS encrypt the whole device. This is the way the old disk was also.

But the live usb apparently uses a newer cryptsetup than was used to make the old disk and its initramfs, and so I built a new cryptsetup by static linking cryptsetup and placing the binary in the bin directory of my initramfs, then I built the kernel to incorporate the new static linked cryptsetup.

It will not boot. I fails the cryptsetup luksOpen command with the following error message:

[frostschutz]

make sure that /dev/random /dev/urandom exists (that /dev devtmpfs is mounted)

[Moriah]

Aha! /dev/random is not in /dev on my initramfs directory:

[Moriah]

I added /dev/random to the initramfs/dev directory, and I still get:

[Hu]

Why are you using df here? You have no mounted filesystems at this stage, so there is nothing for which you can meaningfully measure the amount of disk free.

What is /dev/tempfs? Did you mean the filesystem devtmpfs, which would be present if you mounted it (traditionally on /dev) and absent otherwise?

When you drop to a rescue shell, what is /dev/sda? Does it return what looks like a plausible LUKS header?

[Moriah]

/dev/sda will decrypt using the cryptsetup on my live isb, and that is the same cryptsetup that I used to encrypt /dev/sda before I used LVM2 to do the pvcreate, vgcreate, and lvcreate to set it up for this system. Using the live usb everything works great, but when I try to boot the kernel (with its initramfs), it does as described above.

What exactly do you mean by

[Hu]

On the Live USB, /dev/sda refers to a block device which contains a valid and usable LUKS container. There is no reason to assume that the /dev/sda on the failing system also refers to that same block device. It likely does, unless the kernel is enumerating disks differently, or your use of a manually curated /dev has provided an incorrect device node sda. Since you have a non-working system, I suggest you assume nothing, and verify each prerequisite.

The report that fdisk returns a 2TB drive is encouraging, and from context, I infer the system has no other 2TB drives that sda could wrongly refer to. If so, we can probably assume that sda refers to the block device you intend it to refer to. For checking for a LUKS header, you could use cryptsetup isLuks, or try to do it by hand by manually inspecting the first few dozen bytes of the device. If you suspect a kernel problem, you could check dmesg. You could try using the live kernel's configuration with your custom initramfs. You could try using a non-static cryptsetup with its libraries explicitly included. You could inspect the cryptsetup code to see under what circumstances it produces the errors about RNG initialization, and then try to disprove that those could be occurring in the live environment.

[frostschutz]

cryptsetup code

setup.c :: init_crypto()

[Moriah]

Regarding my non-partitioned /dev/sda, since I use LVM for all my filesystems, I do not partition the disk, as it is unnecessary. I have used this approach since at least 2008, when I built my original backup server. I need to have all filesystems under LVM so that I can take an LVM snapshot of the filesystem before I back it up, thus freezing the filesystem for the backup. Meanwhile, after the snapshot has been created, the machine can be used normally while the backup operation is occurring, and it will not cause any corruption of the backup, as it is a backup of a snapshot of the filesystem frozen in time. Since my root filesystem needs to be snapshotable, it also needs to be based on LVM. I originally started using initrd/initramfs tto get / under LVM back in the days of nash when all initrds were hand crafted. I added the full disk encryption later, around 2012 so that my laptop would be protected against data theft, as I carry it with me on consulting gigs, and it has client proprietary data on it.

Thanks for the sleuthing around for where the error messages I am seeing are generated. Since I now have both /dev/random and /dev/urandom in the initramfs, I am assuming that I must be missing something in the kernel. I did post earlier in this thread a link to a screenshot of the boot failure, and also a link to my .config file. I also checked to make sure that my static linked cryptsetup indeed has no library references:

[Moriah]

I just had a simple idea for a test. I can boot into the live usb, then mount my boot usb that has the cryptsetup problem. I ought to be able to run the static linked cryptsetup manually in the live usb environment and see if it also fails there. If it does, then we know the problem is in the static linked cryptsetup. I will try that tomorrow when I am back in the lab where the laptop is located.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Moriah]

That was a worthwhile test! My static linked cryptsetup works fine when run in the live usb environment, so it must be something about my kernel.

Can somebody knowledgable about the linux-6.6.13-gentoo kernel please take a look at my .config file? It is located at:

http://www.elilabs.com/~rj/config-6.6.13-gentoo

Thanks!
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[pietinger]

I think this is the reason for your problem (but I am not sure): # CONFIG_CRYPTO_USER is not set

Maybe you need also (but I dont think so): # CONFIG_CRYPTO_USER_API_RNG is not set

What I recommend: Enable every accelerated Module, IF you use the plain module AND IF your CPU has this ability - at least:

[sublogic]

[pietinger]

[Moriah]

I had a chance to rebuild my kernel this evening. I only enabled CONFIG_CRYPTO_USER. The idea I had in mind was not to just get this to work, but to find out *EXACTLY* what makes the difference. If this doesn't work, I will enable the other options that pietinger suggested, one by one, until it does work, or until I prove that none of them work.

If nothing works, then I will run strace on cryptsetup in the live usb environment and see what all it calls. Hopefully that will help us zero in on the problem.

I noticed that this new kernel build had to rebuild a lot of files. I guess CONFIG_CRYPTO_USER is a conditional compilation flag that appears in many places.

I am unable to test the new kernel until tomorrow, as I built it by logging in remotely to my lab, which is in another building about 5 miles away from my home where I am working now. I will post again after I see whether it works.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Hu]

Considering how expensive it is for you to test this (visiting the lab in person), perhaps this would warrant a side project of creating a virtual machine, booting it with the Live USB, creating a LUKS volume in that machine, and then rebooting that machine into your test kernels+initramfs combinations. That way, you can run this test on any hardware you like, and not take down a machine anyone else uses when you run your tests.

[Moriah]

Nice idea, but we are in the middle of a move. My lab is still at the old location, all I have at the new location is my wife's W12 laptop running cygwin for ssh tunnellinig into the LAN in my lab at the old location, and VNC to access everything in the lab remotely. My laptop is over there so I have other machines to use to ssh into the laptop and still have access to the rest of my network , including my backup server. Besides the wired business grade connection there is much faster that the wifi connection at the new house. I should be moving the lab to the new house in a few weeks. The construction work on the new lab is just about finished. The move will be pretty disruptive, and I want my laptop working before we start the move. Besides, I am over at the old location every day anyway.
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[Moriah]

Adding CONFIG_CRYPTO_USER did not solve the problem.

I will be adding other suggested tags in the near future
_________________
The MyWord KJV Bible tool is at http://www.elilabs.com/~myword

Foghorn Leghorn is a Warner Bros. cartoon character.

[pietinger]

[Moriah]

Did you mean FUSE_FS, which is currently a module? Is that the tag I should make static?

[pietinger]

[Moriah]

Well, the FUSE_FS did not help, so I added --debug to the cryptsetup call in the init script in the initramfs, but not enough information there to shed any light on the problem either, so I did an strace using the static linked cryptsetup running in the live usb environment, as I could not figure a reasonable way to run strace from the initramfs.

The output of the strace is at http://www.elilabs.com/~rj/cryptsetup_static.strace

Grepping the interesting lines, we get:

[pingtoo]

[Moriah]

See my posts on March 17 where I said /dev/random was missing so I put it in.

Since I am not in my lab now, here is:

[pingtoo]