Lvm-Partitions vanished with Luks setup

[Manu311]

Hello,

I have a pretty bad problem with my Luks/Lvm setup and lost access to my system. Hopefully someone can help me get back my data.

I'll start with my setup:
I needed to have my Gentoo on an (almost) fully encrypted harddrive with as much security as possible, so I went with luks (aes-xts-plain64) with detached headers on an usb-stick. Inside of that luks there's a lvm with two partitions - one ext4 and one swap partition.
Since genkernel couldn't handle that, I switched to dracut (which I had to slighly manipulate so it waits for the usb-device).
I'm aware that the kernel and initramfs aren't encrypted so they are vulnerable, however that's not my problem right now (actually I could live with a bit less security).

So what happened:
The system has been working like this for half a year and I failed to setup any backups (I guess most important data is on servers anyways, so the loss is only 1-2 weeks of work instead of months). Since I had to replace my keyboard, I had to shutdown the notebook, which I did. Since I haven't changed any keyboards on notebooks, I had a couple of trial-and-errors where I cut the power during boot or kernel bootup because the keyboard didn't work (and I didn't have the patience to wait for the password prompt).
Eventually the keyboard worked and I wanted to boot back up - however after entering the password, dracut stopped and dumped me into it's backup shell. I don't recall exactly what I tried there but without much problems I could decrypt the partition again however LVM didn't find anything within it.
I recalled having such problem in the past (like 5 month ago) and that the solution was simply to recreate the Pv with the exact same uuid and restore my config (which I had backuped months ago) - anyways I searched for a tutorial and found this: https://www.golinuxcloud.com/recover-lvm2-partition-restore-vg-pv-metadata/#Step_1_List_backup_file_to_restore_LVM_metadata_in_Linux
which I followed.
At least step 1-4 were no problem, however step 5 didn't work. Neither did anything I have tried after that.

Things I tried:
0. Step 1-4 in the above mentioned tutorial
1. simply mounting: "wrong fs type, bad option, bad superblock on /dev/mapper/......., missing codepage or helper program, or other errors"
2. testdisk deep analysis without partition table: nothing found
3. testdisk deep analysis with intel partition table: found hundreds of partitions it can't recover, most of them bigger than the actual disk (just garbage data)

[alamahant]

Try from a live cd.
Are the lvm partitions visible from livecd?
Please run lsblk
If the partitions are visible maybe your problem is just reinstalling grub.
Can you mount the root partition from a livecd?
Lvm partitions dont just disappear...
_________________

[Manu311]

[alamahant]

I would advice you if your root partition can be mounted to backit up somewhere using rsync and then delete and recreate your pv vg and lvs...and move your backup to your newly created lvm partition.
And maybe if possible avoid taking such drastic actions like the article you are following...

_________________

[Manu311]

[alamahant]

But did you try to mount the root lvm from the livecd?
Is it mountable?
_________________

[Manu311]

[NeddySeagoon]

Manu311,

Try

[Manu311]

[NeddySeagoon]

Manu311,

Either your filesystem has been destroyed or /dev/mapper/censored--encrypted-root_fs does not point to the start of it, that would explain why the alternate superblock mount failed too.

The LVM recovery process only writes PV and LV metadata. filesystems are not harmed.
You also have censored--encrypted-swap_fs,

Does

[Manu311]

[NeddySeagoon]

Manu311,

Do you know some long (the longer the better) filenames you cat search for?

Try

[Manu311]

[NeddySeagoon]

Manu311,

The serching has to happen on the unlocked encrypted space, so the LUKS layer is used.
Searching the raw underlying device will do just that and is the crypto is any good at all, you won't find anything.

You are going to want more tools than there is on the Gentoo minimal CD.
Can you try the adminCD or some other random live distro?

If all that fails, all that left is thrashing through the partition layout, the PV header and the LVM metadata on disk locations to work out where your filesystem should start
Those things are unknown but are
a) constants
b) documented.

What does

[Hu]

You probably should have come to us for help before trying to restore the PV/LV metadata. As I understand your story, at that point, there were no unusual modifications to the data.

UUIDs are Universally Unique IDentifiers. You might consider them secret if you expect that someday someone will see the real UUIDs on your drive, and you want to be able to deny to them that you are Manu311 on the Gentoo forums. Other than that, sharing the UUIDs is probably fine. If you really want to keep them secret, write up a mapping table and apply that when you post. We don't necessarily need the real UUIDs, but as Neddy says, it would be helpful if we can distinguish when two posted UUIDs are the same on your disk.

What is the output of file -s device on each device that you expect to hold a filesystem/swap? If there are recognizable patterns, file can report them, even if the underlying structures are a bit damaged.

Since you are using LUKS, a misspelled command while unlocking will usually just fail to unlock, rather than reporting success and returning garbage.

[Manu311]

NeddySeagoon,

my understand is that the --encrypted_root-fs is just the partition within LVM. While the PV the lvm is on (/dev/mapper/censored-crypt) should also be decrypted. The crypted device should be /dev/nvme0n1p2

I'll download the admin-cd, so I got more tools.

I guess there's also no use in redacting uuids when I can't access my data anymore, so here is the output from the commands you asked for:

[NeddySeagoon]

Manu311,

As you explained it, the LUKS container is on /dev/nvme0n1p2
Once you unlock the container, you have a lvm2 PV with two logical volumes inside the physical volume.
You can do it the other way round too but you have several LUKS containers then.

If thats correct, the on disk layout is your primary GPT copy, /dev/nvme0n1p1, /devnvme0n1p2 then the backup copy of the GPT.
There may be some wasted space too.

/dev/nvme0n1p2 contains the LUKS header, which you say is detached. That does not mean that the space is reallocated, the PV metadata, then LV metadata and your user data.
That's followed by the backup copy of the GPT. There may be same wasted space around there too.

From pvdisplay We know that your PV /dev/mapper/censored-crypt contains 244062, 4.00 MiB extents with 2.00 MiB at the end and its all allocated.
We also know from lvdisplay -m that /dev/censored-encrypted/root_fs is at the end of the PV.
Physical extents 9564 to 244061
In summary, /dev/censored-encrypted/root_fs starts at 234498.5 * 4MiB + Backup GPT + waste space from the end of /dev/nvme0n1.
That's easier than working forward.
We are missing two pieces of information to be able to calculate sb= for the mount command.
The first is your partition table in sectors.
fdisk or parted will show that. We need the output in sectors.
Once we have that, we can work out the offset= value.

This Wikipedia suggests that there is no wasted space. Thus the backup GPT is 2MiB and offset= is 234499 * 4MiB from the end of the volume. The 2MiB GPT and 2MiB unusable space in the PV make up another extent.

Once we have a value for offset=, we can try mounting the filesystem with respect to /dev/nvme0n1. The LUKS container will need to be unlocked but we won't use any other metadata, no partation table, no LVM, just the start of the filesystem from the start of the block device.

-- edit --

The concern about redacted data is the not knowing its redacted, in case you redact something we need to know and miss it.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Manu311]

So, this?

[NeddySeagoon]

Manu311,

The entire /dev/nvme0n1.

We know the filesystem start from the end of the drive but offset= is a count of bytes from the start of the drive.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Manu311]

[DespLock]

testdisk might be of help here.

If you have internet access from your gentoo "live usb" you could simply emerge it. Otherwise SystemRescueCD is a good candidate for another live usb.

You need to open your /dev/nvme0n1p2 with cryptsetup and use testdisk on the /dev/mapper/censored-crypt

read the fine manual before trying to recover. Here is a tutorial:
https://www.cgsecurity.org/wiki/TestDisk_Step_By_Step

[NeddySeagoon]

Manu311,

Its even easier than I though.

The drive is 1024209543168 bytes.
234499 * 4MiB is 983560093696 bytes

The difference is 40,649,449,472 which is exactly divisible by 4096,

Unlock your encrypted volume.
Then

[Manu311]

NeddySeagoon,

you gave me hope. But then my system destroyed it again:

[NeddySeagoon]

Manu311,

nvme0n1 is the the entire block device. I was expecting the kernel to use the decryption layer once the LUKS volume was opened.
It looks like I was mistaken as

[Manu311]

NeddySeagoon,

sadly no luck either: