| View previous topic :: View next topic |
| Author |
Message |
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
 Posted: Wed Apr 03, 2024 5:45 pm Post subject: encrypted bcachefs root partition |
 |
|
After recently getting a new SSD, I wanted to try bcachefs because some interesting patches for the filesystem repair are to be merged soon. I enabled it in the kernel and created an extra partition and then copied my root filesystem over onto that partition.
Then I went to create the initramfs (because previously my root partition was just ext4 and I did not need an initramfs for anything else) from this great tutorial and added all things from lddtree /sbin/bcachefs (I needed to install =bcachefs-tools-9999 because otherwise it fails to read the arguments) to the initramfs_list. When trying to boot this, I noticed that busybox findfs and blkid seem to lack support for bcachefs. I also tried to simply specify the partition as root= kernel command line parameter, but the kernel cannot prompt me for the passphrase so it cannot mount the partition.
While it is possible to use something like dracut, I'd like to avoid using an initramfs especially when nearly everything is already in the kernel. Might it be possible to extract the key via keyctl after unlocking the partition and adding it into the keyring at compile time, as mentioned here? Or maybe modify the kernel so it prompts for a password? |
|
| Back to top |
|
 |
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1910
|
| Posted: Wed Apr 03, 2024 5:51 pm Post subject: Re: encrypted bcachefs root partition |
|
|
| thelordi wrote: | | While it is possible to use something like dracut, I'd like to avoid using an initramfs especially when nearly everything is already in the kernel. Might it be possible to extract the key via keyctl after unlocking the partition and adding it into the keyring at compile time, as mentioned here? Or maybe modify the kernel so it prompts for a password? |
If you need to run any command to prepare the rootfs, then an initramfs will be required. The kernel has no method of accepting input from the user. So, any prompting needs an initramfs and the appropriate tools. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Wed Apr 03, 2024 7:10 pm Post subject: Re: encrypted bcachefs root partition |
|
|
| thelordi wrote: | | [...] I noticed that busybox findfs and blkid seem to lack support for bcachefs. [...] |
They cannot help you; you will need mount.bcachefs ... mybe take a look to:
https://wiki.gentoo.org/wiki/User:Pietinger/temp/test
(attention: I made this article before my switch from split-usr to merged-usr; so some paths will be wrong now)
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Wed Apr 03, 2024 8:31 pm Post subject: |
|
|
I actually want something similar to the setup you describe there: https://wiki.gentoo.org/wiki/User:Pietinger/temp/test.
I followed the suggestion and added the *.bcachefs tools as well as their dependencies. This still fails, because while unlocking works fine, they fail to mount with os error 6, similar to this: https://www.reddit.com/r/bcachefs/comments/1b15jkg/mount_encrypted_bcachefs_fatal_error_no_such/. But when I downgrade bcachefs-tools to 1.6.4-r1 or 1.4.0, (as suggested there) I cannot mount the partitions at all.
Also, when using the regular /sbin/blkid provided by the system, I can get the PARTUUID just fine, but in busybox blkid, both bcachefs partitions don't show up. Might it make sense to take this as well as findfs from the regular system instead of busybox? |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Wed Apr 03, 2024 9:22 pm Post subject: |
|
|
First of all: I have stopped my tests because I am waiting for 6.9. Which kernel version you are running ?
You have said you need masked bcachefs-tools. I would be interested with which tools you have made/formated your partitions ... and which tools have now a problem ? Maybe try to reformat with 1.4.1 and check if mounting works also with 1.4.1 then ?
(TBH: I believe it is too soon to use it for a root partition; maybe test partitions ... with many actual backups  )
(I hope also you didnt used the Use-flag "fuse"  )
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Wed Apr 03, 2024 9:26 pm Post subject: |
|
|
| thelordi wrote: | | [...] but in busybox blkid, both bcachefs partitions don't show up. Might it make sense to take this as well as findfs from the regular system instead of busybox? |
I dont understand for what you need blkid and findfs ... ? In the initramfs you do only the mounting ... with mount.bcachefs ...
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Wed Apr 03, 2024 10:18 pm Post subject: |
|
|
| pietinger wrote: | | Which kernel version you are running ? |
Right now I am using kernel 6.8.2, although it would probably make sense to test this more extensively with 6.9-rc2 tomorrow.
| pietinger wrote: | | I would be interested with which tools you have made/formated your partitions ... and which tools have now a problem ? |
I am at least 80% certain that I already formatted the partitions with the masked version because there was some issues with bcachefs format complaining about invalid arguments when I am fairly certain that the syntax is correct (or has at least changed, since the exact same syntax on the masked version doesn't work with the stable one). Now emerged the stable one, created a new partition, mounted it just fine. Tried the old partition, also works fine. Will try to make an initramfs with this again...
| pietinger wrote: | | TBH: I believe it is too soon to use it for a root partition; maybe test partitions |
I absolutely agree, and I will absolutely keep my ext4 root partition around, but especially with the upcoming promised reliability features I am drawn to at least try to use it, and a perfectly working system is boring (and the reason I could not stand using something like bazzite when I briefly tried it).
| pietinger wrote: | | I hope also you didnt used the Use-flag "fuse" |
I did not, because when it is considered "experimental!" for a filesystem that was merged just one kernel version prior seems a bit to experimental even for me.
| pietinger wrote: | | I dont understand for what you need blkid and findfs ... ? |
The only reason I used findfs is because occasionally on reboots /dev/nvme0n1 and /dev/nvme1n1 switch, and the bcachefs partitions are not the same number on both drives. And I used blkid once after being confused that findfs did not find anything. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Wed Apr 03, 2024 11:24 pm Post subject: |
|
|
| thelordi wrote: | | [...] Now emerged the stable one, created a new partition, mounted it just fine. Tried the old partition, also works fine. Will try to make an initramfs with this again... [...] |
I am very interested in your experiences. If you like, please post an update - either here - or in my Wiki article.
| thelordi wrote: | | [...] and a perfectly working system is boring [...] |
| thelordi wrote: | | [...] The only reason I used findfs is because occasionally on reboots /dev/nvme0n1 and /dev/nvme1n1 switch, and the bcachefs partitions are not the same number on both drives. And I used blkid once after being confused that findfs did not find anything. |
I see ! Yes, just use the original findfs. I have had no problems with the 1.4.1 tools:
| Code: | # ldd /usr/bin/findfs
linux-vdso.so.1 (0x00007ffe23b56000)
libblkid.so.1 => /usr/lib64/libblkid.so.1 (0x00007f30d609d000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007f30d5ebf000)
/lib64/ld-linux-x86-64.so.2 (0x00007f30d6118000)
# blkid
/dev/nvme0n1p3: UUID="7e02d375-1f21-4f7c-97fa-c975d396a283" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="Root" PARTUUID="6979eed7-ffaf-425e-8ac7-2832f6d15e0a"
...
/dev/sda2: UUID="24808e71-3cc0-48db-a567-684eab6b9e14" BLOCK_SIZE="4096" UUID_SUB="7cd5c8bf-5751-4c21-a364-159af14123a0" TYPE="bcachefs" PARTLABEL="primary" PARTUUID="2896efe7-a70e-4205-963a-d28f61d61906"
...
/dev/sda1: UUID="768a9a90-3741-4eef-908d-4abb8e48c4d1" BLOCK_SIZE="4096" UUID_SUB="ebf613be-f720-4c55-a047-243c4211b698" TYPE="bcachefs" PARTLABEL="primary" PARTUUID="0418aa57-c08f-47d4-bbf8-192da2031caf"
...
# findfs PARTLABEL=Root
/dev/nvme0n1p3
# findfs PARTUUID="0418aa57-c08f-47d4-bbf8-192da2031caf"
/dev/sda1
# findfs PARTUUID=0418aa57-c08f-47d4-bbf8-192da2031caf
/dev/sda1 |
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Thu Apr 04, 2024 10:52 am Post subject: |
|
|
Success! Sort of...
I managed to mount the new bcachefs partition just fine, but after some digging and a fsck.bcachefs on the old bcachefs-partition I got an error 6 as well because a buffer overflow occurs. The new partition is already degraded as well... But I can mount it from within the initramfs!
/usr/src/initramfs_list: (For a split_usr system, please check the paths for merged_usr)
| Code: | # Required for embedded initramfs
nod /dev/console 0600 0 0 c 5 1
# General Setup
dir /bin 755 0 0
dir /dev 755 0 0
dir /etc 755 0 0
dir /lib 755 0 0
dir /lib64 755 0 0
dir /mnt 755 0 0
dir /mnt/root 755 0 0
dir /proc 755 0 0
dir /root 700 0 0
dir /sbin 755 0 0
dir /sys 755 0 0
dir /tmp 755 0 0 # Required for mount.bcachefs
dir /usr 755 0 0
dir /usr/bin 755 0 0
# Required in order to find libgcc_s.so.1
dir /usr/lib/ 755 0 0
dir /usr/lib/gcc 755 0 0
dir /usr/lib/gcc/x86_64-pc-linux-gnu 755 0 0
dir /usr/lib/gcc/x86_64-pc-linux-gnu/13/ 755 0 0
dir /usr/lib64 755 0 0
dir /var 755 0 0
# Busybox for most utils
file /bin/busybox /bin/busybox 755 0 0
# The Init
file /init /usr/src/initramfs/init 755 0 0
# Other files called by init
file /sbin/bcachefs /sbin/bcachefs 755 0 0
# Dependencies for /sbin/bcachefs
file /usr/lib64/libblkid.so.1 /usr/lib64/libblkid.so.1 755 0 0
file /usr/lib64/libuuid.so.1 /usr/lib64/libuuid.so.1 755 0 0
file /usr/lib64/liburcu-common.so.8 /usr/lib64/liburcu-common.so.8 755 0 0
file /usr/lib64/liburcu.so.8 /usr/lib64/liburcu.so.8 755 0 0
file /usr/lib64/libsodium.so.26 /usr/lib64/libsodium.so.26 755 0 0
file /usr/lib64/libz.so.1 /usr/lib64/libz.so.1 755 0 0
file /usr/lib64/liblz4.so.1 /usr/lib64/liblz4.so.1 755 0 0
file /usr/lib64/libzstd.so.1 /usr/lib64/libzstd.so.1 755 0 0
file /lib64/libudev.so.1 /lib64/libudev.so.1 755 0 0
file /usr/lib64/libcap.so.2 /usr/lib64/libcap.so.2 755 0 0
file /usr/lib64/libkeyutils.so.1 /usr/lib64/libkeyutils.so.1 755 0 0
file /lib64/libm.so.6 /lib64/libm.so.6 755 0 0
file /usr/lib64/libaio.so.1 /usr/lib64/libaio.so.1 755 0 0
file /lib64/libc.so.6 /lib64/libc.so.6 755 0 0
file /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2 755 0 0
# Needed to be changed because otherwise it wouldn't find the file
file /usr/lib64/libgcc_s.so.1 /usr/lib/gcc/x86_64-pc-linux-gnu/13/libgcc_s.so.1 755 0 0
# Additionally required
slink /sbin/fsck.bcachefs /sbin/bcachefs 777 0 0
slink /sbin/mkfs.bcachefs /sbin/bcachefs 777 0 0
slink /sbin/mount.bcachefs /sbin/bcachefs 777 0 0
# For finding the correct root partition
file /sbin/findfs /sbin/findfs 755 0 0
file /sbin/blkid /sbin/blkid 755 0 0
# Dependencies for /sbin/findfs and blkid (already covered above)
# file /usr/lib64/libblkid.so.1 /usr/lib64/libblkid.so.1 755 0 0
# file /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2 755 0 0
# file /lib64/libc.so.6 /lib64/libc.so.6 755 0 0
# For storing the keys properly
file /bin/keyctl /bin/keyctl 755 0 0
# Dependencies for /bin/keyctl (already covered above)
# file /usr/lib64/libkeyutils.so.1 /usr/lib64/libkeyutils.so.1 755 0 0
# file /lib64/libc.so.6 /lib64/libc.so.6 755 0 0
# file /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2 755 0 0 |
/usr/src/initramfs/init:
| Code: | #!/bin/busybox sh
rescue_shell() {
# The symlinks are not required but it helps tab completion
busybox --install -s
printf 'Starting rescue subshell: %s\nExit with code 5 to replace /init with a shell running as pid 1\n' "$*"
/bin/sh
if [[ $? -eq 5 ]]; then exec /bin/sh; fi
}
### Prepare
# Clear the screen
clear
echo "Mounting proc, sys, securityfs, tmp and devtmpfs ..."
mount -t devtmpfs devtmpfs /dev || rescue_shell "Error: mount /devtmpfs failed !"
mount -t proc proc /proc || rescue_shell "Error: mount /proc failed !"
mount -t sysfs sysfs /sys || rescue_shell "Error: mount /sysfs failed !"
mount -t securityfs securityfs /sys/kernel/security || rescue_shell "Error: mount /sys/kernel/securityfs failed !"
mount -t tmpfs tmpfs /tmp || rescue_shell "Error: mount /tmp failed !"
# Disable kernel messages
echo 0 > /proc/sys/kernel/printk
### Mount root
# Hardcoded UUID
myrootpartition="UUID=12460326-fd2a-4dd2-9241-077588556921" # Change this!!
echo "Searching root partition device name of $myrootpartition ..."
rootdev=`/sbin/findfs $myrootpartition` || rescue_shell "Error with findfs !"
echo "Found $rootdev as your root partition. Will mount it now ..."
/sbin/mount.bcachefs -o ro $rootdev /mnt/root || rescue_shell "Error mounting root partition !"
# Enable kernel messages again
echo 1 > /proc/sys/kernel/printk
echo "Unmounting proc, sys, securityfs, tmp and devtmpfs ..."
umount /proc /sys/kernel/securityfs /sys /tmp /dev
# Switch to the real root and execute init
exec /sbin/switch_root /mnt/root /sbin/init |
This should be able to mount an encrypted root partition at boot time with Kernel 6.8.2.
I'll try some more with version 6.9-rc2 and give another update. Also: what would you recommend for copying the root partition? I used rsync, but it is single-threaded and therefore fairly slow (peaking at around 6/7 MB/s on my M.2 SSD. I did some searching, but all wrappers for multithreading seem horribly out of date... |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Thu Apr 04, 2024 12:49 pm Post subject: |
|
|
| thelordi wrote: | | I'll try some more with version 6.9-rc2 and give another update. [...] |
A big thank you ... for this post ... and for the next one ! 
| thelordi wrote: | | [...] I used rsync, but it is single-threaded and therefore fairly slow [...] |
Hmm ... maybe it is slow ... but ... it is very safe. I am using it also (and therefore I can not suggest another tool). 
(I will safe the link to this article in my wiki article now)
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Thu Apr 04, 2024 2:17 pm Post subject: |
|
|
So I installed Kernel 6.9-rc2 from the release tarball and updated the initramfs as well as my system with it, then reformatted the partitions with:
| Code: | | # mkfs.bcachefs --compression=zstd --encrypted --replicas=2 /dev/nvme0n1p2 /dev/nvme1n1p2 |
And that works. I can mount the partitions on my regular system without any problems. HOWEVER, in the initramfs (with the exact same kernel (aside from the initramfs being enabled) and tools, it still fails with error 6.
I'll try again and bring another update after trying with the -git version of bcachefs-tools as well as the upstream bcachefs source tree. |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Thu Apr 04, 2024 3:24 pm Post subject: |
|
|
So this did not work. Mounting a regular, non-encrypted partition works just fine, but the encrypted one fails, reporting that it does not exist. Even with the newer Kernel / tools. I think something is missing here that is initialized later in the boot process that is needed in order to recognize an encrypted bcachefs-partition, but even that should not really be occuring because I built a module-less kernel, so everything should be there... Right? And creating an encrypted partition also works in the initramfs... I just can't mount it...
Might be worth to report upstream? |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22619
|
| Posted: Thu Apr 04, 2024 4:54 pm Post subject: |
|
|
| Is there anything useful in dmesg when it fails? Since you suppress all kernel messages, you may be missing something important. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Thu Apr 04, 2024 5:09 pm Post subject: |
|
|
| thelordi wrote: | So this did not work. Mounting a regular, non-encrypted partition works just fine, but the encrypted one fails, reporting that it does not exist. Even with the newer Kernel / tools. I think something is missing here that is initialized later in the boot process that is needed in order to recognize an encrypted bcachefs-partition, but even that should not really be occuring because I built a module-less kernel, so everything should be there... Right? And creating an encrypted partition also works in the initramfs... I just can't mount it...
Might be worth to report upstream? |
Only a short question: Have you seen this (in my wiki article) - could it be the same ? =>
| Quote: | After this I had this problem:
# mount.bcachefs /dev/sda1:/dev/sda2 /mnt/bc
ERROR - bcachefs_rust::cmd_mount: Fatal error: Required key not available
I found this: https://lore.kernel.org/all/12360099.O9o76ZdvQC@lichtvoll.de/T/ and did it:
# keyctl link @u @s
# mount.bcachefs /dev/sda1:/dev/sda2 /mnt/bc
Now it worked. I copied a file into it and made a subvolume: |
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Thu Apr 04, 2024 5:42 pm Post subject: |
|
|
| Hu wrote: | | Is there anything useful in dmesg when it fails? |
Unfortunately, there is no mention of anything failing to mount or really anything happening when I mount the partition...
| pietinger wrote: | | Have you seen this (in my wiki article) - could it be the same ? |
When I first saw the error after reading your article, it sounded familiar... but just unlocking the partition works, and according to keyctl, the key gets saved just fine...
Just tried it, still get the error. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Thu Apr 04, 2024 6:05 pm Post subject: |
|
|
Hmmm ... usually error 6 is "#define ENXIO 6 /* No such device or address */" ... strange ... do you have really the mount command with:
| Code: | | mount.bcachefs /dev/nvme0n1p2:/dev/nvme1n1p2 /mnt/root |
with a : between
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Thu Apr 04, 2024 6:31 pm Post subject: |
|
|
This confused me as well...
And yes, I previously had the : in between the two partitions
Now I reformatted for the sake of simplicity so I just have one partition without RAID 1 (because I thought that might cause the error), and of course adjusted my script and tried manually, but it still fails with that error...
ls /dev/ also confirms it is there... just seemingly not usable by bcachefs-tools ? |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
| Posted: Thu Apr 04, 2024 7:26 pm Post subject: |
|
|
Sorry, I cannot help you further because I have no idea what could be wrong ... maybe file a bug ? 
(I would be very glad if you would update this thread if you find out something new)
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Sun Apr 07, 2024 3:01 pm Post subject: |
|
|
So after properly reading this post I found a solution: you can just unlock the partition and then mount it, telling it to wait for the key, like so:
| Code: | bcachefs unlock /dev/nvme0n1p2
bcachefs mount -o ro -k wait /dev/nvme0n1p2 /mnt/root & |
And now the partition is successfully mounted! I have not yet tested whether now booting on it works, but I can see all files on there correctly. Will try to do an rsync over and give an update whether booting works. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5093
Location: Bavaria
|
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Sun Apr 07, 2024 7:50 pm Post subject: |
|
|
So unlocking works now, but booting is still not working, hanging up after unmounting the various things in /...
But that may also just be me failing at correctly modifying a script or me failing to crorrectly rsync...
I'll try some more things and update this when something seems to work. |
|
| Back to top |
|
|
thelordi
n00b
Joined: 16 Sep 2023
Posts: 17
|
| Posted: Thu Aug 08, 2024 12:18 pm Post subject: |
|
|
| Kernel 6.10 was released and I have now successfully moved /home to bcachefs. I will also attempt once again to get the root partition working and post some updates. If anyone wants to attempt the same, DO NOT USE --compression when creating the partition, as this will compress everything on one thread. Use --background-compression instead, as this very much speeds up the initial copying. You can later change this through /sys/fs/bcachefs/$UUID/options/ . |
|
| Back to top |
|
|
|
Kernel & Hardware
