| View previous topic :: View next topic |
| Author |
Message |
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
 Posted: Thu Jan 02, 2025 12:05 am Post subject: ugrd |
 |
|
This thread is for development updates, questions, and information about ugrd: https://github.com/desultory/ugrd
Currently, I'm considering adding support to build in an alternate root, and this may be easier to implement if ugrd doesn't copy files to a build directory before packing a CPIO image.
The original design was built around wiki instructions for building a custom initramfs, so it copied files to a build dir for packing with gen_init_cpio, but this meant device nodes had to be written to disk. To reduce required privileges and improve the build process/checks, I made pycpio and implemented it instead: https://github.com/desultory/ugrd/commit/ee842114c98faad979231f9bbd36275763f80ac6
I kept the "feature" to make a build dir the user can explore, modify, if changes are needed. It's not a core feature, but introduces some additional design challenges and security concerns. I figure it can have similar functionality with an option to extract the CPIO to a specified work dir, possibly with the option to actually deploy the device nodes.
Feel free to share any thoughts or suggestions.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
 |
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Sun Jan 19, 2025 9:07 pm Post subject: Posix |
|
|
I've refactored a lot of the generated shell script, it should be POSIX compatible for anyone interested in trying non-bash shells. The changes are in the latest 9999 and will be part of release 1.31
The next thing I'd like to add is the ability to make init hooks run before/after other hooks. Currently the ordering is mostly hard-coded, but I think it would be nice to make this more dynamic/configurable. I'm not sure if this should be a separate config attr of if stuff like "before" and "After" could be a key in the import hooks
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Mon Mar 31, 2025 9:48 pm Post subject: |
|
|
pingtoo this may be a better place for discussion about using atd with ugrd.
That sounds like a very interesting idea, and I wonder how that would look if used throughout the system. I have no issue with trying to add something like this as a sort of optional interface, to do "event driven" things in a more generic way.
at this point ugrd doesn't have any sort of job control, so im not sure the most sane way to manage running daemonized processes within the initramfs. I guess that's not a big concern if you just expect it's all destroyed after switch_root.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Mon Mar 31, 2025 10:19 pm Post subject: |
|
|
"atd" is very old concept in Unix world. I think go back to SysV3 if not earlier. it was part of cron.
| Quote: | | That sounds like a very interesting idea, and I wonder how that would look if used throughout the system |
I am still working on the idea so don't have concrete example.
time event in init workflow could be happen in following point- as timeout after start of searching root device.
- start a "beacon" so some how it can be detected a node boot up. (some idea I am floating is bluetooth, so may be my phone would know boot happen)
- in parallel to search root device, start network configuration, primary target on dhcp. (not sure about static setup)
- I want to be able load a simple rootfs over network and ran it in memory for recovery or simple system (distcc helper)
- continue above thoughts, I want to use a container based (like CoreOS) environment, so I want a very static Gentoo rootfs and everything else live in container(s)
- so I want to have a way to easily time out and I want to be sure it is not kernel error (therefor node paniced)
Sorry this is not very organized. Just some quick brain dump. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Mon Mar 31, 2025 10:56 pm Post subject: |
|
|
| pingtoo wrote: | "atd" is very old concept in Unix world. I think go back to SysV3 if not earlier. it was part of cron.
| Quote: | | That sounds like a very interesting idea, and I wonder how that would look if used throughout the system |
I am still working on the idea so don't have concrete example.
time event in init workflow could be happen in following point- as timeout after start of searching root device.
- start a "beacon" so some how it can be detected a node boot up. (some idea I am floating is bluetooth, so may be my phone would know boot happen)
- in parallel to search root device, start network configuration, primary target on dhcp. (not sure about static setup)
- I want to be able load a simple rootfs over network and ran it in memory for recovery or simple system (distcc helper)
- continue above thoughts, I want to use a container based (like CoreOS) environment, so I want a very static Gentoo rootfs and everything else live in container(s)
- so I want to have a way to easily time out and I want to be sure it is not kernel error (therefor node paniced)
Sorry this is not very organized. Just some quick brain dump. |
I have been considering ways to make ugrd even faster, I'm not sure if this is the answer, but it's an interesting idea. Currently everything it does is very linear, and that helps make it easy to debug, but there are some cases where it has to just poll or timeout which I think are usually not the best ways to handle something which could be event driven. The "obvious" answer is udev, but that does far more than what I need, and can't really be maintained as an "optional" component to ugrd.
Considering your boot ideas, I actually recently got a kinda interesting card, it's a PCIe based router and I was considering trying to use it to PXE boot my desktop. I've only done PXE booting a few times, but I think that is another area ugrd could be improved, it could possibly use DHCP client option 17 (root-path), not entirely sure. (although I feel it should be possible for the linux kernel itself to handle a lot of this)
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 12:26 am Post subject: |
|
|
PXE/DHCP/BOOTP they are similar and reference to each other.
Just be aware, DHCP options mostly act like hint for the recipient, so if I do dhcp inside the "init" process I will not want to depend on dhcp client implementation on how to call out, but let it post all the hints somewhere and use a separated program to process those hints. because otherwise your flow are tie to one specific implementation and risk lose control when upstream decide to change something.
I don't expect "fast", I want feature rich. so extra 30 seconds or so for booting process is not my design concern,
I try to follow your code, it took me quick some time to figure out how you implement the "init" script because I was expecting there is a script or template somewhere in your code tree. This is where we have fundamental design differences. you are creating a generator that at execution time user will choose one specific target/path therefor a script generated for that path/target. I want to create one solution for all that will work on all scenarios that I can come up so I don't need to rewrite this program for each installation. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 2:15 am Post subject: |
|
|
| pingtoo wrote: |
I try to follow your code, it took me quick some time to figure out how you implement the "init" script because I was expecting there is a script or template somewhere in your code tree. This is where we have fundamental design differences. you are creating a generator that at execution time user will choose one specific target/path therefor a script generated for that path/target. I want to create one solution for all that will work on all scenarios that I can come up so I don't need to rewrite this program for each installation. |
Have you seen the new hello world module I have a pr for? I think that should make some things a bit more clear.
If you want to make a simple module which always does a certain thing using ugrd, that's pretty straightforward. A lot of the complexity comes from attempts to detect configuration and validate it. Certain details, such as the rootfs target uuid, mapped device names, etc, are determined at build time and _may_ be hardcoded into the image, or certain details could be determined at run time using scripts, or configured via kernel command line parameters.
With the goal of a simple ux, most of the current builtin modules do a lot of work to determine system info for later use, that way the actual shell script is kept as simple as possible. For certain things, it probably makes more sense to have that complexity in the "generated" shell script, where all effort could be put instead of worrying about much build time things.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 2:10 pm Post subject: |
|
|
Thanks for information.
I look the pr, I can understand the simple python logic, but I am not able to understand how it relate to actual init flow. But no worries I guess I just need to dig a bit more deeper in to your code base to see how it work.
We have different design goal but I appreciate your effort and sharing. it gave me some more to think about how to work on my project, especially those crypto modules because I never have to work on those concept before. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 3:04 pm Post subject: |
|
|
| pingtoo wrote: | Thanks for information.
I look the pr, I can understand the simple python logic, but I am not able to understand how it relate to actual init flow. But no worries I guess I just need to dig a bit more deeper in to your code base to see how it work.
We have different design goal but I appreciate your effort and sharing. it gave me some more to think about how to work on my project, especially those crypto modules because I never have to work on those concept before. |
More or less, the way it works, is the "toml" definition portion of modules has the "imports" sections. Key names under here are used to say what part of the init process functions are imported to. If specific ordering is needed within an init level, init_order can be used.
| Code: | [imports.init_main]
"hello_world" = [ "hello_world" ]
|
will import the "hello_world" function from the "hello_world" module, and put it in the "init_main" section of the generated init file.
If I made a new module called foo, with this code in foo.py:
| Code: | def foo(self) -> str:
return "echo 'foo'"
def bar(self) -> str:
return """
echo bar
echo baz
"""
|
I could then have the toml do this, so it prints "bar\nbaz\nfoo" during init_main, and then "bar baz" during init_final.
| Code: | [imports.init_main]
"foo" = [ "bar", "foo" ]
[imports.init_final]
"foo" = ["bar"]
|
One additional noteworthy thing is that it turns "bar" into a function because it's multiple lines:
| Code: | # Begin init_main
mount_fstab
bar
echo 'foo'
|
each init "level" will only accept a single instance of a particular line, so if something must be done multiple times, it should handle that within the shell function. One small change I've been considering related to this is to wrap each level into a function, but I'm not sure if that will actually hurt readability, it would force users to look at /etc/profile where all of the actual functions are, which could make things more clear.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 3:29 pm Post subject: |
|
|
"readability"? do you refer to generated "init" script or somewhere else?
the init "level" my interpretation, does it mean a state (stage) of the program flow? If yes, do you have a place (document) how many "level" there are and is there order or control to manage there order? |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 3:34 pm Post subject: |
|
|
| pingtoo wrote: | "readability"? do you refer to generated "init" script or somewhere else?
the init "level" my interpretation, does it mean a state (stage) of the program flow? If yes, do you have a place (document) how many "level" there are and is there order or control to manage there order? |
I mostly mean for the generated init script, I want most of the complexity to be in the python portion if possible (opposed to the POSIX shell portion), nothing is stopping users from making shell specific modules, but I try to keep the built in modules POSIX compatible if possible. I'm much more comfortable with python than any type of shell, so keeping the shell simple also helps me.
init levels are documented here: https://github.com/desultory/ugrd/blob/main/docs/dev_manual.md#init-imports
please let me know if that is confusing, at this point the dev manual is mostly personal notes, if you want the definitive place the init hooks are defined: https://github.com/desultory/ugrd/blob/2.0.0_pre/src/ugrd/initramfs_generator.py#L287
Users can technically just add new init levels, being careful they are inserted correctly in the list. A lot of the design is _very_ open ended. I've tried to set some style standards within the project but they are by no means hard lines.
there is a bit of added complexity because in cases where something like agetty is used, the init is split into 2 files, so this process ensures the primary init file gets the most basic init done, then it jumps to the "main" init which does something else, such as use agetty, or run SSH, could really be anything. That process should eventually die and execution returns to the original init which should finish things.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 3:56 pm Post subject: |
|
|
Thanks for information.
I found it after my post. It is in the "Dev manual". Because it is dev manual so I did not read it first.
| Quote: | | there is a bit of added complexity because in cases where something like agetty is used, the init is split into 2 files, so this process ensures the primary init file gets the most basic init done, then it jumps to the "main" init which does something else, such as use agetty, or run SSH, could really be anything. That process should eventually die and execution returns to the original init which should finish things. |
In order words, it is like programming language, it is subroutine  Thank you I got it.
Overall I think it is fine. I am just not prepare for this design idea. so I was looking for traditional design where there is basic shell/template then stuff were included in external call or subroutine manner.
Do you envision where someone would debug the final result at runtime? if not I think the "readability" consideration could be reduced. instead create a runtime flow printout like dracut create a /run/initramfs/<something>.log. Another idea, could you generate some sort of "flow map" upon create initrd image so at runtime one can review in rescue shell to expect what will happen?
also you mention something about "/etc/profile", I am not sure how it relate to initrd stage, do you create a /etc/profile in the initrd image? |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 4:14 pm Post subject: |
|
|
| pingtoo wrote: | Thanks for information.
I found it after my post. It is in the "Dev manual". Because it is dev manual so I did not read it first.
| Quote: | | there is a bit of added complexity because in cases where something like agetty is used, the init is split into 2 files, so this process ensures the primary init file gets the most basic init done, then it jumps to the "main" init which does something else, such as use agetty, or run SSH, could really be anything. That process should eventually die and execution returns to the original init which should finish things. |
In order words, it is like programming language, it is subroutine Thank you I got it.
Overall I think it is fine. I am just not prepare for this design idea. so I was looking for traditional design where there is basic shell/template then stuff were included in external call or subroutine manner.
Do you envision where someone would debug the final result at runtime? if not I think the "readability" consideration could be reduced. instead create a runtime flow printout like dracut create a /run/initramfs/<something>.log. Another idea, could you generate some sort of "flow map" upon create initrd image so at runtime one can review in rescue shell to expect what will happen?
also you mention something about "/etc/profile", I am not sure how it relate to initrd stage, do you create a /etc/profile in the initrd image? |
If you run ugrd with the "--print-init" arg it lists init functions:
| Code: |
init_pre:
mount_base
make_run_dirs
export_exports
parse_cmdline
set_loglevel
print_banner
load_modules
init_main:
mount_fstab
start_agent
import_keys
crypt_init
init_mount:
btrfs_scan
set_root_subvol
select_subvol
mount_root
mount_late
init_final:
umount_fstab
do_switch_root
|
Making it make a fancier diagram would be interesting, especially if the flow process becomes less linear.
For debugging, I think most users should be able to debug the shell script itself, if there are runtime issues. The process is designed to fail easily during the build process so most of the time the issue is immediately obvious. A lot of this is done through "validation" and "checks", where validation steps generally happen right when things are being detected/set, and checks are run at the very end.
Additionally, the "ugrd.base.debug" module, if added, will open a shell shortly after the most basic init has been done, so a user can use that as a breakpoint to debug it at run time. That module also makes it include extra tools such as the users selected editor.
ugrd puts a lot of info, such as the PATH, LD_PRELOAD, for libraries, and all of the generated functions in the generated /etc/profile (for the initramfs). This keeps all of the complexity/finer details tucked away so the created init file(s) are very short and easy to understand:
| Code: |
> cat /tmp/initramfs_build/init
#!/bin/sh -l
# Begin init_pre
mount_base
edebug Creating run dir: "$(mkdir -pv /run/ugrd)"
export_exports
parse_cmdline
readvar loglevel > /proc/sys/kernel/printk
print_banner
load_modules
# !!custom_init
agetty --autologin root --login-program init_main.sh tty0 tty || rd_restart
# Begin init_final
umount_fstab
do_switch_root
# END INIT
|
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 5:01 pm Post subject: |
|
|
Thanks. So you already thought about my ideas that is great.
The only thing I think (I may be wrong about this) is that my ideas are target at runtime for who may not know about ugRD as in just a third party who inherit the image and at run time need information to troubleshoot. so it would be nice for example the print out of "--print-init" is save somewhere for the user to reference. Just as well the generated init script does not have reference to /etc/profile so it is not intuitive for some one new to ugRD to know where to look.
On the other hand, the generated script are great for me, very easy to understand. (however I been doing this "init" script design on and off for more than a year now)
It would be nice to include you posted script in to document for other to quick pick up of the runtime flow. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 5:19 pm Post subject: |
|
|
| pingtoo wrote: | Thanks. So you already thought about my ideas that is great.
The only thing I think (I may be wrong about this) is that my ideas are target at runtime for who may not know about ugRD as in just a third party who inherit the image and at run time need information to troubleshoot. so it would be nice for example the print out of "--print-init" is save somewhere for the user to reference. Just as well the generated init script does not have reference to /etc/profile so it is not intuitive for some one new to ugRD to know where to look.
On the other hand, the generated script are great for me, very easy to understand. (however I been doing this "init" script design on and off for more than a year now)
It would be nice to include you posted script in to document for other to quick pick up of the runtime flow. |
I did a very basic implementation of your idea, and looking at it now, I just noticed it includes functions which were excluded, as in ones which were skipped for some reason, so it really shows the _imported_ functions at each run level regardless of whether or not they are used.
I'm tempted to color code it, so these are yellow or red or something, so it's clear they were excluded.
The generated init script sorta implies it uses /etc/profile based on the shebang having "-l" in it. It's also a better reference as print-init shows the function names, while the init script will simply use single line "functions" instead of creating a whole function for a single line.
Ideally, a ugrd generated image can kinda be a standalone thing, something which users could possibly manually update or extend by updating the kmods and repacking, that sort of thing. The main original design goal was to have ugrd create an image that is close to how one could be made by following the guides in the wiki, with added automation to make things easier, and checks to make it less error prone.
I think the fact that you're able to understand the final product means the design is aligning with the goals. By design, anyone who knows what they are doing should be able to interpret the init script without a whole lot of effort, especially without needing much knowledge of the build system itself. I'd say the main area for potential confusion is the var system ugrd uses at runtime.
I'll look into improving the dev manual with more examples later.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 5:56 pm Post subject: |
|
|
You are right, the script is easy enough for someone who doing initrd debug should be able to know.
The "sh -l" concept is really smart. only until you point out I realize the /etc/profile is implicit read. although my personally believe is be explicit. So I usually write code either in comment or just code the line even it will happen automatically. because it just make it know without guessing.
It is great to have this discussion. You have wake my coding personality. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 6:02 pm Post subject: |
|
|
| pingtoo wrote: | You are right, the script is easy enough for someone who doing initrd debug should be able to know.
The "sh -l" concept is really smart. only until you point out I realize the /etc/profile is implicit read. although my personally believe is be explicit. So I usually write code either in comment or just code the line even it will happen automatically. because it just make it know without guessing.
It is great to have this discussion. You have wake my coding personality. |
Back when I was first writing it, the functions, PATH, etc were just dumped into the main init file. This sorta worked but started to get really messy once you consider things like re-execing the shell, using multiple portions, etc. I think "sh -l" should be very compatible and helps reduce duplication. It's definitely worth explaining, could probably be a note in the dev manual and comment in the module so it's more clear. The shebang can be manually set using config (like most things in ugrd) if you want to experiment with other methods.
Once slight quirk related to this is that the log functions like "einfo" are used throughout the system and things can get messy/break if they aren't sourced properly. I think in the past, before the "banner" was a discrete function, it was injected into the init script and this quirk was more of an issue. I think making the shebang technically a multiline thing could possibly make sourcing reliable and more clear?
I'm glad you find this topic interesting  I think the concept of this design, basically "front-loading" the flow and info into the image before hand, is an interesting way to make something which can do more to verify behavior and allow for user modifications. While this design presents different challenges, I think the core issue is that it's nearly impossible, or just impossible to try to actually verify the function of something like a dracut image (without running it). There is too much that can happen, and while this dynamic approach is excellent for making something that is more "universal", as in one image _can_ boot a variety of systems, i think the interface (kernel command line parameters) and failure method make it painful for most users to navigate and make full use of.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 6:29 pm Post subject: |
|
|
| Quote: | | making the shebang technically a multiline thing could possibly make sourcing reliable and more clear? |
I don't know what is "multiline" shebang. never encounter one before.
for test drive, I have an idea, use unshare(1) with -p option which will let you run the generated init with pid 1, if you setup the test environment correctly you can even let it run to finish with swich_root. If you want I can try to develop the exact unshare command for you to incorporate into ugRD. I would not able to create as some sort of module because I am not python programmer. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 6:42 pm Post subject: |
|
|
| pingtoo wrote: | | Quote: | | making the shebang technically a multiline thing could possibly make sourcing reliable and more clear? |
I don't know what is "multiline" shebang. never encounter one before.
for test drive, I have an idea, use unshare(1) with -p option which will let you run the generated init with pid 1, if you setup the test environment correctly you can even let it run to finish with swich_root. If you want I can try to develop the exact unshare command for you to incorporate into ugRD. I would not able to create as some sort of module because I am not python programmer. |
I mean right now the shebang exists as a string which gets added to the top of the file, it could technically add multiple lines instead of just one, and later lines could source the profile or similar.
Right now, ugrd has a "--test" arg, which creates a test rootfs (just a basic filesystem with a shell so it can test completion by running an init in there). It uses qemu/kvm to run the tests, and using --test may just work for some systems, but the test code does not have a way to provide input so certain things (like key entry) will fail. Automated tests use this framework so test cases exist as config.toml's. I've strongly considered adding some namespace stuff to ugrd, especially for testing, but to do that natively in python I think you need 3.12+ and these types of test cannot run in portage.
https://github.com/desultory/ugrd/blob/2.0.0_pre/tests/fullauto.toml
this is the most basic test config, it mostly just forces DM detection off, so it'll fail early missing that info. Another hurdle for testing device mapper stuff is whether or not to provision the storage using the user's headers. This makes for a good test but it goes a bit beyond just reading the headers (which is not required but helpful for validation). I'm also not sure if the kernel will get upset at 2 devices beng active with the same uuid, it's not like the user can unmount their root for tests.
Test code: https://github.com/desultory/ugrd/blob/2.0.0_pre/src/ugrd/base/test.py
the test portion actually spawns another ugrd instance, for making the rootfs: https://github.com/desultory/ugrd/blob/2.0.0_pre/src/ugrd/base/test.py#L74 This instance is very stripped down and doesn't include the "base" which handles most of the standard operation, it only includes the "core" which is basically the bare minimum to make something resembling a linux filesytem.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 7:32 pm Post subject: |
|
|
from the fullauto.toml I cannot get the picture on how it work with VM.
--edit--
Sorry, I am being impatient, did not read the test.py
--edit--
but here I come up on using unshare(1) example. I did not test is so please take it as example. DON'T RUN IT.
| step 1: | cd /tmp
INITRD_PATH="/path/to/your/initrd.img" # Replace with the actual path
TEST_ROOT="/tmp/test_rootfs" # Directory to simulate the new root
# 1. Prepare the test environment
rm -rf initrd_test $TEST_ROOT
mkdir -p initrd_test $TEST_ROOT/sbin
# 2. Extract the initrd
cd initrd_test
zcat "$INITRD_PATH" | cpio -idmv |
| step 2: | # Create a fake /sbin/init for test
touch $TEST_ROOT/sbin/init
chmod 755 $TEST_ROOT/sbin/init
cat<<'EOF'>$TEST_ROOT/sbin/init
echo "--- 'init' on Real Rootfs (This is a test) ---"
exit 0
EOF
|
| Test run initrd's init: | | unshare --mount --pid --fork $TEST_ROOT/init |
unshare(1) create a container alike environment so there is no post process require to cleanup after unshare(1) command finish. Except you may want to wipe out $TEST_ROOT.
I understand I am being intrusive, Please forgive me  I am just feel like kid want to play new toy. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 8:20 pm Post subject: |
|
|
| pingtoo wrote: | from the fullauto.toml I cannot get the picture on how it work with VM.
--edit--
Sorry, I am being impatient, did not read the test.py
--edit--
but here I come up on using unshare(1) example. I did not test is so please take it as example. DON'T RUN IT.
| step 1: | cd /tmp
INITRD_PATH="/path/to/your/initrd.img" # Replace with the actual path
TEST_ROOT="/tmp/test_rootfs" # Directory to simulate the new root
# 1. Prepare the test environment
rm -rf initrd_test $TEST_ROOT
mkdir -p initrd_test $TEST_ROOT/sbin
# 2. Extract the initrd
cd initrd_test
zcat "$INITRD_PATH" | cpio -idmv |
| step 2: | # Create a fake /sbin/init for test
touch $TEST_ROOT/sbin/init
chmod 755 $TEST_ROOT/sbin/init
cat<<'EOF'>$TEST_ROOT/sbin/init
echo "--- 'init' on Real Rootfs (This is a test) ---"
exit 0
EOF
|
| Test run initrd's init: | | unshare --mount --pid --fork $TEST_ROOT/init |
unshare(1) create a container alike environment so there is no post process require to cleanup after unshare(1) command finish. Except you may want to wipe out $TEST_ROOT.
I understand I am being intrusive, Please forgive me I am just feel like kid want to play new toy. |
You don't need to apologize about anything, I'm happy to discuss this.
The test process you described is very similar to the current test process. If I were to do something like this, I would lean towards using the builtin namespace methods in python: https://docs.python.org/3/library/os.html#os.unshare Added in 3.12, so very new. Currently, the entire test process is done in python, so it would require a bit of deviation from the current process to use a shell script.
The tests are actually wrapped into unit tests so running "python -m unittest discover tests -v" in the project root will run all tests. The ebuild hooks into this and the github action more or less runs that command.
ugrd uses a fair bit from a common library I maintain called zenlib, this portion in particular could possibly be used for namespace stuff for tests, but depends on python 3.12 so is a bit of a dealbreaker at the time. https://github.com/desultory/zenlib/tree/main/src/zenlib/namespace
If you check this: https://github.com/desultory/ugrd/blob/2.0.0_pre/src/ugrd/fs/test_image.py#L82 you can see how it currently makes the test rootfs. It makes it in a second build dir, similar to how it makes the initramfs image it's testing, and the init just spits out a uuid that the script later checks for after the vm dies.
The way test stuff works is a bit strange, I mean you make ugrd make a test image by using the test module on the config you want to test. Once it finishes making the initramfs, it passes that info to a second generator which uses it to make a suitable test image, then runs the vm using the generated initramfs and filesystem image.
One thing I'm not sure about is how to check kmod loading in a namespace. I feel containers could be great for the build portion, but I'm not sure how to make them properly represent the boot process. In my ideal test, a vm with a kernel unsuitable for booting alone is used, that way it checks that the initramfs is actually working.
Additionally, using namespaces seems to break github actions for tests, and doesn't work in general for portage. I think for portage, trying to use namespaces in tests doesn't work with sanboxing, maybe other things. I believe there were some changes made to even allow /dev/kvm to properly be used within tests, because that's a bit privileged but necessary for proper tests. This is partially because ugrd tries to use the current kernel for tests, so if it was built for your specific CPU and that doesn't work with plain qemu, things break. A potential alternative is to have a separate test route that uses a pre-defined kernel, but I think the tests defaulting to the system kernel and the initramfs built for the system makes the most sense
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 9:03 pm Post subject: |
|
|
Thanks.
I think we are going different direction here again You are thinking incorprate ugRD in part of automation of portage install. Whereas I am thinking someone want to develop their own module(s) therefor they want to test run the process to verify their own project.
In container or namespace, kernel were did not change, so kmod should just work (kind of no-op because they should be already loaded)
Although I think for a initrd project kernel should not be included in testing. May be you have some insight that why you think you should also test kernel together with initrd?
My point being I want my initrd not tie to any kernel. I understand may be the kmod module is to address that need for kernel but isn't that your project expect the process run on the target machine anyway. so module should already be there.
I see your namespace project. So you are well versed for using "namespace". My shell script above just a illustration of sequence. I think with little or no modification you can incorporate that directly in to ugRD.
Also Catalyst in here use namespace too, just for your reference.
I don't know anything about Github Action, so no comment. |
|
| Back to top |
|
|
zen_desu
Apprentice
Joined: 25 Oct 2024
Posts: 220
|
| Posted: Tue Apr 01, 2025 9:23 pm Post subject: |
|
|
| pingtoo wrote: | Thanks.
I think we are going different direction here again You are thinking incorprate ugRD in part of automation of portage install. Whereas I am thinking someone want to develop their own module(s) therefor they want to test run the process to verify their own project.
In container or namespace, kernel were did not change, so kmod should just work (kind of no-op because they should be already loaded)
Although I think for a initrd project kernel should not be included in testing. May be you have some insight that why you think you should also test kernel together with initrd?
My point being I want my initrd not tie to any kernel. I understand may be the kmod module is to address that need for kernel but isn't that your project expect the process run on the target machine anyway. so module should already be there.
I see your namespace project. So you are well versed for using "namespace". My shell script above just a illustration of sequence. I think with little or no modification you can incorporate that directly in to ugRD.
Also Catalyst in here use namespace too, just for your reference.
I don't know anything about Github Action, so no comment. |
The lofty goal for the test module is that it should somewhat transparently and automatically handle tests for any other module. Basically, in theory, you could just make your module and attempt to run "ugrd -m mymodule --test" and it would run tests on it using your kernel and the exact same initramfs image it would be installing. In reality, for more complex setups, the user will likely have to write some of their own tests, but if the were able to integrate them into the existing unit tests, that would make everything very unified and cohesive.
I feel namespace tests could be a very good way to test portions of the shell script in particular. There is a bit of importance in verifying kmods load, and complexity, this is why ugrd doesn't have zfs tests right now.
I agree the initrd shouldn't really be tied to a particular kernel by its nature, but in most test environments, some kmods are required, especially for vms. I don't think there is test coverage for initrd's which have no kmods (basically the no_kmod config option), and testing that in a namespace could make sense.
If you're interested in exploring the idea of running these types of tests in a namespace I could possibly help you make a test module or help debug issues that arise, but for in-project tests, I think there are too many issues making it hard to effectively use containers like this. In some cases, it's easier to just use docker because test environments may expose a docker socket you can use.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1587
Location: Richmond Hill, Canada
|
| Posted: Tue Apr 01, 2025 9:51 pm Post subject: |
|
|
Thank you very much for your offer for help developing test modules.
But I think my design model are too much off from yours. I want to use FSM for code logic and I want my initrd to work for any kernel (so I want a way to dynamically load kmod during initrd run).
Anyway you code gave me much of ideas that I could follow. so much appreciated.
BTW, another difference is my coding choice is tcl/expect instead python. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
