| View previous topic :: View next topic |
| Author |
Message |
zen_desu
n00b
Joined: 25 Oct 2024
Posts: 71
|
 Posted: Thu Dec 26, 2024 4:50 am Post subject: genTree |
 |
|
https://github.com/desultory/genTree
I've been working on this for the last month or so, but more so in the last week.
It's sorta like catalyst, but runs entirely unprivileged in a user namespace. I may add it to GURU soon, and would appreciate feedback.
One nice thing about it is that it builds in layers which should be OCI compatible, and builds packages by default so even if layers can't be reused, packages can.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
 |
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1341
Location: Richmond Hill, Canada
|
| Posted: Thu Dec 26, 2024 5:09 am Post subject: |
|
|
Can you share what is difference between catalyst vs gentree?
How is benefit using gentree?
A quick look it is not relate to container, so what is that layer you talk about? |
|
| Back to top |
|
|
zen_desu
n00b
Joined: 25 Oct 2024
Posts: 71
|
| Posted: Thu Dec 26, 2024 5:26 am Post subject: |
|
|
| pingtoo wrote: | Can you share what is difference between catalyst vs gentree?
How is benefit using gentree?
A quick look it is not relate to container, so what is that layer you talk about? |
It runs the entire process in a user namespace (more or less a container): | Code: | | nsexec(genTree.build_tree) |
where nsexec is a python function i wrote which executes a function in a namespace. It later does some mounts and chroots:
https://github.com/desultory/zenlib/pull/9/files
| Code: | def init_namespace(self):
"""Initializes the namespace for the current config"""
self.logger.info("[%s] Initializing namespace", colorize(self.config.name, "blue"))
self.mount_seed_overlay()
self.mount_system_dirs()
self.bind_mount(self.config.system_repos, self.config.sysroot / "var/db/repos")
self.bind_mount("/etc/resolv.conf", self.config.sysroot / "etc/resolv.conf", file=True)
self.bind_mount(self.config.pkgdir, self.config.sysroot / "var/cache/binpkgs", readonly=False)
self.bind_mount(self.config.build_dir, self.config.build_mount, recursive=True, readonly=False)
self.bind_mount(self.config.config_dir, self.config.config_mount, recursive=True, readonly=False)
self.logger.info("Chrooting into: %s", colorize(self.config.sysroot, "red"))
chroot(self.config.sysroot) |
Concerning containers, it makes image layers as OCI compatible layers, where deleted files are marked as ".wh.<filename>" and if that file is detected, that file is deleted from lower layers when deploying that layer.
https://github.com/opencontainers/image-spec/blob/main/layer.md#whiteouts
https://github.com/desultory/genTree/blob/main/src/genTree/oci_mixins.py
https://github.com/desultory/genTree/blob/main/src/genTree/gen_tree_tar_filter.py
The oci filter is used for importing layers, while the generic tar filter is used for packing.
One advantage is that it's much easier to get started, you can import a stage3 as a "seed" then use that with simple configs to build minimal filesystem images which can be used for containers.
It does not require root to run.
Some output may make it more clear:
| Code: | desu@amazon /mnt/closet/genTree $ genTree nginx.toml
INFO | [nginx] Initializing namespace
INFO | Mounting overlayfs on: /mnt/closet/genTree/seeds/stage3-openrc_sysroot
INFO | [nginx] Mounting system directories in: /mnt/closet/genTree/seeds/stage3-openrc_sysroot
INFO | Mounting /proc over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/proc
INFO | Mounting /sys over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/sys
INFO | Mounting /dev over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/dev
INFO | Mounting /run over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/run
INFO | Mounting /var/db/repos over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/var/db/repos
INFO | Mounting /etc/resolv.conf over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/etc/resolv.conf
INFO | Mounting /mnt/closet/genTree/pkgdir over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/var/cache/binpkgs
INFO | Mounting /mnt/closet/genTree/builds over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/builds
INFO | Mounting /mnt/closet/genTree/config over: /mnt/closet/genTree/seeds/stage3-openrc_sysroot/config
INFO | Chrooting into: /mnt/closet/genTree/seeds/stage3-openrc_sysroot
INFO | Building tree for: nginx
INFO | [nginx.toml] Building base: tini
INFO | [tini.toml] Building base: glibc
INFO | [glibc.toml] Building base: base
WARNING | [base] Skipping build, layer archive exists: /builds/base.tar
WARNING | [glibc] Skipping build, layer archive exists: /builds/glibc.tar
WARNING | [tini] Skipping build, layer archive exists: /builds/tini.tar
INFO | [nginx.toml] Building base: gcc
INFO | [gcc.toml] Building base: base
WARNING | [base] Skipping build, layer archive exists: /builds/base.tar
INFO | [base] Unpacking base layer to build root: /builds/gcc_lower
INFO | [gcc] Mounting build overlayfs on: /builds/gcc
INFO | [gentoo] Setting portage profile: default/linux/amd64/23.0
INFO | [gcc] emerge --root /builds/gcc --jobs 8 --verbose=y --nodeps --usepkg=y --with-bdeps=n sys-devel/gcc
INFO | [gcc] Packing tree: /builds/gcc_upper
INFO | [gcc] Created archive: /builds/gcc.tar (313.49 MB)
WARNING | [nginx] Cleaning root: /builds/nginx
WARNING | [nginx] Cleaning root: /builds/nginx_lower
WARNING | [nginx] Cleaning root: /builds/nginx_work
WARNING | [nginx] Cleaning root: /builds/nginx_upper
INFO | [base] Unpacking base layer to build root: /builds/nginx_lower
INFO | [glibc] Unpacking base layer to build root: /builds/nginx_lower
INFO | [tini] Unpacking base layer to build root: /builds/nginx_lower
INFO | [gcc] Unpacking base layer to build root: /builds/nginx_lower
INFO | [nginx] Mounting build overlayfs on: /builds/nginx
INFO | [nginx] Mounting config overlay: /config/nginx
INFO | [gentoo] Setting portage profile: default/linux/amd64/23.0
INFO | [nginx] emerge --root /builds/nginx --jobs 8 --verbose=y --usepkg=y --with-bdeps=n www-servers/nginx
INFO | [nginx] Unmerging packages: sys-devel/gcc
INFO | [nginx] emerge --root /builds/nginx --unmerge sys-devel/gcc
INFO | [nginx] Packing tree: /builds/nginx
INFO | [nginx] Created archive: /builds/nginx.tar (178.42 MB) |
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
