| View previous topic :: View next topic |
| Author |
Message |
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
 Posted: Sun Jan 05, 2025 10:12 pm Post subject: Binhost inside docker, no chroot |
 |
|
I'm not sure what I want is possible, but it seems possible to me. I'd like to see what the other people think.
My high level goal is to have my laptop do no compilation, and have all packages sourced from a binhost. That seems possible. The part that I'm struggling with is compiling all the binary packages that would be served by that binhost, in a way that is isolate from the dockerized root system.
I'd like to install packages and compile against /binhost instead of /
This way the docker container's root ( / ) is not used when compiling and installing packages in /binhost.
I understand there is a bootstrapping piece where the docker containers root is used to compile the first toolsets like gcc and coreutils for /binhost, but after that I think there would be a way to say "ok, use the binhost tooling and libraries to make packages".
This is sortof like cross-compiling for the same architecture, if that makes sense to you all out there.
This is a piece of my dockerfile that I am using to construct the binhost. Note the use of --root, --config-root, and --sysroot.
| Code: |
# Emerge step1 using system build tools
# Prepare host system with dependencies needed to build toolchain on /binhost
# and build the starting toolchain on /binhost
RUN USE="build" emerge --quiet --root /binhost --config-root /binhost --root-deps @toolchain
# Set environment to use libraries and programs and packages from /binhost
ENV PATH="/binhost/bin:/binhost/usr/bin"
ENV LD_LIBRARY_PATH="/binhost/lib:/binhost/lib64"
ENV PKG_CONFIG_PATH="/binhost/lib/pkgconfig:/binhost/lib64/pkgconfig:/binhost/usr/lib/pkgconfig:/binhost/usr/lib64/pkgconfig:/binhost/usr/share/pkgconfig"
ENV CPPFLAGS="-I/binhost/usr/include"
ENV LDFLAGS="-L/binhost/usr/lib64"
# Emerge step2 using binhost build tools to build out binhost packages
# After this, binhost should have compiled itself, and not be connected to host system toolchain
# Nothing from host system should be used...
RUN emerge --quiet --buildpkg --root /binhost --config-root /binhost --sysroot /binhost --root-deps @world
# Emerge step3 Installs and builds the user-selected packages, using all binhost programs and libs
RUN emerge --quiet --buildpkgonly --root /binhost --config-root /binhost --sysroot /binhost @my_package_set
|
I'd rather avoid chroot, as I feel like that hides linking/environment problems rather than addressing them.
Is is possible to achieve an isolated /binhost like this? I'm happy to post follow up info, but don't want to overload the initial forum post...
I have seen the prefix project but it doesn't seem to do what I want, at least as I understand it.
Bonus question (where I am currently stuck):
How do I tell emerge to use /binhost instead of / as the gnuconfig path? I've set things like CONFIG_SITE=/binhost/path/to/gnnuconfig but it still seem to run the packages configure with /usr/share/gnuconfig as the config directory. Ideally this would be something I can set with command line flags to emerge or environment variables. I've also tried CONFIG_SITE, CONFIG_SUB, CONFIG_GUESS, and EXTRA_ECONF="--config-site=/path/to/custom/gnuconfig" without success. |
|
| Back to top |
|
 |
zen_desu
Tux's lil' helper
Joined: 25 Oct 2024
Posts: 107
|
| Posted: Sun Jan 05, 2025 11:25 pm Post subject: |
|
|
https://github.com/desultory/genTree/tree/main
I'm actively working on this, genTree-server runs a webserver that serves the pkgdir from whatever seed you choose.
It handles all the mounts for you and technically uses a chroot, but does so in a namespace. The pkgdir/etc are mounted into the container, and it uses your system /var/db/repos and resolv.conf by default.
--root is only used for "layers" and is not really used in the binhost mode.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1456
Location: Richmond Hill, Canada
|
| Posted: Sun Jan 05, 2025 11:34 pm Post subject: |
|
|
I am currently working on my own project that may do what you looking for. It is not complete yet and it is very specific to my environment so I can not share exact content with you.
However I will share the design principle.
I use two layer of docker container.
The first layer of docker container is based on alpine which task is to setup storage for the inner docker to run build.
The inner docker container use catalyst to build a rootfs, as side effect of the rootfs build it will also create all packages for the rootfs. so in the end you will have tarball (rootfs) and a "packages" directory that can act like bin-host share.
The inner docker is gentoo stage3 based image with catalyst installed.
This whole initial setup take some effort but once it is done it will do what you are seeking with relative easy to maintain. it also setup possibility with docker volumes-from option so you can have separated web-server container to share the "packages" directory.
Using catalyst benefit that you will have a consisted build environment so modification be come easy.
I hope this give you ideas on your design. Please don't hesitate asking question should you need help. |
|
| Back to top |
|
|
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
| Posted: Mon Jan 06, 2025 1:35 am Post subject: Correction to my thinking... |
|
|
It occurred to me that I don't want to _run_ the binhost toolchain to compile packages for the binhost, because the binhost execs (gcc, clang, etc) will be compiled by the system toolchain with the binhost optimizations. It would be much better for me to run all the compiles with the system toolchains, such that the compiled packages are optimized for my laptop and the laptop is the only place they actually run or are #included from.
It seems best in my case to treat use flags and make.conf the same as a different architecture. I should pretend that I'm not able to use programs or libs that are not for my build system, but that build system can still compile stuff for my target (binhost).
If I am correct in my thinking, optimizations and libraries of the system exec's wont make a difference to the compiled packages for my laptop, because I will be using alternate use flags and make.conf of the binhost (/binhost/etc/portage/) to compile packages for the laptop.
You ever have an idea and end up closer to where you started than after you thought about all the details?  I must have read too much from the LinuxFromScratch manual and got it stuck in my head that I needed to compile my packages with a "clean toolchain" for when the build env goes away.
Testing this out tonight. I should be able to have 0 compile on laptop, yet all packages are binary and optimized for my laptop, with my use personal flags. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2653
|
| Posted: Mon Jan 06, 2025 4:28 am Post subject: Re: Correction to my thinking... |
|
|
| techtruth wrote: | | It occurred to me that I don't want to _run_ the binhost toolchain to compile packages for the binhost, because the binhost execs (gcc, clang, etc) will be compiled by the system toolchain with the binhost optimizations. It would be much better for me to run all the compiles with the system toolchains, such that the compiled packages are optimized for my laptop and the laptop is the only place they actually run or are #included from. |
You cannot run the host toolchain inside the container. It defeats the purpose of containers.
Here's a Dockerfile for something I did recently, I hope it helps:
| Code: | FROM gentoo/stage3:latest
RUN emerge-webrsync
RUN echo "FEATURES=\"${FEATURES} getbinpkg\"" >> /etc/portage/make.conf
RUN getuto
RUN echo "dev-java/openjdk-bin headless-awt" >> /etc/portage/package.use/java
RUN emerge -q dev-java/openjdk-bin
|
You should be able to bring up a whole Gentoo installation this way. Adding buildpkg to FEATURES will make portage store binary packages for everything you build inside the container. Then you should be able to setup a binary packages host.
But are you sure what you're looking for is not discc?
And why don't you want to use the official binhost?
Best Regards,
Georgi |
|
| Back to top |
|
|
zen_desu
Tux's lil' helper
Joined: 25 Oct 2024
Posts: 107
|
| Posted: Mon Jan 06, 2025 4:47 am Post subject: |
|
|
It may be easier to start by making binpkgs with a more generic target, then you can try stuff like setting -mtune. This way you can get the process down, and then work on more complex settings.
_________________
µgRD dev
Wiki writer |
|
| Back to top |
|
|
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
| Posted: Mon Jan 06, 2025 4:51 am Post subject: Re: Correction to my thinking... |
|
|
| logrusx wrote: | | You cannot run the host toolchain inside the container. It defeats the purpose of containers. |
Sorry, got my jargon flopped around. I mean the binhost 'host' not the host system running the docker host. I suppose it would be better said as the docker container runtime (root) and the binhost inside that docker container runtime (binhost).
| logrusx wrote: | | But are you sure what you're looking for is not discc? |
I started there actually!  and that works if I want to issue compile commands in realtime. It doesn't let me update in the background though, like running a binhost in on a different server would.
| logrusx wrote: | | And why don't you want to use the official binhost? |
I want to use my special use flags and common_flags like -mtune and -march. I don't think I can do that with gentoo's default binhost as its not within my control. I'm sure there are a bunch of precompiled packages though. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2653
|
| Posted: Mon Jan 06, 2025 6:08 am Post subject: Re: Correction to my thinking... |
|
|
| techtruth wrote: | | I want to use my special use flags and common_flags like -mtune and -march. |
What's the CPU you want to optimize for?
Best Regards,
Georgi |
|
| Back to top |
|
|
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
| Posted: Mon Jan 06, 2025 6:57 am Post subject: Re: Correction to my thinking... |
|
|
| logrusx wrote: | | What's the CPU you want to optimize for? |
AMD Ryzen 7 7840HS w/ Radeon 780M Graphics
portage make.conf file for my laptop, for more context.
| Code: | GRUB_PLATFORMS="efi-64"
COMMON_FLAGS="-march=znver4 -mshstk --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=1024 -mtune=znver4 -O3 -pipe"
FEATURES="distcc"
DISTCC_HOSTS="192.168.0.5"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
# Local has 16
# DistCC pool has 20 (desktop)
MAKEOPTS="-j36"
USE="dist-kernel wayland sound-server gtk3 udev -pulseaudio pipewire bluetooth dbus -gtk -systemd -elogind -cups -X -xwayland -systemd -x11-backend -xinerama -xkb -qt6 -qt5 -branding -valgrind"
VIDEO_CARDS="amdgpu radeonsi"
ACCEPT_LICENSE="@BINARY-REDISTRIBUTABLE"
|
|
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2653
|
| Posted: Mon Jan 06, 2025 7:02 am Post subject: |
|
|
Do you really need that? Do you use vector graphics or matrix multiplication et.c.? Do you really need to compile on another computer?
Best Regards,
Georgi
p.s. -systemd should be set by your profile(openrc, right?), why is it in your make.conf USE variable? Also it's there twice. |
|
| Back to top |
|
|
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
| Posted: Mon Jan 06, 2025 7:59 am Post subject: |
|
|
| logrusx wrote: | | Do you really need that? Do you use vector graphics or matrix multiplication et.c.? Do you really need to compile on another computer? |
No, I don't think I need to compile on another computer, I just want to do it for my own experimenting and customization. I do have some future projects that could use this, but no pressing needs. I also don't think I mess with vector graphics and matrix multiplications on anything outside maybe SVG images.
| logrusx wrote: | | p.s. -systemd should be set by your profile(openrc, right?), why is it in your make.conf USE variable? Also it's there twice. |
Ahh, good catch on the dupe! My profile is the generic amd64/23.0
| Code: | | /etc/portage/make.profile: symbolic link to ../../var/db/repos/gentoo/profiles/default/linux/amd64/23.0 |
|
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2653
|
| Posted: Mon Jan 06, 2025 8:12 am Post subject: |
|
|
| techtruth wrote: | | I just want to do it for my own experimenting and customization. |
OK, disregard that line, I just needed to dig deeper as oftentimes users end up with a more complex solution than it needs to be.
On the note of running the host toolchain (the host being the binhost running in the container). Yes, this is how you should run it. You can also run your real host's toolchain but I don't know exactly how to do that. You could run portage giving it a different root and it'll build everything there, you can set it up to build binary packages along the way. The only caveat I can think of is you should modify your make.conf during that emerge session to match your target system settings, just like when using distcc.
Best Regards,
Georgi |
|
| Back to top |
|
|
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
| Posted: Mon Jan 06, 2025 11:12 am Post subject: |
|
|
| logrusx wrote: | | OK, disregard that line, I just needed to dig deeper as oftentimes users end up with a more complex solution than it needs to be |
No worries! Reduce when ridiculous is a great optimization path 
| logrusx wrote: | | The only caveat I can think of is you should modify your make.conf during that emerge session to match your target system settings, just like when using distcc. |
I was hoping to leave the host (inside docker) system's make.conf alone, and only make the /binhost/etc/portage/make.conf and have that be used. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2653
|
| Posted: Mon Jan 06, 2025 11:14 am Post subject: |
|
|
I think you can do that too if you just use the stage 3 for the toolchain and emerge in a root.
Best Regards,
Georgi |
|
| Back to top |
|
|
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
| Posted: Sat Jan 25, 2025 8:24 pm Post subject: Progress, but not completed. |
|
|
Whew, I've managed to get pretty far into this, but a few of the major packages I need are giving trouble. I'm using
| Code: | | emerge --buildpkg --root /binhost --config-root /binhost --root-deps package-name |
to install the packages in the binhost directory. The config-root has use flags specific to my target platform (my laptop).
sys-kernel/linux-firmware
gui-wm/wayfire
net-irc/quassel
sys-auth/fprintd
gui-apps/waybar
app-editors/scite
app-arch/file-roller
app-misc/gentoo
app-crypt/gpa
app-office/libreoffice
dev-qt/qtwayland
media-gfx/inkscape
media-gfx/pinta
sys-apps/qdirstat
sys-kernel/gentoo-kernel
www-client/chromium
media-video/mpv
Most of these need use flags set on the host system. Why would these packages need the use flags of the host system to be changed if they are compiling using the /binhost use flags? For instance, the kernel install looks like it required dracut on the host, where I would expect it only needs that in the /binhost. I feel like there is something fundamental about use flags and how they are involved in the compile and merge process... Any pointers? |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1456
Location: Richmond Hill, Canada
|
| Posted: Sat Jan 25, 2025 9:08 pm Post subject: |
|
|
Allow me to redefine some terms.
- buildenv: The environment you run the emerge --root <target dir> ....
- target dir: where you want emerge install to.
your buildenv's portage configuration have different USE flags from your buildenv's USE flags. Which possible lead to <target dir> require additional packages install. However those additional packages may require build time package support therefor you buildenv will need to install additional packages.
when you do emerge --root /binhost ... -pvt ... you can examine the output, some of them will have "to /binhost" and some does not. Those not have the "to /binhost" mean there will be install to you buildenv as some sort of helper to build packages into <target dir> |
|
| Back to top |
|
|
techtruth
n00b
Joined: 01 Jan 2025
Posts: 8
|
| Posted: Sun Jan 26, 2025 6:40 am Post subject: Clarification |
|
|
| pingtoo wrote: | | your buildenv's portage configuration have different USE flags from your buildenv's USE flags. |
Is that supposed to say "your buildenv's portage configuration (/etc/portage) have different USE flags from my target dir's (/binhost/etc/portage) USE flags" ?
| pingtoo wrote: | Which possible lead to <target dir> require additional packages install. However those additional packages may require build time package support therefor you buildenv will need to install additional packages.
when you do emerge --root /binhost ... -pvt ... you can examine the output, some of them will have "to /binhost" and some does not. Those not have the "to /binhost" mean there will be install to you buildenv as some sort of helper to build packages into <target dir> |
Yes, I originally wanted to install no additional packages on the buildenv system, but that was not practical or possible since I am using the buildenv's packages to build the target dir's packages, which have dependencies.
I'm unfortunately still unclear on why the host system would need a use flag (like dracut) to compile packages for the target dir. Since this is inside docker anyway, maybe I should just make the buildenv use flags match what I need for the target dir .... |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2653
|
| Posted: Sun Jan 26, 2025 8:51 am Post subject: Re: Clarification |
|
|
| techtruth wrote: | | maybe I should just make the buildenv use flags match what I need for the target dir .... |
You mean USE in make.conf? That's not a good place to store use flag configuration.
p.s. your can comment your original settings and add temporary ones during the target build. This is what I was doing for distcc.
Best Regards,
Georgi
Last edited by logrusx on Sun Jan 26, 2025 12:41 pm; edited 2 times in total |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1456
Location: Richmond Hill, Canada
|
| Posted: Sun Jan 26, 2025 12:17 pm Post subject: Re: Clarification |
|
|
| techtruth wrote: | | pingtoo wrote: | | your buildenv's portage configuration have different USE flags from your buildenv's USE flags. |
Is that supposed to say "your buildenv's portage configuration (/etc/portage) have different USE flags from my target dir's (/binhost/etc/portage) USE flags" ? |
Yes, you are correct. my mistake.
| Quote: | | pingtoo wrote: | Which possible lead to <target dir> require additional packages install. However those additional packages may require build time package support therefor you buildenv will need to install additional packages.
when you do emerge --root /binhost ... -pvt ... you can examine the output, some of them will have "to /binhost" and some does not. Those not have the "to /binhost" mean there will be install to you buildenv as some sort of helper to build packages into <target dir> |
Yes, I originally wanted to install no additional packages on the buildenv system, but that was not practical or possible since I am using the buildenv's packages to build the target dir's packages, which have dependencies.
I'm unfortunately still unclear on why the host system would need a use flag (like dracut) to compile packages for the target dir. Since this is inside docker anyway, maybe I should just make the buildenv use flags match what I need for the target dir .... |
Since you are using docker, I will share my experience,
I am doing some thing similar as you except I am using two level of docker (docker in docker) + chroot.
First level of docker is just act as platform (I am use alpinelinux for this). So it can be portable. run on any of my SBC.
Second level of docker is the buildenv. initially I use Gentoo stage3 tarball to create base image, from the stage3 base image I then construct (using target dir) a minimal stage1 as my new base image.
from the new-stage1 base image I then build a new stage3 tarball use catalyst. The objective for the new stage3 tarball is optimized to specific target CPU.
from the targeted stage3 tatball I build stage4 also use catalyst with specific set of package for each of my SBC.
currently I am still in the construction of my new-stage1 becuase I want to work out better scripting.
So it is possile to mix docker, chroot (unshare) to get where you want. (Just very complicated) |
|
| Back to top |
|
|
|
Portage & Programming
