View previous topic :: View next topic |
Author |
Message |
Pipeng Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 23 Jul 2013 Posts: 117
|
Posted: Tue Jul 23, 2024 3:09 pm Post subject: Binary host and USEs |
|
|
Hi all, I have a server ,which I want to run a binary host, the idea is instead of build on my laptop just do it all there.
The current binary host project can be found here: https://wiki.gentoo.org/wiki/Binary_package_guide
Which is very complete, but lacks of some details, in particular is not very clear about how USEs works in a binary host....
A binary host will store the binaries with the configured USEs, this causes the next:
If the client request other USEs portage will complains (I asked about this part), so I need go to the server, do the same changes of the client on the server, compile and then install on the client, basically we need to have the same USEs configuration on both of them.
If we add a second client and we perform this, we would need to upgrade the USEs for Client 2, if we also need a new USE in it then if we need update the USEs for the Host, to then upgrade again the USEs on the client 1.... at this point I think is easy to understand the issue here.
So going back, the Host is like a full system, it stores everything that can be installed, so, why we need to keep in the Clinets the track of the USEs? in first place, if we need a new USE, we would need to change the Host, sounds more practical to request the World packages of the Clients to the host with the config the Host have, if we need any USE change the ideal is update the Host and reinstall on the clients looking on changed packages (think equery u foo would show the host uses).
All this can also be applied to any package configuration like accept_keywords.
Maybe I didn't understand well some parts of the binary host, but this issue makes hard to use a binary host, is there any way to simplify it?
Thx! |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Hu Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
Joined: 06 Mar 2007 Posts: 22371
|
Posted: Tue Jul 23, 2024 3:56 pm Post subject: Re: Binary host and USEs |
|
|
Pipeng wrote: | So going back, the Host is like a full system, it stores everything that can be installed, so, why we need to keep in the Clinets the track of the USEs? | Are you proposing that using emerge --getbinpkg should just override local preferences and install packages with the wrong USE flags? Some USE flags are set for good reason, such as VIDEO_CARDS. I would be quite unhappy if emerge --getbinpkg pulling from an nVidia-only binhost quietly installed an nVidia-only Xorg server on my AMD-based system, since it would mean I would not get a working video driver. To guard against this, Portage would raise a conflict telling me that I configured locally for amdgpu, and the binhost cannot satisfy that. That then puts the burden back on me to decide whether I want a non-working Xorg, want to reconfigure the binhost to build an amdgpu-enabled Xorg, or want to build locally.
If you want not to have that check, you could choose to use --binpkg-respect-use=n. I think Portage even suggests this in its output when it rejects a binary package. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Pipeng Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 23 Jul 2013 Posts: 117
|
Posted: Tue Jul 23, 2024 4:11 pm Post subject: |
|
|
Thx for the option!
Just I still don't see it very practical, VIDEO_CARDS, we set it for a client, but if two clients needs different cards, you still need to enable both cards on the host, and both clients will have the needed packages based on that.
Lets pick as example mesa, two clients with different cards, both clients will get the same mesa binary with both use flags enabled, with all deps it means.
So I don't see the difference.....
But I agree is good the client be able to check if their needs match with the server, but with a binhost and several clients, the logic used for this will be different, like "to this packages have at least this uses", "at least this version for this packages".
Seems more complex this than I thought.... |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20330
|
Posted: Tue Jul 23, 2024 7:38 pm Post subject: |
|
|
Keep in mind that a binhost need not use the binaries it creates for clients.
The host OS could use a non-GUI configuration, such as:
"headless server" A) USE="-X -qt -gtk", etc.
The binhost could additionally offer two different sets of binaries. For example:
"gtk desktop" B) USE="X gtk -qt" and
"qt desktop" C) USE="X qt -gtk"
One client could use gtk binaries, the other client could use qt binaries, and the binhost itself uses the binaries it built in A. You can proie as many unique sets of binaries as you can manage to configure, store on disk, and provide sufficient compiling time to produce. Of course, it's easier to provide a single set of "generic" binaries that most clients can use.
In the guide you linked, see "Advanced topics -> Chrooting":
https://wiki.gentoo.org/wiki/Binary_package_guide#Chrooting
To paraphrase NeddySeagoon:
Don't think of Gentoo as a distro. Think of Gentoo as a toolkit you use to design, create, and install your own distro. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Pipeng Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 23 Jul 2013 Posts: 117
|
Posted: Wed Jul 24, 2024 1:54 am Post subject: |
|
|
How is it? lets follow that, we have two clients, one want GTK and the other one QT.
Now lets imagine there is a package that has both options as USE, GTK and QT.
How we want to both clients be enable, means the Host will have GTK and QT enabled.
And now, if the client of GTK want that package, it will look on the Host, but both USEs are enabled, so the GTK client needs to have QT enabled for that package too.
This would also happens with the QT one.
So, both clients needs a minimum of compatibility.
Oks, maybe if we just enable GTK and QT globally on the clients, there will be some packages that will not retrieved from the Host for dependencies, but still need to match at this extent all the clients.. is really painfull.... |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20330
|
Posted: Thu Jul 25, 2024 12:14 am Post subject: |
|
|
Pipeng wrote: | So, both clients needs a minimum of compatibility. | Correct, if you only want one set of binaries to work for different requirements.
Pipeng wrote: | is really painfull.... | I guess I'm not understanding the "pain." Either you build for different requirements, or you build for a common set. The latter is how binary distros work. They build so that most things work most of the time. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mega_flow Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
![](images/avatars/gallery/Star Wars/movie_star_wars_c-3p0.gif)
Joined: 26 Jun 2016 Posts: 92 Location: Belgium
|
Posted: Thu Jul 25, 2024 5:33 am Post subject: |
|
|
portage support multi use flag in binpkg dir but if u use eclean-pkg it will clean any package not build with te use flag. don't use that app to clean te dir
u need to have multi chroot/machinectl/virtual machine . if you want to install different tool kits.
U can create a overlay per build machine for various hosts
For managen the different use flags i use the overlay
my-overlay
├── metadata/
│ └── layout.conf
└── profiles/
└── repo_name/parent ( to set the profile example gentoo:default/linux/amd64/23.0/no-multilib/systemd )
└── repo_name/make.defaults ( for make.conf )
└── repo_name/packages ( what package to install )
└── repo_name/package.use ( for use flags per package set )
└── repo_name/package.accept_keywords ( for unmasking unstable keywords )
this is not the official way to manage your settings. but make it a lot easier. I then sync this overlay to the host, so now i don't have to manual manage all setting on the host
For the VIDEO_CARDS i use package.use to set
x11-base/xorg-drivers -video_cards_nvidia -video_cards_radeonsi
x11-libs/libdrm video_cards_radeon video_cards_intel video_cards_amdgpu
media-libs/mesa video_cards_radeon video_cards_radeonsi video_cards_intel
and make.defaults
VIDEO_CARDS="-* radeonsi nvidia"
For manage various gpu hardware
This make it so i don't install nvidia-drivers on a amd gpu machine _________________ default/linux/amd64/23.0/no-multilib/llvm/gnome/systemd gnome:46 |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Pipeng Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 23 Jul 2013 Posts: 117
|
Posted: Thu Jul 25, 2024 3:30 pm Post subject: |
|
|
@pip the pain is keep sync the host with all the clients! they all must match USEs, accept_keywords, etc, etc. And one change on the host means all clients must change it!
Still all clinets must have the same USEs as the Host, so is not important the Client USEs, change them to the host, or you will need to build locally, you don't want to build locally, then must use the same USEs as the Host..
@mega_flow your idea is great! solve like almost everything about it!
How works the example you post? if a client want a nvidia one, would request x11-base/xorg-drivers nvidia USE... so I don't get it veyr well.
I have no problem on have several chroot ones :3 I learned there is machinectl which seems a great option!
And what do you means with "support multi use flag in binpkg"? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20330
|
Posted: Thu Jul 25, 2024 5:22 pm Post subject: |
|
|
Pipeng wrote: | @pip the pain is keep sync the host with all the clients! they all must match USEs, accept_keywords, etc, etc. And one change on the host means all clients must change it!
Still all clinets must have the same USEs as the Host, so is not important the Client USEs, change them to the host, or you will need to build locally, you don't want to build locally, then must use the same USEs as the Host.. | On my build host, I create the binary packages in a chroot. The chroot represents the "standard" the client should match. When I update the client, I create a tar file of the chroot's /etc/portage. I copy that tar file along with the chroot world file, kernel sources squashfs image, and the ::gentoo repo squahfs image to the client.
It's not an ideal solution, but I keep the "pain" at a low enough such that I'm not often motivated to improve that solution. I have a set of scripts that prepares everything on the build host and I have a script on the client that pulls the files. The work to be done on the client is minimal, but manual. The only major piece that is completely manual are other files in /etc other than /etc/portage.
A much simpler method would be to use NFS, but from past experience, I don't use NFS (it caused problems everywhere I've worked that used NFS).
I don't expect most people would choose my hack of a (partial) solution. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mega_flow Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
![](images/avatars/gallery/Star Wars/movie_star_wars_c-3p0.gif)
Joined: 26 Jun 2016 Posts: 92 Location: Belgium
|
Posted: Thu Jul 25, 2024 7:04 pm Post subject: |
|
|
@Pipeng portage can keep a package build with difference use flags on the same location . portage is smart enough to pick the right one for your host
the nvidia-driver is only pickup with xorg-drivers. So enable nvidia on example mutter wont pull the driver ( not sure on ffmpeg )
U can set this with the video_cards_nvidia or video_cards_(driver) to enable or disable the flags in package.use
( xorg-driver only install x11 driver ,wayland one ar install with mesa )
If u set VIDEO_CARDS or L10N make.defaults make sure u start with "-*" or portage enable all flags
in the local overlay the file packages wil set each application in the @world ( enable with a * example *sys-kernel/gentoo-kernel )
Any others package like nvidia-drivers i set in hostname file in the local overlay and linkt to /etc/portage/sets and emerge @hostname so i can control only host necessary packages. _________________ default/linux/amd64/23.0/no-multilib/llvm/gnome/systemd gnome:46 |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Goverp Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/152613747847c6fd276c31a.jpg)
Joined: 07 Mar 2007 Posts: 2109
|
Posted: Fri Jul 26, 2024 12:54 pm Post subject: |
|
|
I exploit the current behaviour. My binpkg host (in a chroot) omits certain entries from /etc/portage/make.conf 'CPU_FLAGS_X86' since they are supported on only one of my two machines.
This means that when I try to install one or two packages that are sensitive to (and presumably exploit) the relevant flags, portage spurns the binpkg and compiles an host-specific version (and -march=native as well) - notably this applies to ffmpeg, and my little laptop needs all the help it can get with that. _________________ Greybeard |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Pipeng Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 23 Jul 2013 Posts: 117
|
Posted: Wed Aug 07, 2024 2:42 pm Post subject: |
|
|
thx for all the asnwers! taken me some time and I was able to implement the overlay one! well, close, I'm almost be able to do it.
For some reason, the "packages" file, which should set the packages to install, is not being recognized.... all the other parts are working fine.
I tested emerge -a --depclean and always try to remove everything.
the location is: myoverlay/profiles/profile_name/packages
Thx! |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|