Binary package host: when clients dont install all pkgs

[inf1nity]

Hi,

I've been running my own binary package host for some time now and mostly it's working really well.

However, sometimes I have problems when "client" systems don't have all packages installed, that the binary package host has. This leads to strange blockers and or packages that would have to be built on the client.

For example if package A, not installed on a client, requires an older version of package B, thus preventing an upgrade to a newer available version for B on the binary package host, then said package B would be eligible for upgrade on the client system, as A is not installed and blocking the upgrade.

As far as I can figure out, the only way to make it consistent would be to always, after everything has been updated on the binary package host, create a package.mask file, that would list all available newer versions of installed packages.

Does such a script, for creating a list of all available newer versions of installed packages, already exist somewhere and if not, could someone please point me in the right direction? Especially comparing package versions is something I'm unsure on how to do reliably, i.e. does a developer tool for something like this exist, that works with the exact gentoo version format?

[logrusx]

I think emerge --getbinpkgonly, on the client, is what you're looking for.

Best Regards,
Georgi

[eschwartz]

Ideally a binhost builder would have minimal packages in its world file so that it can resolve and install any package without worrying about conflicting requirements.

You can then regularly depclean whenever it has blocker issues, generate a new binary package for whatever the problematic package is, and move on.

[pjp]

Are you keeping the client's repositories, world file and /etc/portage directory structure in sync with the binhost? If not, that can cause some problems.
_________________
Quis separabit? Quo animo?

[NeddySeagoon]

inf1nity,

Build time dependencies will not be installed on binhost clients by default. They are not needed until the client needs to build something.
You can ask for build time dependencies with --with-bdeps=y,
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[inf1nity]

First of all, thank you all for your help, explanations and input!!

I'll try to clarify:

I'm keeping all clients /etc/portage directory in sync (EDIT: with the binhosts), while allowing slight overrides via .local files that aren't synchronized/deleted upon synchronization. One such example is a make.conf.local that helps me use different FEATURES, EMERGE_DEFAULT_OPTS, etc between binhost and client.

Like on the client:

FEATURES="${FEATURES} getbinpkg pkgdir-index-trusted"
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --rebuilt-binaries"

Every time I upgrade the binary package host, I create a snapshot of the /etc/portage and /var/cache/binpkgs directories and sync them to a mirror server. From there the former is rsync'ed and the latter used as a http binrepo.

I have two binary package hosts, one for a server environment and one for desktop. What I don't keep in sync is the world file, however I make sure that it is nearly identical, to avoid unncessary blockers, use flag issues, etc. The clients world file is always a subset of the binhosts.

That works mostly fine, except for cases where not-installed packages lead to different dependency resolutions, which is exactly why I created my post, as I'm searching for a solution.

Regarding the info from @eschwartz, do I understand that correctly, that only a very minimal system should be in the world file and other packages I want to be provided by the binhost are to be installed with some form of build-package-only? Does this also hold true for bigger things like KDE/Plasma?

At the moment I'm just adding all packages that I need in the binhosts world file, so that if the binhost is in a consistent state, I can assume that the clients will be as well ...

I hope that is reasonable.

I'll take a look at emerge --getbinpkgonly later. It could be what I need, but I'm unsure solely from reading the manpages.

@NeddySeagoon: I'm aware, but this was a runtime dependency. Just to make sure I looked in the world file, it's not there, and it is also not being removed by running

emerge -av --depclean --with-bdeps=n

If my approach to using binhosts is off, please let me know ...

Thanks again!

[eschwartz]

[inf1nity]

@eschwartz:

Thank you very much! Now I finally understand how this works.

Though if I were to build packages optimized for different CPU architectures, then I'd need to have separate binhost VMs and binrepos, correct?

[Hu]

You can use a single VM to host more than one binrepo. At worst, you might need the VM to have a separate chroot for each binrepo.

For most people, the overhead of maintaining multiple highly similar binrepos is worse than the tiny performance loss of using a single binrepo that is compatible with all the client machines.

[eschwartz]

The official Gentoo binhost provides baseline -march=x86-64 packages as well as optimized microarchitecture packages for -march=x86-64-v3.

This covers most use cases. You don't really need packages to be compiled with -march=native when you can get very close to that by sticking with common architecture levels such as -v3 that are easily tested for support. And it means you don't have to have 60 different binhosts, one for every unique CPU.

[inf1nity]

[eschwartz]

Yes, that's why the microarchitecture levels were formulated.

[logrusx]