| View previous topic :: View next topic |
| Author |
Message |
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 293
|
 Posted: Mon Dec 16, 2024 11:25 pm Post subject: Rebuild packages with missing binpkg |
 |
|
I set up a build server which creates (and signs) binary packages. Due to a binary package update, the signed binpkg had been removed again before the respective client had run its latest world update. This led to a situation where the client wanted to build the package itself, because the binary package was not available anymore.
I found a thread on this forum (https://forums.gentoo.org/viewtopic-p-8813589.html#8813589) and this commit (https://github.com/gentoo/portage/commit/a7bbb4fc4d38f770fc943f3b856c5de56e315fe4) which discussed to re-generate the signature, but the approach was dismissed. Instead the binpkg are deleted and the commit proposes the following way out | Quote: | | Simply dropping these binpkgs seems like the best solution and to allow the binhost to re-sign it on updating instead. |
Unfortunately, if the timing is bad, this does not always work. The binpkg might have been deleted, no newer package has yet been emerged and the client which still uses on an even older package might try to fetch the current, but missing package.
Hence, I am wondering if there is an option to mitigate the problem on the binhost at least, by identifying missing binary packages and re-emerge the relevant packages, before the client falls into that trap.
Here is what happened (the affected package were llvm:18, llvm:19, mesa, firefox, wine and some other):- The binhost and the clients were all in snyc
- The server synchronized its package list (emerge --sync)
- The server run a world upgrade and built binary packages
- The server synchronized its package list again some days later
- A binary update came in and hence the affected binary packages on the binhost were deleted
- The client synchronized its package list
- The client attempted to run a world upgrade, but some binary package on the binhost were already missing
Unfortunately, a simple "emerge --deep --newuse --update @world" on the binhost did not solve the problem, because from the perspective of the server the packages were installed and up-to-date (only the binary packages were missing). I has to manually re-emerge the affected packages via "emerge --oneshot <package 1> <package 2> ...". Can this process be automated? |
|
| Back to top |
|
 |
nox23
n00b
Joined: 15 Jul 2012
Posts: 42
|
| Posted: Tue Dec 17, 2024 7:00 am Post subject: |
|
|
You can use the command quickpkg to manually create a binpkg from an existing installed package, see :
To achieve it automatically, you'll need to create a script that compare the content of the /var/db/pkg to the directory /var/cache/binpkgs, for example ou via command output, ... |
|
| Back to top |
|
|
freke
Veteran
Joined: 23 Jan 2003
Posts: 1038
Location: Somewhere in Denmark
|
| Posted: Tue Dec 17, 2024 7:50 am Post subject: |
|
|
| nox23 wrote: | You can use the command quickpkg to manually create a binpkg from an existing installed package, see :
To achieve it automatically, you'll need to create a script that compare the content of the /var/db/pkg to the directory /var/cache/binpkgs, for example ou via command output, ... |
Be aware that quickpkg may generate empty files (/etc/) - may not be a problem if the package is already emerged/just being updated on the client, but if the package is a new install on the client you may end up missing files.
https://forums.gentoo.org/viewtopic-t-1171977-highlight-quickpkg.html |
|
| Back to top |
|
|
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 293
|
| Posted: Tue Dec 17, 2024 6:05 pm Post subject: |
|
|
Missing configuration files shouldn't be a problem, because quickpkg can include them. As all.packahes are built in a "clean" chroot-enviroment the respective /etc doesn't contain any individual, confidential settings, but the original configuration files.
The major problem is to reliably automate the process of finding out, which packagea are missing. |
|
| Back to top |
|
|
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1964
|
| Posted: Tue Dec 17, 2024 6:47 pm Post subject: |
|
|
quickpkg has a glaring flaw in my experience in that the binary packages destroy custom settings for configuration files with the default --include-config=n and --include-unmodified-config=n (the latter should be =y by default IMO)
I've had a few files wiped out this way and had to restore from outside backups. The binaries here bypassed the normal protection to ask to merge with tools like dispatch-conf as well. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22877
|
| Posted: Tue Dec 17, 2024 6:50 pm Post subject: |
|
|
| Although the output is not immediately suitable for script use, would emerge --pretend --emptytree --usepkg @world produce a list that, after filtering out the lines that read binary, be correct for what you want? If so, you could probably craft a Python script that collects the equivalent data and prints it in a nice form. |
|
| Back to top |
|
|
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 293
|
| Posted: Tue Dec 17, 2024 7:40 pm Post subject: |
|
|
| Quote: | | quickpkg has a glaring flaw in my experience |
Thank you for the warning. In that case a better re-emerge the packages. (Requires some more time, but its safe.)
| Quote: | | Although the output is not immediately suitable for script use, would emerge --pretend --emptytree --usepkg @world produce a list that, after filtering out the lines that read binary, be correct for what you want? |
That won't work, because on the build server a package is never installed as a binary. So that's not an option.
I was already able to compile this very ugly command line | Code: | | find /var/cache/binpkgs/ -type f -name '*.gpkg.tar' | sed -e 's/^\/var\/cache\/binpkgs\/\([-a-z]\+\)\/\([-a-z0-9]\+\)\/\(-\?[a-z][a-z0-9]*\)*-\([0-9][-._0-9a-z]*\)-[0-9]\+\.gpkg\.tar$/\1\/\2-\4/' |
It provides me with a list of packages and their versions for which binary packages exist. | Code: | | equery list -F '$cpv' -f '.*' |
provides me with a list of installed packages. | Code: | | comm -1 -3 <file 1> <file2> |
should give me what I need. However, I still have a number of false positives. So I am not quite there yet.
PS: Initially, I had some hope that "emerge" might have learned to re-sign already built binary packages which needs to be modified. But that hope seems to be in vain. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22877
|
| Posted: Tue Dec 17, 2024 7:49 pm Post subject: |
|
|
| nagmat84 wrote: | | Quote: | | Although the output is not immediately suitable for script use, would emerge --pretend --emptytree --usepkg @world produce a list that, after filtering out the lines that read binary, be correct for what you want? |
That won't work, because on the build server a package is never installed as a binary. So that's not an option. |
Normally, yes, you would not install a binary on the build server. However, the command I gave would pretend to do that, for the purpose of getting the output. Did you try the command I gave and find it to be unsuitable? |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
