| View previous topic :: View next topic |
| Author |
Message |
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 522
|
 Posted: Sun Jun 30, 2024 6:42 am Post subject: quickpkg: another day, another problem |
 |
|
HI folks
I think I've spent three weeks populating a useless, broken binhost.
I built my packages on a number of different machines, just to spread the load, really. I then consolidated them on a single machine, and used that as the basis of the binhost.
What I didn't realize at first is that 'quickpkg' does not save configuration files by default. When I did hear about this, I figured it applied only to configuration files that an administrator might manually edit. But, no, it seems to apply to all configuration files.
I only discovered my mistake when I found that the system logger wasn't running on the new machine I'm setting up. I eventually tracked down the problem to there being nothing in the /etc/init.d file that starts the service -- just a message about quickpkg not storing configuration.
I later discovered a bunch of missing configuration files for GTK. And no default menus, etc., for Xfce4. And so on.
The worst of it is that I don't know which packages are affected, or even how many. So, even if I could identify the machine that has the original, complete package installed, I don't actually know which packages need to be fixed.
So, if anyone knows a way out of this mess, other than by repeating three weeks of 24/7 compilation, do please share.
In the meantime, is there a way to use compiled binaries from my binhost, but get the configuration files from someplace else? I see there is an `emerge --config`, but I not sure what that does if the binaries already exist. I can't even exclude the binary and do an ordinary `emerge`, accepting the overheads of rebuilding, because it turns out that `emerge` won't overwrite the config files that are just placeholders.
What a mess :/
BR, Lars. |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54578
Location: 56N 3W
|
| Posted: Sun Jun 30, 2024 11:01 am Post subject: |
|
|
lars_the_bear,
reads portages mind to tell what packages are installed.
You can even do something similar to get a list of files that belong to each package.
That will need someone better at one liners than me.
Given the list of packages, you can quickpkg them again, with the installed (modified) config files.
When you emerge -K them, protected config files will be held for etc-update, so you can decide which version you want.
No weeks of rebuilds required. ....
Hmm ... its ugly now as guide.html is no longer supported but https://dev.gentoo.org/~neddyseagoon/HOWTO_DVD11.xml
I'm sure that there is a one liner there to quickpkg a list list of packages.
The bit you need is
| Code: | <pre caption="quickpkg the whole DVD">
while read pkg; do (quickpkg --include-config=y "${pkg}"); done < <(qlist -CI)
</pre> |
The variable pkg is a list of files to package.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 522
|
| Posted: Sun Jun 30, 2024 11:21 am Post subject: |
|
|
| NeddySeagoon wrote: |
Given the list of packages, you can quickpkg them again, with the installed (modified) config files.
|
Thanks, but i don't think I can do that, because I don't know which machine has a 'good' installation for a specific package. All my machines were installed, to some extent or other, from the broken binhost. So I have four (real and virtual machines) with a mixture of 'good' and 'bad' packages.
So the problem lies in finding a machine which doesn't have a broken installation for package X, once I know that package X is broken -- and I'm only finding that out when things don't work as they should.
Sigh :/
BR, Lars. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54578
Location: 56N 3W
|
| Posted: Sun Jun 30, 2024 11:51 am Post subject: |
|
|
lars_the_bear,
That's harder but not impossible.
| Code: | | equery f baselayout |
will show all the files that belong to baselayout.
| Code: | equery f baselayout | wc -l
38 |
Tells how many files that baselayout has installed on that install. Here, its 38
You need test them all on every install, and quickpkg the one with the most files.
Someone with better bash skills than me will be along soon to automate the process for you.
This is Gentoo ... there is always a way.
Oh, I wrote that installing from the DVD guild because I was told that it couldn't be done :)
-- edit --
| Code: | | ls -d /var/db/pkg/*/* | cut -f5,6 -d/ |
will produce an installed package list. by reading portages mind.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1972
|
| Posted: Sun Jun 30, 2024 1:16 pm Post subject: |
|
|
| You can grep for '# empty file because --include-config=n when'. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2435
|
| Posted: Sun Jun 30, 2024 2:41 pm Post subject: Re: quickpkg: another day, another problem |
|
|
| lars_the_bear wrote: |
I built my packages on a number of different machines, just to spread the load, really. I then consolidated them on a single machine, and used that as the basis of the binhost.
What I didn't realize at first is that 'quickpkg'...
|
First, quickpkg is not intended for that purpose. Second, assembling a binhost that way is a lot of manual work. If you insist on continuing doing it that way enable buildpkg and gather packages this way. Judging by your wording, I guess your computers are in the same physical network, that's why I'd recommend distcc to distribute the load and build everything on your binhost. You can configure every node as cliend and server in the same time if I remember correctly, so if you still need to build a package on some of the hosts, that would distribute the load as well. I'm not sure my suggestion would work, but it's worth investigating it.
Regarding your original issue, can't you use Gentoo's binhost to fix most of the installations and then proceed on a case by case basis?
Best Regards,
Georgi
Last edited by logrusx on Sun Jun 30, 2024 2:43 pm; edited 1 time in total |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22681
|
| Posted: Sun Jun 30, 2024 2:42 pm Post subject: |
|
|
As I read the Portage source, and the manual page, only files protected by CONFIG_PROTECT can be excluded by this quickpkg behavior, so that can help to limit space of what files need to be checked for reconstruction. I think counting how many files were installed will not work, since it looks like the "bad" installations have a stub file telling you that the file was not packaged, rather than actually omitting the file.
emerge --config is for running pkg_config in the ebuild. It will not help with your current problem. emerge --noconfmem might, though that is a bit heavy-handed for your use case. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 522
|
| Posted: Sun Jun 30, 2024 3:13 pm Post subject: Re: quickpkg: another day, another problem |
|
|
| logrusx wrote: |
Regarding your original issue, can't you use Gentoo's binhost to fix most of the installations and then proceed on a case by case basis?
|
Thanks, but I'm not entirely sure how.
I can find all the broken config files by doing a grep -r over the whole filesystem for files containing the 'empty file' message. I'll have to do this on multiple machines, but fortunately they all have essentially the same packages.
Then, I guess, I can look up using equery which package each broken file comes from. The I'll have to delete the broken config files, because `emerge` does not overwrite them.
Then... I'm not sure. These aren't files that came from the Gentoo binhost -- they're files I compiled myself, which is why I wanted to preserve them. Maybe the missing config files can come from the binhost packages, but can I download those without overwriting the binaries?
BR, Lars. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20485
|
| Posted: Sun Jun 30, 2024 4:23 pm Post subject: |
|
|
You can extract the contents of binary packages. How probably depends on which format is in use.
If it's an XPAK: | Code: | $ file /var/cache/binpkgs/app-admin/eselect-1.4.27-r1.tbz2
/var/repos/binpkgs/app-admin/eselect-1.4.27-r1.tbz2: Gentoo binary package (XPAK) |
app-portage/portage-utils provides qtbz2 and qxpak to manipulate the contents.
It's been a while since I've used it, but this seems to extract the xpak from the tbz2 file and then a specific file from the xpak (there are other possibilities): | Code: | $ qtbz2 -s python-2.7.15.tbz2
$ qxpak -x python-2.7.15.xpak RDEPEND |
I'm not sure how to work with the newer "gpkg" format, but maybe this has some useful information:
https://www.gentoo.org/glep/glep-0078.html
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2435
|
| Posted: Sun Jun 30, 2024 5:38 pm Post subject: Re: quickpkg: another day, another problem |
|
|
| lars_the_bear wrote: | | logrusx wrote: |
Regarding your original issue, can't you use Gentoo's binhost to fix most of the installations and then proceed on a case by case basis?
|
Thanks, but I'm not entirely sure how. |
The simplest possible way - just use the binary packages from there. The x86_64-v2 binhost should cover almost all hardware currently in use.
Whether you decide to identify all impacted packages or run emerge --emptytree depends on the circumstances. The latter takes a lot of time, despite the fact you're emerging binary files. The 23 profile migration took me ~8 hours for nearly 1600 packages. Some of them had to be compiled of course, but not the bigger ones which I had already taken care the binary packages would be used for them.
You can also use /var/log/emerge.log to parse a list of all the binary packages ever emerged. It includes entries like this:
| Code: | | 1638448082: === (14 of 19) Merging Binary (package name here) |
Best Regards,
Georgi
Last edited by logrusx on Sun Jun 30, 2024 6:43 pm; edited 1 time in total |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Jun 30, 2024 6:09 pm Post subject: |
|
|
| pjp wrote: | | I'm not sure how to work with the newer "gpkg" format |
This is much simpler: You do not need any special tools for that. It is just a .tar file which contains the metadata and the installed package as separate compressed archives. Additionally, it contains an empty file "gpkg-1" and a Manifest file with checksums of the 3 files. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20485
|
| Posted: Sun Jun 30, 2024 8:20 pm Post subject: |
|
|
Thanks for clarifying. I thought that was the case but didn't want to mislead. I need to update that when I migrate to profile 23.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 522
|
| Posted: Mon Jul 01, 2024 7:53 am Post subject: Re: quickpkg: another day, another problem |
|
|
| logrusx wrote: |
The simplest possible way - just use the binary packages from there. The x86_64-v2 binhost should cover almost all hardware currently in use.
|
The problem is that the official Gentoo packages might not be compatible. And if they are, they might not be optimal. Nearly every package I install results in an explanation why a binary can't be used. For example:
| Code: |
!!! The following binary packages have been ignored due to non matching USE:
=media-video/mplayer-1.5_p20230618 -cpu_flags_x86_avx2 -cpu_flags_x86_fma3
|
Is there a way to install only the configuration files from the binary package, and not the actual binaries?
What I could do, I suppose, is install the Gentoo binary, then install my own binary, which won't(?) overwrite the configuration files from the Gentoo binary.
But first I have to identify and remove all the broken configuration files, else the binary install won't overwrite them anyway.
Sigh. I wonder, to be honest, whether it would just be easier to start all over again, and burn another three weeks of CPU time.
BR, Lars. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2435
|
| Posted: Mon Jul 01, 2024 6:22 pm Post subject: Re: quickpkg: another day, another problem |
|
|
| lars_the_bear wrote: |
Is there a way to install only the configuration files from the binary package, and not the actual binaries? |
I can't think of an easy way to do so.
| lars_the_bear wrote: | | What I could do, I suppose, is install the Gentoo binary, then install my own binary, which won't(?) overwrite the configuration files from the Gentoo binary. |
It didn't for me when I was using it.
| lars_the_bear wrote: | Sigh. I wonder, to be honest, whether it would just be easier to start all over again, and burn another three weeks of CPU time.
BR, Lars. |
I highly recommend using as many binary packages from the Gentoo binhost as possible. I used to run highly customized packages but with time I concluded for myself it's not that important. Also if you setup distcc it won't take that much time.
Best Regards,
Georgi |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 522
|
| Posted: Tue Jul 02, 2024 6:26 am Post subject: Re: quickpkg: another day, another problem |
|
|
| logrusx wrote: |
I highly recommend using as many binary packages from the Gentoo binhost as possible. I used to run highly customized packages but with time I concluded for myself it's not that important. |
I wouldn't compile, say, Firefox (which takes two days) just to get a notional improvement in CPU flags. But most of what I've installed would not install as binaries, because I have 'USE=-pulseaudio -systemd -gnome -wayland...'.
I did, at first, override `emerge` choices, and force it to install binaries, with mixed results.
I've always been willing to use the Genoo binaries but, in fact, few of them actually do get used, with my configuration.
I could give in, and install Pulse, Gnome, etc. to make the binary packages more compatible; but, if I do that, I might as well stick with Ubuntu. These are exactly the things I came to Gentoo to get away from.
BR, Lars |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Installing Gentoo
