View previous topic :: View next topic |
Author |
Message |
tckosvic Tux's lil' helper
Joined: 18 Apr 2023 Posts: 138
|
Posted: Sat Sep 07, 2024 1:24 pm Post subject: List of available binary packages |
|
|
After another update that has been running for more than 24 hrs using 5 cpus on an i7 chip, I am going to use binaries. Running update includes qtwebengine (with more than 30000 compilations) and libreoffice, and others.
I think the documentation of how to move to binaries is very well done. I believe I can implement it.
Beforehand, though, I would like to find a list of what packages are available as binaries. Can't seem to find info on where to look at the binary repo.
Any info? thanks, tom kosvic |
|
Back to top |
|
|
asturm Developer
Joined: 05 Apr 2007 Posts: 9272
|
Posted: Sat Sep 07, 2024 1:27 pm Post subject: |
|
|
I can't really answer with a specific number or way to look it up, but afaik it should be equivalent to the amount of stable packages. |
|
Back to top |
|
|
tckosvic Tux's lil' helper
Joined: 18 Apr 2023 Posts: 138
|
Posted: Sat Sep 07, 2024 1:50 pm Post subject: |
|
|
Ok, so all packages are available as binaries. I got the impression from reading that only select packages were available as binaries.
So if I activate binary repo universally then all packages will load as binaries. Then, I would setup to only activate binaries on the packages that give me extended update times and recompile all others.
thanks, tom kosvic |
|
Back to top |
|
|
tckosvic Tux's lil' helper
Joined: 18 Apr 2023 Posts: 138
|
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2420
|
Posted: Sat Sep 07, 2024 4:10 pm Post subject: |
|
|
Setup the binhost.repo and run your emerge update command adding g and v to the switches. Of course p or a so you can take as much time as you want inspecting the output. At the end you'll see a list of packages which cannot be merged due to use incompatibility along with suggestions what can be changed to accommodate them.
Most of the packages are available. Packages like firefox that have a -bin equivalent are not, for obvious reasons. If you're interested in particular package you can browse the URL that you put in binrepos.conf or just issue an emerge -pvg command for it to see if it's available. I believe your concern is packages like before-mentioned qtwebengine. They are mostly available but you might need to change your use configuration and live with some compromises.
One detail worth mentioning is when using binary packages, the build-time dependencies will not be installed. That's why you may be surprised either at depclean or when running regular update after one with binary packages. In the case of deplcean you need to add --with-bdeps=n if I remember correctly, in the case of a regular update, you decide what to do.
Best Regards,
Georgi |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20485
|
Posted: Sat Sep 07, 2024 10:41 pm Post subject: |
|
|
You could browse a mirror to see what's there...
releases/amd64/binpackages/23.0/x86-64/
Doesn't the Packages file contain (in part) the available packages? Code: | $ grep -c ^CPV Packages
5035 |
Off the top of my head, without having installed much, I had to compile grub, and lvm doesn't actually include lvm support, so I also had to compile that. I think there were more, but I don't recall the details. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
eschwartz Developer
Joined: 29 Oct 2023 Posts: 221
|
Posted: Sun Sep 08, 2024 6:53 pm Post subject: |
|
|
logrusx wrote: | Most of the packages are available. Packages like firefox that have a -bin equivalent are not, for obvious reasons. If you're interested in particular package you can browse the URL that you put in binrepos.conf or just issue an emerge -pvg command for it to see if it's available. I believe your concern is packages like before-mentioned qtwebengine. They are mostly available but you might need to change your use configuration and live with some compromises.
|
Packages like firefox that have a -bin equivalent *are* (sometimes) available. Firefox is!
pjp wrote: | You could browse a mirror to see what's there...
releases/amd64/binpackages/23.0/x86-64/
Doesn't the Packages file contain (in part) the available packages? Code: | $ grep -c ^CPV Packages
5035 |
Off the top of my head, without having installed much, I had to compile grub, and lvm doesn't actually include lvm support, so I also had to compile that. I think there were more, but I don't recall the details. |
The Packages file is literally a list of *all* available packages. It is downloaded and used to tell portage what is available for downloading.
Code: |
$ grep PACKAGES: /var/cache/edb/binhost/gentoo.osuosl.org/releases/amd64/binpackages/23.0/x86-64-v3/Packages
PACKAGES: 5057
|
No need to count the number of matches, by the way -- it's part of the file header. |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2420
|
Posted: Mon Sep 09, 2024 5:41 am Post subject: |
|
|
eschwartz wrote: |
Packages like firefox that have a -bin equivalent *are* (sometimes) available. Firefox is!
|
Thanks for letting me know. I hadn't noticed because I used firefox:rapid.
eschwartz wrote: | The Packages file is literally a list of *all* available packages. :) It is downloaded and used to tell portage what is available for downloading.
Code: |
$ grep PACKAGES: /var/cache/edb/binhost/gentoo.osuosl.org/releases/amd64/binpackages/23.0/x86-64-v3/Packages
PACKAGES: 5057
|
No need to count the number of matches, by the way -- it's part of the file header. |
I still think the best answer to this question is to actually run emerge. It has an exact answer as you point out, but it's rarely useful for making decisions. One thing is for sure, the number is big enough to be considered. But the best number is the one you'll actually install and that's only answered by running the query and making your mind about what use changes you're willing to make.
Best Regards,
Georgi |
|
Back to top |
|
|
nikodll n00b
Joined: 26 May 2004 Posts: 19
|
Posted: Tue Sep 10, 2024 6:10 pm Post subject: |
|
|
So far, it does not look very good for the bin packages availability. The very first package I wanted to install on a fresh system is of course the Midnight Commander (app-misc/mc), and there was no binary package for it. Not with the right set of USE flags, simply NONE, no packages for mc. My impression if you really want to be able to merge most of you packages from binaries the Gentoo-based Calculate Linux is still preferred. _________________ Gentoo people |
|
Back to top |
|
|
eschwartz Developer
Joined: 29 Oct 2023 Posts: 221
|
Posted: Tue Sep 10, 2024 6:52 pm Post subject: |
|
|
nikodll wrote: | So far, it does not look very good for the bin packages availability. The very first package I wanted to install on a fresh system is of course the Midnight Commander (app-misc/mc), and there was no binary package for it. Not with the right set of USE flags, simply NONE, no packages for mc. My impression if you really want to be able to merge most of you packages from binaries the Gentoo-based Calculate Linux is still preferred. |
There are many packages available.
https://gitweb.gentoo.org/proj/binhost.git/tree/builders/milou
Each of these directories contains a world file for a different profile + chroot, listing the packages which will be installed. There's pretty decent coverage of the entire gnome, kde, lxqt, mate desktops, together with the flagship DE applications and complete dependency chain, and a selection of cherry-picked popular apps.
That doesn't mean all packages are available. In fact there are 20k packages in gentoo, and the binhost currently only has about 5k packages including USE flag variants. You can install a base system, a desktop, and many of the most common "this package is a tremendous pain to compile from source", which is pretty good to start with!
You can join #gentoo-binhost on IRC and we can discuss adding more packages. Preferably popular packages or packages that take a long time to compile.
I agree with you that "most packages" is not currently something Gentoo Binhost has available. Maybe one day. For now we have laid some pretty good groundwork, which is very suitable for people that would like to use Gentoo but find upgrading literally every package from source to be a huge hassle -- even just having the base system as shipped by new stage3 tarballs is a huge win, never having to compile GCC or LLVM or Rust from source again.
I only recently joined the binhost team, and I've been investigating ways to expand the coverage. |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2420
|
Posted: Tue Sep 10, 2024 7:23 pm Post subject: |
|
|
eschwartz wrote: | even just having the base system as shipped by new stage3 tarballs is a huge win, never having to compile GCC or LLVM or Rust from source again. :) |
We have new stage 3 tarballs that we don't need to compile gcc llvm and rust?
Great news! |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3698 Location: Rasi, Finland
|
Posted: Tue Sep 10, 2024 7:44 pm Post subject: |
|
|
nikodll wrote: | Midnight Commander (app-misc/mc), and there was no binary package for it. | I think compiling package such as mc isn't a problem to most. It's those bigger packages like, llvm/clang, gcc, nodejs, web browsers (web engines)... etc, etc. _________________ ..: Zucca :..
My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
eschwartz Developer
Joined: 29 Oct 2023 Posts: 221
|
Posted: Tue Sep 10, 2024 7:57 pm Post subject: |
|
|
logrusx wrote: | eschwartz wrote: | even just having the base system as shipped by new stage3 tarballs is a huge win, never having to compile GCC or LLVM or Rust from source again. |
We have new stage 3 tarballs that we don't need to compile gcc llvm and rust?
Great news! |
stage3 obviously comes with GCC preinstalled, and less obviously it comes with rust preinstalled. LLVM, actually no.
It is understandably not exciting to anyone, to reinstall from a stage3 tarball just to avoid compiling an update to GCC... |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20485
|
Posted: Tue Sep 10, 2024 11:22 pm Post subject: |
|
|
nikodll wrote: | So far, it does not look very good for the bin packages availability. ... Not with the right set of USE flags, simply NONE | Then why did you choose the wrong USE flags?
nikodll wrote: | My impression if you really want to be able to merge most of you packages from binaries the Gentoo-based Calculate Linux is still preferred. | Calculate Linux has a binary for every possible USE flag combination? I don't understand how your issue with USE flags is resolved by using Calculate Linux and their binaries? _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2420
|
Posted: Wed Sep 11, 2024 4:33 am Post subject: |
|
|
nikodll wrote: | So far, it does not look very good for the bin packages availability. The very first package I wanted to install on a fresh system is of course the Midnight Commander (app-misc/mc), and there was no binary package for it. Not with the right set of USE flags, simply NONE, no packages for mc. My impression if you really want to be able to merge most of you packages from binaries the Gentoo-based Calculate Linux is still preferred. |
Let me explain you something. If you have a package with 5 configuration options (use flags in Gentoo's case), there must be 32 different binaries for that package to satisfy all possible configurations of that package. Now extrapolate that for all 20k packages in tree. No distribution, not Gentoo, not Calculate Linux builds so many binaries just to satisfy all possible configuration preferences. It would take insane amount of time and space.
If you want "the right set of USE flags", do not use binary packages.
Best Regards,
Georgi |
|
Back to top |
|
|
nikodll n00b
Joined: 26 May 2004 Posts: 19
|
Posted: Wed Sep 11, 2024 7:59 am Post subject: |
|
|
Maybe I was not expressing myself very clear, but my point is that there is no binary package for app-misc/mc available. Regardless of your USE-flag settings. And because MC is such a popular and widely used app, I consider this something that can be improved in the future for a better binary installation support.
Also, this is my first attempt (and quite successful after all, actually) to install Gentoo from binaries, so I may have missed something. I believe the mc package is entirely missing, because the corresponding dir on the file mirror is absent (e.g.: https://mirror.leaseweb.com/gentoo/releases/amd64/binpackages/23.0/x86-64/app-misc/ ), as well as because portage doesn't complain / offer different set of USE-flags for it. Sorry if I am wrong, feel free to educate me on how binary packages are resolved. _________________ Gentoo people |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2420
|
Posted: Wed Sep 11, 2024 8:28 am Post subject: |
|
|
nikodll wrote: | Maybe I was not expressing myself very clear, but my point is that there is no binary package for app-misc/mc available. |
I apologize, I missed the part "Simply NONE", but it's also a bit convoluted sentence for an international support forum. Not all of us speak English natively. Actually I'd bet on the minority of users do not.
I'd also like to mention mc is not that big of a package. It can easily be compiled on very old hardware. It took 48 seconds on my current 16 thread/32GB RAM laptop, I'd say it would get 10 times slower on my old Intel Q6600 4Thread/8GB RAM computer and I would round it up to 10 minutes, just to be within the boundary of what would actually be possible on that machine with a spinning disk.
Best Regards,
Georgi
Last edited by logrusx on Wed Sep 11, 2024 6:29 pm; edited 1 time in total |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3698 Location: Rasi, Finland
|
Posted: Wed Sep 11, 2024 9:12 am Post subject: |
|
|
logrusx wrote: | I'd also like to mention mc is not that big of a package. | Exactly.
@nikodll: I clocked building mc on my slowest box that has Intel Celeron 2955U as its CPU. 2 core, no hyperthreading. Compilation took 2m 40s.
What is it with mc that you want it as binary package? Or do you have many other packages that are not available as binary packages in any configuration? _________________ ..: Zucca :..
My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
psycho Guru
Joined: 22 Jun 2007 Posts: 542 Location: New Zealand
|
Posted: Wed Sep 11, 2024 11:01 am Post subject: |
|
|
Is it only the size/time of package compilation that matters? I was thinking perhaps they could help with broken systems... am updating an ancient one (old profile and many hundreds of outdated packages) and was seriously tempted (when I saw the massive piles of conflicts and other issues) to try updating world with all the available binaries just so that they'd already be in place to resolve problems before attempting compilation. Fortunately it turned out to be pretty straightforward to resolve all the issues just by fixing a dozen or so key ones... it's happily building update 103 of 830 as I type this... but couldn't binaries be useful to push through circular deps etc.?
Maybe not, I don't think I've used the binary repos myself yet so don't know... but if binaries can be useful in that way then maybe it's not unreasonable to suggest that a comprehensive binary repo (with all the stable packages, regardless of size) could be useful even if many of the packages have trivial compile times. In fact I'd have thought that going systematically through the stable tree building everything would be easier to automate than figuring out what to build and what to skip. I don't know: all I know is that the binaries give me a sense of security that I can potentially fix an urgently needed package without restoring from a backup... I'm much more concerned with stability and resilience to updates than with compilation times. |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2420
|
Posted: Wed Sep 11, 2024 1:18 pm Post subject: |
|
|
psycho wrote: | I was thinking perhaps they could help with broken systems... but couldn't binaries be useful to push through circular deps etc.? |
Yes! It might have its specifics, but it would save a lot of trouble.
psycho wrote: | Maybe not, I don't think I've used the binary repos myself yet so don't know... but if binaries can be useful in that way then maybe it's not unreasonable to suggest that a comprehensive binary repo (with all the stable packages, regardless of size) could be useful even if many of the packages have trivial compile times. In fact I'd have thought that going systematically through the stable tree building everything would be easier to automate than figuring out what to build and what to skip. I don't know: all I know is that the binaries give me a sense of security that I can potentially fix an urgently needed package without restoring from a backup... I'm much more concerned with stability and resilience to updates than with compilation times. |
You've said it yourself:
psycho wrote: | Fortunately it turned out to be pretty straightforward to resolve all the issues just by fixing a dozen or so key ones... |
It's the same with binary packages. Take care of the key ones and you won't worry about the others. Or at least you'll worry less. And the selection is pretty good, a system like mine nearing 1700 packages can use 2/3 binary packages!
Code: | 1521 reinstalls, 1119 binaries |
Current coverage is pretty good and over time it's getting even better!
Best Regards,
Georgi |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20485
|
Posted: Wed Sep 11, 2024 6:24 pm Post subject: |
|
|
nikodll wrote: | Maybe I was not expressing myself very clear | Thank you for clarifying. I now understand that post. Although I'm still confused about wanting to use Gentoo but not wanting to build even small packages. But I also haven't used Calculate. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
eschwartz Developer
Joined: 29 Oct 2023 Posts: 221
|
Posted: Thu Sep 12, 2024 3:20 pm Post subject: |
|
|
psycho wrote: | Is it only the size/time of package compilation that matters? I was thinking perhaps they could help with broken systems... am updating an ancient one (old profile and many hundreds of outdated packages) and was seriously tempted (when I saw the massive piles of conflicts and other issues) to try updating world with all the available binaries just so that they'd already be in place to resolve problems before attempting compilation. Fortunately it turned out to be pretty straightforward to resolve all the issues just by fixing a dozen or so key ones... it's happily building update 103 of 830 as I type this... but couldn't binaries be useful to push through circular deps etc.? |
Binary packages do not need to satisfy build dependencies, which means that the dependency graph is simpler and can resolve conflicts that would otherwise not be able to be satisfied.
Circular dependencies are harder because if they are RDEPEND then they still cause a block.
psycho wrote: | Maybe not, I don't think I've used the binary repos myself yet so don't know... but if binaries can be useful in that way then maybe it's not unreasonable to suggest that a comprehensive binary repo (with all the stable packages, regardless of size) could be useful even if many of the packages have trivial compile times. In fact I'd have thought that going systematically through the stable tree building everything would be easier to automate than figuring out what to build and what to skip. I don't know: all I know is that the binaries give me a sense of security that I can potentially fix an urgently needed package without restoring from a backup... I'm much more concerned with stability and resilience to updates than with compilation times. |
I would love this! I am not sure it's practical though. The current builder setup installs a lot of packages all at the same time, and runs a world update once a day to build updates for all those packages. Systematically going through the tree building all packages is a nice idea, but even relying on existing binpackages to avoid much of the compilation, portage is particularly fast at merging. Just look at how long it takes to merge a set of acct-*/ packages. Doing a world update in an existing container means a lot of that work can be skipped, but it also means there's a lot more room for conflicts to appear.
I did implement it anyway: https://gitweb.gentoo.org/proj/binhost.git/commit/?id=d06884c8ce125cf6ee217afbb38cdb5b3fe1d101
It builds one package at random with heavy use of autounmasking, to augment all the permanent packages we always build. I believe there's meaningful overhead for this and doing it 10k times without batching is impractical, but I intend to play with the parameters a bit. |
|
Back to top |
|
|
|