| View previous topic :: View next topic |
| Author |
Message |
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1236
Location: Richmond Hill, Canada
|
 Posted: Tue Sep 17, 2024 10:47 am Post subject: |
 |
|
| lars_the_bear wrote: | | I've heard reference to 'stage 4' images and, although I don't fully understand what that entails, perhaps that's what I need rather than just a binhost? But I guess that's not going to help me with reducing the burden of regular updates. |
"Stage 4" is reference to use Gentoo's tool "Catalyst" to prepare a root filesystem with your selection of packages. In the build processes you will also get binary packages (optional) for entire build.
However I have feeling this is not the kind of thing you want to do. Because this usually take several days or weeks to complete the processes. The benefit is that you get a rootfs that is kind like Gentoo stage3 tarball except the tatball also have your exact target setup. i.e. /etc/portage/* + all-selected-packages and because the process take very long time so by design it sort of stop "rolling" natural of Gentoo until you finish the entire process.
A brief guide for Catalyst stages |
|
| Back to top |
|
 |
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 11:51 am Post subject: |
|
|
Aargh! I've just realized that this profile change doesn't just take effect during installation. I have (again) completely misunderstood how Gentoo works. I thought that I was having a problem when setting up a new machine, because its profile defaults didn't match the binhost I had created. But, in fact, I've just updated a different machine and found that it, too, wants to recompile qtwebengine and qtmultimedia because USE=qml has changed.
Sigh. I'm not sure I will ever understand Gentoo.
BR, Lars. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9262
|
| Posted: Tue Sep 17, 2024 11:56 am Post subject: |
|
|
I'm really not sure how else you could have understood what has been talked about here.
I mean, just think about the insane fragmentation caused if this only affected "starters". Gentoo would be completely unmaintainable for developers and impossible to administer for users.
Profile changes are effective immediate and global. And, I'm repeating myself: Stop overthinking it regarding minimalism; especially when you depend on official binpkgs, do not go against the flow, or it will hurt you continuously. If you don't have local USE overrides set, and the binhost lags a bit behind the latest profile change, just wait another day. The more USE overrides you have, the more you'll have to look into why it is that you can't use a binpkg. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 12:13 pm Post subject: |
|
|
| asturm wrote: | I'm really not sure how else you could have understood what has been talked about here.
|
Don't underestimate my ignorance 
The idea that profile defaults could change at any time, with no warning, just seems crazy to me. I guess it must make sense to the maintainers but, jeez...
The effect is that any part of all my Gentoo installations could need to be updated, at any time, even if no actual packages have changed. The changes can be extensive and, worse, they could preclude my use of my own binhost and and the official Gentoo binhost (which my configuration means I get little use of, even in ideal conditions). This is exactly what happened in this case.
I can only imagine that my use of Gentoo is just completely different from everybody else's, and totally out of line with the way the Gentoo maintainers expect it to work. I don't run the kind of hardware that can routinely compile monsters like qtwebengine. I thought I had this situation kind-of under control, but it's clear now that I don't.
BR, Lars. |
|
| Back to top |
|
|
Nowa
Developer
Joined: 25 Jun 2014
Posts: 429
Location: Nijmegen
|
| Posted: Tue Sep 17, 2024 12:16 pm Post subject: |
|
|
| Quote: | | The changes can be extensive and, worse, they could preclude my use of my own binhost and and the official Gentoo binhost |
No it does not, as we've already explained if you want the ensure that one machine is always able to compile binpkgs for another machine then you should ensure that the state of the repository is the same on both machines.
_________________
OS: Gentoo 6.10.12-gentoo-dist, ~amd64, 23.0/desktop/plasma/systemd
MB: MSI Z370-A PRO
CPU: Intel Core i9-9900KS
GPU: Intel Arc A770 16GB & Intel UHD Graphics 630
SSD: Samsung 970 EVO Plus 2 TB
RAM: Crucial Ballistix 32GB DDR4-2400 |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 12:36 pm Post subject: |
|
|
| AndrewAmmerlaan wrote: |
No it does not, as we've already explained if you want the ensure that one machine is always able to compile binpkgs for another machine then you should ensure that the state of the repository is the same on both machines. |
That's not the point, I think. I spent weeks-to-months setting up a machine to use as a binhost, and I just wanted to use those binaries indefinitely, until I really, really needed to use a later version of something. I want (need?) to keep the kernel and system-level libraries up to date, but I don't feel any need to update the end-user applications I actually use. They don't update very often, anyway.
Keeping the repositories in sync means (IIUC) that I could use one machine to build binaries for another now. But that isn't what I want -- I want to just keep using the binaries I built over that long time period, until I need later versions.
I can use package.mask and whatnot to prevent packages being upgraded from the versions I have built, and which I know work for me. But if a profile setting changes, then emerge wants to rebuild some of them even though the version is the same. I can work around this by fiddling the USE-flags on a per-package basis. But the implications are hard for me to assess, and it's a nuisance.
I guess none of this is an issue if you're OK with non-stop compiling, or you have the kind of hardware that does it quickly.
BR, Lars. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9262
|
| Posted: Tue Sep 17, 2024 12:51 pm Post subject: |
|
|
| lars_the_bear wrote: | | That's not the point, I think. I spent weeks-to-months setting up a machine to use as a binhost, and I just wanted to use those binaries indefinitely, until I really, really needed to use a later version of something. I want (need?) to keep the kernel and system-level libraries up to date, but I don't feel any need to update the end-user applications I actually use. They don't update very often, anyway. |
If that was your idea, you have much more to worry about than profile changes: slot rebuilds due to ABI/API breaks. You are asking Gentoo to do things for you that are simply incompatible with rolling release.
| lars_the_bear wrote: | | I guess none of this is an issue if you're OK with non-stop compiling, or you have the kind of hardware that does it quickly. |
You'll decide how often you sync. In general, doing it more often is advisable to get security updates, especially for browsers. Neglecting that, you can also sync just once a year. But you'll be an annual thread starter in these forums because the amount of changes accumulating over a year can be massive, and solving the resulting dependency resolution issues can be daunting especially if you are not trained by doing it more frequently. There are a few of such candidates in these forums; and the problems they need help with solving are always technically the same. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 1:17 pm Post subject: |
|
|
| asturm wrote: | | But you'll be an annual thread starter in these forums because the amount of changes accumulating over a year can be massive, and solving the resulting dependency resolution issues can be daunting especially if you are not trained by doing it more frequently. |
Actually, I had planned to be an annual reinstaller. My plan was to get my machines all set up the way I wanted, then keep using them until I needed a new version of something that was incompatible, then reinstall from scratch. That's pretty-much how I use Fedora and Ubuntu, anyway.
I thought using a shared, private binhost was a reasonable way to set that up, and perhaps it would have been if I had set up all the machines within days of one another. My new plan is simply to rsync the whole root filesystem from the working system to the new one, and just hack on the things that have to be changed. It's ugly, but I suspect it's no more error-prone that the hacks I'm already using to control the amount of compilation I have to endure. And, after all, I'm copying from a system that works perfectly well for me.
I've learned the hard way that there's simply no practical way I can keep a Gentoo installation remotely up to date, with the kind of hardware I have.
BR, Lars. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9262
|
| Posted: Tue Sep 17, 2024 1:28 pm Post subject: |
|
|
Sounds painful. You could have set up your systems any time you like. They just need to rsync from your binhost instead of Gentoo mirrors, as you were already told.
You do you, I guess. But do know that I am running Gentoo from binpkgs on an Intel Core m7-6Y75 laptop with 1.2 GHz and no active cooling. That is slow even by 2010 standards, yet for some reason it can do the things you say your 2015 hardware can't, like running Plasma 6 Wayland without losing any sleep about minimising every last USE flag. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 1:57 pm Post subject: |
|
|
| asturm wrote: | | That is slow even by 2010 standards, yet for some reason it can do the things you say your 2015 hardware can't, like running Plasma 6 Wayland without losing any sleep about minimising every last USE flag. |
Can you run FreeCAD on it? Or Darktable? Or KiCAD? I've only been able to make these things run effectively on my old laptops by stripping the Linux installation to bare bones. The irritating thing, as I suspect I might have complained before, is that these applications worked perfectly well on these machines with Ubuntu in 2015. If that were not the case, I might have concluded that what I was trying to do was unreasonable.
BR, Lars. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9262
|
| Posted: Tue Sep 17, 2024 2:02 pm Post subject: |
|
|
| That's not what it was made for, and micro-optimisations won't get me there either. I could try, but I'm not really interested in these packages. The main performance difference to Ubuntu or Fedora will not lie in the way you build your packages but the lack of default enabled runtime components or modern way of package encapsulation that aren't there on a Gentoo system in the first place unless you go through the trouble of it. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 2:09 pm Post subject: |
|
|
| asturm wrote: | | I could try, but I'm not really interested in these packages. |
Fair enough; no reason you should be. But I hope you'll understand that I am interested in those packages and, in fact, they're what I use a computer for. Everything else I can do on my cellphone.
And, as I said, with sufficient care I can make these things work effectively under Gentoo. It's just very painful.
BR, Lars.
Last edited by lars_the_bear on Tue Sep 17, 2024 2:14 pm; edited 1 time in total |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9262
|
| Posted: Tue Sep 17, 2024 2:11 pm Post subject: |
|
|
| Did you ever try running them in Gentoo without going full on minimalist? |
|
| Back to top |
|
|
Nowa
Developer
Joined: 25 Jun 2014
Posts: 429
Location: Nijmegen
|
| Posted: Tue Sep 17, 2024 2:17 pm Post subject: |
|
|
| Quote: | | Actually, I had planned to be an annual reinstaller. My plan was to get my machines all set up the way I wanted, then keep using them until I needed a new version of something that was incompatible, then reinstall from scratch. That's pretty-much how I use Fedora and Ubuntu, anyway. |
I am confused, if you don't plan on updating these machines anyway, then why is it a problem if the profiles change? Simply get all machines in the same state now, build your binpkgs, and then don't update the system.
_________________
OS: Gentoo 6.10.12-gentoo-dist, ~amd64, 23.0/desktop/plasma/systemd
MB: MSI Z370-A PRO
CPU: Intel Core i9-9900KS
GPU: Intel Arc A770 16GB & Intel UHD Graphics 630
SSD: Samsung 970 EVO Plus 2 TB
RAM: Crucial Ballistix 32GB DDR4-2400 |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 2:24 pm Post subject: |
|
|
| AndrewAmmerlaan wrote: | | Quote: | | Actually, I had planned to be an annual reinstaller. My plan was to get my machines all set up the way I wanted, then keep using them until I needed a new version of something that was incompatible, then reinstall from scratch. That's pretty-much how I use Fedora and Ubuntu, anyway. |
I am confused, if you don't plan on updating these machines anyway, then why is it a problem if the profiles change? Simply get all machines in the same state now, build your binpkgs, and then don't update the system. |
That was my plan. But the profile changed between setting up the first and the second. So, by the time I can to set up the second, the binpkgs I had built were no longer compatible with the changed profile.
Had I known that the profile defaults could change, I might have done things differently. It simply never occurred to me for a moment that they might. Oh, well -- now I know.
BR, Lars. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9262
|
| Posted: Tue Sep 17, 2024 2:25 pm Post subject: |
|
|
| But *any number of* packages will also have changed dependencies, USE flags and available versions. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 2:29 pm Post subject: |
|
|
| asturm wrote: | | But *any number of* packages will also have changed dependencies, USE flags and available versions. |
Yeah. But the packages I really care about don't seem to show such changes very often. At least, that's been my experience so far. Maybe I've just been lucky, and my luck finally ran out.
BR, Lars. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Tue Sep 17, 2024 2:39 pm Post subject: |
|
|
| asturm wrote: | | Did you ever try running them in Gentoo without going full on minimalist? |
Fair point, and the answer is: no, I didn't. I did try simulating this in a VM, with the same memory settings, and with default USE-flags for the profile (I think it was desktop/gnome). The machine I run Darktable on has only 8Gb RAM, and it's not upgradeable. The results were, um... unspectacular.
But, to be fair, I don't know whether it would have been better, or worse, on real hardware. I can simulate a small amount of RAM, but I can't simulate a 2015 CPU on a 2024 machine. At least, I don't think I can. On the other hand, the real hardware doesn't have all the virtualization and what-not.
I've found that I can run Darktable on this machine under Ubuntu if I strip it right down. That is, run Xfce4 as a desktop, get rid of Pulse, background updaters, file indexing stuff, and any unused systemd bits. This is a real hack. I was originally drawn to Gentoo because I thought it would provide an elegant, supportable way to do the same thing. Which it does -- if you don't mind compiling a lot.
But you make a perfectly good point: I really ought to try it.
BR, Lars. |
|
| Back to top |
|
|
eschwartz
Developer
Joined: 29 Oct 2023
Posts: 214
|
| Posted: Tue Sep 17, 2024 3:05 pm Post subject: |
|
|
| lars_the_bear wrote: | Thank you all for the responses, but I remain completely confused about the best way to proceed.
I've complained repeatedly about the amount of compilation involved in running Gentoo, on ten-year-old laptops. Some packages take literally days to compile, and my choice of global USE flags (particularly -pulseaudio -wayland) means that the Gentoo official binhost is often not of much help.
So I thought I could build all the packages I want on one machine, and use it as a binhost for others. They all have similar hardware and the same USE settings. I really expected this to relieve me of a load of compilation.
And it worked fine, until yesterday, when I tried to install FreeCAD on a machine I was setting up. This required an installation of qtwebengine, which I'd already spent nearly four days building a couple months ago. So I really expected to be able to use my pre-existing binary on this new installation. |
It should work fine in general, yes, but... I thought the original idea for you was to use the gentoo official binhost? You can ALSO use your own laptop to run a second binhost, and have your second laptop use both binhosts. That way you have *two* choices of pre-existing binaries to use. Two is better than one. For something like qtwebengine it feels worth it to just tweak USE flags so that you get the existing binpackage, even if you have to tweak those USE flags once or twice in the future as well.
| lars_the_bear wrote: |
But I couldn't, because the binary was built on a system that did not have the 'qml' USE-flag set. My new installation -- based on the same profile and global settings as the old one -- does have this flag set. It wasn't set by me -- @eschwarz confirms that this was a deliberate change in the profile. I worked around the problem by adding USE=-qml to qtwebengine and, although I'm not sure what the implications are, FreeCAD seems to work fine, albeit using a version of qtwebengine from an incompatible set-up. I don't use Qt for anything else than FreeCAD, so I don't really care how broken it is in other respects.
|
The consequences of modifying a USE flag for your personal use are never a "broken system". Portage will refuse to install if it would break other packages.
By disabling the qml USE flag for your personal use, you may end up in a situation where installing other packages that you said you aren't installing anyway, would depend on qml and cause a rebuild of the entire qt stack. But you aren't using those packages, so it's fine.
This is independent of whether it's a good generic default for the average user.
| lars_the_bear wrote: |
I'm not sure whether what I've seen here -- unexpected profile changes -- is a weird and rare fluke, or something I have to be prepared to deal with on a regular basis. If profile USE flags can change at any time without warning, then the idea of using one machine as a binhost for later installations starts to look unattractive. I really did think this was going to help with my compilation burden, but perhaps not.
|
It *is* uncommon. It's just not "once every 5 years" uncommon. More like once every few months, and cases where programming languages such as python or ruby change the default major implementation are gonna happen anyway regardless of whether it's done by a profile. For example, python updates to a new feature version once a year, and gentoo updates the default python once a year too (not immediately). But you also get an "eselect news" post telling you about the change and how to defer it if you want.
Ultimately, Gentoo is not an LTS distro. A stable Fedora or Debian release has no major changes once you install, no matter what. Gentoo, as a rolling release distro, does have changes at some point. You have a degree of flexibility in choosing to accept or defer those changes (for example, by customizing USE flags or PYTHON_TARGETS defaults) but Gentoo moves on and doesn't just backport a handful of security patches to the set of packages from 2018.
There's no reason someone couldn't maintain a Gentoo derivative that does precisely this LTS concept, if they wanted to. https://archive.fosdem.org/2015/schedule/event/providing_an_lts_distro_with_gentoo_prefix/
Maintaining a distro is quite a bit of work and I don't know that anyone is currently doing this publicly, unfortunately. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22618
|
| Posted: Tue Sep 17, 2024 3:13 pm Post subject: |
|
|
| Performance tests in a VM can be highly misleading, depending on how the VM is configured. There are graphics-intensive games that I could run well on real hardware years ago, which struggle today on a VM hosted by modern hardware, if the VM isn't configured to do all the right graphics tricks. Similarly, I/O-bound programs can be horrible in a VM that is not using a good configuration for its virtual block devices. |
|
| Back to top |
|
|
wjb
l33t
Joined: 10 Jul 2005
Posts: 629
Location: Fife, Scotland
|
| Posted: Tue Sep 17, 2024 3:22 pm Post subject: |
|
|
Sorry, I may have missed it in all the threads, but have we established why a 6th gen i7 with 9G ram (or whatever) is taking such a ridiculous amount of time to build qtwebengine? Because that's what seems to me to be driving all this effort to avoid building stuff.
Bill |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Wed Sep 18, 2024 7:40 am Post subject: |
|
|
| wjb wrote: | Sorry, I may have missed it in all the threads, but have we established why a 6th gen i7 with 9G ram (or whatever) is taking such a ridiculous amount of time to build qtwebengine? Because that's what seems to me to be driving all this effort to avoid building stuff.
|
I think because it's a laptop, and the CPU has to throttle right down to control the heat. But I'm not sure. But I think qtwebengine is generally a tedious thing to compile, from previous discussions. I don't think I'm the only Gentoo user who finds it a nuisance.
BR, Lars. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9262
|
| Posted: Wed Sep 18, 2024 7:51 am Post subject: |
|
|
| You *think* so, but you have direct control over how much load and memory pressure you put onto your machine via emerge. And if the burden on memory is too big, swapping will cause a huge increase in build time. |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 517
|
| Posted: Wed Sep 18, 2024 8:05 am Post subject: |
|
|
| eschwartz wrote: |
It *is* uncommon. It's just not "once every 5 years" uncommon. More like once every few months, and cases where programming languages such as python or ruby change the default major implementation are gonna happen anyway regardless of whether it's done by a profile. For example, python updates to a new feature version once a year, and gentoo updates the default python once a year too (not immediately). But you also get an "eselect news" post telling you about the change and how to defer it if you want.
|
Thanks. I think I began my Gentoo journey with a really fundamental misconception about updates. I expected to have to compile everything at the start, and I anticipated this taking a long time (which it did). I thought that maybe a new kernel would come along every few months, and I'd have to recompile that. And maybe some changes to system libraries at similar intervals. I always anticipated that the software packages I actually use, I would fix at the installed versions, and never update, unless they stopped working.
What I didn't anticipate was that there would be an update to something substantial every few days. At least, that's what I'm finding at present. Maybe I should have anticipated this -- I don't know.
My chosen USE configuration limits my use of the Gentoo official binhost, although there have been some improvements in this area. So I maintain my own binhost, which I based on the first machine I set up, in the hope that this would supply binaries to my other machines. What I didn't anticipate was that these binaries would become out of date so quickly.
So, on the whole, keeping Gentoo up to date has been a battle, and it's a battle I don't think I can win, because I'm chasing my own tail the whole time (sorry, that's a mixed metaphor). It's a shame, because Gentoo comes the closest to meeting my needs, of all the various Linux distributions I've tried. So when I've got this latest machine working as I want it, I will rsync it to my other machines, and make no further updates. If it gives me a year of use before I have to reinstall from scratch, that's fair enough.
BR, Lars. |
|
| Back to top |
|
|
wjb
l33t
Joined: 10 Jul 2005
Posts: 629
Location: Fife, Scotland
|
| Posted: Wed Sep 18, 2024 10:01 am Post subject: |
|
|
| They're reasonable PCs, they should not be throttling just because they're being worked a bit. Have they been dusted out recently? |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
