View previous topic :: View next topic |
Author |
Message |
ipic Guru
Joined: 29 Dec 2003 Posts: 377 Location: UK
|
Posted: Sun Mar 24, 2024 4:32 pm Post subject: |
|
|
There used to be a group - @system - that was intended for this.
Seriously - forcing a rebuild on a system where it would take the best part of a 24 hour day should never be an option.
"Enforcing" things has never been the Gentoo way either. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 21724
|
Posted: Sun Mar 24, 2024 5:11 pm Post subject: |
|
|
No, @system is the set of packages that all ebuilds can assume are installed. It may contain non-program packages, such as virtuals, which do not need to be rebuilt here. It usually will omit at least some packages that deserve a rebuild on a toolchain improvement, because those packages are not system critical. Thus, it fails in both directions. sam_'s comment about searching the VDB is what I had in mind, and yes, that is a somewhat ugly approach, but I cannot think of an alternative that will (1) find all packages which can benefit, (2) exclude all packages that clearly cannot benefit (documentation, scripts), (3) avoid trying to rebuild things which are not packages (/usr/local/bin/ programs). It's still not perfect, since it will flag for inclusion things which distribute ELF files prebuilt upstream (such as firefox-bin), but it excludes quite a bit that --emptytree would include. It might be possible to abuse some of the ebuild's QA_ variables to identify prebuilt files and exclude them from the package's effective contents before deciding whether it ships anything worth rebuilding.
As for forcing - users are of course free to ignore the recommended path. This path is recommended because it is the simplest way to ensure that all packages which can benefit are rebuilt so that they do benefit. Identifying specific packages out of the VDB would be nice, but I expect that most packages excluded that way are cheap to build. If you have a long total build time, it is likely because you have many packages which are compiled by your toolchain, so excluding the unnecessary ones will not help you much. It would mainly be a savings on write cycles for limited life storage devices. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54317 Location: 56N 3W
|
Posted: Sun Mar 24, 2024 5:19 pm Post subject: |
|
|
ipic,
Nothing is enforced, only recommended. The further you stray from the beaten path, the less sympathy and help you are likely to get. That's not a reason to not do it, as long as you understand you can keep the pieces if it breaks.
I still use a static /dev, separate /usr and root in LVM, all without an initrd. I'm old any cynical, I know that that when it breaks, I won't even get a shoulder to cry on. :)
I had to add an initrd to mount /usr the other day but I was warned beforehand, so when it broke, it was just another entry on the syslinux menu.
"Because I can" is an excellent reason to do something with Gentoo :) _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
ipic Guru
Joined: 29 Dec 2003 Posts: 377 Location: UK
|
Posted: Sun Mar 24, 2024 5:30 pm Post subject: |
|
|
Hu wrote: | No, @system is the set of packages that all ebuilds can assume are installed. |
So how about coming up with a different one aimed at this problem - like @toolchain for example.
sam_ wrote: |
.....
I can also imagine that if we get to the point where we can make CET enforced by default, it might be required, but that's a long way off.
....
|
Looks a lot to me like enforcing is on the table. |
|
Back to top |
|
|
sam_ Developer
Joined: 14 Aug 2020 Posts: 1685
|
Posted: Sun Mar 24, 2024 5:31 pm Post subject: |
|
|
Enforcing has a specific meaning wrt security measures. In this case, it means "[allowing users to] ask glibc to enforce it" (and testing that a lot, and fixing any breakage, and documenting it), not enforcing it upon all users (i.e. making users use CET whether they like it or not).
That is, enforcing of the measure onto applications, not onto users. Why would I want to "enforce" anything on our users?
I have no idea if we could get to a point where that was done on all CPUs without opting-in at any point, and even then, it's not like it'd be mandatory. |
|
Back to top |
|
|
ipic Guru
Joined: 29 Dec 2003 Posts: 377 Location: UK
|
Posted: Sun Mar 24, 2024 5:40 pm Post subject: |
|
|
NeddySeagoon wrote: | ipic,
Nothing is enforced, only recommended. The further you stray from the beaten path, the less sympathy and help you are likely to get. That's not a reason to not do it, as long as you understand you can keep the pieces if it breaks.
I still use a static /dev, separate /usr and root in LVM, all without an initrd. I'm old any cynical, I know that that when it breaks, I won't even get a shoulder to cry on.
I had to add an initrd to mount /usr the other day but I was warned beforehand, so when it broke, it was just another entry on the syslinux menu.
"Because I can" is an excellent reason to do something with Gentoo |
Exactly so.
What I'm finding confusing about this is that people on this thread really seem to think that one can get support from Gentoo when something breaks.
This has never been the case. If an e-build fails to build, one can submit a bug that may, eventually, be looked at by someone, who is 90% likely to tell you it's your problem. In the mean time, you find patches and sort it yourself.
If a runtime fails to work - looking for Gentoo support is by definition pointless - upstream is where the help or otherwise will come from.
And, packages such as Freecad, Kicad, Libreoffice, Chromium, anything to do with Qt, etc, etc take many, many hours to *each* to build. Not sure where the "you wouldn't save much time over --emptytree" comes from either. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 21724
|
Posted: Sun Mar 24, 2024 6:12 pm Post subject: |
|
|
ipic wrote: | So how about coming up with a different one aimed at this problem - like @toolchain for example. | That's exactly what I was wishing for above. However, it doesn't exist today. ipic wrote: | What I'm finding confusing about this is that people on this thread really seem to think that one can get support from Gentoo when something breaks.
This has never been the case. If an e-build fails to build, one can submit a bug that may, eventually, be looked at by someone, who is 90% likely to tell you it's your problem. In the mean time, you find patches and sort it yourself. | Support means different things to different people. Nobody expects that they can get a helpful response by a guaranteed deadline after submission, as might be available with a commercial support contract issued by a major vendor. Reports about build failures may or may not get aid here or on the bug tracker. As a notable counterexample, I refer you to Forced to update the old system, where a concerted effort from multiple forum posters managed to update a system that had been neglected for years, and the system owner was not able to solve the update problems unaided. ipic wrote: | If a runtime fails to work - looking for Gentoo support is by definition pointless - upstream is where the help or otherwise will come from. | That depends on why it fails to work. If Gentoo broke it with a bad customization, Gentoo may be the place to see it fixed. In some cases, the Gentoo developer will have a better understanding of the package and its upstream than you do, and can facilitate seeking upstream help. ipic wrote: | And, packages such as Freecad, Kicad, Libreoffice, Chromium, anything to do with Qt, etc, etc take many, many hours to *each* to build. Not sure where the "you wouldn't save much time over --emptytree" comes from either. | All of those are things you build with the toolchain, so a hypothetical @toolchain would rebuild them, just as --emptytree @world does rebuild them. The @toolchain target would skip rebuilding Python documentation, reinstalling pure-Perl packages, etc. That saves a bit of time, but not much, because most of the time is spent emerging the packages that @toolchain will choose to rebuild. |
|
Back to top |
|
|
ipic Guru
Joined: 29 Dec 2003 Posts: 377 Location: UK
|
Posted: Sun Mar 24, 2024 7:09 pm Post subject: |
|
|
ipic wrote: | Take a note of all CHOST related settings as per the wiki page on changing CHOST
Follow the instructions up to step 15.
Check the CHOST settings as above, and confirm that NOTHING HAS CHANGED.
If that is the case, instead of an --emptytree rebuild, do this:
Code: |
emerge -uNDpv --with-bdeps y @world
emerge -uND --with-bdeps y --jobs --keep-going --quiet-fail y @world
|
reboot
|
Just to report that I have used this approach on my bare metal machine - the one with the 2000 odd packages.
The approach resulted in around 40 packages being emerged. GCC was the one that took the longest, and avoiding doing it twice saved a lot.
The points that I could not test elsewhere were the kernel + modules boot (it worked) and all the packages with MULTILIB - of which I have a metric ton.
Pleased to report that system rebooted clean.
All QEMU VMs started clean
Things that need MULTILIB are all running as before (Wine mainly).
Apps I regularly use all running clean.
So, I'm going to sign off now. Some people here do not want to see the problem, or do anything about it - which is their right, but talking to them just pisses me off. So, bye. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 21724
|
Posted: Sun Mar 24, 2024 7:27 pm Post subject: |
|
|
That only built packages which were out of date or which had a pending USE flag change. Per earlier remarks in the thread, any packages for which you have the current version, but which were built with an older toolchain, have not been rebuilt and are not benefiting from the improvements built into the new profile. You will eventually rebuild those with the new toolchain as updates become available, but for some projects, that could be quite a ways in the future. |
|
Back to top |
|
|
UlvHare n00b
Joined: 09 Sep 2015 Posts: 20 Location: USSR
|
Posted: Sun Mar 24, 2024 8:02 pm Post subject: |
|
|
Do I need to move to merged-usr ( Code: | [27] default/linux/amd64/23.0/desktop/plasma (stable) | ) from my current Code: | [8] default/linux/amd64/17.1/desktop/plasma (exp) | if I have /usr in the same partition as /? Are there any benefits in it for "normal user"? I mean not a developer, just work (R, QGIS) and enjoy ebooks and films. But widely using ebuilds from zugaina. |
|
Back to top |
|
|
flysideways Guru
Joined: 29 Jan 2005 Posts: 438
|
|
Back to top |
|
|
flysideways Guru
Joined: 29 Jan 2005 Posts: 438
|
Posted: Sun Mar 24, 2024 8:27 pm Post subject: |
|
|
Four of the six provided links are broken here
Code: | [1] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_transition
[2] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_timeline
[3] https://www.gentoo.org/support/news-items/2019-06-05-amd64-17-1-profiles-are-now-stable.html
[4] https://www.gentoo.org/support/news-items/2022-12-01-systemd-usrmerge.html
[5] https://wiki.g.o/wiki/Project:Toolchain/23.0_update_table
[6] https://wiki.g.o/wiki/Changing_the_CHOST_variable#Verifying_things_work |
from https://wiki.gentoo.org/wiki/Project:Toolchain/23.0_update_instructions .
Otherwise, the instructions were adequate for me to follow to a successful resolution.
Thanks @Gentoo Devs |
|
Back to top |
|
|
houtworm Guru
Joined: 08 Mar 2003 Posts: 391 Location: Den Haag, Netherlands
|
Posted: Sun Mar 24, 2024 8:46 pm Post subject: |
|
|
I am almost at the end of the news message about this topic. Now I have to rebuild world. I have a lot of binary packages because this pc also builds for other pc's. Can I still use these binary packages, or do I have to rebuild everything? _________________ niemand is onbekwamer, dan een timmerman zonder hamer
Kees |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 21724
|
Posted: Sun Mar 24, 2024 8:49 pm Post subject: |
|
|
UlvHare wrote: | Do I need to move to merged-usr ( Code: | [27] default/linux/amd64/23.0/desktop/plasma (stable) | ) from my current Code: | [8] default/linux/amd64/17.1/desktop/plasma (exp) | if I have /usr in the same partition as /? Are there any benefits in it for "normal user"? I mean not a developer, just work (R, QGIS) and enjoy ebooks and films. But widely using ebuilds from zugaina. | As I explained above, the filesystem on which /usr resides is the "separate /usr" discussion. The "split /usr" vs "merged /usr" discussion is about whether /bin is a symlink to /usr/bin. As to whether you should move, I suggest that you follow the upgrade table, and handle "merged /usr" when the Gentoo developers explicitly instruct you to do so, and not sooner. |
|
Back to top |
|
|
CaptainBlood Advocate
Joined: 24 Jan 2010 Posts: 3638
|
Posted: Sun Mar 24, 2024 10:25 pm Post subject: |
|
|
flysideways wrote: | Four of the six provided links are broken here
Code: | [1] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_transition
[2] https://wiki.g.o/wiki/Project:Toolchain/23.0_profile_timeline
[3] https://www.gentoo.org/support/news-items/2019-06-05-amd64-17-1-profiles-are-now-stable.html
[4] https://www.gentoo.org/support/news-items/2022-12-01-systemd-usrmerge.html
[5] https://wiki.g.o/wiki/Project:Toolchain/23.0_update_table
[6] https://wiki.g.o/wiki/Changing_the_CHOST_variable#Verifying_things_work | from https://wiki.gentoo.org/wiki/Project:Toolchain/23.0_update_instructions .
Otherwise, the instructions were adequate for me to follow to a successful resolution.
Thanks @Gentoo Devs | Indeed
The "wiki.g.o" should be replaced by "wiki.gentoo.org".
Maybe done on purpose?
However unconfortable imo.
Thks 4 ur attention, interest & support. _________________ USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
Back to top |
|
|
mkyral Apprentice
Joined: 06 May 2007 Posts: 185 Location: Czech Republic
|
Posted: Sun Mar 24, 2024 11:03 pm Post subject: |
|
|
It looks like openrc binpkg installs binaries to wrong place (/usr/sbin) and kernel can't find /sbin/openrc to run and boot fails. I had to force portage to compile openrc package. Then correct folders are used and my system boots again.
I'm on default/linux/amd64/23.0/split-usr/desktop/plasma profile, split-usr USE flag is set, but there is no such flag on openrc. |
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 2965 Location: Edge of marsh USA
|
Posted: Mon Mar 25, 2024 4:39 am Post subject: |
|
|
Sam_, your explanations and your patience are greatly appreciated. I haven't found the instructions difficult to follow. But, I have a question.
On an old x86 openrc system (CHOST="i686-pc-linux-gnu" with an AMD Phenom CPU) migrating from [5] to [28], and having just compiled binutils, gcc, and glibc with no CHOST or version changes, would it be sound to use selected excludes in the --emptytree rebuild in order to speed things up.
Code: | emerge --ask --emptytree --exclude "sys-devel/gcc sys-devel/binutils sys-libs/glibc sys-kernel/linux-firmware sys-kernel/gentoo-sources sys-kernel/linux-headers sys-apps/baselayout dev-build/libtool" @world |
On this system, with no binary packages or other complications, just rebuilding gcc takes two hours, so I can probably save quite a few electrons with those excludes.
Finally, after migration, shouldn't the kernel be rebuilt also?
ADDED: Notes at 15:04 20240326
1. I did recompile the kernel because it just seemed wrong to not do so.
2. I did use all the excludes noted above.
3. After 24 hours doing emerge --emptytree, it failed on Audacity. I did emerge --skipfirst --resume and the remainder finished in another two hours.
4. Rebooted, did emerge --sync and normal updates and everything appears to be as it should. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi
Last edited by figueroa on Tue Mar 26, 2024 7:04 pm; edited 1 time in total |
|
Back to top |
|
|
e8root Tux's lil' helper
Joined: 09 Feb 2024 Posts: 77
|
Posted: Mon Mar 25, 2024 8:30 am Post subject: |
|
|
Great article.
Good to know I need to rebuild with --emptytree after changing CFLAGS. Numbers of packages reported by neofetch and when emerging world didn't add up... now with --emptytree they do
Emerging world with --emptytree seems like it will take some more hours compared to normal world because number of packages increased drastically but it also looks like these mostly are small packages that sit on configure phase most of the time and so overall CPU utilization is very low so not as bad as just chromium or libreoffice alone. _________________ Unix Wars - Episode V: AT&T Strikes Back |
|
Back to top |
|
|
UlvHare n00b
Joined: 09 Sep 2015 Posts: 20 Location: USSR
|
Posted: Mon Mar 25, 2024 5:40 pm Post subject: |
|
|
Hu wrote: | As to whether you should move, I suggest that you follow the upgrade table, and handle "merged /usr" when the Gentoo developers explicitly instruct you to do so, and not sooner. |
Thank you very much! |
|
Back to top |
|
|
logrusx Veteran
Joined: 22 Feb 2018 Posts: 1584
|
Posted: Mon Mar 25, 2024 7:42 pm Post subject: |
|
|
sam_ wrote: | It can be done right now via grepping the VDB but that feels gross. |
I already finished the migration but still I think whoever wants to get their hands dirty should know how to do it, gross or not. Also I'm curious. Could you please tell me how?
Best Regards,
Georgi |
|
Back to top |
|
|
Havin_it Veteran
Joined: 17 Jul 2005 Posts: 1247 Location: Edinburgh, UK
|
Posted: Tue Mar 26, 2024 8:35 pm Post subject: |
|
|
figueroa wrote: | Sam_, your explanations and your patience are greatly appreciated. I haven't found the instructions difficult to follow. But, I have a question.
On an old x86 openrc system (CHOST="i686-pc-linux-gnu" with an AMD Phenom CPU) migrating from [5] to [28], and having just compiled binutils, gcc, and glibc with no CHOST or version changes, would it be sound to use selected excludes in the --emptytree rebuild in order to speed things up.
Code: | emerge --ask --emptytree --exclude "sys-devel/gcc sys-devel/binutils sys-libs/glibc sys-kernel/linux-firmware sys-kernel/gentoo-sources sys-kernel/linux-headers sys-apps/baselayout dev-build/libtool" @world |
On this system, with no binary packages or other complications, just rebuilding gcc takes two hours, so I can probably save quite a few electrons with those excludes.
|
This is what I'm planning to do. Seems no sense to re-emerge packages with no binaries to build. With that in mind I'm screening out some populous groups of non-binary ebuilds with some use of qlist from portage-utils:
Code: | emerge -eav --keep-going --exclude "gcc glibc binutils libtool linux-firmware linux-headers gentoo-sources intel-microcode baselayout $(qlist -IC -- -bin) $(qlist -IC acct-) $(qlist -IC virtual/)" @world |
(You may want to run those subcommands individually to ensure they don't include any compiling packages on your system.)
There are probably other categories you could target if you want to go all-out: PHP PEAR packages, if you have those; apps compiled entirely from langs other than GCC's (Java, OCaml, ...). Others I'm not sure about ... do any fonts make use of a compiler? Man pages? Probably not but I'm not certain. And past a certain point the time you spend identifying excludes outweighs the time you'd save
figueroa wrote: |
Finally, after migration, shouldn't the kernel be rebuilt also?
|
Sounds like it's not essential but like you I'll probably do it anyway. The decision taken must be applied also to any external modules of course (e.g. virtualbox-modules in my case).
figueroa wrote: |
3. After 24 hours doing emerge --emptytree, it failed on Audacity. I did emerge --skipfirst --resume and the remainder finished in another two hours.
|
I have had that build failing on me the last 2-3 (~monthly) upgrade cycles. I haven't needed the app in some time so haven't cared but maybe one of us should file a bug at some point, when there's nothing else to do |
|
Back to top |
|
|
thrashed Apprentice
Joined: 01 Sep 2004 Posts: 294
|
Posted: Tue Mar 26, 2024 9:23 pm Post subject: |
|
|
yesterday i migrated from "default/linux/amd64/17.1" to "default/linux/amd64/23.0"
i read https://www.gentoo.org/support/news-items/2024-03-22-new-23-profiles.html calmly and in detail.
and to be honest i skipped point 16. --> Rebuild world: "emerge --ask --emptytree @world".
instead of point 16, i just updated world, like i always did all the years and on this update the successfull build of sys-apps/baselayout-2.14-r2 broke my system. after this successfull build, every other emerge was not possible anymore (PORTAGE_BZIP2_COMMAND setting is invalid: 'bzip2' error messages and so on). i figured out, that /sbin and /bin were missing in my PATH variable. adjusting the PATH variabel in /etc/profile did not help, because every env-update freshly writes the PATH variable in /etc/profile. In /etc/env.d/50baselayout i found, that /sbin and /bin were missing in the PATH variable and in ROOTPATH variable. after adapting /sbin and /bin to the variables in /etc/env.d/50baselayout everyhting worked again.
actually i run the recommended "emerge --ask --emptytree @world", hope after this is everything done for the successfull profile change.
because I was very interested in it, i tried to figure out what caused my problem.
in baselayout-2.14-r2.ebuild you can read from line 198-204
Code: | if ! use split-usr && ! use prefix-guest; then
sed \
-e 's|:/usr/sbin:|:|g' \
-e 's|:/sbin:|:|g' \
-e 's|:/bin:|:|g' \
-i etc/env.d/50baselayout || die
fi |
do you know, if it was a bad coincidence and an emerge of sys-apps/baselayout-2.14-r2 would have broken my system anyway or did my "emerge -avuDN --with-bdeps y world" instead of the recommended "emerge --ask --emptytree @world" after changing my profile from 17.1 to 23.0 broke my system?
thank you all! |
|
Back to top |
|
|
pietinger Moderator
Joined: 17 Oct 2006 Posts: 4249 Location: Bavaria
|
Posted: Tue Mar 26, 2024 9:32 pm Post subject: |
|
|
thrashed wrote: | [...] do you know, if it was a bad coincidence and an emerge of sys-apps/baselayout-2.14-r2 would have broken my system anyway or did my "emerge -avuDN --with-bdeps y world" instead of the recommended "emerge --ask --emptytree @world" after changing my profile from 17.1 to 23.0 broke my system? |
I think none of both ... the reason of your problem was this:
thrashed wrote: | yesterday i migrated from "default/linux/amd64/17.1" to "default/linux/amd64/23.0" |
Coming from an OpenRC-split-usr profile needs switching to default/linux/amd64/23.0/split-usr ... you have done the migration to 23.0 AND the migration from split-usr to merged-usr together (=not good) ...
( See more here: https://wiki.gentoo.org/wiki/Merge-usr ) _________________ https://wiki.gentoo.org/wiki/User:Pietinger
Last edited by pietinger on Tue Mar 26, 2024 9:35 pm; edited 1 time in total |
|
Back to top |
|
|
GDH-gentoo Veteran
Joined: 20 Jul 2019 Posts: 1549 Location: South America
|
Posted: Tue Mar 26, 2024 9:34 pm Post subject: |
|
|
thrashed wrote: | yesterday i migrated from "default/linux/amd64/17.1" to "default/linux/amd64/23.0" |
You should have chosen default/linux/amd64/23.0/split-usr instead. Your original profile was a split /usr one. The 23.0 ones that don't have "split-usr" in the name have merged /usr by default.
That's explicitly mentioned in step #6 of the migration instructions. _________________
NeddySeagoon wrote: | I'm not a witch, I'm a retired electronics engineer |
Ionen wrote: | As a packager I just don't want things to get messier with weird build systems and multiple toolchains requirements though |
|
|
Back to top |
|
|
figueroa Advocate
Joined: 14 Aug 2005 Posts: 2965 Location: Edge of marsh USA
|
Posted: Tue Mar 26, 2024 9:38 pm Post subject: |
|
|
I suspect that you should have migrated to the split-usr profile:
Code: | [45] default/linux/amd64/23.0/split-usr (stable) |
If so, I think you will continue to have trouble until such time as you either migrate to non-split-usr layout or migrate to the correct profile.
ADDED: That was either easy, or great minds (to which I make no claim) think alike. _________________ Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|