Updating to GCC 3.4/4? Want to use 2006.1 profile? READ THIS

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Hi all,

(You are reading version 1.19 of this guide.)

ABSTRACT: How to recompile your entire system / toolchain with MINIMUM PROCESSING TIME EFFORT.

If you plan to do a major compiler update for your Gentoo box, such as from GCC 3.3 to GCC 3.4, or from GCC 3.3/3.4 to GCC 4, then read on. (This will typically be the case when upgrading your Gentoo installation to the new 2006.1 profile.)

The problem is that in those cases the C++ ABI of the GCC changes, which requires recompiling most of your system.

In other words: You have in fact to recompile your ENTIRE SYSTEM in order to be sure no "undesired" side-effects may arise from your compiler upgrade.

This is especially true when using KDE/QT and other programs/libraries which extensively make use of the C++ language.

And here is the reason for this posting: While I have found many postings how to best recompile your entire Gentoo system in such cases, or how to rebuild your "toolchain" best, I found none of them being optimal.

The problem: Actually all of them require much more time to execute then is really necessary.

Because in my opinion the amazing truth is: It is in fact UNNECESSARY to recompile any package, including the new GCC, more than exactly once when rebuilding your entire system! (For those who cannot believe this, see my argumentation in the related article https://forums.gentoo.org/viewtopic-p-3548628.html#3548628.)

Using my guide (and the accompanying utility script) presented here, you will be able to rebuild your entire Gentoo system with an absolute minimum of effort.

Compared to other guides which suggest recompiling the entire system up to 6 times "to be sure" (I will prove them wrong - their approach is not safer than my approach presented here, it only takes more time), my approach can save you days or even weeks of processing time!

And as weeks of unnecessary processing time equates days/weeks of unnecessary power consumption as well, this will certainly save you a few bucks on your next electicity bill.

I feel obliged to mention at this point that badpenguin has also written a guide how to recompile your entire system, taking about the same running time. Although not for the faint of heart (because it requires a stage 1 reinstallation of your system), I consider it to be a viable hardcore alternative to my guide. You can find it at http://badpenguins.com/gentoo-build-test/.

OK; let's be short: Here is my guide as of 2006-12-05:

===================

Recompile Entire System Helper

Copy the script from the end of this article and save it to a text file. (Go to the section "HOW TO OBTAIN THE SCRIPT" of this article to get it.) Be sure to follow the instructions below before you use it!

Script version as of $Date: 2006-09-26T04:22:55.640944Z $

Written in 2006 by Guenther Brunthaler

This script will generate another script to be run by you. That other script will then recompile each and every package of your whole current system in the correct order.

This will typically be required on a major GCC upgrade.

IMPORTANT: Do not execute this script before all of the following prerequisites are met:

Portage tree is up-to-date (emerge --sync)
Your system is up-to-date (emerge --ask --update --deep --newuse world)
gentoolkit is available. (The script uses it.) If you are unsure, just do an 'emerge --update gentoolkit' and it will be emerged unless it is already installed.
OPTIONAL. Review the /var/lib/portage/world list of installed packages and consider unmerging packages you are no longer using. It's a good time cleaning up your packages: The less packages are installed, the faster the recompilation of the entire system will be finished.
OPTIONAL. Now it might also be a good time to run "emerge --ask --depclean" in order to uninstall packages representing outdated old dependencies no longer used. Be sure to read the manual page for emerge before trying this!
Run "revdep-rebuild". If there are problems found and fixed, repeat running "revdep-rebuild" until no more problems are reported.
OPTIONAL. If you want to change your Gentoo system profile to a new one using "eselect profile", it is now also the right time to do it. Also do an env-update after changing the profile. (Be sure your kernel version is supported by the new profile! If necessary, upgrade the kernel, install it, reboot and continue following this guide only after that.)
Do an "emerge --oneshot --update --newuse gcc". It is quite common that emerge will detect that the current GCC is already up-to-date at this point, and will therefore emerge nothing. (But for the less common situations, this step will prevent troubles later.)
At this point, the new compiler you want should already be *installed*. (No packages need have to be recompiled with it yet. It also need not be the currently selected default compiler version yet.) As GCC allows multislot installations, it is not a problem in Gentoo to have both your current and a new compiler be installed at the same time. This guide even depends on that fact!
If all the above conditions are met, and no more packages need to be compiled in order to have an up-to-date system, set the desired new system compiler as the new system default compiler using "gcc-config". (This will make the new GCC 3.4 or GCC 4 the new system compiler.)
For those who have never used gcc-config before: "gcc-config --list-profiles" displays a list of all GCC versions which are currently installed on your box. The compiler you are currently using is marked with an asterisk. In order to change this "current" compiler to a different one from the list, remember the number between the brackets on the left side of the line which contains the compiler version you want to use. Then use "gcc-config <number>" to change the system default compiler to the list entry <number>, where <number> is the the number you remembered in the previous step.
Do an "env-update". Then run "gcc-config --get-current-profile". Verify that the new compiler version is really displayed as the current system compiler.
REBOOT! Many troubles when recompiling the system with a new compiler are due to the fact that the updated environment variables may not have been propagated to all active shell windows which you might be using for command input. A reboot ensures that every process in the system now uses the updated environment variables.
Are you using ebuilds which install kernel modules? Typical examples are special ALSA drivers, closed source video drivers, virtual machine networking drivers, add-on encryption engines. Not sure? Then better assume this is the case, and do this: Optionally update your kernel first. Then recompile your kernel with the (newly installed) GCC. Or, if you are using genkernel, just re-emerge genkernel. This will eliminate potential build failures later when the kernel module ebuilds will be emerged as part of the system rebuild. For instance, Karsten1973 reported build failures for the nvidia drivers when following an older version of this guide which did not include this point.
Download my utility script now if not already done. The link can be found at the beginning of this article. A good place where to put the script after downloading may be /usr/local/sbin/. (This is only a suggestion. Any other place will do as well.)
Run it!
The script will do nothing dangerous on its own. Not yet. Instead, it will use gentoolkit's services in order to construct a list of packages to be re-emerged when recompiling your entire system, and also determines the correct order in which those packages must be emerged. It also filters out unnecessary or duplicate emerges.
Then it will construct a shell script which, when run by you, will emerge all the packages in the right order until every package in your system has been recompiled as required.
The generated script will also allow to be aborted and resumed at any point. That is, you do not need to keep it running for days or weeks. The script remembers the last package successfully emerged, and will skip any packages already recompiled when run again later. This allows incrementally rebuilding your system. You can even shut down your system and continue recompiling the other day!
Run the generated script to actually start recompiling your entire system.
Run "revdep-rebuild". If there are problems found and fixed, repeat running "revdep-rebuild" until no more problems are reported. (Thanks go to devsk who identified possible problems when omitting this step.)
Reboot. (Just to be sure. It's not strictly necessary.)

OK, that's the guide so far.

If my guide served you well, you might perhaps also be interested in my article https://forums.gentoo.org/viewtopic-t-495997-highlight-.html where a promising new approach to prelinking is discussed.

Now some warnings and comments about the above guide. Although not necessary, it may be wise to read them also.

The script is now tolerant when packages fail to compile. In this case, the failing packages will be skipped and their names will be recorded in a file. Then, after all remaining packages have been recompiled, the script attempts to recompile those failing packages again. This will be repeated until all packages have been recompiled, or until none of the remaining packages could be recompiled successfully. This new feature allows an interruption-free rebuild of the system, without a need for the administrator to periodically monitor the build progress. The idea for this very useful feature was contributed by count_zero. Thanks a lot!
A different log file containing all of the screen output from the generated script will be created each time it is run. Together with the new interruption-free rebuild behaviour, this effectively turns system rebuilds into long-running batch-jobs.
The generated script assumes your portage tree will remain in a consistent state during the rebuild process. So please don't do an "emerge --sync" again before your system has been recompiled completely! (This is because the generated script contains specific package version numbers which were extracted at the time the script was generated. If the portage tree was updated before the script finished, package dependencies might change and the assumptions of the script regarding package dependendies might thus become void.)
In former versions of this guide, I strongly recommended running this script from the text mode interface, without an X-server running. Due to the experiences reported by Lloeki in his posting https://forums.gentoo.org/viewtopic.php?p=3553978#3553978, I consider this no longer to be a prerequisite, as you follow the general guidelines from the referenced posting.
Repeating from that post, those guidelines are: If you are heavily using KDE or other C++ based applications, start all such applications you may ever need during the update before running the script from my guide. Keep those applications running and use them until the update is finished. But don't ever close them, or there are chances you won't be able to start them again until the update is complete! Then, after the update, quit all applications, log out and reboot. (The trick here is that shared libraries which are already loaded will be kept im memory, even if their on-disk files are being replaced by different versions during the update. As long as you don't close those applications, the old versions will remain in memory, and all will remain consistent.)
The above trick allows you to keep working with your applications using C++ shared libraries during the update, which normally would run you into troubles. If you have a second machine to work on during the update, it may still be better to shut down the X server during the update and keep the machine running in text mode.
If you don't know how to get to the text mode, here is a short guide: First log out of your X session if necessary, so that you see the X login-widget of your kdm, xdm or gdm. Instead of entering your credentials here, press [Ctrl]+[Alt]+[F1] now. This should switch from the X login screen to the text console where you can log in as root. Then run "/etc/init.d/xdm stop" in order to shut down the X server.
Although the approach presented in the above guide greatly reduces the amount of time required to rebuild your entire system in comparison to all the other guides I am aware of, rebuilding your system will nevertheless take bloody ages. (At least unless you have a distcc compiler farm at your disposal. Or unless you are a "Bot-Master", controlling thousands of hijacked Windows PCs serving as distcc-slaves... )

So far for my comments - for now.

HOW TO OBTAIN THE SCRIPT

Here are the instructions how to get the utility script mentioned in the text of the article above.

First select the text between (not including) the "---BEGIN SCRIPT---" and "---END SCRIPT---" lines (can be found at the very end of this article text) with the mouse.
Then copy the selected lines into a text editor, using the mouse or copy/paste.
Save the copied text to some file, such as "genscript.sh".
Launch a command shell and go to the directory where you saved the text file.
Run the command
Code:

sh genscript.sh
This should decompress the actual script and leave it in the current directory as a file "recompile-entire-system-20060926".

Final remarks:

I wrote this guide not only as a small contribution to the Gentoo community, but also to the environment.

Yes, to the environment! Thousands and thousands of Gentoo users, all wasting weeks worth of processing power when following inefficient "recompile entire system"-style compiler upgrade guides, will waste an enormous amount of kilowatt hours of electrical power.

This energy not only costs money, but will also have to be produced, where it will contribute to even more pollution of air or water.

And that additional amount of completely unnecessary environmental pollution can easily be avoided by following the above guide... so that's why I wrote it.

Greetings from Vienna/Austria,

Guenther Brunthaler

==============================

Nick C · Posted: Fri Sep 01, 2006 6:42 pm Post subject:

can you not just post the script in plain text inside code bb tags? Would save anyone wanting to follow your guide a bit of hassle.

_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved.
www.monkeydust.net

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

devsk · Posted: Fri Sep 01, 2006 7:11 pm Post subject:

please post a text version, no matter how long it is. A useful thing is never too long to cut&paste....

I was hoping someone actually wrote such a script. Thanks for doing that.

Nick C · Posted: Fri Sep 01, 2006 7:14 pm Post subject:

aslong as you use the code tags whitespace will be preserved
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved.
www.monkeydust.net

Earthwings · Posted: Fri Sep 01, 2006 7:18 pm Post subject:

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

MaBu-Gentoo · n00b Joined: 28 Sep 2005 Posts: 22 Location: Slovenia

I have put the script on pastebin. I think it is a great script, but I have one question, when should I update glibc?
I konw have sys-libs/glibc-2.3.6-r4 and the new one is 2.4-r3.
_________________
[IMG]http://img101.imageshack.us/img101/2447/gentoo27li.gif[/IMG]

devsk · Posted: Fri Sep 01, 2006 7:55 pm Post subject:

devsk · Posted: Fri Sep 01, 2006 7:58 pm Post subject:

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Due to several requests, here is a line-numbered copy of the generator script which will be generated if the script from my article at the top is copied/pasted, saved and then executed.

This version is for review purposes only, please use the version from the article at the top if you actually want to download it.

The reason: Much can easily go wrong with copy/paste; and this is especially dangerous in a language like Perl where a single missing period can cause a script to reformat your hardisk instead of doing some useful job :-)

Which basically means, just carelessly copying/pasting can be dangerous. One must also be sure that the pasted text is identical (and not just similar) to the original text.

Which can best be achieved with checksums.

For this reason, the original script from my article contains a copy of the script below (in compressed form), and also an MD5 checksum. When the script is run, it expands the script below to your current directory, verifies its MD5 checksum and will warn you if the checksum does not match.

So please use the code below for review purposes only, but for the sake of security, use the version from the article at the top for downlading.

Abit667 · n00b Joined: 05 Nov 2003 Posts: 31 Location: PA

Wish I had seen this a little earlier..now that I'm halfway into that stupid emerge -eav system and world crap from the guide. That always seemed so insanely inefficient to me...

staffan · n00b Joined: 31 Oct 2004 Posts: 29 Location: Sweden

stahlsau · Posted: Sat Sep 02, 2006 12:29 pm Post subject:

hi,
thanks for the script, i think it's really valuable in this world of "i-recompile-my-system-a-thousand-times-to-be-uptodate" people. I can't say i'd need it since i've never encountered severe problems after updating gcc (i'm running 4.1.1 atm), but i think many people will find this useful.

KD-120RD · Posted: Sat Sep 02, 2006 1:45 pm Post subject:

Hi

I have a question here: What is the purpose of switching the profile?

I thought gentoo is a meta-distribution. Just update your packages and configs (ie. baselayout) and there you are.
Besides I'm using gentoo for about two years now and as I recall I have not manually swicthed my profile.
When I take a look at my /etc/make.profile it is linked against ../usr/portage/profiles/default-linux/x86/2006.0/
Is that possible?

grimm26 · Posted: Sat Sep 02, 2006 4:17 pm Post subject:

i don't understand the significance of my system profile matching my kernel. It's just default use flags and stuff, right?
_________________
"Blessed is he who finds happiness in his own foolishness, for he will always be happy".

jure1873 · Apprentice Joined: 09 Feb 2006 Posts: 183

This script seems very interresting to me, but how much does this approximately take? days?
I mean, I have and sempron 3000 with 1gb of ram but if I remember installing gentoo from stage3 wasn't all that fast (I mean to get to kde)
Upgrading gcc probably recompiles everything so this is basically a stage1 install? Could I grab a newer gentoo cd and get binary packages for some of the files from that cd and recompile the other files to make it faster?

CPUguy387 · Posted: Sun Sep 03, 2006 2:44 am Post subject:

Hi,

I performed all the steps in the instructions before running the script, and everything worked great! Now I'm trying to run your script, but it quits quickly with an error:

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna

Guenther Brunthaler · Apprentice Joined: 22 Jul 2006 Posts: 217 Location: Vienna