CPU ISA level is lower than required (32bit cross-compiled)

[sliwowitz]

I have a x86-64 machine that builds binpkgs for a 32bit Pentium 4 machine. The build machine tyr-cross-master is a 64bit gentoo installation in a VM. When doing updates, I:

• Log in, update the packages in the VM
  
• Do a bunch of bind mounts and switch to a 32bit chroot with linux32 chroot /usr/i686-pc-linux-gnu /bin/bash --login
  
• Run emerge world -DNu in the chroot. This builds binpkgs and stores them on a NFS mount.
  
• Fire up tyr-staging VM which runs a 32bit system
  
• Update the staging system from the binpkgs on the NFS mount
  
• Check that everything works
  
• Log in to the physical 32bit Pentium 4 machine tyr
  
• Update the physical machine via the same binpkgs
  

So far this worked fine, but today, glibc-2.40-r5 was built which doesn't work on the 32bit hardware

In the 32bit build chroot, I have:

[sam_]

I can't look at this more right this moment, but let me dump some notes to help you (and maybe anyone else here).

There's two things to keep in mind when it comes to ISA markers:

• The ISA markers can be a pain when building binaries even without -march=native, see https://wiki.gentoo.org/wiki/User:Sam/ISA_markers_vs_startup_files. I don't think this is the case for you here.
  
• We had a bug a few months ago where this marker was wrongly applied in some cases for x86: https://bugs.gentoo.org/933764.
  

But I don't think either of those things are your issue - just noting them to hopefully avoid red herrings, and also to give a better understanding of the mechanism at play here.

[sliwowitz]

Yeah, I've seen that bug report, and after a while figured it probably doesn't apply to me exactly (also the other bug report about livecds that was about glibc-2.39-r9 which worked for me). IM[/quote]HO the -march=pentium4 should do, as that machine is an actual Pentium 4. That also means it has SSE2, so many other reports where this error came on x86 due to missing SSE2 don't apply to me either.

One think that changed between the last update (cca a month ago, though the glibc-2.39-r9 package is even older, built on Aug 16) and this one is that this time there was a gcc update, so before everything was built with gcc-13, now I rebuilt with gcc-14 (did an emerge -e world). I've come across a couple recent reports that also used gcc-14, though it might be just a coincidence (and their other parameters were also different from my situation).

[sam_]

Can you diff the objdump output for each of the binaries installed by both glibcs? I assume you have binpkgs of them both available, possibly to share if necessary.

It's possible that one of the other files does differ (like ld-linux-* or one of the startup files).

[sliwowitz]

[sam_]

Start with just readelf -n, actually, then can look at the proper binaries if needed (can email them or something) later on (easier than trying to do it all via proxy).

[sliwowitz]

[sliwowitz]

Since the physical Pentium 4 machine's OS won't boot anymore, I have to access it via a LiveUSB stick. From there I can only copy files over to the hard drive because chroot won't work either. Just to be sure - if I try to make a new glibc binpkg, can I then just extract the contents of the binpkg image to the root of the target machine filesystem, and expect it to work or is there more to it?
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra

[sam_]

You can raw untar the image. It's not great to do and it's better to use emerge --root= from outside but in a pinch, untarring is alright, just it might make a mess and obviously won't remove any old files which got left behind by the previous version (so you might have dead symlinks or similar).

Will check the link in a bit, thanks.

[sliwowitz]

I got into the machine, and replaced the glibc, but it didn't help. Then I noticed (in /var/log/portage/elog) that emerge actually died in the prerm phase of emerging coreutils which came right after glibc. So that's where the issue might have been actually coming from. I'm investigating now, trying to replace coreutils with an old binpkg too.
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra

[sliwowitz]

OK. For some reason, coreutils and ncurses were broken. I emerged both again (and the glibc too) on the build machine (this is imho weird since the earlier emerge -e world should have done that), and again on the physical machine from the new binpkgs, they started working.

Then went to I install gcc-14 on the physical machine, and found out its binaries also started throwing the ISA error - though here it somehow made sense since earlier I actually ran emerge -e world --exclude=sys-devel/gcc, (when a new gcc is out, I always upgrade gcc itself first, then rebuild everything except the gcc because compiling gcc takes about half of the time of the full upgrade). After rebuilding gcc-14 on the build machine, and installing the new binpkg on the physical machine, it started working.

Then I tried adding some random nonessential packages to see if I'd have to rebuild everything once more. Some worked (ncdu) some didn't (pciutils), so to be on the safe side, I'm just rebuilding everything again with the newly rebuilt gcc-14 now.
_________________
The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim.
-- Edsger Dijkstra

[sliwowitz]

Yeah. Seems the issue was gcc after all. Originally, I:

• logged in to a VM with gcc-13 and glibc-2.39-r9
  
• built gcc-14 with glibc-2.39-r9,
  
• built glibc-2.40-r5, with gcc-14,
  
• then built all remaining packages with gcc-14 and glibc-2.40-r5.
  

This gcc-14 (and some other packages) were bad - they worked in the VM, but not on the real Pentium 4.

Then I rebuilt gcc-14 (with glibc-2.40-r5 already installed), and it started working on the real Pentium 4 too.

readelf shows the gcc binaries are differnt