PGO LTO optimize: compile with chroot on fast CPU

[GreenNeonWhale]

Hi,

I hope that I've posted this in the correct sub-forum.

My question involves the following scenario:
Overall: compile code on faster CPU for installation and use on slower CPU
Method:
- mount entire filesystem from Slow CPU on FAST CPU system.
- chroot
- execute compiles, emerge/portage etc., and compile the code
- -march=amdfam10 set in make.conf in chroot, and on native system.
- -march=native has never been set in make.conf anytime during this.
- entire chroot used as native environment on Slow CPU system.
Assumption: Fast CPU has all the features of Slow CPU. In my case, Fast CPU is a 2nd Gen Bulldozer, and Slow CPU is a Turion II Dual-Core (march=amdfam10). So far, no crashes or execution problems.

My understanding is that both PGO and LTO optimizations involve executing code during compilation, and analyzing the results.

My Question: Given the above scenario, if I turn on PGO and/or LTO optimizations, which run on the faster and somewhat different CPU, and those results are used by the compiler to optimize, will I get code that will end up with poor optimizations when executed on the slower CPU as it is a bit different? Basically, am I wasting my time with PGO and/or LTO turned on, when I'm compiling on a non-native CPU like this? Please note, that in this scenario, substantially longer compile times are acceptable to me if I gain faster code in the end.

I'd appreciate any thoughts and/or insight you folks can offer.

Thank You!

[stefantalpalaru]

[Hu]

Link Time Optimization does not run the built code as part of its operation. It is merely an expedient way to give the compiler visibility into the whole program, so that it can perform optimizations across translation units. PGO does run instrumented code, but as far as I know, it is only looking at which paths are executed most often, not at how quickly the CPU can execute them. Therefore, results obtained on a fast system should not be skewed relative to what you would get running on a slow system (unless the test involves timing sensitive decisions, like "Run this loop as many times as possible in 2 seconds"). In my opinion, such tests should be rare.

[molletts]

Hi,

As I understand it:

• LTO doesn't execute any of the generated code - it effectively defers (some of) the optimisation phase of compilation until link time, when all of the code to be linked together is available for inspection, allowing better choices to be made by the optimiser than it would have been able to make by looking at small chunks of code in isolation.
  
• PGO does execute the generated code during the profiling phase but, as far as I am aware, it doesn't rely on precise timings, either at the instruction level or at function level. Instead, it collects statistics about things like how many times functions get called and from where, how many times loops go around (and how often they get run), which branch of "if" statements gets taken most often (the "if this then do this" case or the "otherwise do this" case) and suchlike. This allows the optimiser to make more informed decisions about whether it's worth inlining a function or unrolling a loop and whether it might be beneficial to reorganise an "if" statement so that the most likely outcome results in the code running straight through instead of jumping to elsewhere then coming back.

I haven't read the code, though, so I may be wrong.

I've been doing what you describe for some years now on this basis and haven't encountered any major problems (except when I accidentally forgot that I had put -march=native in make.conf on one of the "client" systems while experimenting on it...) I think I've seen one or two packages whose build systems generate native code regardless of the -march=... setting but I can't remember which ones offhand. I don't think they were big ones so I was able to just rebuild them locally. They've either been fixed or I'm not using them any more (or they haven't been updated for a long time) because it's quite a while since I last had to do this.

My slower systems all have a small hand-built initramfs which can be booted instead of the main system from the grub menu, which exports the main system drive via nbd (the Network Block Device). I then mount this on the build host (usually either my main PC, an FX-9590, or my test-lab server, a dual Opteron 6380, both -march=bdver2) and chroot into it to run updates. I use this method on systems including a Pentium M laptop, an Athlon Neo (-march=k8-sse3) mini-PC and my 1st-gen Core i5 (-march=nehalem) work laptop. The only system I can't update from these is an Atom-based "set top box"-type PC which I have to update from a VM in the Xeon-based cluster at work because it has the BSWAP instruction which isn't available on AMD Piledriver. Interestingly, the Core i5 laptop can't be updated from my Core i7 desktop at work because the i7 doesn't have AES-NI!

If anyone with deeper knowledge of the internal workings of the toolchain can correct any misunderstandings (or confirm that this is broadly correct), I'd certainly be grateful.

Thanks,
Stephen

[GreenNeonWhale]

Hu & molletts,

Thank you for your replies. They were quite helpful to me, and quite informative.

molletts,

I think you and I have come to some similar ideas in regards to computing. Thus, I thought I'd share in case you (or anyone else reading this) is interested.

I too have decided, at some point, to try out a 9590. I haven't gotten there yet, as its still in its box, but I really hope to soon. I have a STRONG distaste for management engines. If I'm not mistaken, it is one of the fastest CPUs available that doesn't have one. Raptor Computing's POWER9 systems look fantastic, but they're rather expensive. I'm also planning trying my hand at running coreboot/libreboot on the Opteron board that they fully support. Hopefully, it will work out.

Out of curiosity, if you feel like sharing, do you air cool your 9590, or water cool it?

While I haven't build a custom initramfs like you have, what I have built and used, is a variation on that idea. I've built a custom boot/rescue/backup OS on a separate device, either a USB flash drive with hardware write protect (Kanguru), one without, or an SD card, on which I've installed a very basic setup. Said basic setup is for the purposes of booting a full disk encryption OS on the primary drive, using tripwire to scan the primary OS (and the boot drive), perform backup operations while the primary OS isn't running, and other such rescue/maintenance tasks. When I'm running the primary OS, I usually disconnect the boot drive. That way the bootloader, the kernel, and the entire tripwire setup, is completely inaccessible to the primary OS most of the time. Furthermore, the write protect capable flash drive gives me just a little more protection, as I can keep it read-only, enforced by hardware, except after I've verified the system, and temporarily disconnected the network. Thus, hopefully, keeping everything somewhat more secure. So far, its worked out rather well. It is more work to use and maintain, but I like the extra security of it all.

Again, my thanks to you both for your replies and help. I really appreciate it.

[molletts]

[GreenNeonWhale]

Stephen

Thanks for the info, that was helpful. I saved it in my notes for when I get the chance to tackle my 9590 project.

[Goverp]

[GreenNeonWhale]

Goverp,

Thank you for your write up and further explanation. That will be quite helpful in making future compile/optimization decisions.