Portage config for building with -O3, Graphite, and LTO

InBetweenNames

Hi everyone,

I'm new to the Gentoo Forums, but I have received some positive feedback about a project I've started with Gentoo and I figured I would share it with the Gentoo community as a whole.

I've been silently working on a Portage configuration for building with O3, Graphite, and LTO by default since about the start of the year. My configuration and results are in this GitHub repo: https://github.com/InBetweenNames/gentooLTO

My goals of this project are as follows:

* Identify packages which do not play nice with LTO and fix them if possible
* LTO, O3 and Graphite optimizations will help reveal the use of Undefined Behaviour in C and C++ programs (see https://en.wikipedia.org/wiki/Undefined_behavior )
* Perhaps, in the best case, even get a performance improvement on the whole system. Benchmarks are sorely needed for this.
* Identify cases where -O3 and Graphite optimizations actually have a performance regression versus O2. These would make for good GCC bug reports. Especially with Graphite!
* In all cases, let the compiler decide which transformations to apply. Never override the compiler's cost function for applying a certain transformation (no -funroll-all-loops -- ever).

This is definitely advised for expert users only. Breakages do happen and you will probably need a strong programming and administration background in order to resolve them.

I have a lot more information in the README about this which is accessible via the GitHub link. If you are interested, contributions are very welcome!

duby2291 · Guru Joined: 17 Oct 2004 Posts: 583

I can see a lot of the configuration you've chosen is specific to your personal preferences. But all in all it does give a great reference to start figuring it out. Thanks for your effort!

Maitreya · Guru Joined: 11 Jan 2006 Posts: 445

I have to agree with duby2291 here.
For example:

gentooLTO/env/addncurses.conf

in the repo is a fix (which I use too) but is not related to gentooLTO, so if you want people to be able to easily fork your project make it only about gentooLTO and put the fix for ncurses in a seperate/personal overlay/repo.

InBetweenNames · Posted: Wed Sep 13, 2017 3:00 pm Post subject:

Thanks for the feedback! Would you consider contributing to the issue about this in the gentooLTO github? Someone suggested that things be abstracted out into an actual Gentoo profile such that people could more easily try out these settings for themselves. It sounded like a great idea to me, but I'm not sure about what technical details are involved.

I have heard from a few people that a more generic repository would be ideal. It's unlikely that I will change this main repository to be more generic one as this is actually synced from my own /etc/portage, but I fully support the idea of a generic configuration springing from this one, and I'd be willing to help with that. If the profile idea works, I'd even switch to that profile myself and contribute my relevant configuration updates there.

Maitreya · Guru Joined: 11 Jan 2006 Posts: 445

Yes, I would. And I think many would.
You do the kickoff and you'll see people are suddenly interested.
This project is not very old and it already has quite some followers and stars.
I don't think that is because of reputation but more "finally someone is doing it"-mentality

But for that idea to succeed you need a clear "goal" and/or "target".
So if you would remove that fix for ncurses, it becomes more generic and useable for a broader audience without conflicting other overlays/builds.

So I would really recommend either starting a new repo for your "personal" stuff or the "new idea" stuff. Actually, strongly recommend because I do not see it succeed if it's a mingle of functionality and random system patches.

But do be careful, because with LTO + O3 + graphite you are stacking optimizations and other new stuff at the same time. Which could put people off.
If you look at the forums, you can see that there is a lot of stuff happening around LTO + graphite for years. So I bet when you remove O3 (or put it in a different overlay) those people in those topics will join you really quickly.
(now everybody has their own collection of profile.env and overrides)

InBetweenNames · Posted: Thu Sep 14, 2017 2:49 pm Post subject:

What would the Gentoo community think of an "lto-overlay" with just the LTO-relevant parts of this configuration, packaged as profile? Is it possible to put package.env overrides in profiles? I'm unfortunately a bit time limited today, but if someone could point me in the right direction/give feedback on this idea, I'd be glad to work on it and make such an overlay.

Maitreya · Guru Joined: 11 Jan 2006 Posts: 445

I use them in my overlay. And as a workaround I just make a symlink /var/lib/layman/REPONAME/profiles/PATH/env in /etc/portage/env/REPONAME and for /etc/profile/package.env likewise.
Not pretty but it gets the job done until it is supported. (I have no idea btw, why that is, I'll ask around on irc)

InBetweenNames · Posted: Thu Sep 14, 2017 8:25 pm Post subject:

I've got an issue on the issues page that discusses a potential "LTOize" script to help users migrate to an LTO configuration. I don't have time to work on it today but I found the idea promising. It would be a simple lightweight script that you could use to retrieve the latest LTO workarounds and configurations, and advise you how to modify your make.conf to have LTO working well. I would have to make a small number of changes to the repo for it to work nicely, but those are changes I would be willing to do. If someday "env" support is added to Portage, we could make it a profile very easily that way. Thoughts/suggestions? I figure we can discuss it for a bit to make sure we get a good solution, then when I get some time I'll try it out.

InBetweenNames · Posted: Sat Sep 16, 2017 7:58 pm Post subject:

For anyone interested: I have updated the repo and converted it into an overlay. I also have a minimal 'ltoize' script that helps set things up for you. Check it out and lets get bug hunting!

tranquilcool · Veteran Joined: 25 Mar 2005 Posts: 1246

InBetweenNames · Posted: Tue Sep 19, 2017 4:22 pm Post subject:

Thanks--I'll file an issue about this in the issue tracker.

EDIT: just a quick question, can you elaborate on the sys-libs/binutils-libs? I can't find any references to it in my repo except in @ltoknownworking

EDIT 2: Nevermind, I found it. Issue filed.

InBetweenNames · Posted: Tue Sep 19, 2017 8:11 pm Post subject:

Fixed as of HEAD. Thanks for bringing that to my attention!

42n4 · n00b Joined: 10 Feb 2015 Posts: 23

Thanks a lot!
It works as I describe in other threads, but compiled with gcc 8.1.0 and flags -march=native some programs e.g. opencl R packages crashes.
Thus, I compiled with -march=core2 and lto graphite all my opencl programs, R packages based on RCPP, even RInside C++ files (with embedded R).

And after:

steveL