Gentoo future and modern packaging

[friesia]

Recently GCC depelopers improved "function multi-versioning" in GCC which allows software to make use of all available CPU instructions without need for compiling with "-march=native -mtune=native".
https://lwn.net/Articles/691666/

Source-based distros will probably lose the advantage of execution speed in future.

There are also projects like Flatpak which allow distribution-independent packaging (not for all installed software, of course, but still). Maybe they will gain popularity. I like the idea perhaps, but it won't fit into Gentoo philosophy.

Gentoo offers flexibility with USE flags, but in other distros it's also possible to build a set of packages from source if the default build doesn't suit you...

I've been a Gentoo user for many years but I'm trying re-consider benefits it currently offers and whether I need them.

I like the simplistic approach you can have in Gentoo (not installing unnecessary deps) and in Arch (overall KISS philosophy). Will probably stay with them for a while.

What do you think will change about building and packaging in upcoming years in Linux ecosystem?

[eccerr0r]

Most build their systems suboptimally anyway, like I do, mainly for compatibility. Probably would be better for me.

However this will give binary bloat? This would require complete routines to be kept around, fine grain instruction selection doesn't work as capabilities branches themselves would waste cpu cycles.

Hopefully there's a way to prune out irrelevant code to reduce object file sizes. The resultant binary will be smaller and thus at least save disk space. Cache resident size is a fallacy as unused code will not be cached anyway. Then again people tend not to care about binary disk space consumption, assuming it's the "cost of doing business" and get that bigger hard drive. Sigh.

The "KISS" philosophy bugs me as perhaps it's simpler for the end user, but it is clearly NOT simpler for the developer to keep so many test cases around to support the end user. So who should it be simpler for? Why should the developer keep on offering OSS if it costs them effort to support options that they do not care for?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[Ant P.]

I like having a distro that makes its own decisions and lets me make my own. This has nothing to do with execution speed; if I wanted that I'd just upgrade my CPU more than twice a decade and use Windows.

[Naib]

[krinn]

I just care how gcc authors will implement this, making a code that can execute specific instruction when running is nothing really new, what is new is that it's gcc that will build that specific code part.

I just hope they will think about code size and that this option could be turn off, and that any "running section" will be built as "one running section of that cpu" when the option is off.
If not, i would see it as a regression, you have no real gain, as you can already do yourself running section for specific feature (mmx must be the most common seen in program), but bloating code of a running program for code segment that will never be trigger would be really a regression, specially on embed devices.

It's also arguable about the value added, while it will ease dev works, some feature will gave hard time to choose which one would do the job better.
Look at software raid, it execute specific running section to benchmark what feature will gave the better result, and on two cpu with the same features, depending on cpu and family, results may differ.

[cue-33]

Sorry if this is a really noobish question. But aside from the obvious answer (of application developers being too lazy or neglectful) wouldn't most issues be solved if every application specified exact library versions and we quit using generic "library.so -> library.so.X.Y" symlinks?

It's my understanding that the difficulty of building a package manager or ensuring that a Linux distro runs smoothly comes down to whether or not the library versions on the system play nice with the software linking to them, since we pretty much only ever have a single version of any library at a time. Is that correct or am I mistaken?
It's also my understanding that these new package managers like "nix" or "snappy" intend to solve this issue by basically giving every application a complete set of libraries (which would lead to us having a whole bunch of redundant libraries). Is this correct?

Assuming I'm not missing something here, wouldn't it be beneficial to just allow multiple versions of each library to be installed and just create a new "Linux standard" for software to always specify the exact library version number?
Sure that might take up extra disk space, but isn't that better than the "snaps" solution of giving us potentially several copies of the exact same library version?
and isn't it better than all these dependency headaches and questions of whether a piece of software can run on one Linux distro and not another?

Then package managers can just keep track of when no packages use a specific library version and remove it.

Again, sorry if this is really ignorant and I'm completely missing something here.

[The Doctor]

Well, your noobness is missing the topic, but that is fine.

You are talking about linking, which isn't what is being discussed. Each processor speaks a slightly different language. They all speak x86_64, but intel teaches their cpu a few special words and amd teaches their cpus a few different special words. Telling the compiler to use these special words speeds up program execution for the program but will cause a crash on the wrong cpu. The new GCC feature allows programs to speak both languages and choose which one at run time.

They are worried that this would waste disk space since both instructions are needed.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.

[cue-33]

Ok, sorry about that. I thought since the OP mentioned flatpak and USE flags that it was just a general discussion about package management.

[Roman_Gruber]

[laineg]

gcc -march=native knows everything about your CPU, and can optimize accordingly, like avoiding instructions that are slower than normal on your CPU. Function multi-versioning only enables stuff like AVX while not breaking support for older CPUs. If your CPU has a slower-than-normal <some instruction>, FMV won't help you.
_________________
/sbin/fsck: error while loading shared libraries: /lib/libc.so.6: invalid ELF header

[gwr]

I am confused. When is function multi-versioning done? If I don't specify -march/-mtune? If I specify -march does it compile in the proper functions without doing the initial runtime lookup?

[John R. Graham]

The OP's linked article is actually a pretty good introduction. It's done at three different places & times:

1. At development time, when the programmer calls out critical functions that should be compiled against multiple architectures. Alternatively, the developer can provide multiple implementations of the same function that are optimized for different architectures. Functions are effectively overloaded with multiple implementations.
   
2. At compile time, when the compiler provides the infrastructure to automatically detect and call the correct implementation.
   
3. At run time, when this detection and automatic dispatch takes place.

The no free lunch aspect is that all implementations are compiled and exist in the resultant binary (i.e., code size gets bigger), and that the function calls themselves become slightly less efficient; presumably it's worth it most of the time because the implementation makes up for the loss of call/return efficiency. However (someone correct me if I'm wrong), this overhead appears to only occur where multiple implementations are explicitly called for in the source code. So, although the lunch isn't free, it appears to be at a reasonable cost for some use cases.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

[gwr]

[John R. Graham]

It's more like you have to opt in. If the actions are not taken at development time, then there is no overhead at runtime.

Edit: Although true, this is simplistic because it appears that many OSS developers are adding optimized snippets to their code that utilize this feature, so you might get some of it whether you want it or not. I haven't been able to confirm this yet, but "-Os" in CFLAGS will likely disable this feature.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.