Why does it seem like dynamic linking is going out of style?

[Zeault]

I typed this title into my search engine hoping to find some discussion but nothing really relevant came up so I am posting here.

When I look at the trends in the broader linux/unix world I see a lot of technology just doing away with dynamic linking of executable code. Correct me if I am wrong but here are some examples:

• Rust by default statically links all dependent libs.
  
  
• Go (the google one) statically links everything by default.
  
  
• One of the killer features of the rapidly growing NixOS is how it ships each package with its own set of libs instead of installing all shared libs into the system folder. The executables may still be compiled with dynamic linking enabled but since they will only ever link to the libs that they are shipped with I consider this to effectively be a form of static linking. (Or at least this is how it was explained to me. I still have not used NixOS yet so I would love if someone could chime in start a more informed discussion.)
  
  
• The growing popularity of containers which also ship with their own set of libs independent from the system set.
  
  
• The shift of flatpak from being an alternative method of installing software that may not be supported by your distro to being THE preferred method of installing software by some people (e.g. some GNOME developers).
  

I am not really against any of technologies above. I use all of them! I just want to hear some perspectives from developers on why things are going this way. I guess I would say that I prefer dynamic linking because it makes for a small efficient system. I am aware that dependency management and especially ABI management can be a really annoying thing for packagers ("Dependency Hell" still has it's own Wikipedia article). I've never had that problem on Gentoo, but of course that's probably because we rebuild our packages all the time.

I like flatpak and its competitors. I don't really see the big advantage of one over the other besides flatpak being the most popular and widely supported. I do think it is a bit ironic and silly though how some flatpaks pull in other dependent flatpaks. Like damn are they gonna reinvent dependency hell or what.

I find it annoying how Rust and Go packages on Gentoo with the meson build system will check to see if their dependent libs are installed and then go ahead and redownload their own version of it anyways to statically link with. I don't understand this. I wish they could just use the library I already have installed.

What are some engineering problems that prevent the continued usage of plain old dynamic linking? Do programming languages need a more unified system of specifying ABI and maintaining compatibility through updates? Does the old Executable and Linkable Format (ELF) need an upgrade? Do object file formats need new features? Do the various runtime linkers used across different systems need to behave more similarly?

Share your thoughts, experiences, and opinions!

[flexibeast]

Interesting topic; i'm looking forward to reading people's thoughts on this.

My feeling is that it's at least partly driven by the sort of issues described in this 2015 post by Michael Orlitzky: "Motherfuckers need package management".

Regarding:

[szatox]

I suppose it's another mark of open source development being taken over by companies.
Static linking (and app containers) provides convenience of build-once-use-everywhere and removes a bunch of variables like versions of libraries installed on your system. This translates to more control (by developers, not by you) and predictability, and thus easier project management (within company structures), as well as reliable (and paid) tech support.

Basically, the old ecosystem used to be like:
A user creates/modifies a piece of software which scratches his own itch, and shares it with the world because copying is free
and now it's shifting towards:
A company modifies software it doesn't need or intend to use, hoping someone else will look at a shiny thing, pay for it, and then pay again for a dude from India to walk him through an instruction manual, and shares the code because the original project they hijacked was already released on GPL and it's cheaper to suck it up than start from scratch.


In the first model, the original developer doesn't care what setup you have, because it's not his problem. At the same time, interfaces don't change every day, so multitudes of unsupported setups will work just fine, and package managers tend to do pretty good job navigating the dependency hell.
In the second model the developer doesn't care about your setup, because he forces the only supported option on you. It's your responsibility as the user to provide the extra storage, and if libs provided by the app happen to be outdated and buggy... Well, it sucks to be you. You'll update your dependencies when the app's dev decides to update the dependencies.
_________________
Make Computing Fun Again

[GDH-gentoo]

Relevant thread from Debian's developers mailing list. Popcorn worthy in places.

Note: it was started by someone who saw this in Debian bookworm's release notes:

[flexibeast]

[GDH-gentoo]

This one has a funny part as well:

[e8root]

Coming from Windows world I found it convenient if application had all dependencies included. For binaries either statically linked or right there in the installation folder so that I didn't need to install program again and just keep it in folder with other such programs. Likewise having source or prebuilt libs in source repository means it is easy to modify and rebuild application.

Open source projects focusing on Windows have such setup more often than those focusing on Linux.
Now something which on Linux needs bazzilion apt/else commands to build is usually trivial (though not bullet proof - and especially after some time has passed). On Windows it is genuine hell to deal with such projects.

Then there is an issue of reproducibility. It is nice to have known working starting point even when wanting to update dependencies.

Now we come to the crux of the issue - current/mainstream way its handled by Linux distros is too much focused on single use case which is having all packages (binary or source) in package manager and all dependencies defined and then optimizing for providing fast updates and saving disk space by not keeping old and potentially buggy/insecure code. This is all great and all but only when it works and I cannot blame people for designing the system in such a way to keep it working like that forever and ever. The issue is that sooner or later it does lead to dependency hell and lack of reproducibility. In Gentoo we have added USE flags so we crafted additional ring to our dependency hell - not saying its an issue for me or that it is unmanageable (break it in to smaller chunks by rebuilding the world weekly like Google does and it can even be considered as form of fun ) and rather that unlike our world which we rebuild ourselves the real world often has different needs. Even Linus himself said it that distributing binary packages on Linux is terrible. This is the same issue I had with building programs on Windows but on steroids by adding dependency on distribution maintainers to make sure they update all the dependencies to minimum required versions and that all dependencies do not break anything. Otherwise you might release new version of your application but you won't see it released in all distros. They (distro maintainers) might try to build your app but when it fails then they will need to spend effort in to making it work delaying things and possibly delaying them forever.

Solution for developers is to do it in such a way you just clone repository and it all just compiles with least outside dependencies. And if someone doesn't like what you do they can fork your project and apply patches. For them it means their applications will be easier to get deployed - be it in distributions or even by just providing binaries directly.

The solution for us Gentoo users is to either suck it up and use programs like upstream designed them or to find people doing backports of applications made with statically linked code to make them build in our linuxy style way of doing things - which BTW is what I think how it already works and why some packages have USE flags to use system versions of libraries.

If we do it like that it is us users who have choice to give in to trends or to keep things nice and tidy in good old Linux package management way. In the end if done right (read: someone spends effort to do it right) it gives us less dependency hell. For example we might find it hard to build package with external dependencies but due to conflicts and whatnot but be able to make statically (or with libs in program directory) linked version.

In fact imho it would be nice to always have this very option! What I mean by that is to be able to build self-sufficient packages like some package managers (Guix, Nix, etc) allow us to do. Of course we can always use these already existing solutions.but it might be worth considering to add similar capabilities to Gentoo.

My 2 cents.
_________________
Unix Wars - Episode V: AT&T Strikes Back

[Hu]

[wanne32]

[pa4wdh]

As far as i know there are two main reasons for dynamic linking:
1) Save some disk space because all code is present only once. E.g. all programs that need crypto just use openssl.
2) Security. E.g. if a bug is found in openssl it is updated and all programs that use the shared library immediately benefit

Reason 1 might not be very valid anymore. TB's are cheap so no one cares about a few MB used extra because of some code duplication.

Reason 2 i think is still very valid. Consider all types of "static" linking listed in the OP, and now assume a vulnerability has been found in a library included in those programs:
1) Do you know which programs are affected? Keep in mind that real static linking makes it hard to figure out which libs are originally included
2) How can you make sure all programs that need to be updated are updated? Keep in mind that the programs version doesn't change, it's just one shipped library that has changed
3) And what if the project is dead the dev doesn't make the update?
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com

[GDH-gentoo]

[dmpogo]

It is about control during distribution. The downside is that it breeds a proliferation of slightly different and potentially incompatible versions of the libraries

[Hu]

[dmpogo]

[szatox]

[Zucca]

[GDH-gentoo]

[Zucca]

[GDH-gentoo]

[Leonardo.b]

I don't mind static linking.

[Leonardo.b]

[Hu]

[dmpogo]

[dmpogo]

[Leonardo.b]