Is my idea any good?

[leyvi]

Hello Gentoo community.
Portage is by far the coolest package manager to ever grace us GNU/Linux users with its presence, but it has one flaw:
Its really goddam slow.
I think that I may have a solution, but I'm a high-school student, and I don't have any experience contributing to open-source software, and I would like some feedback on whether or not this idea of mine is practical, feasible, etc.

I was re-emergeing `@world` with the LLVM toolchain, when an idea occurred to me:
What if packages were distributed as LLVM IR/bitcode?
This would probably shrink the file sizes of packages, and thus the download time (a huge issue for me)
but more importantly, it would reduce the amount of time needed to compile and install an application; some
of us use the LLVM toolchain anyway, and not having to use clang, rustc, clang++, etc. and just using the LLVM
backend could reduce the time needed to build packages by quite a bit. In addition, the core principals of portage
would still be there. The IR can still be optimized on the end-user's machine, USE flags can still do their thing, etc.
The only thing that wouldn't be possible is editing the source code before it gets built and installed, but who
does that anyway? Regardless, one could still do that if needed (its open-source, after all).

Some things I'm less sure about, but could still be interesting:
1) Proprietary software could be distributed as LLVM IR, as its a lot like binary executables, which software companies distribute all the time, so things like games could be optimized for performance on the end-user's machine.
2) IR/bitcode is more compressible than source code, so smaller file sizes.
3) Maybe sub components of packages could be separated by USE flag, i.e. base install (USE flag independent), Wayland, X, udev, kde, etc., so that only IR necessary for your USE flags is downloaded, etc.

I think that if this idea is feasible and practical, it could be made into its own distro, as in a fork of Gentoo (this new distro should totally be called Peregrine Linux).
It would be mostly the same as Gentoo, but focused more on speed (as a general policy), oriented around LLVM, and hopefully a bit more accessible to tech novices, and people with less free time in general.
For the most part, portage is versatile enough that most of these ideas could be implemented on Gentoo as a proof of concept of sorts, before being migrated to a dedicated fork.
I would love to participate in such a project.

Sorry if that was a bit of a rant, but it's a general outline, not a complete idea, and I'm tired.
What do you guys think? Leave a vote in the poll below :

[NeddySeagoon]

leyvi,

You are missing a yes but ... option.

Portage is slow as its certainly solving a NP-hard problem and maybe an NP-complete problem, see https://en.wikipedia.org/wiki/NP-hardness.
That means that in theory, portage has to compute all possible solutions to find the best one. The classic example is the travelling salesman problem. I'll let you search for that.

Trying to distribute precompiled binaries, in any form, misses the point of Gentoo. Gentoo is not a distro. It's a toolkit that you use to design and build your very own distro.
You make leyvi LInux :) I have NeddySeagoon Linux and so on. We all use the same tools, we do not have the same installs.
Gentoo is the portage package manager and the ::gentoo ebuild repo. That's all. Everything else is {$UPSTREAM}

To take one example of the design aspect. app-office/libreoffice is affected by USE="accessibility base bluetooth +branding clang coinmp +cups custom-cflags +dbus debug eds firebird googledrive gstreamer +gtk java kde ldap +mariadb odk pdfimport postgres test valgrind vulkan" That's 25 USE flags. If they are all independent that's 2^25 different ways to build that one package.
When you use prebuilt binaries, you don't choose the USE flags settings.

Have a look at Experimental binary package host to see what is already being done.
Its somebody elses Gentoo, not yours. That's not to say its a bad thing. Use it if it does what you need. After all, an operating system and the installed packages are a tool to do a job' not an end in themselves.

As for download times, portage has a mostly forgotten option

[Hu]

First, I think you need to quantify what is slow. Are you complaining about (1) the time Portage requires to perform dependency analysis, (2) the time packages take to build once Portage starts the build, (3) the time packages take to install after building completes, (4) other?

For (1), your proposal does nothing to help with this, because Portage still needs to solve the dependency chain before it knows what packages to build.

For (2), distributing packages in a ready-to-build configuration could help. Will your proposal actually make it meaningfully faster? What benchmarks have you run to validate this?

For (3), like (1), your proposal does not help.

For (4), please describe what you think is slow.

With that out of the way, I think this will not work. Packages are distributed in the form upstream prefers, except when that form is so much trouble to manage that the Gentoo developers repackage the archive and point the ebuild to that. So in the general case, implementing this requires convincing the relevant upstreams to distribute the required IR bitcode, or asking the Gentoo developers to repackage every relevant release as IR bitcode.

Next, most build systems are strongly oriented towards compiling source code into object code. If you want to distribute an input that is not source code, you need to adjust the build system to handle that. This is potentially a per-package task, which means it is a huge amount of work. Worse, if not merged upstream and maintained there, it could break with every release that adds new source files or changes how existing ones are built.

[leyvi]

[leyvi]

[leyvi]

[szatox]

Ok, so distributing binary packages is a good idea but it doesn't fit Gentoo.
In fact, this idea is so good most linux distributions do that as the primary means of distributing software from their repositories.
Gentoo is an outlier made for different purposes. Nothing is free; in case of Gentoo you have more flexibility, and the price of said flexibility has been dumped into build times. Yes, it sucks, deal with it.

[Goverp]

In some respects this suggestion is to do what Java does, and ship everything as byte codes for some virtual machine (in this case, one that compiles). Advantages: consistent packaging;probably smaller files; less work on the target machine. Drawbacks: many packages are written in languages not supported by LLVM - Java, python, ruby, haskell, etc; AFAIK the conversion to IR takes account of USE flag settings, so the choice is now fixed on the target machine, or you make 2**n versions of the package, one for each combination of n USE flags; portage itself, rather than the C compiler, still has the same amount of work (as above); IIUC quite a bit of work, often single-threaded, is in the "make" stage; the final compilation from IR to machine code, and especially optimization, remains quite expensive - conversion from C to IR is usually quite cheap.

The net is the savings will be a lot less than it first appears. It's certainly a workable approach - as mentioned, it's the Java infrastructure - but it's not Gentoo. You could probably call it GenThree if you wanted . Gentoo is source; if it's not source, it's not Gentoo, so the approach cannot make Gentoo faster.

Gentoo has no built-in limitation that stops someone building packages that ship as IR or binary (indeed, there are many foo-bin packages, such as firefox-bin, rust-bin - 103 in total) and an ebuild that compiles to machine code and installs. It would be comparatively easy to set up a repository and populate it. I'm not sure how much use it would get though - it depends on whether the chosen USE flag settings are sufficiently general, and whether users would put up with the bloat - for example, support for both Gnome and KDE in all packages.
_________________
Greybeard

[Zucca]

[NeddySeagoon]

Zucca,

USE=doc affects what is built too. Being somewhat err excentric :) I have USE=doc in make.conf in by Raspberry Pi 4 build host,
That finds build time bugs. Then I have a list of exceptions in package.use/nodocs to get things to build.

FEATURES="nodocs" affects the install time. So I don't install docs to the the Pi. That throws them away at install time.
There is also FEATURES="noinfo noman" if you want to be lean and mean, even if there is not separate USE flags to avoid the build.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[sam_]

[ChrisJumper]

Hi leyvi,

the reason why i use gentoo is to have in some case, not 99%. The possibility to cook the software by myself. So i have not to trust this "i already have prepared a binary" Package or Finished cooked or baked food. Gentoo is about sharing some kind of knowledge about software. You are right - hardware optimization could do wtf optimization on the software layer too. But we have to trust someone else on a level.

You can try to distribute your compiled binary Packages for your domain of computers with same hardware.. but it is every time a solution aimed between pros and cons of self compilation.