View previous topic :: View next topic |
Would you like to see Portage rewritten in C? |
Yes |
|
45% |
[ 73 ] |
No |
|
22% |
[ 35 ] |
Don't care |
|
32% |
[ 51 ] |
|
Total Votes : 159 |
|
Author |
Message |
jhon987 Guru
Joined: 18 Nov 2013 Posts: 301
|
Posted: Wed Dec 10, 2014 6:58 pm Post subject: Why not migrate Portage to C language? |
|
|
Hi,
Just to clarify things out, I'm asking this from a user perspective, i.e. presuming it's technically feasible... but first:
Why?
As a user, I find many apps, plugins, etc... written in Python are considerably slower than those written in many other languages such as javascript or better yet C.
Besides that, when Portage is written in Python it's dependent on a specific Python version, for instance 2.7, because Python versions aren't (for the most part) backward compatible.
This, in turn, may require users to have more than one Python version installed on their machines at all times, since other apps might require different Python version.
To sum up, the main reasons IMHO Portage would gain by being rewritten in other programing language (namely C) are:
- Python is slow, rewriting Portage in other language might make it faster
- Python isn't backwards compatible, thus rewriting Portage would rid the user of Python specific version dependency...
So far I've found 2 threads concerning that issue yet, both are fairly old and aren't directly asking the herein question - https://forums.gentoo.org/viewtopic-t-54862-postdays-0-postorder-asc-start-50.html
- https://forums.gentoo.org/viewtopic-t-311291-postdays-0-postorder-asc-highlight-portagec-start-50.html?sid=3462e3cdb24530af6c93c8378cdbfa61
This is my opinion, does anyone agree? |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10590 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Dec 10, 2014 7:15 pm Post subject: |
|
|
Why don't you try out sys-apps/paludis? It's a Portage tree compatible package manager written in C++.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3186
|
Posted: Wed Dec 10, 2014 7:58 pm Post subject: |
|
|
"i don't care" option is missing. I use it every few days and let it run while I do other things. E.g. go sleep. Or go for a walk. Or play games. Does it matter if I do 1 or 2 rounds in that time? No, I will play 3 or 4 (or 20 ), so I wouldn't even notice the difference anyway
Do you sit there watching emerge spin that / - \ / - \ / - \ / - \ / I-don't-know-when-i'm-done progress mark? |
|
Back to top |
|
|
Naib Watchman
Joined: 21 May 2004 Posts: 6053 Location: Removed by Neddy
|
Posted: Wed Dec 10, 2014 8:23 pm Post subject: Re: Why not migrate Portage to C language? |
|
|
jhon987 wrote: |
- Python is slow, rewriting Portage in other language might make it faster
|
Bad python is slow like bad C++ is slow. While python is a scripted language and thus is slower than a compiled, it isn't painfully slow. Equally python is a very good boilerplate langage if the actual speed concerns are correctly quantified and then that code dropped to C
jhon987 wrote: |
Python isn't backwards compatible, thus rewriting Portage would rid the user of Python specific version dependency...
| Python is very backwards compatible with the py2 and py3 being the obvious exception as the python dev's broke compatibility to fix some initial architectual issues. a py3.3 application will work perfectly fine on py3.4
If you use py3.4 specific extensions then yes you can only use py3.4 but portage as a whole doesn't use anything that exotic _________________
Quote: | Removed by Chiitoo |
Last edited by Naib on Wed Dec 10, 2014 9:36 pm; edited 1 time in total |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10590 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Dec 10, 2014 9:01 pm Post subject: |
|
|
szatox wrote: | "i don't care" option is missing. | @jhon987, I can add an "I don't care" option to the poll if you like.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
jhon987 Guru
Joined: 18 Nov 2013 Posts: 301
|
Posted: Wed Dec 10, 2014 9:04 pm Post subject: |
|
|
John R. Graham wrote: | szatox wrote: | "i don't care" option is missing. | @jhon987, I can add an "I don't care" option to the poll if you like.
- John |
sure, why not...
And thanks for the tip (Paludis, the Other Package Mangler) |
|
Back to top |
|
|
jhon987 Guru
Joined: 18 Nov 2013 Posts: 301
|
Posted: Wed Dec 10, 2014 9:08 pm Post subject: |
|
|
szatox wrote: | "i don't care" option is missing. I use it every few days and let it run while I do other things. E.g. go sleep. Or go for a walk. Or play games. Does it matter if I do 1 or 2 rounds in that time? No, I will play 3 or 4 (or 20 ), so I wouldn't even notice the difference anyway
Do you sit there watching emerge spin that / - \ / - \ / - \ / - \ / I-don't-know-when-i'm-done progress mark? |
I have intel i7 8-core, so usually I do "sit there watching emerge spin that / - \ / - \ / - \ / - \ / I-don't-know-when-i'm-done progress mark?"
(except when I emerge a web-browser, libre-office or other really huge package which takes approx 40 mins to compile) |
|
Back to top |
|
|
ct85711 Veteran
Joined: 27 Sep 2005 Posts: 1791
|
Posted: Wed Dec 10, 2014 9:11 pm Post subject: |
|
|
I wouldn't mind if portage was rewritten in C IF it would improve the performance. However, it's been said in numerous threads, the part in portage that takes the most time, is the dependency tree. That isn't going to be any faster in which ever language it is written in. Beyond that, I see no benefits where portage would be noticeably faster.
Like Naib said, portage is hardly fixed on one version of Python. Since I've used Gentoo, I've updated python numerous times, and never had an issue recompiling portage to use the new python (it always supported the new versions of python almost immediately after it is released)
If anything, I think you should be complaining why so many packages still only support python 2.7, and never updated to support 3 (so that we won't have multiple copies of python). |
|
Back to top |
|
|
The Doctor Moderator
Joined: 27 Jul 2010 Posts: 2678
|
Posted: Wed Dec 10, 2014 9:16 pm Post subject: |
|
|
Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while C, C#, C++, lisp, haskell, etc. would make it more difficult (or, in the case of haskell, more lazy.) Actually, haskell might not be a bad language to use, except for the rewriting required. That would be a side effect, which violates too many rules to consider. _________________ First things first, but not necessarily in that order.
Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box. |
|
Back to top |
|
|
The_Great_Sephiroth Veteran
Joined: 03 Oct 2014 Posts: 1602 Location: Fayetteville, NC, USA
|
Posted: Wed Dec 10, 2014 10:16 pm Post subject: |
|
|
I would like to see it in C++ personally, but C is good too. My reasons are simply, I am damn tired of scripting languages being used for everything, bloating my systems with interpreters. It has always been my firm belief that an OS and its tools should be done in a good, binary-compiled language. I could then cut crap like Python out of my OS.
With that said, I do not see Portage and its tools as "slow". The compilation process takes time and the system does not affect that. Hardware affects that! I am happy with Gentoo and its tools, and if they stay the way they are, I won't mind a bit. Now if they switch things around so the entire OS is Python and all of the software is suddenly Python, I am outta' here! :p _________________ Ever picture systemd as what runs "The Borg"? |
|
Back to top |
|
|
ulenrich Veteran
Joined: 10 Oct 2010 Posts: 1480
|
Posted: Thu Dec 11, 2014 12:19 am Post subject: |
|
|
The Doctor wrote: | Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while ... |
+1
And I think this poll is dumb and blind in an obvious way:
The "Gentoo Comunity" is of maintainers and developers. Gentoo forum users do not decide anything. It is docracy! Thus you do not need a poll but a coder!
If the solving part of portage takes the most of the time, you could think of a more narrow solution of your performance issue: Make the solver part of portage pluggable like Debian does to provide alternatives. There was a french akademic project, financed by European Union science funds, that provided some more advanced algos. I think also openSUSE took some of that for their super fast zypper tool. |
|
Back to top |
|
|
Yamakuzure Advocate
Joined: 21 Jun 2006 Posts: 2285 Location: Adendorf, Germany
|
Posted: Thu Dec 11, 2014 10:25 am Post subject: Re: Why not migrate Portage to C language? |
|
|
Naib wrote: | If you use py3.4 specific extensions then yes you can only use py3.4 but portage as a whole doesn't use anything that exotic | And if you rely on features introduced with -std=gnu11, it won't compile using -std=c99. _________________ Important German:- "Aha" - German reaction to pretend that you are really interested while giving no f*ck.
- "Tja" - German reaction to the apocalypse, nuclear war, an alien invasion or no bread in the house.
|
|
Back to top |
|
|
WWWW Tux's lil' helper
Joined: 30 Nov 2014 Posts: 143
|
Posted: Thu Dec 11, 2014 7:49 pm Post subject: |
|
|
There's already a protage in C:
PALUDIS |
|
Back to top |
|
|
WWWW Tux's lil' helper
Joined: 30 Nov 2014 Posts: 143
|
Posted: Thu Dec 11, 2014 7:53 pm Post subject: |
|
|
ulenrich wrote: | The Doctor wrote: | Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while ... |
+1
And I think this poll is dumb and blind in an obvious way:
The "Gentoo Comunity" is of maintainers and developers. Gentoo forum users do not decide anything. It is docracy! Thus you do not need a poll but a coder!
If the solving part of portage takes the most of the time, you could think of a more narrow solution of your performance issue: Make the solver part of portage pluggable like Debian does to provide alternatives. There was a french akademic project, financed by European Union science funds, that provided some more advanced algos. I think also openSUSE took some of that for their super fast zypper tool. |
Nothing beats eix. Try find, filter updates etc, in debian. |
|
Back to top |
|
|
Fitzcarraldo Advocate
Joined: 30 Aug 2008 Posts: 2038 Location: United Kingdom
|
Posted: Thu Dec 11, 2014 8:05 pm Post subject: |
|
|
The Doctor wrote: | Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. | +2.
It's not that I would mind if Portage were written in C, so I voted "Don't care", but rewriting something as complex as Portage in any language would likely introduce new bugs, which would take time to shake out. No doubt Portage has bugs currently, but just think what re-implementing the whole thing in a different language could do in the short term. _________________ Clevo W230SS: amd64, VIDEO_CARDS="intel modesetting nvidia".
Compal NBLB2: ~amd64, xf86-video-ati. Dual boot Win 7 Pro 64-bit.
OpenRC systemd-utils[udev] elogind KDE on both.
My blog |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sat Dec 13, 2014 7:59 pm Post subject: |
|
|
I couldn't care less about what Portage is written in. But I sure would like reading ebuild metadata and arbitrary bash script execution to be separate operations. |
|
Back to top |
|
|
hasufell Retired Dev
Joined: 29 Oct 2011 Posts: 429
|
Posted: Tue Dec 16, 2014 8:24 pm Post subject: |
|
|
The Doctor wrote: | Performance really isn't the most important feature of a package manager. It needs to be robust, reliable, and maintainable. Python offers this while C, C#, C++, lisp, haskell, etc. would make it more difficult (or, in the case of haskell, more lazy.) Actually, haskell might not be a bad language to use, except for the rewriting required. That would be a side effect, which violates too many rules to consider. |
You wouldn't "rewrite" any imperative program in haskell. You re-design it and then implement it. At which point there's not much similarity left, even in the algorithms (which should be reverse-engineered and optimized for FP).
Also, whether your functions/algorithms are actually lazy depends on a lot of things and there are lots of ways to break laziness without knowing. Haskell on it's own doesn't make all your functions lazy.
For the topic... it implies that there is anything particularly useful/reusable/understandable in the portage code. Since I don't find any indication that this is the case, I don't see a reason to reimplement a broken piece of code in a random language. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9709 Location: almost Mile High in the USA
|
Posted: Wed Dec 17, 2014 11:30 pm Post subject: |
|
|
I always thought, though portage takes a while to compute, emerging packages will equally be slow on that same machine. So it doesn't really matter that much in the whole scheme of things unless the typical usage model is to emerge packages one at a time.
As far as I can tell, it's the disk database that's the slowest thing. Having to read in the disk database and then traversing it is not going to be fast no matter what language - at least when I compare portage on NFS, portage on mechanical disk, and portage on SSD. Having said that, I'm not sure if a SQL/SQLite database holding the portage tree is the right solution either, easily peeking into the ebuild is nice too.
I guess it is somewhat consistent at least, speed does not seem to be a concern to most Gentoo users... or at least the solution is "solve it with hardware." Yes, agreed, portage is virtually a no-op on my SSD i7 too.
But my poor, poor 486 it takes a two minutes or so to run "emerge" to get the other white meat ... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
mistbow n00b
Joined: 13 Apr 2019 Posts: 4
|
Posted: Fri Oct 18, 2019 6:32 pm Post subject: |
|
|
I'd rather rewrite Portage on Golang. |
|
Back to top |
|
|
Muso Veteran
Joined: 22 Oct 2002 Posts: 1052 Location: The Holy city of Honolulu
|
Posted: Fri Oct 18, 2019 7:11 pm Post subject: |
|
|
mistbow wrote: | I'd rather rewrite Portage on Golang. |
Go for it. _________________ "You can lead a horticulture but you can't make her think" ~ Dorothy Parker
2021 is the year of the Linux Desktop! |
|
Back to top |
|
|
axl Veteran
Joined: 11 Oct 2002 Posts: 1144 Location: Romania
|
Posted: Sat Oct 19, 2019 4:04 am Post subject: |
|
|
C wouldn't be a good programming language to write portage in.
portage makes heavy use of strings and regexp expressions. there's no way to get out of the interpreter box.
python|perl|php
these are the choices. well, the old tested choices. using C here is just weird.
what you CAN do is optimize the portage tree, the /var/db/pkg files, maybe the distfiles. these things contribute to the overall speed of how portage behaves. |
|
Back to top |
|
|
axl Veteran
Joined: 11 Oct 2002 Posts: 1144 Location: Romania
|
Posted: Sat Oct 19, 2019 4:24 am Post subject: |
|
|
If I would vote for portage to change, I would vote for multi-cpu feature. smp. multi-core. whatever you wanna call it.
As it stands, we're 20 years into the future, and portage still uses one core (FOR ITSELF... I'm not suggesting it compiles with one core. it functions with one core). Check signatures. equery belongs. emerge. think of any command... it runs in a single core. which in 2020 soon... it's kinda... uhm, unperfect. unideal. unoptimized.
I remember the 90's when I bought my first smp computer. one motherboard. 2 cpu's. a smp kernel. back then we didn't have multicore. which came after hyperthreading. but they did come, one after another and now ... applications which run in a single thread are ... very old-fashioned.
that aside, there's another issues with portage. the stupid memory usage. sorry for the offtopic, but it's kinda ontopic.
i'm talking about portage on raspberry pi.
you run emerge... emerge runs some stuff, makes a list of packages to ebuild, and when you actually get to start building anything, the emerge command/python command already has 200-300 megs of ram consumated. not doing anything. once you have a list of packages, you can just ebuild {suite} (whatever suite is.. configure, compile, package or merge or both or whatever). which is another thing that you might want to do. maybe you want to build 2-3 packages at the time with 4-8 threads for each package. portage can't do that. an admin can, but portage can't.
I'm pretty sure it would be a monumental impossible task, but on the other hand, it's 2020. and emerge is single core. can you ... ? I can't. I'd like that to be different. and it can be done. python has pthread. perl has. hell, even php can mime it with a little bit of help from some statically build executables/scripts.
But let's start with the easy stuff. let go of the memory. (ON PI) when you figure out what needs to be upgraded & stuff, just run a script. why is emerge 200-300Mb of memory. just emerge itself. the python behind emerge. can't we free() all that memory and just run the list? |
|
Back to top |
|
|
axl Veteran
Joined: 11 Oct 2002 Posts: 1144 Location: Romania
|
Posted: Sat Oct 19, 2019 4:39 am Post subject: |
|
|
I'll put it another way. I feel like emerge should just "forget" all the dependency stuff that it has calculated, once it starts building.
I know it doesn't do that. Because you can just look at top. That's what would be a good first step toward a better portage.
And a multi-core portage. multi-cpu. multi-threading. I don't care how you call it as long as it hauls ass. My old computer has 8 cores. My Pi has 4. I am ashamed to say out loud how many cores just stand and do nothing.
And lastly, use tmpfs/ramfs and often backups ... but that's up to the admins. |
|
Back to top |
|
|
pa4wdh l33t
Joined: 16 Dec 2005 Posts: 815
|
Posted: Sat Oct 19, 2019 6:48 am Post subject: |
|
|
In my general opinion a package manager should have the least number of dependencies possible so that if something on your system breaks you have the best chance your package manager still works to fix it.
As a result of that I'd like to see portage in C (or any other compiled language) with the option to build it with all it's dependencies statically linked in. this will eliminate huge dependencies such as the python language.
I still remember an update many years ago which broke wget, which caused the download for the fix to fail ... _________________ The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world
My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com |
|
Back to top |
|
|
Goverp Advocate
Joined: 07 Mar 2007 Posts: 2023
|
Posted: Sat Oct 19, 2019 9:20 am Post subject: |
|
|
From my experience, rewriting in a new language rarely makes much performance improvement, and of course it introduces bugs - especially in a language lacking many, many features of the one currently in use, so the rewrite is non-trivial. When there's a performance issue, (a) IMHO you want at least 3* better speed to be worthwhile (this is for applications; infrastructure gains of 10% are good 'cos they're cumulative) (b) it's invariably about the algorithms, and the starting point is instrumentation to detect the hotspots.
Starting with the assumption that it will be faster in C is doomed; find out what's actually slow, and see if there's a better way of doing it.
<edit>
Oh, I find myself contradicted, in an interesting way:
https://www.phoronix.com/scan.php?page=news_item&px=Solaris-Old-C-Code-To-Python
</edit> _________________ Greybeard |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|