New Fast, Light Portage Using Sqlite?

Gavinv · n00b Joined: 05 Aug 2004 Posts: 10

I just found this .. looks really interesting, and I see lots of activity in the CVS source, with 5+ developers:

http://sourceforge.net/projects/portage-c/

Voltago · Posted: Wed Sep 29, 2004 6:13 pm Post subject:

Whatever does the job...

gwlinden · Posted: Wed Sep 29, 2004 6:26 pm Post subject:

The current portage implementation seems to struggle with the large number of packages. It's getting to be annoyingly slow, especially over NFS, but also when using a local file system.

I'm all for a hashtable-like implementation, as long as there is a way to rebuild the hashtable, in case it gets corrupted. I would not favour a single all singing all dancing portage tool, but rather suggest a small set of simple core programs, and a number of scripts to build the full-fletched tools. Python would be my personal preference, also given the Gentoo heritage.
_________________
Gwendolyn

colonel_dolphin · n00b Joined: 12 Jan 2004 Posts: 39

Need both:
1) exclude broad sets of things that will always be irrelevant (e.g. no X/Qt/KDE/Gnome stuff for a Gentoo server), and
2) "quick" emerge sync grabbing only fresh copies of the installed package ebuilds and their dependencies.

https://bugs.gentoo.org/show_bug.cgi?id=5857

spb · Posted: Wed Sep 29, 2004 6:45 pm Post subject:

Personally, I'm in favour of what's being planned for portage-2.1 and eventually 3.0.

A clean API with a core set of modules, neatly seperated, and emerge being *just* a front-end.

Ruzbeh · Apprentice Joined: 23 Jun 2004 Posts: 223

Those guys, instead of creating a seperate project, they should just help the portage developers! I mean we all want portage to be fast and everything.

castorilo · Apprentice Joined: 25 Dec 2002 Posts: 157

The poll lacks options. I would like it to be a bunch of python scripts with a light DB back end.

It is my understanding portage-ng is headed that way.

Voltago · Posted: Wed Sep 29, 2004 7:36 pm Post subject:

AFAIK there is already db backend support implemented in portage, although it is undocumented.

Pythonhead · Posted: Wed Sep 29, 2004 8:34 pm Post subject:

Database backend is already in portage:
https://forums.gentoo.org/viewtopic.php?t=175461

portage-ng is dead.

richk449 · Guru Joined: 24 Oct 2003 Posts: 345

Ummmm, the question says "Should portage be A or B" and my options are "Yes" or "No". I went with "No" to indicate that whoever put the poll together has NO clue.

Naughtyus · Posted: Thu Sep 30, 2004 5:13 am Post subject:

Ya, what is the poll asking? The answers don't make sense.

siti · Posted: Thu Sep 30, 2004 5:38 am Post subject:

Well I am the main developer of portage-c. If you want to check it out make sure you have sqlite2 & sqlite3 (I don't have optional dependices with automake - At the moment), Portage 2.0.51 (because I use the new cache), and glib 2. And grab the the latest from CVS - hopefullyl it compiles

Already you can see how insanly fast c/sqlite is

Also if anyone wants to help out please send me a message/email.

MooktaKiNG · Guru Joined: 11 Nov 2002 Posts: 326 Location: London, UK

Would be nice to have a website that shows the speed improvements for portage-c, and how to install the thing.

I just want to know how good it is before i start compiling

_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router

Trevoke · Advocate Joined: 04 Sep 2004 Posts: 4099 Location: NY, NY

Should it be ... OR should it be ...

The test returns 1.
_________________
Votre moment detente
What is the nature of conflict?

colonel_dolphin · n00b Joined: 12 Jan 2004 Posts: 39

miqorz · Posted: Wed Oct 06, 2004 8:38 pm Post subject:

I vote for anything that doesn't use python so it isn't slow as fizuck.

placeholder · Advocate Joined: 07 Feb 2004 Posts: 2500

In my experiences with Python and C/C++, the speeds are generally the same. It is all the things that the python scripts must take into account that makes it take so long, and C/C++ binaries would inherit the same latency because they are still searching through all of those directories.

I am for taking the need for bash out and just using more standard scripting using the $SHELL variable. I personally do not like needing to keep bash on my system since I use Zsh and am a clean freak about my installation(never to the point of reinstalling it though).

miqorz · Posted: Wed Oct 06, 2004 8:58 pm Post subject:

colonel_dolphin · n00b Joined: 12 Jan 2004 Posts: 39

Also run "top" and watch how much memory is consumed ..

placeholder · Advocate Joined: 07 Feb 2004 Posts: 2500

firephoto · Posted: Wed Oct 06, 2004 9:21 pm Post subject:

So you say the speeds are the same between c++ and python, show us your trials and tests that brought you to this conclusion. And I do believe this topic was about a portage database backend rather than python scripts so I'm not sure where the c++ came from.

Don't float your ideas as factual and then turn around and ask for proof from someone else, makes you look bad.

There is a slowness to portage that doesn't seem to be necessary, a database makes sense and also only updating what has changed in the tree rather than the whole tree itself. Last I checked emerge sync was bringing in 95,000+ files each time. I've also noticed that sometimes it's slower than other times and it's behavior is different. Sometimes it's gets the 95,000 files and updates the cache while other times it has lots of output of doing things and talking back and forth to the server.

Anyway I'm all for something that is more efficient which should speed things up a little.

Dolio · l33t Joined: 17 Jun 2002 Posts: 650

'emerge -s' isn't slow because of python. It's slow for the same reason that 'find /' is slower than 'locate'.

If you want 'emerge -s' to go fast, emerge esearch. It builds a search index like locate, so searches are almost instantaneous.

And it's written in python.
_________________
They don't have a good bathroom to do coke in.

colonel_dolphin · n00b Joined: 12 Jan 2004 Posts: 39

# time eupdatedb -v

real 4m24.486s
user 1m28.255s
sys 0m12.527s

If "emerge sync" required zero seconds, then using esearch/eupdatedb still requires 4+ minutes every time I sync.

Hmmm ... is this good coding practice?

eupdatedb, line 28:tmpfile = "/tmp/esearchdb.py.tmp"

Storing data as the root user in a well-known location that is potentially writable by ordinary users is usually frowned on ..

Dolio · l33t Joined: 17 Jun 2002 Posts: 650

plbe · l33t Joined: 01 May 2004 Posts: 661

I like the idea of portage but as others have said its getting very slow with the large amount of packages. I used freebsd for 3 years and found ports to be very fast, an emerge -uD world takes some time while a portupgrade -a is instant yeah a know portupgrade isn't _part_ of ports but even installing a regular application is instant no waiting for calculating dependencies or nothing. Devs should really look at this and maybe rewrite portage in a different language or something since python isn't known for being all that speedy