revdep-rebuild is slow -- hopes/ideas

[user11]

When I update system, much time is spent on revdep-rebuild. The more CPUs I have, the more noticable is single-threaded revdep-rebuild compared to compiling.

Some ideas ('hopes for ideas', because I did not look into revdep-rebuild sources, so I am not sure if they are possible):

* Allow multiple --library options to search for any of the libraries specified. It seems that is almost the same with specifying regexp 'library1|library2', so should be easy to implement. Alternative way (no programming requred): give the users explicit regexp hints, like emerge sometimes reporting something like that:

[Kollin]

Well revdep-rebuild is a bash script, I think that the only way to see some speed improvements is some nice guy with good programming skills to port it to C or Python
_________________
"Dear Enemy: may the Lord hate you and all your kind, may you be turned orange in hue, and may your head fall off at an awkward moment."
"Linux is like a wigwam - no windows, no gates, apache inside..."

[slis]

Hi!

Thats interesting, what you say, Kollin.
I wrote small scipt in Python, that collects all binaries and libraries from system, and checks them for broken libraries, and indeed, it's more that twice faster than revdep! And there is just "raw" reading and calling ldd program. Thats impressive.

Time for revdep-rebuild (terminated after checking dynamic linking consistency)
real 1m25.551s
user 1m22.488s
sys 1m31.948s

and time for my little script:
real 0m39.776s
user 0m37.347s
sys 0m39.501s


So, you've right. I'm wondering, why do they wrote revdep-rebuild as bash script, instead of in python? The more, emerge is wrotten in it.
Nevertheless, if someone would be interested, I could finish python version and make it public.

Regards,
Sławek

[dol-sen]

slis Both the current maintainers of revdep-rebuild would like to re-write it in python or c. The portage dev's are also wanting to change portage and add abi_slots to an eapi level which will complete/replace the @preserved-libs feature and make revdep-rebuild obsolete.

join #gentoo-portage on irc, there things can be discussed more.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

[Kollin]

[Shining Arcanine]

[user11]

I think that gentoo team will rewrite revdep-rebuild completely at the time that matches their plans. Otherwise, we (non-team) have a risk to create a program that will become unsupported and useless after a month or so (just think of complexities like revdep-rebuild control files...).

Beyond those plans, I suggest making changes to existing script that are likely to be painlessly accepted by present revdep-rebuild maintainers. Little efforts (and little maintenance headache) for the sake of moderate results.

As for the first glance at the sources wrt my simple ideas:

(1) multiple --library options: At least, we could just remind the user that regexps are allowed. No sources modification needed then. Anyway, my very rough patch against app-portage/gentoolkit-0.2.4.6.1-r1 is here: http://pastebin.com/LPuYH3a5

(2) multithreading: revdep-rebuild allows parallel invocations if run non-root. Some coding is needed to allow the user specify such behavior for root as well. Finally, having (1) resolved, no (2) is so needed.

(3) assiging files to packages: There is massive 'grep' invocation, which is known for being VERY (really very) slow for UTF-8 locale. Assigning-files-to-packages phase speeds up much just by running 'LANG=C revdep-rebuild'. I don't know if it is safe to introduce LANG=C into revdep-rebuild itself, if not, it would anyway be worth notifying the user that LANG=C is recommended.

[user11]

[user11]

Sławek, could you explain why you have could real<user+sys? Does it mean you have multithreaded version of revdep-rebuild? I wish I had timing like that

[Shining Arcanine]

[slis]

Hello!

All of you got some rights. The most important factor is hard disc. But, it can be used more or less efficient.
First step is to find all libraries, and this is, what we cannot do faster. But there is disc cache created for.

One tools are good, but other are better. As someone told before, ldd is slower than scanelf. And revdep uses ldd.

Another thing is that, revdep is shell script, so there is no optimizations, and there are just program calls. As we know, to run any program, we need some overhead.

@user11 timings are small for two reason: it's fresh installation (under one month), and I'd interrupted it just after it checks consistency

And one more thing, it's ok to wait two or even five minutes for revdep, but it's not so rare that you have to call it several times. And there we've got headache.


I've done some work and I've prepared script that checks my system for broken libraries, list them and selects packages. In other words, there is something like revdep, but in his very, very alpha It does nothing more for now.

Here are some timings:

[user11]

Sławek,
my opinion is that your script's performance is fine.
Is it a full replacement for revdep-rebuild?

[slis]

No, it's not.
For now it's only for testing purposes. As I said before, it collects all binaries and libraries from system, checks for broken libraries and matches packages for those. It does nothing more than that.

I have to add some improvements and compatibility (especially command line parameters). Finally, your requests have to be written, such as multiple libraries to check.

When I made some more tests, I'll do my script public, so some volonteers may test it.

Regards


/edit
As i promised yesterday, there are timings from another (older gentoo installation) machine:

AMD Phenom(tm) II X4 965 Processor, 8GB RAM, 4xSATA (RAID-0):

[user11]

Sławek,
Sorry for my english in previous question.
I just want to ask, *will* your script become a full replacement to revdep-rebuild?
If yes, I will wait for that. If no, I will probably refine my patch for revdep-rebuild.

[slis]

It depends, what do you mean as full replacement. As for now, it won't work excactly the same way as original revdep, so: it does not create any of reveps' files, and output is little different.
But the core is the same: it finds out any broken libraries and give you packages.
If you want, priv me and leave your e-mail, so I'll send you my working wersion so you could check if it fullfits your demands.

Regards,
Sławek

[Ant P.]

There already is a faster replacement - it's in paludis:

[sera]

And takes multiple --library arguments as in