Prelinking KDE - some solutions - some problems

r2dtu · Tux's lil' helper Joined: 23 Aug 2002 Posts: 87

Since I've rebuilt my gentoo with glibc 2.3.1 i decided to try to prelink it. (I mean Jakub Jelinek's prelink ftp://people.redhat.com/jakub/prelink/ ).

I've found following problems:
1) some apps (i.e. designer from qt) would not prelink complaining about libGLU.so.1 being non-PIC shared library
2) all KDE apps would not prelink complaining about libqt-mt.so.3 being non-PIC shared library.

The first problem was easly solved. Xfree default configuration is forcing libGLU.so to be non-PIC with "BuildLibGlxWithoutPIC YES" in xc/config/cf/linux.cf due to performance reasons. Since I'm not using GL apps I do not care about its performance so adding:

Evangelion · Posted: Mon Nov 04, 2002 12:32 pm Post subject:

any comments regarding performance?

r2dtu · Tux's lil' helper Joined: 23 Aug 2002 Posts: 87

Well I run it on P4 1.7G, 512MB RAM notebook, so things were already fast. But startup times are much improved. Konqueror starts in a blink of an eye (well maybe two), compared to about two seconds before.
I can't give you any specific numbers, cause I'm reemerging kde now. Since prelink changes binaries I un-prelinked my system so emerge would see right MD5 sums.

Lovechild · Posted: Mon Nov 04, 2002 1:02 pm Post subject:

So who's with me, we need prelinking directly in portage - just how hard would this be to implement?

I would like to offer my limited programming skills and my unrivaled bold foolishness to this project.

I gather from reading Jakub' mailinglist posts that the best approce(the only?) would be to:
1) prelink on install.
2) run a cronjob that checks say once a week, did we change the database of installed ebuilds.
3) if so rerun the prelinker.

The problems would then be
1) Keeping md5 sums in check
2) Implementing a USE flag for this (or would it be better just to force this on everyone and keep it in unstable untill everything compiles)

Mind you I've never played with prelinking in glibc 2.3.1 (and people.redhat.com is down for me so I can't get the tools)

Evangelion · Posted: Mon Nov 04, 2002 1:09 pm Post subject:

Lovechild · Posted: Mon Nov 04, 2002 1:12 pm Post subject:

r2dtu · Tux's lil' helper Joined: 23 Aug 2002 Posts: 87

Evangelion · Posted: Mon Nov 04, 2002 1:31 pm Post subject:

Lovechild · Posted: Mon Nov 04, 2002 2:36 pm Post subject:

r2dtu · Tux's lil' helper Joined: 23 Aug 2002 Posts: 87

I upgraded to kde 3.1rc2 and now no kde app works when prelinked. It either ends with segmentation fault or dcop communication error. Strange, all qt-only based apps work great and load faster when prelinked.
Maybe my idea to force -fPIC flag on whole XFree was not good. I'll try recomipilng XFree and qt and kde with only forcing libXinerama. This is what redhat is doing (Jakub Jelonek's patch). I'll keep you informed.

r2dtu · Tux's lil' helper Joined: 23 Aug 2002 Posts: 87

I tried diffrent approach to libXinerama problem. I simply added "-no-xinerama" to configure command in qt ebuild. Rebuilt qt and kde and now all kde programs work ok when prelinked!

Lovechild · Posted: Sun Nov 10, 2002 12:20 pm Post subject:

Lovechild · Posted: Sun Nov 10, 2002 12:45 pm Post subject:

Doing a checkrun on those config change now, I'll get back to you tomorrow with the results.

Lovechild · Posted: Mon Nov 11, 2002 10:23 pm Post subject:

btw... no big succes with those instrutions..... killed X....

fkurth · Posted: Mon Nov 18, 2002 5:53 pm Post subject: results ?

Hi
Are there results in the meantime ?

Does anybody got it to work now ? And how ?
Can anyone post a patch for the newer portage version ?
_________________
--
Felix Kurth

splooge · l33t Joined: 30 Aug 2002 Posts: 636

"Hi ppl, I have make a draft version of the portage changes needed.

I am not a python programmer, or want to be! The code can be optimised a
bit but I believe that prelink was well programmed so all checks for
file type etc are best handled by C based prelink, as long as os.system
is fast.

I have created code which adds support for the FEATURE flag prelink.
If it exists it uses prelink to verify the md5sums.
Also it prelinks files as they are installed.

Most importantly it works for me! Happy testing!

code:
download from
http://cvs.gentoo.org/~cretin/portage.py
This is based on the latest CVS portage

TODO:
* Safety checks for existence of /usr/sbin/prelink
* optimise to minimise number of calls to prelink
* see if a less IO intensive md5sum method exists (of course it does!)

now to sleep .....

--
Stefan Jones <cretin@gentoo.org>
Gentoo Linux"

Lovechild · Posted: Tue Nov 19, 2002 8:01 am Post subject:

AWESOME... thx, now I¨m looking forward to reinstalling Linux

AlterEgo · Veteran Joined: 25 Apr 2002 Posts: 1619

Lovechild · Posted: Tue Nov 19, 2002 11:28 am Post subject:

Fro the look of it, you need to

1) Get needed tools, glibc 2.3.1, lastest binutils, lastest GCC (else compiling gcc after compiling glibc will fail)
2) get the prelinking tool
3) then set prelink as a FEATURE flag in make.conf
4) now upgrade Portage to the CVS version and replace with the python file our kind geek provided
5) Install shit
6) enjoy prelinked shit
7) watch in horror as it breaks shit

Now we might need some workarounds for some packages, like QT and X as stated in this thread already, but kind geeks code looks fairly solid (I'm not a python geek either so what do I know) so I think there should be a good chance of this being relatively painless upgrade at some later point in portage history and once again, the Gentoo users shall roam free with their fast as hell distro...

fkurth · Posted: Tue Nov 19, 2002 11:51 am Post subject:

dont set prelink feature befor installing shit !
at least nasm fails to build with prelinked libs.

If you're installing from scratch, you have to remerge zlib too.
-no-xinerama to qt works fine, but that schould become a use var !?!

just compiling...compiling...compiling...compiling...compiling

What about collecting all infos here and write a short Paper about
setting up prelinking ?

btw.
(for kde-prelink or mozilla you have to edit /etc/prelink.conf)
_________________
--
Felix Kurth

AlterEgo · Veteran Joined: 25 Apr 2002 Posts: 1619

OK I'll start one from scratch. On a new disk

So far, I started with the stage-1 tarball and just after bootstrapping, I emerged prelink (and copied the patched portage.py files over from another install), and then finished the emerge -system.

For comparision, I am building identical boxes with and without prelink.
Example:
[without prelink]
LD_DEBUG=statistics tar
09152:
09152: runtime linker statistics:
09152: total startup time in dynamic loader: 815171 clock cycles
09152: time needed for relocation: 389953 clock cycles (47.8%)
09152: number of relocations: 139
09152: number of relocations from cache: 8
09152: time needed to load objects: 314093 clock cycles (38.5%)

[with prelink]
LD_DEBUG=statistics tar
09154:
09154: runtime linker statistics:
09154: total startup time in dynamic loader: 416912 clock cycles
09154: time needed for relocation: 4061 clock cycles (.9%)
09154: number of relocations: 0
09154: number of relocations from cache: 71
09154: time needed to load objects: 301529 clock cycles (72.3%)

Promising

and I guess it's working problem-free until now

I'll keep you posted

Cretin · Retired Dev Joined: 11 Nov 2002 Posts: 23 Location: UK

Just found this topic on the stuff I'm working on, so I thought I might as well popup.

I will try to fix-up the code this evening to make it work better, I have a few ideas.

The portage.py file is a replacement for the portage version,
/usr/lib/python2.2/site-packages/portage.py

In future I will post diffs.

Also regarding the -fPIC flag, I have run into this a few times and am changing ebuilds as I go though, feel free to contact me with any additional changes that are required.

Will change Xfree ebuild soon.

Till then, bye

Lovechild · Posted: Tue Nov 19, 2002 3:27 pm Post subject:

How about setting up a page that shows this:

Prelinking OK:
These ebuilds work out of the box with prelinking

Prelinking HACK:
These ebuilds currently need these(url) hacks to work with prelinking

Prelinking SHOWSTOPPERS:
These ebuilds have no known workarounds to enable prelinking, or running them causes the ever popular segfault. etc.

Now couldn't we use the FEATURE flag to disable prelinking in SHOWSTOPPER ebuilds... then again... maybe not a good idea...

Cretin · Retired Dev Joined: 11 Nov 2002 Posts: 23 Location: UK

Good idea, there should be a place where all the info is kept.

The easiest way I think would be to create a bug at bugs.gentoo.org
and put all the info there, and that way anyone can add infomation to the cause.

As a dev I could own the bug and put the fixes into gentoo, is this OK everyone?

Cretin · Retired Dev Joined: 11 Nov 2002 Posts: 23 Location: UK

patch: http://cvs.gentoo.org/~cretin/portage-prelink.patch
diffed against the latest portage.py

I think the patch included here is close to the final product, or at
least the principles involved. Some of it can still be polished a bit
more by the gurus, email me with any ideas plz!

Prelink features are only enabled if you have prelink in FEATURES in
make.conf. You also must have prelink installed at /usr/sbin/prelink
(the default location)

What it does:

On unmerges it uses prelink to calculate the md5sums.

On emerge a list of all files installed with the tokens "bin" or "lib"
in their path or name is made. (ie /bin/ls or /opt/something/bin/ls or
libblibble.la ) An attempt is then made to prelink these. This is dirty
but safe. Patches welcome for better code.

How to install:

patch /usr/lib/python2.2/site-packages/portage.py with this patch

/bin/cp -f portage.py /usr/lib/python2.2/site-packages/

/bin/rm -f /usr/lib/python2.2/site-packages/portage.py[co]

/usr/bin/python -c "import py_compile;
py_compile.compile('/usr/lib/python2.2/site-packages/portage.py')"
/usr/bin/python -O -c "import py_compile;
py_compile.compile('/usr/lib/python2.2/site-packages/portage.py')"

add prelink to the FEATURES list and emerge prelink

Have fun!

--
Stefan Jones <cretin@gentoo.org>
Gentoo Linux