Re: shared library loading performance and prelinking

[Andreas Jaeger <aj at suse dot de>]

Robert Schweikert <rjschwei at cox dot net> writes:

> Hi all,
>
> I have a question about library load performance. We have a large
> application that has 125 shared libraries and a very large number of
> symbols, lots of them from C++ code. One of the issue we are having is
> application start up. Not all libraries are loaded at start up. Most
> libraries are loaded while the user interacts with the program.
>
> Since loading is an issue I have been trying to learn about the issues
> that go along with shared object loading on Linux. One thing that we are
> trying is to use export lists as suggested in Ulrich's paper. We are
> still having some problems with it but that's a topic for another post.
>
> In my search for answers I have run across prelinking and I was trying
> to figure out how it works and what it will do for me as far as load
> performance is concerned. Everything I found said the feature is
> available only in glivc 2.3.x, thus I am waiting for RedHat 8.1 since
> earlier distributions still have glibc 2.2.x. Anyway, can someone give
> me an idea what is happening when prelinking is used, what kind of
> performance gains one can expect and how to use the prelinking feature?

First thing is to measure how long relocations at startup use at all.
glibc comes with LD_DEBUG for this, e.g. try (this is a glibc 2.2.5):
$ LD_DEBUG=statistics /bin/ls  df 
12832:
12832:  runtime linker statistics:
12832:    total startup time in dynamic loader: 964121 clock cycles
12832:              time needed for relocation: 661825 clock cycles (68.6%)
12832:                   number of relocations: 275
12832:        number of relocations from cache: 101
12832:             time needed to load objects: 207215 clock cycles (21.4%)
/bin/ls: df: No such file or directory

Other stuff to try is LD_BIND_NOW, it increases the startup time since
all relocations will be done at start-up time but it should give you
some statistics:
$ LD_DEBUG=statistics LD_BIND_NOW=1 /bin/ls  df
13612:
13612:  runtime linker statistics:
13612:    total startup time in dynamic loader: 1788165 clock cycles
13612:              time needed for relocation: 1484874 clock cycles (83.0%)
13612:                   number of relocations: 1016
13612:        number of relocations from cache: 101
13612:             time needed to load objects: 207482 clock cycles (11.6%)
/bin/ls: df: No such file or directory

Now test this with your application - now you know how much time is
spend in the relocations and how much prelink could bring at all
(bringing down that time by around 90 % according to my tests).

I don't have timings for this but to see the effect of prelinking have
a look:
# LD_DEBUG=statistics ls /tmp/q
     27689:                      number of relocations: 125
     27689:           number of relocations from cache: 8
ls: /tmp/q: No such file or directory
     27689:
     27689:     runtime linker statistics:
     27689:                final number of relocations: 187
     27689:     final number of relocations from cache: 8
# prelink /bin/ls
# LD_DEBUG=statistics ls /tmp/q
     27704:                      number of relocations: 0
     27704:           number of relocations from cache: 46
ls: /tmp/q: No such file or directory
     27704:
     27704:     runtime linker statistics:
     27704:                final number of relocations: 0
     27704:     final number of relocations from cache: 46

prelink comes with documentation, just read the manual page or run
prelink --help
to get started.

Btw. Debian has switched to glibc 2.3.1 and SuSE 8.1 uses 2.3.2 with
prelink support.

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj at suse dot de
   private aj at arthur dot inka dot de
    http://www.suse.de/~aj