A collection of LD_AUDIT bugs that are important for tools

[John Mellor-Crummey]

I was encouraged to notify this list about several LD_AUDIT bugs that have significant impact on performance tools that we are developing for Linux in general and US Department of Energy (DOE) parallel supercomputers in particular.

My team develops the HPCToolkit performance tools (https://hpctoolkit.org <https://hpctoolkit.org/>, https://github.com/HPCToolkit/hpctoolkit <https://github.com/HPCToolkit/hpctoolkit>) under funding from the DOE. We have been modifying our measurement subsystem to interpose itself between an application and the OS using glibc’s LD_AUDIT capability. 

For such tools to succeed, we need many of LD_AUDIT’s features to work. Over the last eight months, we identified six bugs on systems we use. We believe that four are still problems in upstream glibc and two may be fixed (as noted). 

We would like confirmed fixes for the following bugs in upstream glibc:

Issue
Priority
When using an auditor, there is an unacceptable performance degradation of over 10x for PLT calls to small procedures even when neither la_pltenter or la_pltexit is present.
VERY HIGH
When auditing, a dlmopen of a shared library causes a SEGV.
HIGH
la_symbind isn't always called when appropriate. We observed that glibc 2.26 calls la_symbind  when appropriate; glibc 2.28 does not.
HIGH
glibc does not save all necessary registers (e.g. X8 - the indirect result register, truncated SIMD registers) when auditing on aarch64 since the beginning of time.
HIGH
When auditing, a dlopen of a shared library that uses R_X86_64_TLSDESC causes a SEGV. This is reportedly fixed in glibc 2.34.
LOW
An auditor added to an executable at link time with --audit=auditor.so and noted in the DT_AUDIT entry of the dynamic section is not called at runtime. This is reportedly fixed in glibc 2.32. 
LOW

A repository of reproducers for these bugs can be found here: https://github.com/hpctoolkit/auditor-tests <https://github.com/hpctoolkit/auditor-tests>. 

A detailed writeup of everything known about each of these bugs, including links to Red Hat and Sourceware Bugzilla entries, if any are known to exist, can be found here: https://docs.google.com/document/d/1dVaDBdzySecxQqD6hLLzDrEF18M1UtjDna9gL5BWWI0/edit?usp=sharing <https://docs.google.com/document/d/1dVaDBdzySecxQqD6hLLzDrEF18M1UtjDna9gL5BWWI0/edit?usp=sharing> 

Technical stakeholders for platforms that are HPCToolkit’s principal targets under DOE funding, especially the exascale computing program:

Intel
Prime contractor on Aurora exascale system at Argonne National Laboratory
IBM
Prime contractor and processor vendor for Summit and Sierra supercomputers at Oak Ridge National Laboratory and Lawrence Livermore  National Laboratory.
ARM
Stakeholder who wants all ARM Linux platforms to succeed, including Sandia National Laboratory’s Astra supercomputer and SUNY Stony Brook’s A64FX-based Ookami.
AMD
Processor vendor for Frontier and El Capitan exascale supercomputers at Oak Ridge and Lawrence Livermore National Laboratories.
SuSE
Linux distribution provider for Cray systems to be delivered to Oak Ridge and Lawrence Livermore National Laboratories and the A64FX-based system installed at SUNY Stony Brook.
Red Hat
Linux distribution provider for Oak Ridge National Laboratory’s Summit, Lawrence Livermore National Laboratory’s Sierra.
Cray
Prime contractor and system vendor for Oak Ridge and Lawrence Livermore National Laboratories, and SUNY Stony Brook; system vendor for Argonne National Laboratory.


For reference, here is a pointer to the portion of our tool that uses the LD_AUDIT interface: https://github.com/HPCToolkit/hpctoolkit/blob/master/src/tool/hpcrun/audit/auditor.c <https://github.com/HPCToolkit/hpctoolkit/blob/master/src/tool/hpcrun/audit/auditor.c> 

Here are some of the capabilities of LD_AUDIT that we need to work and why:
We use LD_AUDIT’s la_objopen and la_objclose to track what objects are in an application’s address space so that our measurement subsystem can unwind the call stack when a profiling signal is received. Tracking libraries by wrapping dlopen is problematic for several reasons. For instance, a wrapper would need to implement RPATH and RUNPATH semantics because glibc does not provide an alternate dlopen interface (like _dlsym) so that a wrapper can provide the return address in the requesting library as an argument which glibc needs to determine the R_PATH and RUNPATH to use when trying to find the library and its dependencies.
We want to use LD_AUDIT’s la_symbind32 and la_symbind64 to interpose wrappers around key functions, e.g. pthread_create. This enables a tool to intercept functions invoked through pointers obtained with dlsym, which preloaded wrappers can’t do. (Note: We don’t use la_symbind for interposition yet, but we plan to when it works everywhere.)
We need auditing to work when an application or a tool library (e.g., Intel’s GT-Pin) opens a shared library with dlmopen.

We need auditing to work when opening a dynamic library with TLS dialect gnu2 relocations on x86_64 (R_X86_64_TLSDESC). We don’t have any special interest in such relocations; at present, they cause a SEGV when auditing and that must be avoided.
We want to add an auditor to an application at link time, noted in the DT_AUDIT entry of the dynamic section. Loading the DT_AUDIT entry as a program is launched enables our profiler to be injected into an application’s address space without a wrapper script that sets LD_AUDIT and LD_PRELOAD.
LD_AUDIT needs to work on aarch64, which is an important target for our tools. The fact that _dl_runtime_profile does not save register x8 (the indirect result register) is often fatal for applications, which makes LD_AUDIT unusable for any purpose on aarch64.
LD_AUDIT needs to support auditing of inter-object calls on aarch64 when SVE registers are in use. 

As a final thing to consider: we understand that there is a tension between security and auditability. We are concerned that changes being considered for security may compromise observability for tools. For tools, we would need a way to authorize full observability even in the cases when that may theoretically reduce security. Perhaps setting DT_AUDIT could be considered as authorizing full observability.
--
John Mellor-Crummey		Professor
Dept of Computer Science	Rice University
email: johnmc@rice.edu		phone: 713-348-5179