Re: Adding systemtap probe points in pthread library

[Ray Ho <rho at redhat dot com>]

Two more updates related to the pthread library probes:

1) Besides benchmarking on a modern x64 box, I've also benchmarked on an
older PowerPC G5 machine. I wrote the following microbenchmark for
finding the overhead of the probes in the pthread library:

#include <pthread.h>

main()
{
  pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
  int i;

  for (i=0;i<100000000;i++)
  {
    pthread_mutex_lock(&mutex);
    pthread_mutex_unlock(&mutex);
  }
}

Discarding the fastest & slowest times for each one:

Original (in seconds):    31.424, 31.428, 30.929, 30.608

With probes (in seconds): 30.935, 30.791, 30.935, 30.790

So with the probes in the pthread library, there is in fact a speedup of
1.0076, which could be noise, or could be the benefit of better loop
alignment, which can make the the G5 dispatch group work better.


2) To show the benefit of including the probes in libpthread, I just
want to show how easy we can detect deadlocks by taking advantage of the
probes:

Deadlock Code:
=============
#include <pthread.h>

main()
{
 pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER ;

 pthread_mutex_lock(&lock);
 pthread_mutex_lock(&lock);
}

System Tap code:
===============
#! /usr/bin/env stap

global lock_thread_array

probe
process("/home/rayson/s/glibc_bld3/lib/libpthread.so.0").mark("mutex_block") {
      lock_thread_array[$arg1] = tid();
}

probe
process("/home/rayson/s/glibc_bld3/lib/libpthread.so.0").mark("mutex_release") {
      lock_thread_array[$arg1] = 0;
}

probe
process("/home/rayson/s/glibc_bld3/lib/libpthread.so.0").mark("mutex_entry") {
     if (lock_thread_array[$arg1] == tid())
     {
       printf("dead lock detected %p\n", $arg1);
     }
}

Output:
======
[root@computer ~]# stap dl.stp -x 9837
dead lock detected 0x00007fffe82bde70


A more advanced version can detect 2 threads waiting for each other for
locks.

Deadlock Code - 2:
==================
#include <pthread.h>

pthread_mutex_t lock1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t lock2 = PTHREAD_MUTEX_INITIALIZER;

thread_main()
{
 pthread_mutex_lock(&lock2);
 sleep(1); /* hack, but good enough for demos */
 pthread_mutex_lock(&lock1);
}

main()
{
 pthread_t tid;

 pthread_create(&tid, NULL, thread_main, NULL);

 pthread_mutex_lock(&lock1);
 sleep(1);
 pthread_mutex_lock(&lock2);

}

System Tap code - 2:
====================
#! /usr/bin/env stap

global lock_thread_array
global wait_thread_array

probe
process("/home/rayson/s/glibc_bld3/lib/libpthread.so.0").mark("mutex_block") {
      wait_thread_array[$arg1] = 0;
      lock_thread_array[$arg1] = tid();
}

probe
process("/home/rayson/s/glibc_bld3/lib/libpthread.so.0").mark("mutex_release") {
      lock_thread_array[$arg1] = 0;
}

probe
process("/home/rayson/s/glibc_bld3/lib/libpthread.so.0").mark("mutex_entry") {
     wait_thread_array[$arg1] = tid();

     # see which thread is holding this lock
     owner = lock_thread_array[$arg1];

     # see if we have a lock that this thread is waiting for
     foreach ([lock] in wait_thread_array)
     {
       if (wait_thread_array[lock] == owner)
       {
         if (lock_thread_array[lock] == tid())
           printf("dead lock detected: thread = %d, lock owner = %d,
lock addr = %p\n", tid(), owner, $arg1);
       }
     }
}

Output - 2:
===========
[root@computer ~]# stap dl2.stp -x 11133
dead lock detected: tid = 11570, lock owner = 11133, lock addr =
0x0000000000600ba0

Ray



On Fri, 2010-07-23 at 11:55 -0400, Rayson Ho wrote:
> Thanks Roland & Chris for the feedback.
> 
> Roland -- I set the mail client to plain text. My first email to this list was blocked by the list, so I will need to find out what needs to be set additionally.
> 
> Chris - I will google the uprobes / perf userspace tracing you mentioned. I am currently developing on Fedora 13 with the latest glibc git tree. So I will find out what needs to be upgraded on my machine.
> 
> Rayson
> 
> 
> ----- Original Message -----
> From: "Christoph Hellwig" <hch@lst.de>
> To: "Rayson Ho" <rho@redhat.com>
> Cc: libc-alpha@sourceware.org
> Sent: Friday, July 23, 2010 9:01:43 AM GMT -05:00 US/Canada Eastern
> Subject: Re: Adding systemtap probe points in pthread library
> 
> On Wed, Jul 21, 2010 at 11:56:14AM -0400, Rayson Ho wrote:
> > Purpose 
> > ======= 
> > The changes here are systemtap probe points that would enable systemtap understand pthread behavior -- esp. mutex and read/write lock & unlock operations. 
> 
> Care to do it so it works with the upstream uprobes / perf userspace
> tracing instead of non-mainline code that in this form doesn't have any
> chance to be merged?
>