RFC stap/java backtracing

[Lukas Berk <lberk at redhat dot com>]

Hey,

I've attached my patch for native java backtraces for systemtap.

How to use it:

I've added a java_backtrace() function which returns the backtrace as a
string.  One would use it as any other stap function, ie;

[lberk:~/code/tmp/classes]$ stap -DMAXSTRINGLEN=6000 -ve
'probe java("org.my.AppMain3").class("org.my.AppMain3").method("printMessage(String)")
{log("hit ".user_string($arg1)." ".java_backtrace())}'
    Pass 1: parsed user script and 99 library script(s) using 212540virt/28056res/2996shr/25632data kb, in 110usr/10sys/109real ms.
    Pass 2: analyzed script: 4 probe(s), 8 function(s), 2 embed(s), 1 global(s) using 367904virt/30900res/4204shr/27088data kb, in 10usr/60sys/79real ms.
    Pass 3: translated to C into "/tmp/stapImwsw0/stap_fa9d62d13587f9ebca29e3411b8fa6a0_4608_src.c" using 365428virt/30876res/4260shr/27088data kb, in 10usr/70sys/83real ms.
    Pass 4: compiled C into "stap_fa9d62d13587f9ebca29e3411b8fa6a0_4608.ko" in 1560usr/170sys/1758real ms.
    Pass 5: starting run.
    hit foobar java.lang.Throwable
            at org.systemtap.byteman.helper.HelperSDT.STAP_BACKTRACE(HelperSDT.java:9)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.jboss.byteman.rule.expression.MethodExpression.interpret(MethodExpression.java:315)
            at org.jboss.byteman.rule.Action.interpret(Action.java:144)
            at org.systemtap.byteman.helper.HelperSDT_HelperAdapter_Interpreted_1.fire(toium-5999.btm)
            at org.systemtap.byteman.helper.HelperSDT_HelperAdapter_Interpreted_1.execute0(toium-5999.btm)
            at org.systemtap.byteman.helper.HelperSDT_HelperAdapter_Interpreted_1.execute(toium-5999.btm)
            at org.jboss.byteman.rule.Rule.execute(Rule.java:682)
            at org.jboss.byteman.rule.Rule.execute(Rule.java:651)
            at org.my.AppMain3.printMessage(AppMain3.java)
            at org.my.AppMain3$1.run(AppMain3.java:27)
     

How it works:

To start, java_backtrace() sets a new pragma:java_backtrace, which adds
a #define JAVA_BACTRACE in the kernel module.

Within parsing the probe point the user specified, I've added another
parameter to be passed to stapbm, signaling if a backtrace is going to
be needed.  This is done through the _bt() pure c function in java.stp
as well.  If its not, then carry on as usual.  However, if it is, we
add another helper method directly before calling our 'original' native
helper method, STAP_BACKTRACE('rulename').   STAP_BACKTRACE simply
triggers the e.printStackTrace() call, converting the output to a single
string, and passes said string through to a new native function
(METHOD_STAP_BT) along with the rulename.  The new native function is
required to make sure we don't overwrite/mangle a possible probe point
that only has on parameter (similar to the exapmle I used above), and
has the added advantage of being able to immediately call the correct
jni functions to exract the string, no guessing/trail and error
required.  

From systemtap's perspective, we watch for this information by adding
another probe point, ie
'process("/path/to/libHelperSDT_*").mark("method__bt"), and then passing
the string back to the probe point specified by the user via a global
variable, which is then the return value of java_backtrace().

I think the main issue is with possible collisions with the global
variable and multiple, concurrent probe points requesting stacktraces.
On a second look it may be possible to ditch the pragma, and trigger
adding #define JAVA_BACKTRACE simply via an internal variable, which
could allow the global to be internally defined as well (with a unique
runtime name, specific to the probe point).  The issue then would be to
have the java_backtrace() function return the proper global variable to
the original probe point (perhaps by returning the result of another,
internally defined function, ie __java_backtrace:string(), which is
passed the uniquely named global variable).  If there is a better or
more simple way please let me know.


Other Improvements:

- Add rulename check in the method__bt probe
- Iterate over the backtrace and pass it line by line through stap to
  avoid needing to set -DMAXSTRINGLEN (or just require it?)
- Make the method__bt probe point optional in cases where no backtrace
  is required (currently its always defined)

Any comments/thoughts/concerns are welcome and much appreciated!

Cheers,

Lukas

Attachment: java_backtrace.patch
Description: Text document

Attachment: pgpdF1bLV6Gu9.pgp
Description: PGP signature