Re: systemtap java hotspot backtrace support

[jon dot vanalten at redhat dot com]

Hi,

Gave this a go, seems to work as advertised (for better or worse).  I tend to agree that the dependency on info extracted in the vm_init probe is less than desireable.  From a usefulness perspective, this is especially because an attempt to target an already-running java process will fail horribly.  Even so, once the x86_64 blocker is resolved and the unreleased systemtap fixes you mentioned are released, I'd also like to see this in IcedTea with the tapsets already there.  It might then be a good idea to note the systemtap version dependency in the comments at the top of the script.

cheers,

jon


----- "Mark Wielaard" <mjw@redhat.com> wrote:

> Hi,
> 
> Attached is a systemtap helper script to go with the other tapsets
> (hotspot.stp and hotspot_jni.stp) in icedtea. (And some
> build/configury
> patches to slot it in the build).
> 
> At the moment on x86_64 it is blocked on systemtap bug #11034, but
> when
> that is solved I like to check this in as a first step (it works fine
> on
> i386 with either client or server libjvm.so and a smaller version -
> with
> some of the frames smarts removes works fine on x86_64). You will
> need
> the latest systemtap from git for the moment though, since it relies
> on
> some recent bug/feature fixes on git trunk.
> 
> With this you can get java backtraces, with or without full method
> signatures and with or without native frames in between, from any
> hotspot probe context. You can directly print the stack to the trace
> log, or collect it as a space separated string to inspect (and if
> needed
> tweak some parameters to say how many frames you want and whether or
> not
> to include method signatures and/or native frames). There is
> documentation in the jstack.stp script about the various jstack()
> function variants.
> 
> There are a couple of points that can use improvements (suggestions
> how
> to do it very welcome):
> 
> - Collecting of frames isn't as useful as it looks since they are
>   limited by MAXSTRINGLEN.
> - The server (c2) compiler can trash the frame pointer, so we try to
>   catch up by inspecting the stack. This mostly works, but can miss a
>   frame. It would be nice to be able to retrieve the register map for
>   the frame somehow.
> - Similar to the above, we collect "native frames", while it would be
>   nice to somehow collect the "virtual frames" (so inlined code
> expands
>   to the corresponding java frames again).
> - We are using the server libjvm.so dwarf info to extract all
> structure
>   info, this is actually wrong, we should use the client libjvm.so
> when
>   the client libjvm.so is probed (but the key structures are the
> same).
> - The @casts in the actual jstack_call() function look somewhat ugly
>   because they need to explicitly reference the absolute libjvm.so
> path.
> - Related to the above, $vars cannot be used in stap functions
> (unlike
>   @cast), which means we need to extract some info in a global probe
>   (hotspot.vm_init_ended). Some way to associate an helper function
> with
>   the probed module would be really nice.
>   This creates two major limitations:
>   - When starting to trace after this global probe has triggered
>     jstack() just doesn't work.
>   - Only one java process at a time can be traced since multiple
>     global variables will conflict otherwise (this impacts tracing
>     eclipse and netbeans, which start in "stages" running multiple
>     java processes).
> - For native frames it helps to add -d .../.../libjava.so also to see
>   the core library jni helper functions. Would be nice to somehow
>   include this (and maybe other libraries under jre/lib)
> automatically.
> 
> If anybody uses the jstack script please let me know how it works
> out.
> And if you hit any of the above limitations or have ideas how to
> resolve
> them, please also let me know.
> 
> Cheers,
> 
> Mark