Re: proposed instruction trace support in SystemTap

[fche at redhat dot com (Frank Ch. Eigler)]

Dave Nomura <dcnltc@us.ibm.com> writes:

> [...]  The single_step and branch_step syntax identifies the handler
> code and needs to be set up before the user probe code that turns
> on//off instruction tracing.

That goes without saying.  Whatever setup need be done must be done before
explicit on/off operations become available to script code.

> I am assuming that since you have access to the pid() function in
> your *.stp script, that the user program is invoked after all of the
> probes have been processed, so I'm not sure I understand how the
> actual instruction tracing events would get lost.  

Consider the scenario where tracing cannot be toggled on and off
atomically, and yet the probe handle requesting this toggling prefers
not to block.  So we may have an enqueued "turn tracing back on"
request that may take a little while to satisfy.  During this time, we
may lose events.

True, this might not apply to uprobes if we choose to take advantage
of its handlers' ability to sleep.  But it would apply to other types
of probes.



> > Is the "raw trace data" a well-defined thing?  Why would this sort of
> > hard-coded data set be desirable, as opposed to letting the programmer
> > write something explicit like:
> >    printf("%2b%8b%4b", cpu(), get_ticks(), pc())
> > (Of course this can be hidden in a function of his own, or in an
> > inspectable tapset.)
> >
> The user could do the simple printf that you suggest.  The proposed
> callout to itrace_output() would only by used if you wanted more
> detailed information (like timestamp) as required by a tool like
> qtrace [...]

It sounds like what we may need is a collection of functions that
print tracing data in formats compatible with specific post-processing
tools.  It's not "raw trace data" in the itrace sense - it is trace
data in the *qtrace* sense - and that of other tools.


> > Is this sort of thing done/doable in kernel space also, or just on
> > user-space threads?  Is there an existing kernel API for management of
> > these registers/fields?

> [...] There is basically one single instruction trap handler that
> the stap translator will generate with logic to figure out what
> handler code to run [...]

The "existing kernel API" is the key issue here.  How exactly does one
activate single-stepping traps on each of the interesting
architectures, and on multiple different kernel generations (RHEL)?
How does one hook into the handling system correctly (avoiding
interference to other consumers of trap data like gdb, uprobes)?

These questions need answers in order for systemtap to generate code
to implement this.


> One issue that Jim Keniston identified is that we would want some
> way to not trace any of the nstructions in the kernel code
> associated with stap probes, trap handler, etc.  [...]

It's a complicated area with potential to easily bring a system down.
Let's stay away from it until we get some more experience with
user-space.


- FChE