Re: Looking over LTT instrumentation

[Tom Zanussi <zanussi at us dot ibm dot com>]

William Cohen writes:
 > Tom Zanussi wrote:
 > > William Cohen writes:
 > >  > This afternoon I looked over the LTT-0.9.6-pre4 kernel patch
 > >  > from http://www.opersys.com/ltt/index.html.  This was to see what event
 > >  > and which locations in the kernel are insrumented, what kind of
 > >  > functionality could be provided with the existing kprobe
 > >  > instrumentation, what things need additional help to implement, and
 > >  > what things are unlikely to be instrumentable.  I will refer to things
 > >  > by the LTT event define in most cases.
 > >  > 
 > >  > There are some LTT probe points that can be implemented with jprobes,
 > >  > probe at function entry and just using arguments passed into function:
 > >  > 
 > >  > 	LTT_EV_PROCESS_WAIT
 > >  > 	LTT_EV_TIMER_EXPIRED
 > >  > 	LTT_EV_MEMORY_PAGE_FREE
 > >  > 	LTT_EV_SOCKET_SEND
 > >  > 	LTT_EV_SOCKET_RECEIVE
 > >  > 	do_page_fault() entry
 > >  > 
 > >  > There are some that could implemented as regular function exit probes:
 > >  > 	do_page_fault() exit
 > >  > 
 > >  > There are some that are basically exit probes but use an argument
 > >  > passed into the function:
 > >  > 	LTT_EV_SOCKET_CREATE (needs type arg)
 > >  > 	LTT_EV_MEMORY_PAGE_ALLOC (needs order arg)
 > >  > 
 > >  > 
 > >  > However, many LTT probes are the middle of functions, e.g. ones for
 > >  > scheduling and exit logging. Commonly, there is some checking that the
 > >  > operation can be done. After all the checks the operation is
 > >  > guaranteed to be sucessful. The probe is inserted after those checks
 > >  > to avoid writing a log entry for something that didn't actually
 > >  > happen. Quite often there is a function call do the operation after
 > >  > the checks, e.g. LTT_EV_MEMORY_SWAP_IN in
 > >  > mm/filemap.c:wait_on_page_bit() immediately before swapin_readahead().
 > >  > Maybe specify have kernel functions specify points where the error
 > >  > checking is all done in the function and a probe could/should go
 > >  > there, with a list of variables live there.
 > >  > 
 > >  > Other alternatives spec of probe points:
 > >  > 1) have instrument callee functions and have predicates to specify
 > >  > 	caller
 > >  > 2) allow instrumentation to specify probe point before a call to
 > >  > 	a particular function in the function.
 > >  > 	Could do instruction after call, but need to be careful because
 > >  > 	the following instruction could be in a different basic block
 > >  > 
 > >  > 
 > >  > The do_traps logging in LTT will not work because kprobes uses traps
 > >  > and it would be recursive.
 > >  > 
 > >  > LTT systemcall logging appears to collect do all or no
 > >  > syscalls. Similarly for traps, all or nothing.  Given that traps and
 > >  > syscall do a wide variety of things, finer grain control of these
 > >  > would be desired.
 > >  > 
 > >  > Need to get data transfer mechanism in place. printks are not going to
 > >  > work well for the volume of data that LTT produces. However, it would
 > >  > be hoped that some of the data analysis could be done in the probe and
 > >  > reduce the need to produce a trace that is analyzed offline.
 > >  > 
 > >  > Wednesday I am going to see about implementing some of the LTT probe
 > >  > points with kprobes.
 > >  > 
 > > 
 > > I believe the LKST project may have already done something along these
 > > lines - I'm cc:ing Karim Yaghmour, the LTT maintainer, who would know
 > > more.  Following the recent lkml thread on the subject, I believe LKST
 > > and LTT have also been combining efforts on a common framework for
 > > high-speed tracing; it would seem to make sense to work with them and
 > > get some synergy between the different tracing tools.
 > 
 > Ahh, more information at http://lkst.sourceforge.net/
 > 
 > > What level of integration with LTT were you planning on?  Would an
 > > 'ltt provider' make sense?  LTT provides a set of events and an API to
 > > enable/disable them - the provider could just use these directly and
 > > not worry about instrumenting anything i.e. I don't see why systemtap
 > > would care whether they're instrumented statically or via kprobes -
 > > that's already taken care of by LTT.  As is the data transfer
 > > mechanism - it uses relayfs for that; in the most recent versions of
 > > relayfs, the file operations are exported so they can be used in other
 > > filesystems - we should be able to put the ltt trace files into
 > > 'tapfs' directories if that's the way we want to go.  (BTW, relayfs
 > > has been pared down significantly in order to 'do one thing and do it
 > > well', which is high-speed kernel->user data transfers and therefore
 > > would't be useful as a general kernel<->user mechanism for systemtap
 > > i.e. it's strictly one-way and doesn't support read(2) or anything
 > > really besides mmap()).  I believe new versions of LTT will also
 > > support a generic filtering callback which can be used to filter the
 > > event stream in the kernel if you want to.
 > 
 > I was mainly looking at LTT to see what points are instrumented and 
 > seeing how hard it would be to get compariable information from kprobes. 
 > I was thinking more along the lines of have the data reduction in the 
 > probes themselves. I am not sure how much would be gained by providing 
 > an LTT provider. The graphical browser for LTT doesn't show the 
 > interesting relationships. It is either showing "the entire forest" or 
 > "a single tree". Are there other tools that make use of the LTT data stream?
 > 

Well, LTT seems to do a lot of what you're thinking of doing already
so IMHO it would make sense to reuse whatever you could rather than
re-doing it in systemtap.  For instance, if you hooked into the LTT
event filtering callback, you could use that to 'dispatch' events to
individual handlers for each event, which would of course be
implemented as systemtap script handlers, but without being tied to an
actual probe.  Same result, different mechanism.  The 'provider'
concept should be able to handle this - I think DTrace 'static'
providers probably work in a similar way.

As for the LTT user-space tools, as I understand it, there are
currently people on the LTT team working on improvements to the
graphical tools - I'm sure they would be more than happy to hear
suggestions.

Also, awhile ago, I created a prototype tool that plugged into the LTT
daemon and made the events available to Perl script handlers.  This
was used to monitor the trace stream in real-time, which could be
useful for some types of applications, but it could also be used
offline to analyze a complete trace stream.  Since it's pretty easy to
quickly write graphical applications using Perl or Tcl/Tk or similar,
providing a userspace interface to the event stream for high-level
scripting languages allows you to write quick and dirty (or more
substantial) applications in cases where you're not happy with the
provided tools.

http://www.listserv.shafik.org/pipermail/ltt-dev/2004-August/000649.html

Tom