Re: [RFC] SystemTap future direction

[Masami Hiramatsu <masami dot hiramatsu dot pt at hitachi dot com>]

Mark Wielaard wrote:
> On Thu, 2010-08-05 at 21:27 +0900, Masami Hiramatsu wrote:
>> Mark Wielaard wrote:
>>>>>> - Port SystemTap on the perf/ftrace but drop embedded-C support.
>>>>>>  This will enhance perf/ftrace to support enough flexible data
>>>>>>  filter/modifier (including fault injection feature). In this case,
>>>>>>  SystemTap scripts will handle the data in user-space (not on-line).
>>>>> I think the "not on-line" part is a bit of a showstopper. Since that
>>>>> kills the main idea of having powerful scriptable observability. Simple
>>>>> filters are too restrictive IMHO. It might be enough for simple
>>>>> profiling, where you analyze the data off-line afterwards. But that
>>>>> isn't an option for everybody (you need to store/push the data
>>>>> somewhere), and not very efficient some cases.
>>>> The efficiency is the key, and perf and systemtap aim to
>>>> different efficiency. SystemTap focuses on the efficiency of
>>>> transporting data, but perf focuses on the efficiency of
>>>> probing time. What they are trying to is reducing the overhead
>>>> of recording data to buffers, because it is less disturbance for
>>>> the performance of target processes.
>>> Right. It just comes down to priorities of the different goals.
>>> Profiling (with offline analysis) versus scriptable tracing (with some
>>> debugging elements). But making either efficient will help both cases.
>>> We just have to be careful not to trade in one completely for the other,
>>> or we kill useful use cases at probing time.
>> Hmm, could you find any useful use-case of scriptable tracing (on-line)?
>> I know generally scriptable interface provides us great flexibility, but
>> does it have to be done on-line? I mean, perf already has scriptable
>> interface(python and perl) for off-line analysis. Why isn't that enough?
> 
> But you should know already! :) You wrote the kprobe-based event tracer!
> Allowing to do online filtering based on simple register and memory
> values on-line. So you don't need to do the filtering afterwards
> off-line.

Hm,

> SystemTap just extends that basic idea with making on-line tracing
> decision based on global variables, associate arrays, statistical
> variables and some context gathering functions. All so you can easily
> zoom into what is really important in your particular environment.
> 
> The off-line analysis through perf/python/perl is fine if you can afford
> recording all the data you might be using at all the probe trace events.
> And for some kinds of statistical profiling that is precisely what you
> need anyway. Dump all data and analyse it off-line.

Agreed.

> But this obviously has a cost. Say, you are interested in getting a
> backtrace for when a certain event triggers, in the reference counting
> gnome example when the count goes up to > 1024 for a particular object
> type, but only if allocated from a particular module after startup,
> which you mark as when the number of different objects created passes a
> certain threshold. You can easily express that with systemtap using a
> global variable and associate array. Only when the "context" is right do
> you gather all the context data at the probe point, dump a stack trace
> and the aggregated data. You can make that decision on-line before
> gathering all the possibly relevant data at the probe point.

Yeah, that's a good case when the systemtap on-line analysis works fine!

> If you need to post-process off-line and don't have "on-line storage"
> for keeping the statistics/associate arrays then you need to dump the
> full context with every probe, including for example the backtraces
> (which is expensive) since you don't know whether or not you might need
> it. The post-processing off-line tools can of course then filter out all
> unneeded data for you. But you still need to generate all the data
> first, and then process it all (from a different context - context
> switches aren't cheap either). That does disturb your system, especially
> with larger data sets. If you are unlucky you could actually get into a
> situation where the data dumping & post-processing tools generate more
> i/o, syscalls, memory usage, etc. than the issue you are trying to
> observe in the first place.

So we might better start with useful concrete examples which show
the actual performance differences between on-line and off-line.

Thank you for giving us good ideas! :)

> 
> Cheers,
> 
> Mark
> 
-- 
Masami HIRAMATSU
2nd Research Dept.
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com