Re: [PATCH tracing/kprobes 0/7] tracing/kprobes: kprobe-based event tracer update and perf support

[Frederic Weisbecker <fweisbec at gmail dot com>]

On Thu, Sep 10, 2009 at 07:52:58PM -0400, Masami Hiramatsu wrote:
> Hi Frederic,
> 
> This series fixes bugs and upgrades kprobe-based event tracer
> as a dynamic event tracer on ftrace/perf tools. This also enhances
> tracer output format to show each argument name and event name on
> each entry.
> 
> With this series, users can add trace events dynamically on ftrace
> and use those events with perf tools as below.
> 
> (Step.1) Define new events under new group
> 
> $ echo p:mygroup/myprobe do_sys_open dfd=a0 filename=a1 flags=a2 mode=a3 \
>  > /debug/tracing/kprobes_events
> $ echo r:mygroup/myretprobe do_sys_open rv >> /debug/tracing/kprobes_events
> $ cat /debug/tracing/kprobes_events
> p:myprobe do_sys_open+0 dfd=a0 filename=a1 flags=a2 mode=a3
> r:myretprobe do_sys_open+0 rv=rv
> 
> (You can see that each argument has its name.)
> 
> 
> (Step.2) Perf shows new events
> 
> $ perf list
> ...
>    mygroup:myretprobe                         [Tracepoint event]
>    mygroup:myprobe                            [Tracepoint event]
> ...
> 
> 
> (Step.3) Record events with perf
> 
> $ perf record -f -e mygroup:myprobe:record -F 1 -a ls
> ...
> [ perf record: Captured and wrote 0.081 MB perf.data (~3544 samples) ]
> 
> 
> (Step.4) Perf trace shows the result
> 
> $ perf trace
> version = 0.5
>             perf-1405  [000]     0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=810d3f7 flags=98800 mode=1
>             perf-1405  [000]     0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=bff7650c flags=8000 mode=1b6
>             perf-1405  [000]     0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=bff7650c flags=98800 mode=bff7450c
>             perf-1405  [000]     0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=bff7650c flags=8000 mode=1b6


Nice!

May be another step in the todo-list that would be nice: define the format
for a type. Like it's done from ftrace events.


> 
> 
> (Step.5) You can also use return probes.
> 
> $ perf record -f -e mygroup:myretprobe:record -F 1 -a ls
> ...
> [ perf record: Captured and wrote 0.081 MB perf.data (~3544 samples) ]
> $ perf trace
> version = 0.5
>             perf-1408  [000]     0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=b
>             perf-1408  [000]     0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=c
>             perf-1408  [000]     0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=d
>             perf-1408  [000]     0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=c
> 
> 
> TODO:
> - Implement perf kprobe command to help defining new probes.



Yeah!

I wonder what could be the best workflow to use it.

Imagine the following steps:

- perf kprobe = define kprobes using C expression
- perf record -e our_kprobes
- perf trace

That's way too much.
Especially it's sad to be forced to define a kprobe, then
get back its name, use it with record, and eventually
unsheathe perf trace.

I guess we should choose between the low level, very granular
but uninviting method "kprobe + record + trace" and also an all
in one quick approach.

And that could be chosen from perf kprobe:

Low level:

perf kprobe --define-only [-p|-r] [probe_name] -a1 [arg1] -a2 [arg2] \
	    --format="%s %...."

perf record -e kprobes:probe_name
perf trace

Quick:

perf kprobe -p probe_name -a1 ..... cmdline| -a

And after the profiled task is finished, it could launch perf trace
by itself (or wait for a Ctrl + C if -a/wide profiling)

Well, it's just a brainstorming, having the low level method first
would be already a very nice thing.

I'm really looking forward seeing this C expression-like kprobe creation
tool.

It seems powerful enough to replace printk + kernel rebuild.

No need anymore to write some printk to debug, worrying,
sweating, feeling guilty because we know we'll need yet another
printk() after the reboot, and we even already know where while
it is compiling.

We would build less kernels, then drink less coffee, becoming
less nervous, more friendly. Everyone will offer flowers in
the street, the icebergs will grow back and white bears will...

And eventually we'll be inspired enough to write perf love,
the more than expected tool to post process ftrace "love" events.

Thanks,
Frederic.