This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: [TOOL] kprobestest : Kprobe stress test tool
- From: Masami Hiramatsu <mhiramat at redhat dot com>
- To: Frederic Weisbecker <fweisbec at gmail dot com>
- Cc: Ingo Molnar <mingo at elte dot hu>, Steven Rostedt <rostedt at goodmis dot org>, lkml <linux-kernel at vger dot kernel dot org>, Ananth N Mavinakayanahalli <ananth at in dot ibm dot com>, Avi Kivity <avi at redhat dot com>, Andi Kleen <ak at linux dot intel dot com>, Christoph Hellwig <hch at infradead dot org>, "Frank Ch. Eigler" <fche at redhat dot com>, "H. Peter Anvin" <hpa at zytor dot com>, Jason Baron <jbaron at redhat dot com>, Jim Keniston <jkenisto at us dot ibm dot com>, "K.Prasad" <prasad at linux dot vnet dot ibm dot com>, Lai Jiangshan <laijs at cn dot fujitsu dot com>, Li Zefan <lizf at cn dot fujitsu dot com>, PrzemysÅawPaweÅczyk <przemyslaw at pawelczyk dot it>, Roland McGrath <roland at redhat dot com>, Sam Ravnborg <sam at ravnborg dot org>, Srikar Dronamraju <srikar at linux dot vnet dot ibm dot com>, Tom Zanussi <tzanussi at gmail dot com>, Vegard Nossum <vegard dot nossum at gmail dot com>, systemtap <systemtap at sources dot redhat dot com>, kvm <kvm at vger dot kernel dot org>, DLE <dle-develop at lists dot sourceforge dot net>
- Date: Thu, 20 Aug 2009 21:00:07 -0400
- Subject: Re: [TOOL] kprobestest : Kprobe stress test tool
- References: <20090813203403.31965.20973.stgit@localhost.localdomain> <4A847E30.9050903@redhat.com> <20090820184331.GA6078@nowhere> <4A8DA7CE.2040702@redhat.com> <20090821000108.GC6078@nowhere>
Frederic Weisbecker wrote:
Most of them can be fixed just by adding __kprobes.
Some of them which are already in the another section, kprobes
should check the symbols are in the section.
You mean the blacklist?
I also fear that putting bad kprobed functions into the kprobe
section or into the blacklist may hide some kprobe internal bugs.
Doing so is indeed mandatory for functions that trigger tracing
recursion of things like that, but what if kprobe has an internal
bug that only triggers while probe a certain class of function.
Ie: it would be nice to identify the reason of the crash for
each culprit in these lists.
>
> That may even help to find the others in advance.
Indeed, actually I've found some bugs while making jump-optimization
patches by using this stress test.
But some of them are obviously what we just forget to add __kprobes,
since those will be called from kprobes int3 handling functions.
And also, many lock-related code has been changed. I think
kprobes should use raw_*_lock, or prohibit to probe lock monitoring
functions like lockdep, because it will cause recursive call.
Also kprobes seems to be a very fragile feature (that's what
this selftest unearthes at least for me).
And it really needs a recursion detection that stops every kprobing
while reaching a given threshold of recursion. Something
that would dump the stack and the falling kprobe structure.
Hmm, kprobes already has recursion detection(kp->nmiss), so
maybe, we can check it.
That would avoid such hard lockups and also help to identify
the dangerous symbols to probe.
The problem is that I don't have any serial line in this
box then I can't catch any crash log.
My K7 testbox also died in my arms this afternoon.
But I still have two other testboxes (one P2 and one P3),
hopefully I could reproduce the problem in these boxes
in which I can connect a serial line.
Thank you for helping me to find it!
I've pushed your patches in the following git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/fgrederic/random-tracing.git \
tracing/kprobes
So you can send patches on top of this one.
Great! I've found another trivial bugs, so I'll fix those on it.
Cool :)
Btw, here is the result of your stress test in a PIII (attaching the log
and the config).
Thanks, I'll check that.
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: mhiramat@redhat.com