Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso <tytso at MIT dot EDU>]

On Mon, Dec 22, 2008 at 03:37:48PM -0500, Frank Ch. Eigler wrote:
> > Yes, but how long will people need to wait before debuginfo quality
> > will improve?  And when such a hypothetical improved compiler be
> > available and trusted to compile the kernel correctly?  [...]
> 
> You misinterpret the status quo.  In reality, debuginfo quality with
> current gcc has been satisfactory for many purposes.  That is what
> enables user-space debuggers to work as well as they do, and indeed
> enables systemtap to work as well as it does.  The major current
> debuginfo quality shortcoming involves only those intra-function
> statement probes.

I can only speak from my personal experience, which is given the
kernel default compilation options, and the inline functions and
static functions in the parts of the kernel that I normally play with
(i.e., ext3/ext4 filesystem code), in nearly every single time where
I've had to try to place a Systemtap probe, if it hasn't been at a
non-static function entry/exit point, the vast majority of the time,
it Simply Doesn't Work.

With user-space debuggers, it's not so great, either, but normally
when I do debugging e2fsprogs, I recompile w/o optimizations to avoid
these frustrations.  Furthermore, e2fsprogs doesn't have the same
large number of static functions which gcc is far too happy to
in-line, and which currently makes the debuginfo largely useless.

Worse yet, the error handling with Systemtap is so uniformly awful (to
the point that even Roland has complained --- "stap -vvvvvvv" anyone?)
that when it doesn't work, it's highly frustrating to figure out what
is going on.  Basically, I've written off anything other than
non-static function entry/exits as being too frustrating to even try using.

It might be that the right answer is to tell kernel developers to
compile their development kernels without optimization.  OK, it that
goes against the Systemtap religion that you must never need to
recompile or use special built options.  But that mantra is causing
massive frustration, since after multiple times of trying to set probe
points inside ext4 code, and watching it Utterly Fail, it causes me to
curse debuginfo, and Systemtap's reliance on it, as Utter Crap.

You can go ahead, and dismiss my reports --- after all, I'm one of the
few kernel developers who's even bothering to try System Tap at this
point, and adding markers into the kernel --- and the first North
American kernel developer and CTO of the Linux Foundation can't
possibly know anything about, you know, kernel development
workflows....

> If you're still talking about debuginfo per se, I am led to believe
> that only a small number of kernel developers are negatively affected:
> those that build large kernel configurations but on such miniscule
> hardware that the sheer disk I/O of the -g data is deemed intolerable.

Heh.  Only a very few number of kernel developers are even *trying* to
use Systemtap at this point.  Trying to keep the few remaining kernel
developers happy might be considered a good thing.  You saw the number
of hands that went up at the Kernel Summit.  After the Plumber's
Conference and the Kernel Summit, Aryan van der Ven told me and a
group of of kernel developers that as a senior manager, he planned to
pull Intel's last resources working on SystemTap off the project,
based on SystemTap developers not being willing to listen to input.
And other kernel engineers employed at Red Hat told me that they
thought the Red Hat SystempTap developers were totally out of control
and they had but very little respect for the project.  You are in a
*hole*, brother.  And the first rule is, when you are in a hole, stop
digging....

> > The process of getting userspace probes in will go slowly (if at
> > all).  I haven't seen a new version of the patches submitted in
> > months --- and the 2.6.29 merge window will be opening soon.  [...]
> 
> Roland?  Please advise.

Note that the best way to get kernel patches in is to handle the
comments, and repost new versions quickly and often with the comments
addressed.  Look at how Eric Paris from Red Hat posted fanotify
patches; multiple versions, addressing comments, every few days.  And
even though many people are dubious about his stated goal of helping
the virus scanner partner's of Red Hat, he's made enough core
infrastructure improvements to other notification systems, such as
inotify and dnotify, that those patches will likely make it into the
mainline.

Quite frankly, the pace at which new versions of the patches are
published in response to comments reminds me doomed patches such as
reiserfs4....

> > This means no modules, since debuginfo is a disaster with modules
> > (so helping people create configurations that don't use modules
> > would be helpful);
> 
> Please elaborate what you mean.  Why systemtap is forced to use
> (create) modules at all has been explained often.

What I mean is that if you build a kernel with a large number of
modules, the bloat of systemtap becomes disastrous.  With a
distro-style kernel, the compilation time literally goes up by a
factor of 4-5 or so, and the disk space utilization goes up by a
factor of 8 or so.  It's not so bad if you only the device drivers you
into a single kernel, and not create any unneeded modules.  This is
often not the common way at least some kernel developers build
systems, since (a) modules make it easier to load and unload
subsystems without rebooting, and this can shorten the
compile-edit-debug cycle times, and (b) some kernel developers want to
be sure they know whether any infrastructural changes they might cause
some random device driver or other random subsystem to break.
Unfortunately, building a distro-style kernel is a horrible disaster
with debuginfo enabled, and so warning people off about this
particular issue would be helpful.

> Sure, upstream changes in the location of Modules.marker are something
> we can easily adapt to.  Are you aware of a current problem with that?
> On the other hand, we plan to work on also connecting straight to
> tracepoints, and to various slices of ftrace, for which a file such
> the troubled-history Modules.markers will not be necessary.

Only that "make install" or "make modules_install" doesn't currently
install Modules.marker anywhere useful.  And there is no standard
location for where Modules.marker will be installed given current
distribution packaging tools.  Attention to consumability of SystemTap
given a variety of different kernel installation schemes is a Really
Good Idea.  It's not just Fedora RPM-style macros and "make install".
There are many other ways that people create and install kernels; and
this kind of attention to detail will stop people from getting
Frustrated with SystemTap.

> > 3) Actually try to find a way for Systemtap to be more useful
> > *without* waiting for [new gcc]
> 
> As I said above, the main thing we're expecting from a future gcc is
> improvement on statement (arbitraty source line) probing.  Otherwise
> quality is fine and we're not having to wait.  (As everyone knows,
> quantity (compression) is not great but at least this is only a
> usability rather than functionality matter.)

It's a rather important usability issue.  Part of the reason why
SystemTap is so frustrating is precisely because there hasn't been a
big enough focus on usability, IMHO.  Maybe systemtap developers get
used to stap -vvvvvv to figure out why something doesn't work, but it
still strikes me as a reminder of how awful and kldugy the technology
is.

> Nor do we.  In fact, we've been as unimposing as possible upon both
> kernel and user-space build systems -- perhaps to a fault.
> 
> > [...] the only advantage SystemTap has over ftrace is (a) Systemtap
> > has access to the function parameters, and (b) Systemtap allows you
> > to filter logging information based on conditions, which is more
> > efficient than filtering in userspace.
> 
> Beyond those, there is really plenty of difference in capability.
> Line-by-line function tracing?  User-space?  Concurrent users?  The
> list goes on...

Line-by-line function tracing doesn't work for me.  I've given up on
it as useless.   Yawn.

User-space tracing --- is a patch that I don't forsee going mainline
anytime soon, given a lack of understanding about how to respond to
comments and how to get patches upstream.  (Again, there are people in
Red Hat who know how to do this; why isn't Roland reaching out to
people like Eric Paris, David Jones, Alan Cox, etc.?)

Concurrent users -- doesn't matter to kernel developers.  And exactly
who do you expect is going to be adding markers to the kernel sources,
if you continually blow off the concerns of kernel developers such as
James and myself?

> > [...] is to simply generate the debuginfo information on demand so
> > the function parameters on entry can be decoded.  We have the header
> > files after all, so it shouldn't be hard to generate a pseudo function:
> > 
> > extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
> >        void *ptr)
> > {
> >         return 0;
> > }
> 
> It has some potential.  On first blush this seems to require parsing
> all the C header files (accounting for arch/kconfig cpp macros) to
> look for a matching function declaration.  Then the parameter/return
> types may require listing additional C headers.  As you know, kernel
> headers are complicated and rich with #ifdefs and macros that may
> expand to inlines inlines, so a naive text processing inspired by the
> following may not be worthwhile.  How do you imagine it working?

Well, the original way I was thinking about doing this involved
support from the kernel sources --- so changes to kbuild to generate
debugging information that is far more compact, only optimized to
provide kernel function parameter information --- since as I have
mentioend, line-by-line probing in practice doesn't work for me, so
why pay for a factor 8 explosion of debuginfo overhead when most of it
seems largely useless given standard kernel compile options.

However, it might be doable without having made any kernel changes.
Finding which header file a function parameter can be found is pretty
simple.  There are some places where the function names is created
using magic cpp concatenation games, but these tend to be inline
functions which systemtap can't probe anyway.  The hard part is
knowing which other header files might be needed.  This can be easily
extracted from source files --- so the question is whether Systemtap
ships a database of which header files are needed before it's safe to
include a particular .h file containing the function prototype; or
whether we need a kbuild patch to extract this information; or do both.

	   	  	       	  	       		    - Ted