Re: Test suite results for ARM with uprobes

[Wade Farnsworth <wade_farnsworth at mentor dot com>]

Hi Mark,

Thanks for your analysis!

Mark Wielaard wrote:
> On Tue, 2011-12-06 at 12:45 -0700, Wade Farnsworth wrote:
[...]
> > First, four tests are causing hangs, panics or other kernel errors.
>
> In general I found that arm kernels pre-3.0 (2.6.40 in fedora speak)
> were somewhat unstable. There were lots of kprobe cleanups in 3.0.

Ah, yes.  You did mention that previously.  I'll see about retesting 
with something more recent.

[...]
> > FAIL: debugpath-good (eof) [This one puzzles me.  I've built with
> > CONFIG_DEBUG_INFO enabled and the build directory exists in
> > /lib/modules/`uname -r`/build.  Is there something else that I need to
> > do to get systemtap access to the debug info?]
>
> It fails because:
> spawn env SYSTEMTAP_DEBUGINFO_PATH=/lib/modules/2.6.37.6-yocto-standard+/build s
> tap -e probe kernel.function("vfs_read") {} -wp2
> semantic error: missing arm kernel/module debuginfo under '/lib/modules/2.6.37.6
> -yocto-standard+/build' while resolving probe point kernel.function("vfs_read")
> Pass 2: analysis failed.  Try again with another '--vp 01' option.
> FAIL: debugpath-good (eof)
>
> It PASSes for me with:
> spawn env SYSTEMTAP_DEBUGINFO_PATH=/usr/lib/debug stap -e probe kernel.function(
> "vfs_read") {} -wp2
> # probes
> kernel.function("vfs_read@fs/read_write.c:306") /* pc=_stext+0x1445e0 */ /*<- k
> ernel.function("vfs_read") */
> PASS: debugpath-good
>
> The debugpath.exp testcase does:
>
> # Guess where debuginfo is installed
> if [file isdirectory /usr/lib/debug] {
>    set debuginfo_path "/usr/lib/debug"
> } elseif [file isdirectory /lib/modules/$uname/build] {
>    set debuginfo_path "/lib/modules/$uname/build"
> } else {
>    set debuginfo_path "/lib/modules/$uname"
> }
>
> So, I guess it is guessing wrongly?

Hmm, I do have the debuginfo-enabled vmlinux in 
/lib/modules/2.6.37.6-yocto-standard+/build, which stap seems to 
otherwise be able to find.  For some reason it seems to only occur when 
SYSTEMTAP_DEBUGINFO_PATH is used.  I'll do some more digging.

[...]
> > FAIL: inlinedvars-m32-O
> > FAIL: inlinedvars-m32-O2
> > [line 1: expected "call (22,84)"
> > Got "  (84,22)"]
>
> That is interesting, it switches the numbers around?
> It obviously doesn't run for me. It would be interesting to see the
> debuginfo debug-dump of the created binary, the disassembly of function
> "m" and the generated stap script C source code. Could you create a bug
> report with that info in it?

Done.  http://sourceware.org/bugzilla/show_bug.cgi?id=13485

[...]
> > FAIL: sdt -O2  uprobe
> > FAIL: sdt c89  uprobe
> > FAIL: sdt c99  uprobe
> > FAIL: sdt c99 -pedantic uprobe
> > FAIL: sdt gnu99  uprobe
> > FAIL: sdt gnu99 -pedantic uprobe
> > FAIL: sdt c++98  uprobe
> > FAIL: sdt c++98 -pedantic uprobe
> > FAIL: sdt gnu++98  uprobe
> > FAIL: sdt gnu++98 -pedantic uprobe
> > FAIL: sdt c++0x  uprobe
> > FAIL: sdt c++0x -pedantic uprobe
> > FAIL: sdt gnu++0x  uprobe
> > FAIL: sdt gnu++0x -pedantic uprobe
> > [semantic error: unable to find local 'arg1' near pc 0x8408  in  call1
> > /home/root/stuff/systemtap/testsuite/systemtap.base/sdt.c (
> > (alternatives: $a): identifier '$arg1' at
> > /home/root/stuff/systemtap/testsuite/systemtap.base/sdt.stp:8:18.]
> >
> > FAIL: sdt_va_args base
> > FAIL: sdt_va_args c89
> > FAIL: sdt_va_args c99
> > FAIL: sdt_va_args gnu99
> > FAIL: sdt_va_args c++98
> > FAIL: sdt_va_args gnu++98
> > FAIL: sdt_va_args c++0x
> > FAIL: sdt_va_args gnu++0x
> > [similar to the sdt failures above]
>
> I am unable to run these tests, but it would be interesting to see
> debuginfo dump, disassembly and generated stap script C source for
> these.

The failure occurs before the C source is generated I have uploaded the 
other info to:
http://dl.dropbox.com/u/40714612/stap-debug-files.tar.gz

[...]
> > FAIL: vta-test-m32-O
> > FAIL: vta-test-m32-O2
> > [semantic error: failed to retrieve location attribute for local 'a'
> > (dieoffset: 0x181): identifier '$a' at
> > /home/root/stuff/systemtap/testsuite/systemtap.base/vta-test.stp:2:27]
>
> I cannot run this tests. It would be interesting to see the debuginfo
> dump for the generated binary.

Also in the tarball on dropbox.

[...]
> > FAIL: buildok/seventeen.stp [semantic error: unable to find local
> > 'nfs_program' near pc 0xc01bc714  in  nfs_fsync_dir]
>
> This one does PASS for me. We should probably compare generated
> debuginfo for our kernels.

Kernel debuginfo is also in the tarball.

[...]
> > FAIL: semok/config_number.stp [semantic error: probe point mismatch at
> > position 0]
>
> This one PASSes for me.
> What is CONFIG_NR_CPU set to for your kernel?
> It is set to 2 for me.

CONFIG_SMP is not set, so CONFIG_NR_CPU doesn't exist in my config file. 
 Maybe stap treats that as being equal to zero?

I'll keep poking at the other failures.  Thanks again!

Regards,

-Wade Farnsworth