Re: Segmentation Fault: Task died at uprobe probepoint

[Gaurav Dhiman <dimanuec at gmail dot com>]

On Mon, Jun 14, 2010 at 9:56 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Gaurav Dhiman <dimanuec@gmail.com> writes:
>
>> [...]
>> I have recently installed SystemTap on my Ubuntu 9.04 based system. I
>> have Linux kernel 2.6.29.1 with utrace support [...]
>
> Great.
>
>> [...]
>> However, when i actually run uprobe-test, although both the probes
>> fire successfully, the task generates a segmentation fault and dies on
>> the 'test_func' probe. I see the following message on doing dmesg:
>> [ 1709.650827] Task died at uprobe probepoint: ?pid/tgid = 3934/3934,
>> probepoint = 0x400517
>
> Almost certainly it's a uprobes or utrace or systemtap bug, rather
> than something you might have done wrong. ?Can you collect more
> information, such as systemtap git version, stap -vvvt -DDEBUG_UPROBES
> traces, disassembly of your probe-target executable?

The following commit is on the head of the git log I have (I don't
know how to get the exact systemtap version):

commit b275171e26c0cf9441858209ac62e70b8ddda71d
Author: Josh Stone <jistone@redhat.com>
Date:   Thu Jun 10 16:05:29 2010 -0700

    PR11690: Remove a dtrace debug-print, and fix probe_perf

    Stan's commit 278c975 fixes the core issue; I'm just polishing...

    * dtrace.in: Comment out the debug-print before calling gcc.
    * scripts/probe_perf/bench.sh: Pass -DSTAP_SDT_V2 to dtrace.

Following are the logs from the Pass 5 of the run with stap -vvvt
-DDEBUG_UPROBES:

Pass 5: starting run.
Running /usr/local/bin/staprun -v -v -u
/tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko
staprun:main:273
modpath="/tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko",
modname="stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686"
staprun:init_staprun:211 init_staprun
spawning: /bin/grep -q unregister_uprobe /proc/kallsyms
staprun:insert_module:60 inserting module
staprun:insert_module:79 module options: _stp_bufsize=0
staprun:init_ctl_channel:31 Opened
/sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/.cmd
(3)
staprun:close_ctl_channel:50 Closed ctl fd 3
execing: /usr/local/libexec/systemtap/stapio -v -v -u
/tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko
stapio:main:37 modpath="/tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko",
modname="stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686"
stapio:init_stapio:317 init_stapio
stapio:init_ctl_channel:31 Opened
/sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/.cmd
(3)
stapio:stp_main_loop:494 in main loop
stapio:init_relayfs:238 initializing relayfs
stapio:init_relayfs:262 attempting to open
/sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/trace0
stapio:init_relayfs:262 attempting to open
/sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/trace1
stapio:init_relayfs:268 ncpus=1, bulkmode = 0
stapio:init_relayfs:345 starting threads
stapio:stp_main_loop:571 probe_start() returned 0
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process
uprobe-test[25763] addr 0000000000400517 pp
process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")
test begin!!
test_func called!
i=0x5
stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process
uprobe-test[25763] reloc 0000000000400517 pp
process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")

The program receives a SIGSEGV signal at this point.

By the probe target executable do you mean, the .ko file that is
generated? The disassembled file of that is pretty big, close to 17000
lines. Do you want that or something else?

Thanks,
-Gaurav