Re: Cross instrumentation with user space tracing enabled

[h patel <hpatel05 at gmail dot com>]

Thanks for the clarifications.
Now I am facing kernel crash issue.
Systemtap's Kernel profiling scripts were working fine until I enabled
utrace support on target.
After applying three utrace patches and enabling utrace support, as
soon as I run any script on target, it gives me following on serial
and then it goes for reboot. Am I missing something ? :

BUG: unable to handle kernel NULL pointer dereference at 00000003
IP: [<c04ea10e>] __list_add+0xa/0x5c
*pdpt = 0000000036814001 *pde = 0000000000000000
Oops: 0000 [#1] SMP
last sysfs file: /sys/module/xt_tcpudp/sections/.text
Modules linked in: topsys xt_mark xt_MARK xt_TCPMSS ipt_REJECT xt_tcpudp iptable
_filter iptable_mangle ipt_vlan_routing ip_gre 8021q atsfilter atsvif force_frag
 ipt_CLSFY ip_tables x_tables rebrid_ioctl rebridging wncreg reg reg_table kcomm
2(P) meru_debug meru_util iTCO_wdt iTCO_vendor_support bonding igb

Pid: 705, comm: monit Tainted: P        W  (2.6.32.10 #20) To Be Filled By O.E.M
.
EIP: 0060:[<c04ea10e>] EFLAGS: 00010286 CPU: 0
EIP is at __list_add+0xa/0x5c
EAX: f1c797dc EBX: f1c79778 ECX: ffffffff EDX: f1c30660
ESI: f1c30660 EDI: f1c797dc EBP: f25c9f54 ESP: f25c9f48
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process monit (pid: 705, ti=f25c8000 task=f756d350 task.ti=f25c8000)
Stack:
 f1c79778 f1c30648 f1c79778 f25c9f64 c04842d4 f1c79778 f1c30648 f25c9f74
<0> c048438b 00000002 f2347d18 f25c9f50 c05b5538 f25c9fac c05b657e 00000002
<0> 00000001 00000000 00000002 00000001 00000000 00000000 c04e68c8 b7806424
Call Trace:
 [<c04842d4>] ? __d_instantiate+0x1b/0xaa
 [<c048438b>] ? d_instantiate+0x28/0x36
 [<c05b5538>] ? sock_attach_fd+0x5f/0xab
 [<c05b657e>] ? sys_socketcall+0x55/0x178
 [<c04e68c8>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c0402898>] ? sysenter_do_call+0x12/0x36
Code: f3 ff 83 c4 14 8b 13 8b 43 04 89 42 04 89 10 c7 43 04 00 02 20 00 c7 03 00
 01 10 00 8b 5d fc c9 c3 55 89 e5 57 89 c7 56 89 d6 53 <8b> 41 04 89 cb 39 d0 74
 17 51 50 52 68 f1 51 72 c0 6a 1a 68 a6
EIP: [<c04ea10e>] __list_add+0xa/0x5c SS:ESP 0068:f25c9f48
CR2: 0000000000000003
---[ end trace 7c8b87fe60b0346b ]---


On Tue, Dec 7, 2010 at 3:29 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
> hpatel05 wrote:
>
>> [...]
>> My setup is one of the most complex use case for systemtap :)
>> - Cross Instrumentation with user space trace support required.
>> - Host : Running CentOS 5.4 with 2.6.18-164.el5 i686
>> - Target : Self compiled Kernel 2.6.32.10 i686, image size max 512 mb.
>
> Sounds fine.
>
>> My queries are as follows :
>> 1) I was able to run normal kernel scripts successfully on target, but
>> user space script is having issue.. [...]
>
> You found the uprobes.ko bit. ?git systemtap contains a fix to the
> client/server code to automate this aspect too.
>
>> 2) Is it possible to use for Systemtap without debuginfo ? e.g. Only
>> with Kernel Markers support and User space traces enabled.
>
> Kernel markers & tracepoints don't require kernel-debuginfo on the
> systemtap compile (host) side. ?Modern user-space markers (sys/sdt.h)
> usually don't require userspace debuginfo either. ?General user-space
> probes may require userspace debuginfo (for the probed processes).
>
>> As of now, I am working the long way of keeping debuginfo for target
>> on Host box itself, I can't afford debuginfo on target as it goes
>> beyond max image size.
>
> Of course. ?None of this is needed on the target.
>
>> 3) This is with performance related. Pardon my kernel knowledge. How
>> the dynamic filtering of traces actually works?
>
> Each kernel & user-space marker/tracepoint is individually controlled.
> Only those you attach to with live stap scripts represent any run-time
> overhead.
>
>
> - FChE
>