This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: Differences of stack traces between linux kernel and systap
- From: Andy Gospodarek <gospo at redhat dot com>
- To: Peter Teoh <htmldeveloper at gmail dot com>
- Cc: systemtap at sources dot redhat dot com
- Date: Fri, 2 May 2008 13:15:13 -0400
- Subject: Re: Differences of stack traces between linux kernel and systap
- References: <804dabb00805020935u643adf07s7225caeac93d9e92@mail.gmail.com>
On Sat, May 03, 2008 at 12:35:41AM +0800, Peter Teoh wrote:
> Notice the diference in the stack trace between systap:
>
> 0xc047ecf1 : slab_pad_check+0x1/0xb9
> 0xc064dd69 : kretprobe_trampoline_holder+0x18/0x38
> 0xc064dd69 : kretprobe_trampoline_holder+0x18/0x38
> 0xc064dd69 : kretprobe_trampoline_holder+0x18/0x38
> 0xc0434e33 : send_signal+0x4a/0x10e
> 0xc0435476 : __group_send_sig_info+0x6e/0x8c
> 0xc0435a8e : group_send_sig_info+0x43/0x58
> 0xc048d42a : send_sigio+0xe3/0x133
> 0xc048d4c9 : __kill_fasync+0x4f/0x52
> 0xc048dc78 : vfs_ioctl+0x0/0x6c
> 0xf8943ef5 : snd_timer_user_tinterrupt+0x11f/0x13c [snd_timer]
> 0xf8942c74 : snd_timer_interrupt+0x1de/0x23f [snd_timer]
> 0xf896667d : snd_pcm_period_elapsed+0xf0/0x134 [snd_pcm]
> 0xf8bfef49 : azx_interrupt+0x60/0xa4 [snd_hda_intel]
> 0xc0459eab : handle_IRQ_event+0x26/0x4d
>
>
> And that of the linux kernel (latest patch):
>
> [<c042bad6>] warn_on_slowpath+0x46/0x56
> [<c0415a33>] ? apic_wait_icr_idle+0x16/0x1d
> [<c0415243>] ? __send_IPI_dest_field+0x50/0x54
> [<c04020e5>] ? send_IPI_mask+0xd/0xf
> [<c046773c>] ? get_pageblock_flags_group+0x50/0x6e
> [<c046777e>] ? get_pageblock_migratetype+0x24/0x27
> [<c0468472>] ? free_hot_page+0xf/0x11
> [<c0468494>] ? __free_pages+0x20/0x2b
> [<c047f471>] ? __free_slab+0xac/0xb4
> [<c0480754>] kmem_cache_destroy+0xfe/0x108
> [<f8d337c0>] nf_conntrack_cleanup+0x53/0x7a [nf_conntrack]
> [<f8d3766d>] nf_conntrack_standalone_fini+0x1c/0x1e [nf_conntrack]
> [<c044b56f>] sys_delete_module+0x177/0x1af
> [<c0472c00>] ? remove_vma+0x31/0x53
> [<c0473468>] ? do_munmap+0x182/0x19c
> [<c0404bae>] sysenter_past_esp+0x6a/0x90
> [<c0640000>] ? pci_scan_bridge+0x1dc/0x2eb
>
> The difference lies in the "?" in front of the function, which is to
> indicate that this function has just been executed and returned, so it
> is no longer on the stack.
>
I'm pretty sure that on x86 arch's the kernel's stack traces are really
more like intelligent guesses rather than guarantees. The kernel
searches back though it's stack and checks for possibly valid function
addresses and prints those out.
Other arches (like some PPC flavors) do this much better since they have
a dedicate register for storing stack frames and pre-defined stack frame
sizes and parameters. Those are much more trustworthy, but I find the
backtraces on x86 to be quite usable.
> Is it possible to modify the systap stacktrace dump to be inlined with
> that of the kernel? It this not more informative?
Can you quantify the performance impact of something like this running
all the time?