Experiences getting SystemTap work on Nokia N800

[Eugene Teo <eteo at redhat dot com>]

Experiences getting SystemTap work on Nokia N800
http://eugeneteo.livejournal.com/7484.html

Setup of SystemTap on Host Machine

The existing systemtap-20070712 CVS snapshot does not allow cross-compiling of systemtap
scripts between different architectures. In order to bypass the architecture check (for
now), systemtap was modified to include the translate.diff patch[1] in the systemtap
bugzilla PR 4186[2]. The stp_check.in[3] file was modified to use sh instead of bash,
because N800 uses busybox (ash), and bash is not installed. Also, if you are using a CVS
snapshot prior to July 4th, you will need to patch systemtap with the #ifdef CONFIG_CPU_CP15
diff[4] I committed. The installation prerequisites and procedure are described in the
README[5]. Do make sure that you install it within scratchbox ARM environment[6].

The stock maemo kernel needs to be patched as well, so that systemtap is able to translate
and precompile systemtap modules. By default, the stock kernel does not have kprobes
support. It does not enable CONFIG_RELAY and CONFIG_DEBUG_FS as well. You need to patch the
kernel with the ARM kprobes kernel patch[7]. See "Kprobes and Maemo (N800)"[8] on how to
patch and compile the kernel. You need to modify the .config file to include

CONFIG_RELAY=y
CONFIG_KPROBES=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_INFO=y

Enabling CONFIG_DEBUG_INFO option will compile the kernel with -g flag. One way to find that
out is to run readelf -e vmlinux after you finish make zImage. You should see

$ readelf -e ./kernel-source-rx-34-2.6.18/vmlinux | grep debug
[12] .debug_abbrev PROGBITS 00000000 2b2128 096c24 00 0 0 1
[13] .debug_info PROGBITS 00000000 348d4c 12b6bd5 00 0 0 1
[14] .debug_line PROGBITS 00000000 15ff921 12906f 00 0 0 1
[15] .debug_pubnames PROGBITS 00000000 1728990 01c89a 00 0 0 1
[16] .debug_str PROGBITS 00000000 174522a 077e65 01 MS 0 0 1
[17] .debug_aranges PROGBITS 00000000 17bd08f 0060e0 00 0 0 1
[18] .debug_frame PROGBITS 00000000 17c3170 053a08 00 0 0 4
[19] .debug_ranges PROGBITS 00000000 1816b78 01f7a8 00 0 0 1

Once you are done with the patching and configurations, compile the kernel and flash the
zImage to the N800. Also do the following on your host machine

ln -s /path/to/kernel-source-rx-34-2.6.18/vmlinux /boot/vmlinux-2.6.18-omap1
mkdir -p /lib/modules/2.6.18-omap1/kernel
ln -s /path/to/kernel-source-rx-34-2.6.18 /lib/modules/2.6.18-omap1/build
ln -s /path/to/kernel-source-rx-34-2.6.18 /lib/modules/2.6.18-omap1/source

Once you did these, you are ready to precompile systemtap modules for the N800.

Setup of SystemTap on Nokia N800

After you have compiled systemtap on your host machine, tar the /usr/local (default install
directory), transfer it to the N800. and untar it in /usr.

host# tar cvzf local.tar.gz /usr/local
n800# tar xvzf local.tar.gz -C /

Setup $PATH to include /usr/local/bin

export PATH=$PATH:/usr/local/bin

Precompile SystemTap scripts on Host Machine

I will be using the following example to demonstrate how you can precompile the systemtap
script. This script attempts to print out information on any process that tries to open a
file. So, to precompile the script, you do

$ stap -r 2.6.18-omap1 -e 'probe kernel.function("sys_open") { printf("%s\n", probefunc()) }'
/home/eteo/.systemtap/cache/ef/stap_ef7c58450bc9dec9284bd24ba394b5c5_7253.ko

Once you done that, transfer the systemtap module
stap_ef7c58450bc9dec9284bd24ba394b5c5_7253.ko to the N800.

Run Precompiled SystemTap scripts on N800

# staprun stap_ef7c58450bc9dec9284bd24ba394b5c5_7253.ko
sys_open NULL, O_RDONLY|O_CREAT|O_EXCL|O_LARGEFILE|O_NONBLOCK|O_SYNC, 023224
sys_open NULL, O_RDONLY|O_CREAT|O_EXCL|O_LARGEFILE|O_NONBLOCK|O_SYNC, 0600002
sys_open NULL, O_RDONLY|O_CREAT|O_EXCL|O_LARGEFILE|O_NONBLOCK|O_SYNC, 01101
sys_open NULL, O_RDONLY|O_CREAT|O_EXCL|O_LARGEFILE|O_NONBLOCK|O_SYNC, 01101
[...]
^C
#

It is painful to get SystemTap working on N800 if you are not familiar with scratchbox, so
hopefully this short write-up would help you get started. There's still a lot of work to
make sure that SystemTap works well on ARM, so don't be surprised that some tapsets or code
do not work as it should.

I wrote this late at night (read: tired and sleepy!), so hopefully everything makes sense.
Write me a comment if you need help or give me some feedback :)

[1] http://sources.redhat.com/bugzilla/attachment.cgi?id=1618&action=view
[2] http://sources.redhat.com/bugzilla/show_bug.cgi?id=4186
[3] http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~//src/stp_check.in?cvsroot=systemtap
[4] http://sourceware.org/ml/systemtap-cvs/2007-q3/msg00021.html
[5] http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~//src/README?cvsroot=systemtap
[6] http://maemo.org/development/documentation/tutorials/Maemo_tutorial_bora.html#settingup
[7] http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=4447
[8] http://eugeneteo.livejournal.com/6172.html