Re: Systemtap cannot probe user programs in docker

[Arkady <arkady dot miasnikov at gmail dot com>]

Containers run in different namespaces. For example, inside of a
Docker container PIDs start from 1.
I would suggest to hook the Docker kernel module.


On Sat, Dec 23, 2017 at 12:48 PM, lifeibiren@gmail.com
<lifeibiren@gmail.com> wrote:
> HI,
>
>     I try to probe user program in docker but it doesn't return anything at
> exit. But it works in the host which runs the docker container. When I probe
> kernel in docker, it works fine.
>
>     I google for half a day and got nothing. There are also some other guys
> who meet the same problems and can't resolve it. Can anybody help? Thanks.
>
>
> ----------------------------Execution----------------------------------
> Run docker by: 'docker run -it --privileged fa57ca962f13 /bin/bash'
>
> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
> WARNING: function _start return probe is blacklisted: keyword at
> para-callgraph.stp:24:1
>  source: probe $1.return { trace(-1, $$return) }
>
>
> While in host:
> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
> WARNING: function _start return probe is blacklisted: keyword at
> para-callgraph.stp:24:1
>  source: probe $1.return { trace(-1, $$return) }
>          ^
>      0 test2(22954):->_start
>     15 test2(22954): ->__libc_csu_init
>     19 test2(22954):  ->_init
>     22 test2(22954):  <-_init
>     24 test2(22954):  ->frame_dummy
>     28 test2(22954):   ->register_tm_clones
>     31 test2(22954):   <-register_tm_clones
>     32 test2(22954):  <-frame_dummy
>     34 test2(22954): <-__libc_csu_init
>     36 test2(22954): ->main
>    102 test2(22954):  ->dummy
>    105 test2(22954):  <-dummy
>    107 test2(22954): <-main
>    111 test2(22954): ->__do_global_dtors_aux
>    114 test2(22954):  ->deregister_tm_clones
>    117 test2(22954):  <-deregister_tm_clones
>    119 test2(22954): <-__do_global_dtors_aux
>    121 test2(22954): ->_fini
>    123 test2(22954): <-_fini
>
> -----------------------------Source-------------------------------
> $ cat para-callgraph.stp
> #! /usr/bin/env stap
> function trace(entry_p, extra) {
>   %( $# > 1 %? if (tid() in trace) %)
>   printf("%s%s%s %s\n",
>          thread_indent (entry_p),
>          (entry_p>0?"->":"<-"),
>          ppfunc (),
>          extra)
> }
> %( $# > 1 %?
> global trace
> probe $2.call {
>   trace[tid()] = 1
> }
> probe $2.return {
>   delete trace[tid()]
> }
> %)
> probe $1.call   { trace(1, $$parms) }
> probe $1.return { trace(-1, $$return) }
>
>
> ---------------------------Enviroment-------------------------
> Host OS: Ubuntu 16.04 LTS
>
> $ uname -r
> 4.4.0-98-generic
>
> $ stap -V
> Systemtap translator/driver (version 2.9/0.165, Debian version 2.9-2ubuntu2
> (xenial))
> Copyright (C) 2005-2015 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> enabled features: AVAHI LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>
> $ docker version
> Client:
>  Version:      17.09.0-ce
>  API version:  1.32
>  Go version:   go1.8.3
>  Git commit:   afdb6d4
>  Built:        Tue Sep 26 22:42:18 2017
>  OS/Arch:      linux/amd64
>
> Server:
>  Version:      17.09.0-ce
>  API version:  1.32 (minimum version 1.12)
>  Go version:   go1.8.3
>  Git commit:   afdb6d4
>  Built:        Tue Sep 26 22:40:56 2017
>  OS/Arch:      linux/amd64
>  Experimental: false