Error removing module: Device or resource busy

[Chris Dunlop <chris at onthe dot net dot au>]

Hi,

Linux-3.0.13, systemtap 6.1 and now HEAD(b6a3da4)

I've just started using systemtap to help diagnose a problem with zfsonlinux[1].
Whenever I run stap[2], the output ends with:

  Error removing module 'stap_fcac0085842418e34d8094455dc203e8_1_21605': Device or resource busy.

(obviously the module name changes) and the module is still loaded:

  # lsmod | grep stap
  stap_fcac0085842418e34d8094455dc203e8_1_21605  2896285 1027571582 [permanent]

Using "rmmod -f" on that module results in a kernel NULL pointer dereference[3].

Any suggestions about how to avoid the unloadable module?

Cheers,

Chris

[1] https://github.com/zfsonlinux/zfs/issues/503

----------------------------------------------------------------------
[2] stap usage
----------------------------------------------------------------------
# stap  \
  -c "getfattr -d file" \
  para-callgraph.stp \
  'module("zfs").function("*@module/zfs/*.c")' \
  'module("zfs").function("zpl_xattr_list")'

# cat para-callgraph.stp
#! /usr/bin/env stap
#
# http://sourceware.org/systemtap/examples/general/para-callgraph.stp
#
function trace(entry_p, extra) {
  %( $# > 1 %? if (tid() in trace) %)
  printf("%s%s%s %s\n",
         thread_indent (entry_p),
         (entry_p>0?"->":"<-"),
         probefunc (),
         extra)
}

%( $# > 1 %?
global trace
probe $2.call {
  trace[tid()] = 1
}
probe $2.return {
  delete trace[tid()]
}
%)

probe $1.call   { trace(1, $$parms) }
probe $1.return { trace(-1, $$return) }

----------------------------------------------------------------------
[3] kernel message
----------------------------------------------------------------------
kernel: [  292.856177] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
kernel: [  292.856392] IP: [<ffffffff810cf584>] free_percpu+0xa4/0x112
kernel: [  292.856531] PGD 611554067 PUD 5e1622067 PMD 0
kernel: [  292.856754] Oops: 0000 [#1] SMP
kernel: [  292.856927] CPU 5
kernel: [  292.856980] Modules linked in: ext4 mbcache jbd2 crc16 loop fuse nfsd exportfs zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate sha256_generic aesni_intel cryptd aes_x86_64 aes_generic cbc dm_crypt dm_mod nfs lockd auth_rpcgss nfs_acl sunrpc bridge stp llc sg sd_mod usbhid hid uhci_hcd mpt2sas psmouse scsi_transport_sas raid_class scsi_mod i2c_i801 i2c_core ioatdma ehci_hcd igb dca button processor thermal_sys [last unloaded: stap_152d5b1673c7d1a7749e69173a687f9a_12_6436]
kernel: [  292.859810]
kernel: [  292.859886] Pid: 6444, comm: rmmod Tainted: P  R         3.0.13-otn-00016-g1306024 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH
kernel: [  292.860152] RIP: 0010:[<ffffffff810cf584>]  [<ffffffff810cf584>] free_percpu+0xa4/0x112
kernel: [  292.860319] RSP: 0018:ffff8805d4371e58  EFLAGS: 00010046
kernel: [  292.860406] RAX: 0000000000000000 RBX: 0000000000000287 RCX: ffff880be0261dde
kernel: [  292.860498] RDX: 00003ffffffff000 RSI: ffff880000000000 RDI: 0000000000000000
kernel: [  292.860591] RBP: ffff8805d4371e78 R08: ffffffff8117c1ab R09: ffffffff8100da91
kernel: [  292.860684] R10: ffffffff8107487f R11: ffff88062763dfa0 R12: ffff880c3ffdde40
kernel: [  292.860777] R13: ffff8805e0221dde R14: ffffffffa0728a28 R15: 0000000000000008
kernel: [  292.860870] FS:  00007f9150eb8700(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000
kernel: [  292.860989] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: [  292.861078] CR2: 0000000000000020 CR3: 000000060ffa1000 CR4: 00000000000006e0
kernel: [  292.861171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: [  292.861263] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
kernel: [  292.861356] Process rmmod (pid: 6444, threadinfo ffff8805d4370000, task ffff88062155c350)
kernel: [  292.861476] Stack:
kernel: [  292.861551]  ffffffff8117c1ab ffffffffa0728a18 ffffffffa0728830 ffffffffa0728a18
kernel: [  292.861853]  ffff8805d4371ea8 ffffffff810758f1 ffffffffa0728830 ffffffffa0728880
kernel: [  292.862154]  0000000000000002 ffff8805d533e670 ffff8805d4371ed8 ffffffff81075a6f
kernel: [  292.862456] Call Trace:
kernel: [  292.862539]  [<ffffffff8117c1ab>] ? kobject_del+0x38/0x38
kernel: [  292.862631]  [<ffffffff810758f1>] module_unload_free+0xca/0xd3
kernel: [  292.862721]  [<ffffffff81075a6f>] free_module+0x175/0x1c6
kernel: [  292.862810]  [<ffffffff8107725b>] sys_delete_module+0x1f3/0x226
kernel: [  292.862903]  [<ffffffff812c2312>] system_call_fastpath+0x16/0x1b
kernel: [  292.862992] Code: c2 85 d2 4c 0f 45 25 04 52 54 00 eb 22 48 8b 05 9b f7 3e 00 4c 89 ef 65 8b 14 25 f8 db 00 00 48 63 d2 48 03 3c d0 e8 eb e4 00 00 <4c> 8b 60 20 44 89 ee 41 2b 74 24 18 4c 89 e7 e8 dc fd ff ff 41
kernel: [  292.866166] RIP  [<ffffffff810cf584>] free_percpu+0xa4/0x112
kernel: [  292.866302]  RSP <ffff8805d4371e58>
kernel: [  292.866383] CR2: 0000000000000020
kernel: [  292.866464] ---[ end trace 2b4d7238c919f6da ]---
----------------------------------------------------------------------