A 48 byte kernel module/cfi load mystery

[Mark Wielaard <mjw at redhat dot com>]

Hi Roland,

I was hoping you (or any other systemtap hacker of course) could help me
solve a mystery 48 byte offset I am seeing when using the CFI of kernel
modules. For some reason it looks like the FDE initial location is
always 48 bytes beyond the actual code that it covers. You can also see
this by hand. e.g:

$ eu-objdump --disassemble /lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko
/lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko: elf64-elf_x86_64

Disassembly of section .text:

       0:    55                       push    %rbp
       1:    48 89 e5                 mov     %rsp,%rbp
       4:    e8 00 00 00 00           callq   0x9
       9:    48 8b 87 90 02 00 00     mov     0x290(%rdi),%rax
      10:    49 89 d0                 mov     %rdx,%r8
      13:    48 8b 50 60              mov     0x60(%rax),%rdx
      17:    8b 78 10                 mov     0x10(%rax),%edi
      1a:    8b 52 14                 mov     0x14(%rdx),%edx
      1d:    48 29 d6                 sub     %rdx,%rsi
      20:    48 85 c9                 test    %rcx,%rcx
      23:    74 0a                    je      0x2f
      25:    48 89 f0                 mov     %rsi,%rax
      28:    31 d2                    xor     %edx,%edx
      2a:    48 f7 f7                 div     %rdi
      2d:    89 11                    mov     %edx,(%rcx)
      2f:    4d 85 c0                 test    %r8,%r8
      32:    74 0b                    je      0x3f
      34:    48 89 f0                 mov     %rsi,%rax
      37:    31 d2                    xor     %edx,%edx
      39:    48 f7 f7                 div     %rdi
      3c:    41 89 00                 mov     %eax,(%r8)
      3f:    c9                       leaveq
      40:    c3                       retq

Our first function starts at zero in the .text section and is 0x41 bytes
long. So look at the CFIs:

$ eu-readelf --debug-dump=frames /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko.debug

DWARF section [47] '.debug_frame' at offset 0x475ca8:

 [     0] CIE length=20
   CIE_id:                   18446744073709551615
   version:                  3
   augmentation:             ""
   code_alignment_factor:    1
   data_alignment_factor:    -8
   return_address_register:  16

   Program:
     def_cfa r7 (rsp) at offset 8
     offset_extended_sf r16 (rip) at cfa-8
     nop
     nop
     nop
     nop
     nop

 [    18] FDE length=52 cie=[     0]
   CIE_pointer:              0
   initial_location:         0x30
   address_range:            0x41

   Program:
     advance_loc4 1 to 0x1
     def_cfa_offset 16
     offset_extended_sf r6 (rbp) at cfa-16
     advance_loc4 3 to 0x4
     def_cfa_register r6 (rbp)
     advance_loc4 60 to 0x40
     restore r6 (rbp)
     def_cfa r7 (rsp) at offset 8
     nop
     nop
     nop
     nop
     nop
     nop

So our first FDE covers a location starting at 0x30 (covering these
first 0x41 bytes).

I cannot figure out why the FDE initial_location is shifted by 48 bytes.
If in the stap unwinder I just pretend the kernel module has been loaded
a little earlier:

-    vm_addr = s->static_addr;
+    vm_addr = s->static_addr - 0x30;

all the CFIs line up and I can unwind perfectly through the kernel
modules. But this mysterious magic 48 bytes bothers me. It is the same
on i686 btw. Any idea where they might be coming from?

Thanks,

Mark