Re: tracking memory map changes

[Jim Keniston <jkenisto at us dot ibm dot com>]

On Fri, 2008-05-30 at 11:42 -0500, David Smith wrote:
> One of the requirements of doing user-space probing is being able to
> follow user-space program memory map changes.  I've been poking around
> and I thought I'd share what I found and the possibilities of where to
> go from here.
> 
> I did all of the following on f8 and looking at 2.6.24.4 kernel source.
>  I took a peek at the source for 2.6.26, but didn't see any drastic
> differences in this area.
> 
> Let me start by describing what is composed by a memory map.

...

Yes, your description is consistent with my understanding.

> Frank, here are some initial questions.
> 
> Q1: What information will the runtime need from each vm_area_struct?
> I'd guess the path, vm_start, and vm_end at a minimum.
> 
> Q2: Will the runtime want to know only about new text sections being
> added or all sections?

It'd also be good to watch for vm areas going away.  I think Anil
Keshavamurthy mentioned that this can happen for things like Java.
Uprobes can theoretically get into trouble if you register a uprobe,
then the underlying vma gets unmapped, then a new vma gets mapped in the
same address range.  We should catch the munmamp/mremap and unregister
any associated uprobes.
  
> 
> Q3: Will the runtime want to know about any of the vm_area_structs not
> associated with a file?
> 
> When /bin/cat, gets exec'ed, the /lib/ld-2.7.so and /bin/cat files are
> already mapped in.  As /bin/cat runs, it loads in /lib/libc-2.7.so.
> This means that we've got 2 related problems: enumerating the sections
> when first attaching to a thread (either by being exec'ed or by
> attaching to an existing thread) then tracking memory map changes as
> they occur (as in loading /lib/libc-2.7.so or by a thread calling dlopen()).
> 
> Enumerating the existing vm_area_structs seems easy enough.  Tracking
> new vm_area_structs as they get added is harder.  Finding the right
> point and the right method is the problem.
> 
> The sys_mmap2() system call is a wrapper around
> mm/mmap.c:do_mmap_pgoff(). do_mmap_pgoff() does lots of error checking,
> then calls mm/mmap.c:mmap_region() to actually add a new vm_area_struct.
>  Toward the end of mmap_region(), vm_stat_account() is called (if
> CONFIG_PROC_FS is on).
> 
> So, where/how to track memory map changes?  Here are a few ideas:
> 
> 1) Set a kretprobe on sys_mmap2()/do_mmap_pgoff()/mmap_region().  One
> problem here is that kretprobes are limited in quantity.

Kretprobe_instances are preallocated per-kretprobe, so they're the
limited quantity.  But quantity is a concern only if there are multiple
instances of that particular probed function running concurrently.  We
can play with maxactive to come up with a "safe" number... then triple
that (or make it a -D option ;-)).

A problem with a k[ret]probe is that the handler can't block, so that
limits what we can do.
 
> 
> 2) Set a kprobe on vm_stat_account().  This would require that the
> kernel was configured with CONFIG_PROC_FS and that vm_stat_account() is
> getting called in the correct place in all the kernels we're interested in.
> 
> 3) Turn on utrace syscall return tracing for that thread and wait for
> mmap calls to return.  This is probably the easiest route, but it forces
> every syscall for that thread to go through the slow path.  A big
> advantage here is that an all utrace solution wouldn't require any
> debugging info for the kernel to be present on the system.

Yes.

> 
> Does any have any better ideas or preferences here?

"mmap change event" is already on Roland's utrace TODO list.  That'd be
my preference.  It's be lightweight, and you could block if necessary to
do your magic when notified.  You obviously know the terrain well, so
maybe you could funnel that work into the utrace enhancement.

> 
> In all of the above methods the code won't know what was added, just
> that a new vm_area_struct might exist, so I'll have to figure out a way
> to track changes.
> 
> Finally, the shortest path to something somewhat useful would be to
> first work on providing notification of existing vm_area_structs.  This
> might help move user-space tracing along while I work on the harder
> problem of tracking memory map changes.  Providing notification of
> existing vm_area_structs might allow attaching to an existing thread
> (which already has all its shared libraries loaded and doesn't call
> dlopen()) and being able to figure out the right address to probe.
> 

You don't explicitly say so, but I infer from this that you're making
progress on probing shlibs.  That'll be huge.

Jim