Re: [PATCH] Add Documentation/kprobes.txt

[Suparna Bhattacharya <suparna at in dot ibm dot com>]

On Tue, Aug 02, 2005 at 03:20:06PM -0700, Jim Keniston wrote:
> The enclosed patch creates Documentation/kprobes.txt, a guide to using
> the existing Kprobes facility for dynamic kernel instrumentation.
> Please apply.
> 
> Jim Keniston
> 
> Acked-by: Prasanna S Panchamukhi <prasanna@in.ibm.com>
> Signed-off-by: Jim Keniston <jkenisto@us.ibm.com>
> 
> 

> --- linux.old/Documentation/kprobes.txt	1969-12-31 16:00:00.000000000 -0800
> +++ linux.new/Documentation/kprobes.txt	2005-08-02 14:02:43.000000000 -0700
> @@ -0,0 +1,588 @@
> +Title	: Kernel Probes (Kprobes)
> +Authors	: Jim Keniston <jkenisto@us.ibm.com>
> +	: Prasanna S Panchamukhi <prasanna@in.ibm.com>
> +
> +CONTENTS
> +
> +1. Concepts: Kprobes, Jprobes, Return Probes
> +2. Architectures Supported
> +3. Configuring Kprobes
> +4. API Reference
> +5. Kprobes Features and Limitations
> +6. Probe Overhead
> +7. TODO
> +8. Kprobes Example
> +9. Jprobes Example
> +10. Kretprobes Example
> +
> +1. Concepts: Kprobes, Jprobes, Return Probes
> +
> +Kprobes enables you to dynamically break into any kernel routine and
> +collect debugging and performance information non-disruptively. You
> +can trap at almost any kernel code address, specifying a handler
> +routine to be invoked when the breakpoint is hit.
> +
> +There are currently three types of probes: kprobes, jprobes, and
> +kretprobes (also called return probes).  A kprobe can be inserted
> +on virtually any instruction in the kernel.  A jprobe is inserted at
> +the entry to a kernel function, and provides convenient access to the
> +function's arguments.  A return probe fires when a specified function
> +returns.
> +
> +In the typical case, Kprobes-based instrumentation is packaged as
> +a kernel module.  The module's init function installs ("registers")
> +one or more probes, and the exit function unregisters them.  A
> +registration function such as register_kprobe() specifies where
> +the probe is to be inserted and what handler is to be called when
> +the probe is hit.
> +
> +The next three subsections explain how the different types of
> +probes work.  They explain certain things that you'll need to
> +know in order to make the best use of Kprobes -- e.g., the
> +difference between a pre_handler and a post_handler, and how
> +to use the maxactive and nmissed fields of a kretprobe.  But
> +if you're in a hurry to start using Kprobes, you can skip ahead
> +to section 2.
> +
> +1.1 How Does a Kprobe Work?
> +
> +When a kprobe is registered, Kprobes makes a copy of the probed
> +instruction and replaces the first byte(s) of the probed instruction
> +with a breakpoint instruction (e.g., int3 on i386 and x86_64).
> +
> +When a CPU hits the breakpoint instruction, a trap occurs, the CPU's
> +registers are saved, and control passes to Kprobes via the
> +notifier_call_chain mechanism.  Kprobes executes the "pre_handler"
> +associated with the kprobe, passing the handler the addresses of the
> +kprobe struct and the saved registers.
> +
> +Next, Kprobes single-steps its copy of the probed instruction.
> +(It would be simpler to single-step the actual instruction in place,
> +but then Kprobes would have to temporarily remove the breakpoint
> +instruction.  This would open a small time window when another CPU
> +could sail right past the probepoint.)
> +
> +After the instruction is single-stepped, Kprobes executes the
> +"post_handler," if any, that is associated with the kprobe.
> +Execution then continues with the instruction following the probepoint.
> +
> +1.2 How Does a Jprobe Work?
> +
> +A jprobe is implemented using a kprobe that is placed on a function's
> +entry point.  It employs a simple mirroring principle to allow
> +seamless access to the probed function's arguments.  The jprobe
> +handler routine should have the same signature (arg list and return
> +type) as the function being probed, and must always end by calling
> +the Kprobes function jprobe_return().
> +
> +Here's how it works.  When the probe is hit, Kprobes makes a copy of
> +the saved registers and a generous portion of the stack (see below).
> +Kprobes then points the saved stack pointer at the stack-copy, points
> +the saved instruction pointer at the jprobe's handler routine, and
> +returns from the trap.  As a result, control passes to the handler,
> +which is presented with the same register and stack contents as the
> +probed function.  When it is done, the handler calls jprobe_return(),
> +which traps again to restore processor state and switch back to the
> +probed function.
> +
> +gcc assumes that the callee owns its arguments.  To prevent unexpected
> +modifications to the probed function's stack, Kprobes presents the
> +jprobe handler with a copy of the stack.  Up to MAX_STACK_SIZE bytes
> +are copied -- e.g., 64 bytes on i386.

IIRC, we save and restore the stack, rather than pass a copy of the stack
to the handler. Thus, while jprobes does make a copy of MAX_STACK_SIZE
bytes, the handler still operates on the original stack (e.g. stack
addresses are unchanged) and the stack contents are restored
before returning control to the probed routine.

> +
> +Note that the probed function's args may be passed on the stack
> +or in registers (e.g., for x86_64 or for an i386 fastcall function).  
> +The jprobe will work in either case, so long as the handler's
> +prototype matches that of the probed function.
> +
> +1.3 How Does a Return Probe Work?
> +
> +When you call register_kretprobe(), Kprobes establishes a kprobe at
> +the entry to the function.  When the probed function is called and this
> +probe is hit, Kprobes saves a copy of the return address, and replaces
> +the return address with the address of a "trampoline."  The trampoline
> +is an arbitrary piece of code -- typically just a nop instruction.
> +At boot time, Kprobes registers a kprobe at the trampoline.
> +
> +When the probed function executes its return instruction, control
> +passes to the trampoline and that probe is hit.  Kprobes' trampoline
> +handler calls the user-specified handler associated with the kretprobe,
> +then sets the saved instruction pointer to the saved return address,
> +and that's where execution resumes upon return from the trap.
> +
> +While the probed function is executing, its return address is
> +stored in an object of type kretprobe_instance.  Before calling
> +register_kretprobe(), the user sets the maxactive field of the
> +kretprobe struct to specify how many instances of the specified
> +function can be probed simultaneously.  register_kretprobe()
> +pre-allocates the indicated number of kretprobe_instance objects.
> +
> +For example, if the function is non-recursive and is called with a
> +spinlock held, maxactive = 1 should be enough.  If the function is
> +non-recursive and can never relinquish the CPU (e.g., via a semaphore
> +or preemption), NR_CPUS should be enough.  If maxactive <= 0, it is
> +set to a default value.  If CONFIG_PREEMPT is enabled, the default
> +is max(10, 2*NR_CPUS).  Otherwise, the default is NR_CPUS.
> +
> +It's not a disaster if you set maxactive too low; you'll just miss
> +some probes.  In the kretprobe struct, the nmissed field is set to
> +zero when the return probe is registered, and is incremented every
> +time the probed function is entered but there is no kretprobe_instance
> +object available for establishing the return probe.
> +
> +2. Architectures Supported
> +
> +Kprobes, jprobes, and return probes are implemented on the following
> +architectures:
> +
> +- i386
> +- x86_64 (AMD-64, E64MT)
> +- ppc64
> +- ia64 (Support for probes on certain instruction types is still in progress.)
> +- sparc64 (Return probes not yet implemented.)
> +
> +3. Configuring Kprobes
> +
> +When configuring the kernel using make menuconfig/xconfig/oldconfig,
> +ensure that CONFIG_KPROBES is set to "y".  Under "Kernel hacking",
> +look for "Kprobes".  You may have to enable "Kernel debugging"
> +(CONFIG_DEBUG_KERNEL) before you can enable Kprobes.
> +
> +You may also want to ensure that CONFIG_KALLSYMS and perhaps even
> +CONFIG_KALLSYMS_ALL are set to "y", since kallsyms_lookup_name()
> +is a handy, version-independent way to find a function's address.
> +
> +If you need to insert a probe in the middle of a function, you may find
> +it useful to "Compile the kernel with debug info" (CONFIG_DEBUG_INFO),
> +so you can use "objdump -d -l vmlinux" to see the source-to-object
> +code mapping.
> +
> +4. API Reference
> +
> +The Kprobes API includes a "register" function and an "unregister"
> +function for each type of probe.  Here are terse, mini-man-page
> +specifications for these functions and the associated probe handlers
> +that you'll write.  See the latter half of this document for examples.
> +
> +4.1 register_kprobe
> +
> +#include <linux/kprobes.h>
> +int register_kprobe(struct kprobe *kp);
> +
> +Sets a breakpoint at the address kp->addr.  When the breakpoint is
> +hit, Kprobes calls kp->pre_handler.  After the probed instruction
> +is single-stepped, Kprobe calls kp->post_handler.  If a fault
> +occurs during execution of kp->pre_handler or kp->post_handler,
> +or during single-stepping of the probed instruction, Kprobes calls
> +kp->fault_handler.  Any or all handlers can be NULL.
> +
> +register_kprobe() returns 0 on success, or a negative errno otherwise.
> +
> +User's pre-handler (kp->pre_handler):
> +#include <linux/kprobes.h>
> +#include <linux/ptrace.h>
> +int pre_handler(struct kprobe *p, struct pt_regs *regs);
> +
> +Called with p pointing to the kprobe associated with the breakpoint,
> +and regs pointing to the struct containing the registers saved when
> +the breakpoint was hit.  Return 0 here unless you're a Kprobes geek.
> +
> +User's post-handler (kp->post_handler):
> +#include <linux/kprobes.h>
> +#include <linux/ptrace.h>
> +void post_handler(struct kprobe *p, struct pt_regs *regs,
> +	unsigned long flags);
> +
> +p and regs are as described for the pre_handler.  flags always seems
> +to be zero.
> +
> +User's fault-handler (kp->fault_handler):
> +#include <linux/kprobes.h>
> +#include <linux/ptrace.h>
> +int fault_handler(struct kprobe *p, struct pt_regs *regs, int trapnr);
> +
> +p and regs are as described for the pre_handler.  trapnr is the
> +architecture-specific trap number associated with the fault (e.g.,
> +on i386, 13 for a general protection fault or 14 for a page fault).
> +Returns 1 if it successfully handled the exception.
> +
> +4.2 register_jprobe
> +
> +#include <linux/kprobes.h>
> +int register_jprobe(struct jprobe *jp)
> +
> +Sets a breakpoint at the address jp->kp.addr, which must be the address
> +of the first instruction of a function.  When the breakpoint is hit,
> +Kprobes runs the handler whose address is jp->entry.
> +
> +The handler should have the same arg list and return type as the probed
> +function; and just before it returns, it must call jprobe_return().
> +(The handler never actually returns, since jprobe_return() returns
> +control to Kprobes.)  If the probed function is declared asmlinkage,
> +fastcall, or anything else that affects how args are passed, the
> +handler's declaration must match.
> +
> +register_jprobe() returns 0 on success, or a negative errno otherwise.
> +
> +4.3 register_kretprobe
> +
> +#include <linux/kprobes.h>
> +int register_kretprobe(struct kretprobe *rp);
> +
> +Establishes a return probe for the function whose address is
> +rp->kp.addr.  When that function returns, Kprobes calls rp->handler.
> +You must set rp->maxactive appropriately before you call
> +register_kretprobe(); see "How Does a Return Probe Work?" for details.
> +
> +register_kretprobe() returns 0 on success, or a negative errno
> +otherwise.
> +
> +User's return-probe handler (rp->handler):
> +#include <linux/kprobes.h>
> +#include <linux/ptrace.h>
> +int kretprobe_handler(struct kretprobe_instance *ri, struct pt_regs *regs);
> +
> +regs is as described for kprobe.pre_handler.  ri points to the
> +kretprobe_instance object, of which the following fields may be
> +of interest:
> +- ret_addr: the return address
> +- rp: points to the corresponding kretprobe object
> +- task: points to the corresponding task struct
> +The handler's return value is currently ignored.
> +
> +4.4 unregister_*probe
> +
> +#include <linux/kprobes.h>
> +void unregister_kprobe(struct kprobe *kp);
> +void unregister_jprobe(struct jprobe *jp);
> +void unregister_kretprobe(struct kretprobe *rp);
> +
> +Removes the specified probe.  The unregister function can be called
> +at any time after the probe has been registered.
> +
> +5. Kprobes Features and Limitations
> +
> +As of Linux v2.6.12, Kprobes allows multiple probes at the same
> +address.  Currently, however, there cannot be multiple jprobes on
> +the same function at the same time.
> +
> +In general, you can install a probe anywhere in the kernel.
> +In particular, you can probe interrupt handlers.  Known exceptions
> +are discussed in this section.
> +
> +For obvious reasons, it's a bad idea to install a probe in
> +the code that implements Kprobes (mostly kernel/kprobes.c and
> +arch/*/kernel/kprobes.c).  A patch in the v2.6.13 timeframe instructs
> +Kprobes to reject such requests.
> +
> +If you install a probe in an inline-able function, Kprobes makes
> +no attempt to chase down all inline instances of the function and
> +install probes there.  gcc may inline a function without being asked,
> +so keep this in mind if you're not seeing the probe hits you expect.
> +
> +A probe handler can modify the environment of the probed function
> +-- e.g., by modifying kernel data structures, or by modifying the
> +contents of the pt_regs struct (which are restored to the registers
> +upon return from the breakpoint).  So Kprobes can be used, for example,
> +to install a bug fix or to inject faults for testing.  Kprobes, of
> +course, has no way to distinguish the deliberately injected faults
> +from the accidental ones.  Don't drink and probe.
> +
> +Kprobes makes no attempt to prevent probe handlers from stepping on
> +each other -- e.g., probing printk() and then calling printk() from a
> +probe handler.  As of Linux v2.6.12, if a probe handler hits a probe,
> +that second probe's handlers won't be run in that instance.
> +
> +In Linux v2.6.12 and previous versions, Kprobes' data structures are
> +protected by a single lock that is held during probe registration and
> +unregistration and while handlers are run.  Thus, no two handlers
> +can run simultaneously.  To improve scalability on SMP systems,
> +this restriction will probably be removed soon, in which case
> +multiple handlers (or multiple instances of the same handler) may
> +run concurrently on different CPUs.  Code your handlers accordingly.
> +
> +Kprobes does not use semaphores or allocate memory except during
> +registration and unregistration.
> +
> +Probe handlers are run with preemption disabled.  Depending on the
> +architecture, handlers may also run with interrupts disabled.  In any
> +case, your handler should not yield the CPU (e.g., by attempting to
> +acquire a semaphore).
> +
> +Since a return probe is implemented by replacing the return
> +address with the trampoline's address, stack backtraces and calls
> +to __builtin_return_address() will typically yield the trampoline's
> +address instead of the real return address for kretprobed functions.
> +(As far as we can tell, __builtin_return_address() is used only
> +for instrumentation and error reporting.)
> +
> +If the number of times a function is called does not match the
> +number of times it returns, registering a return probe on that
> +function may produce undesirable results.  We have the do_exit()
> +and do_execve() cases covered.  do_fork() is not an issue.  We're
> +unaware of other specific cases where this could be a problem.
> +
> +6. Probe Overhead
> +
> +On a typical CPU in use in 2005, a kprobe hit takes 0.5 to 1.0
> +microseconds to process.  Specifically, a benchmark that hits the same
> +probepoint repeatedly, firing a simple handler each time, reports 1-2
> +million hits per second, depending on the architecture.  A jprobe or
> +return-probe hit typically takes 50-75% longer than a kprobe hit.
> +When you have a return probe set on a function, adding a kprobe at
> +the entry to that function adds essentially no overhead.
> +
> +Here are sample overhead figures (in usec) for different architectures.
> +k = kprobe; j = jprobe; r = return probe; kr = kprobe + return probe
> +on same function; jr = jprobe + return probe on same function
> +
> +i386: Intel Pentium M, 1495 MHz, 2957.31 bogomips
> +k = 0.57 usec; j = 1.00; r = 0.92; kr = 0.99; jr = 1.40
> +
> +x86_64: AMD Opteron 246, 1994 MHz, 3971.48 bogomips
> +k = 0.49 usec; j = 0.76; r = 0.80; kr = 0.82; jr = 1.07
> +
> +ppc64: POWER5 (gr), 1656 MHz (SMT disabled, 1 virtual CPU per physical CPU)
> +k = 0.77 usec; j = 1.31; r = 1.26; kr = 1.45; jr = 1.99
> +
> +7. TODO
> +
> +a. SystemTap (http://sourceware.org/systemtap): Work in progress
> +to provide a simplified programming interface for probe-based
> +instrumentation.
> +b. Improved SMP scalability: Currently, work is in progress to handle
> +multiple kprobes in parallel.
> +c. Kernel return probes for sparc64.
> +d. Support for other architectures.
> +e. User-space probes.
> +
> +8. Kprobes Example
> +
> +Here's a sample kernel module showing the use of kprobes to dump a
> +stack trace and selected i386 registers when do_fork() is called.
> +----- cut here -----
> +/*kprobe_example.c*/
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/kprobes.h>
> +#include <linux/kallsyms.h>
> +#include <linux/sched.h>
> +
> +/*For each probe you need to allocate a kprobe structure*/
> +static struct kprobe kp;
> +
> +/*kprobe pre_handler: called just before the probed instruction is executed*/
> +int handler_pre(struct kprobe *p, struct pt_regs *regs)
> +{
> +	printk("pre_handler: p->addr=0x%p, eip=%lx, eflags=0x%lx\n",
> +		p->addr, regs->eip, regs->eflags);
> +	dump_stack();
> +	return 0;
> +}
> +
> +/*kprobe post_handler: called after the probed instruction is executed*/
> +void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags)
> +{
> +	printk("post_handler: p->addr=0x%p, eflags=0x%lx\n",
> +		p->addr, regs->eflags);
> +}
> +
> +/* fault_handler: this is called if an exception is generated for any
> + * instruction within the pre- or post-handler, or when Kprobes
> + * single-steps the probed instruction.
> + */
> +int handler_fault(struct kprobe *p, struct pt_regs *regs, int trapnr)
> +{
> +	printk("fault_handler: p->addr=0x%p, trap #%dn",
> +		p->addr, trapnr);
> +	/* Return 0 because we don't handle the fault. */
> +	return 0;
> +}
> +
> +int init_module(void)
> +{
> +	int ret;
> +	kp.pre_handler = handler_pre;
> +	kp.post_handler = handler_post;
> +	kp.fault_handler = handler_fault;
> +	kp.addr = (kprobe_opcode_t*) kallsyms_lookup_name("do_fork");
> +	/* register the kprobe now */
> +	if (!kp.addr) {
> +		printk("Couldn't find %s to plant kprobe\n", "do_fork");
> +		return -1;
> +	}
> +	if ((ret = register_kprobe(&kp) < 0)) {
> +		printk("register_kprobe failed, returned %d\n", ret);
> +		return -1;
> +	}
> +	printk("kprobe registered\n");
> +	return 0;
> +}
> +
> +void cleanup_module(void)
> +{
> +	unregister_kprobe(&kp);
> +	printk("kprobe unregistered\n");
> +}
> +
> +MODULE_LICENSE("GPL");
> +----- cut here -----
> +
> +You can build the kernel module, kprobe-example.ko, using the following
> +Makefile:
> +----- cut here -----
> +obj-m := kprobe-example.o
> +KDIR := /lib/modules/$(shell uname -r)/build
> +PWD := $(shell pwd)
> +default:
> +	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
> +clean:
> +	rm -f *.mod.c *.ko *.o
> +----- cut here -----
> +
> +$ make
> +$ su -
> +...
> +# insmod kprobe-example.ko
> +
> +You will see the trace data in /var/log/messages and on the console
> +whenever do_fork() is invoked to create a new process.
> +
> +9. Jprobes Example
> +
> +Here's a sample kernel module showing the use of jprobes to dump
> +the arguments of do_fork().
> +----- cut here -----
> +/*jprobe-example.c */
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/fs.h>
> +#include <linux/uio.h>
> +#include <linux/kprobes.h>
> +#include <linux/kallsyms.h>
> +
> +/* 
> + * Jumper probe for do_fork. 
> + * Mirror principle enables access to arguments of the probed routine 
> + * from the probe handler.
> + */
> +
> +/* Proxy routine having the same arguments as actual do_fork() routine */
> +long jdo_fork(unsigned long clone_flags, unsigned long stack_start,
> +	      struct pt_regs *regs, unsigned long stack_size,
> +	      int __user * parent_tidptr, int __user * child_tidptr)
> +{
> +	printk("jprobe: clone_flags=0x%lx, stack_size=0x%lx, regs=0x%p\n",
> +	       clone_flags, stack_size, regs);
> +	/* Always end with a call to jprobe_return(). */
> +	jprobe_return();
> +	/*NOTREACHED*/
> +	return 0;
> +}
> +
> +static struct jprobe my_jprobe = {
> +	.entry = (kprobe_opcode_t *) jdo_fork
> +};
> +
> +int init_module(void)
> +{
> +	int ret;
> +	my_jprobe.kp.addr = (kprobe_opcode_t *) kallsyms_lookup_name("do_fork");
> +	if (!my_jprobe.kp.addr) {
> +		printk("Couldn't find %s to plant jprobe\n", "do_fork");
> +		return -1;
> +	}
> +
> +	if ((ret = register_jprobe(&my_jprobe)) <0) {
> +		printk("register_jprobe failed, returned %d\n", ret);
> +		return -1;
> +	}
> +	printk("Planted jprobe at %p, handler addr %p\n",
> +	       my_jprobe.kp.addr, my_jprobe.entry);
> +	return 0;
> +}
> +
> +void cleanup_module(void)
> +{
> +	unregister_jprobe(&my_jprobe);
> +	printk("jprobe unregistered\n");
> +}
> +
> +MODULE_LICENSE("GPL");
> +----- cut here -----
> +
> +Build and insert the kernel module as shown in the above kprobe
> +example.  You will see the trace data in /var/log/messages and on
> +the console whenever do_fork() is invoked to create a new process.
> +(Some messages may be suppressed if syslogd is configured to
> +eliminate duplicate messages.)
> +
> +10. Kretprobes Example
> +
> +Here's a sample kernel module showing the use of return probes to
> +report failed calls to sys_open().
> +----- cut here -----
> +/*kretprobe-example.c*/
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/kprobes.h>
> +#include <linux/kallsyms.h>
> +
> +static const char *probed_func = "sys_open";
> +
> +/* Return-probe handler: If the probed function fails, log the return value. */
> +static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
> +{
> +	// Substitute the appropriate register name for your architecture --
> +	// e.g., regs->rax for x86_64, regs->gpr[3] for ppc64.
> +	int retval = (int) regs->eax;
> +	if (retval < 0) {
> +		printk("%s returns %d\n", probed_func, retval);
> +	}
> +	return 0;
> +}
> +
> +static struct kretprobe my_kretprobe = {
> +	.handler = ret_handler,
> +	/* Probe up to 20 instances concurrently. */
> +	.maxactive = 20
> +};
> +
> +int init_module(void)
> +{
> +	int ret;
> +	my_kretprobe.kp.addr =
> +		(kprobe_opcode_t *) kallsyms_lookup_name(probed_func);
> +	if (!my_kretprobe.kp.addr) {
> +		printk("Couldn't find %s to plant return probe\n", probed_func);
> +		return -1;
> +	}
> +	if ((ret = register_kretprobe(&my_kretprobe)) < 0) {
> +		printk("register_kretprobe failed, returned %d\n", ret);
> +		return -1;
> +	}
> +	printk("Planted return probe at %p\n", my_kretprobe.kp.addr);
> +	return 0;
> +}
> +
> +void cleanup_module(void)
> +{
> +	unregister_kretprobe(&my_kretprobe);
> +	printk("kretprobe unregistered\n");
> +	/* nmissed > 0 suggests that maxactive was set too low. */
> +	printk("Missed probing %d instances of %s\n",
> +		my_kretprobe.nmissed, probed_func);
> +}
> +
> +MODULE_LICENSE("GPL");
> +----- cut here -----
> +
> +Build and insert the kernel module as shown in the above kprobe
> +example.  You will see the trace data in /var/log/messages and on the
> +console whenever sys_open() returns a negative value.  (Some messages
> +may be suppressed if syslogd is configured to eliminate duplicate
> +messages.)
> +
> +For additional information on Kprobes, refer to the following URLs:
> +http://www-106.ibm.com/developerworks/library/l-kprobes.html?ca=dgr-lnxw42Kprobe
> +http://www.redhat.com/magazine/005mar05/features/kprobes/


-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India