[RFC-v2] Enhance backtrace for microsoft system DLL calls

["Pierre Muller" <muller at ics dot u-strasbg dot fr>]

  I wrote a i386_skip_noop function.

  It currently only tests for 'nop' and 'mov %edi,%edi'
instructions, but the way it is written, other
instructions should be easy to add.
  I also tried to explain the reason of the presence 
of the 'mov %edi,%edi' instruction in the win32 system DLL prologue,
as explained by Pedro. 
  

  Tested on cygwin target, no regressions found.
The patch allows to get the backtrace of the main thread of gdb 
to come up to the  functions that called the systems DLL.
If I use ./gdb ./gdb with 'set new-console on'
and use Ctrl-C on the debuggee gdb window.
Without the patch, the backtrace only shows
3 levels in ntdll.dll and kernel32.dll

Questions:
 1) Is the 'nop' test useful or should it be removed?

 2) Should we add other possible no-ops?
  As said in my previous email, the number of
possible no-ops is big, and it is probably not wise to test all of 
them.
 
3) this call is used for all i386 targets, but it
is probably useless for all operating systems but Microsoft Windows,
so should it be called only for that OS, and if yes, how should
we code this?

4) Any suggestions to make the comment clearer will be 
most appreciated.

Pierre Muller



ChangeLog entry:

2008-01-14  Pierre Muller  <muller@ics.u-strasbg.fr>

	* i386-tdep.c (i386_skip_noop): New function.
	(i386_analyze_prologue): Call i386_skip_noop function.


Index: gdb/i386-tdep.c
===================================================================
RCS file: /cvs/src/src/gdb/i386-tdep.c,v
retrieving revision 1.248
diff -u -p -r1.248 i386-tdep.c
--- gdb/i386-tdep.c	11 Jan 2008 13:20:00 -0000	1.248
+++ gdb/i386-tdep.c	14 Jan 2008 08:11:27 -0000
@@ -632,6 +632,51 @@ struct i386_insn i386_frame_setup_skip_i
   { 0 }
 };
 
+
+/* Check whether PC points to a no-op instruction.  */
+static CORE_ADDR
+i386_skip_noop (CORE_ADDR pc)
+{
+  gdb_byte op;
+  int check = 1;
+
+  read_memory_nobpt (pc, &op, 1);
+
+  while (check) 
+    {
+      check = 0;
+      /* Ignore `nop' instruction.  */
+      if (op == 0x90) 
+	{
+	  pc += 1;
+	  read_memory_nobpt (pc, &op, 1);
+	  check = 1;
+	}
+      /* Ignore no-op instruction `mov %edi, %edi'.
+	 Microsoft system dlls often start with
+	 a `mov %edi,%edi' instruction.
+	 The 5 bytes before the function start are
+	 filled with `nop' instructions.
+	 This pattern can be used for hot-patching:
+	 The `mov %edi, %edi' instruction can be replaced by a
+	 near jump to the location of the 5 `nop' instructions
+	 which can be replaced by a 32-bit jump to anywhere
+	 in the 32-bit address space.  */
+
+      else if (op == 0x8b)
+	{
+	  read_memory_nobpt (pc + 1, &op, 1);
+	  if (op == 0xff)
+	    {
+	      pc += 2;
+	      read_memory_nobpt (pc, &op, 1);
+	      check = 1;
+	    }
+	}
+    }
+  return pc; 
+}
+
 /* Check whether PC points at a code that sets up a new stack frame.
    If so, it updates CACHE and returns the address of the first
    instruction after the sequence that sets up the frame or LIMIT,
@@ -817,6 +862,7 @@ static CORE_ADDR
 i386_analyze_prologue (CORE_ADDR pc, CORE_ADDR current_pc,
 		       struct i386_frame_cache *cache)
 {
+  pc = i386_skip_noop (pc);
   pc = i386_follow_jump (pc);
   pc = i386_analyze_struct_return (pc, current_pc, cache);
   pc = i386_skip_probe (pc);