This is the mail archive of the mailing list for the GDB project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[RFC] Tighten and tweak ptrace argument checks

Hi Mark,

I discovered recently that your rewrite of the ptrace argument type
checks had busted some (not-yet-contributed, sorry) changes to use long
long for ptrace calls on MIPS n32.

This made me take a closer look at the tests.  It turns out that
neither the return type checks nor argument type checks work on
GNU/Linux.  The return type check will work with oldish versions of
GCC, but not with 4.0 (or 3.4, I think).  The argument type checks
won't work at all.  Here's the problem:

  extern long int ptrace (enum __ptrace_request, ...);

() won't match ...; this is important because of different calling
conventions on some platforms.

On i386-pc-linux-gnu the defaults work so this is merely an annoyance.
When the real type is long long instead of long, well, it wasn't

So here's a proposed patch.  It handles the GNU/Linux case.  It handles
long long.  It also is violently fatal if it fails, instead of making
up something sensible - this is because I wasted several days trying to
figure out what was wrong with GDB when it was casting all the ptrace
arguments to the wrong type.  I'm sure it'll break the build in a lot
of places but I think it's worth fixing if you want to use autoconf for
this at all!

Any comments on the patch?

I also noticed considerable PTRACE_ARG3_TYPE vs PTRACE_TYPE_ARG3
confusion in the sources.  I think that the last time I looked at this,
I convinced myself that it was possible to get them out of sync, and
the results would be pretty gruesome.  I didn't touch that for now.

Daniel Jacobowitz
CodeSourcery, LLC

2005-09-09  Daniel Jacobowitz  <>

	* Improve tests for the return type and argument
	list of ptrace.  Exit if the tests fail.
	* configure: Regenerated.

RCS file: /big/fsf/rsync/src/src/gdb/,v
retrieving revision 1.24
diff -u -p -r1.24
---	25 Jul 2005 15:08:40 -0000	1.24
+++	9 Sep 2005 21:20:01 -0000
@@ -486,12 +486,28 @@ AC_CHECK_DECLS(ptrace, [], [
   : ${gdb_cv_func_ptrace_ret='int'}
   : ${gdb_cv_func_ptrace_args='int,int,long,long'}
 ], $gdb_ptrace_headers)
-# Check return type.
-AC_CACHE_CHECK([return type of ptrace], gdb_cv_func_ptrace_ret,
-  AC_TRY_COMPILE($gdb_ptrace_headers,
-    [extern int ptrace ();],
-    gdb_cv_func_ptrace_ret='int',
-    gdb_cv_func_ptrace_ret='long'))
+# Check return type.  First try without an argument list.  But
+# any argument list containing stdargs is incompatible with a missing
+# argument list, and recent (e.g. 4.x) versions of gcc will reject
+# them.  So list every known stdargs argument list too.
+AC_CACHE_CHECK([return type of ptrace], gdb_cv_func_ptrace_ret, [
+for gdb_args in '' 'unsigned int, ...'; do
+  for gdb_ret in 'int' 'long' 'long long'; do
+    AC_TRY_COMPILE($gdb_ptrace_headers,
+      [extern $gdb_ret ptrace ($gdb_args);],
+      [gdb_cv_func_ptrace_ret='int'; break 2])
+  done
+if test "${gdb_cv_func_ptrace_ret+set}" != set; then
+  AC_MSG_ERROR(unknown)
+# Currently the only systems known to use stdargs expect four arguments.
+if test x"$gdb_args" = x'unsigned int, ...'; then
+  gdb_cv_func_ptrace_args="unsigned int,int,$gdb_ret,$gdb_ret"
 AC_DEFINE_UNQUOTED(PTRACE_TYPE_RET, $gdb_cv_func_ptrace_ret,
   [Define as the return type of ptrace.])
 # Check argument types.
@@ -517,8 +533,9 @@ gdb_cv_func_ptrace_args="$gdb_arg1,$gdb_
-# Provide a safe default value.
-: ${gdb_cv_func_ptrace_args='int,int,long,long'}
+if test "${gdb_cv_func_ptrace_args+set}" != set; then
+  AC_MSG_ERROR(unknown)
 ac_save_IFS=$IFS; IFS=','
 set dummy `echo "$gdb_cv_func_ptrace_args" | sed 's/\*/\*/g'`

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]