[RFC] To fork or to vfork

[Mark Kettenis <kettenis at gnu dot org>]

It's now pretty clear to me why HP-UX 11.xx needs -Dvfork=fork.  In
order to set up a child to be traced using the HP-UX ttrace(2) system
call, it needs to do some handshaking.  It uses two pipes to do this.
The child writes to one end to indicate that it's ready to be traced.
When the parent is able to read from the other end it will set things
up such that the child is properly traced.  It then indicates that the
child may continue by writing on the other pipe.  However, since the
parent is suspended while the child is in the vforked state, this
doesn't work, and the debugger hangs.

Now I think we all agree that the -Dvfork=fork is pretty ugly.  Here's
an alternate approach.  It's pretty likely that if PRE_TRACE_FUN is
non-null things are complicated enough that it's unlikely that a vfork
will work correctly.  I therefore propose the attached patch.

If there are no objections, I'll check this in in a few days.

Mark

Index: ChangeLog
from  Mark Kettenis  <kettenis@gnu.org>
 
	* fork-child.c (fork_inferior): Fork instead of vfork if
	PRE_TRACE_FUN is non-null.

Index: fork-child.c
===================================================================
RCS file: /cvs/src/src/gdb/fork-child.c,v
retrieving revision 1.23
diff -u -p -r1.23 fork-child.c
--- fork-child.c 30 Sep 2004 20:15:39 -0000 1.23
+++ fork-child.c 23 Nov 2004 22:15:03 -0000
@@ -274,10 +274,18 @@ fork_inferior (char *exec_file_arg, char
   if (pre_trace_fun != NULL)
     (*pre_trace_fun) ();
 
-  /* Create the child process.  Note that the apparent call to vfork()
-     below *might* actually be a call to fork() due to the fact that
-     autoconf will ``#define vfork fork'' on certain platforms.  */
-  if (debug_fork)
+  /* Create the child process.  Since the child process is going to
+     exec(3) shortlty afterwards, try to reduce the overhead by
+     calling vfork(2).  However, if PRE_TRACE_FUN is non-null, it's
+     likely that this optimization won't work since there's too much
+     work to do between the vfork(2) and the exec(3).  This is known
+     to be the case on ttrace(2)-based HP-UX, where some handshaking
+     between parent and child needs to happen between fork(2) and
+     exec(2).  However, since the parent is suspended in the vforked
+     state, this doesn't work.  Also note that the vfork(2) call might
+     actually be a call to fork(2) due to the fact that autoconf will
+     ``#define vfork fork'' on certain platforms.  */
+  if (pre_trace_fun || debug_fork)
     pid = fork ();
   else
     pid = vfork ();