Small getpt() problem for BSD-style pty's

[Mark Kettenis <kettenis@wins.uva.nl>]

I've discovered some problems and inconsistencies in the current BSD
getpt() implementation (sysdeps/unix/bsd/getpt.c).  The implementation
is used by the Hurd, but doesn't work right now.  This implementation
is also used for Linux if Unix98 pty's are not available.

 * The current implementation uses isatty() to check if the opened
   file descriptor really is a master pseudo terminal.  This is wrong,
   since isatty() checks for tty-ness and a master pseudo terminal is
   not a tty.  AFAIK isatty() on a master pseudo terminal returns a
   nonzero value only on Linux.  In fact this could mean that the
   implementation of isatty() on Linux is wrong.  In any case I think
   this check should be removed (if a user fucks up his installation
   and makes /dev/pty?? something different from a master pseudo
   terminal device, he/she deserves to be punished, and would notice
   it when he/she starts using the file descriptor anyway).

 * Right now getpt() returns ENFILE if it could not open a master
   pseudo terminal.  According to the documentation this means:

   ``There are too many distinct file openings in the entire system.''

   this is not really helpfull.  The BSD openpty() function returns
   ENOENT in this case which seems more logical to me.  This would be
   very convenient since getpt() is used to implement openpty() so we
   can avoid converting errno there.

 * Right now we stop looking for master pty's when open() returns any
   error except EIO.  Indeed this is the error that Linux returns when
   a master pseudo terminal is already in use, but this also means
   that the implementation is tightly coupled to the Linux kernel.  In
   fact the Hurd (which uses the same implementation) returns EBUSY in
   this case.  Stopping on ENOENT seems a better thing to do.  Again
   this is what BSD does.

The following patch adresses these issues.  If we agree that this is
indeed the right thing I have a patch for the documentation to correct
the return value of getpt() available.

Mark


1999-03-13  Mark Kettenis  <kettenis@gnu.org>

	* sysdeps/unix/bsd/getpt.c (__getpt): Do not use `isatty' to check
	if the opened master pty really is a pty.  `isatty' checks for
	tty-ness and a pty is not a tty.  Return ENOENT instead of ENFILE
	if we are out of pty's.


--- /home/kettenis/CVS/libc/sysdeps/unix/bsd/getpt.c	Thu Sep 24 23:31:59 1998
+++ libc/sysdeps/unix/bsd/getpt.c	Wed Feb 24 18:21:41 1999
@@ -1,4 +1,4 @@
-/* Copyright (C) 1998 Free Software Foundation, Inc.
+/* Copyright (C) 1998, 1999 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
    Contributed by Zack Weinberg <zack@rabi.phys.columbia.edu>, 1998.
 
@@ -64,20 +64,14 @@
 
 	  fd = __open (buf, O_RDWR);
 	  if (fd != -1)
-	    {
-	      if (__isatty (fd))
-		return fd;
-
-	      __close (fd);
-	      continue;
-	    }
+	    return fd;
 
-	  if (errno != EIO)
+	  if (errno == ENOENT)
 	    return -1;
 	}
     }
 
-  __set_errno (ENFILE);
+  __set_errno (ENOENT);
   return -1;
 }
 weak_alias (__getpt, getpt)