[newlib-cygwin/main] Cygwin: Ensure unused fd available for open()

[Jon Turney]

https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=31bf91f867c5fadd7deb408cf06fe3af8e86bb74

commit 31bf91f867c5fadd7deb408cf06fe3af8e86bb74
Author: Mark Geisert <mark@maxrnd.com>
Date:   Wed May 27 22:42:44 2026 -0700

    Cygwin: Ensure unused fd available for open()
    
    The existing logic for open() assumes an fd is always available in
    the fdtable for a created file.  This leads to a situation where, if
    there is no fd available due to the OPEN_MAX limit being hit, the
    file is created but cannot be referenced by a Cygwin fd.
    
    Move the fd reservation code to an earlier location within open().
    
    Reported-by: Christian Franke <Christian.Franke@t-online.de>
    Addresses: https://cygwin.com/pipermail/cygwin/2026-May/259664.html
    Signed-off-by: Mark Geisert <mark@maxrnd.com>
    Fixes: e859706578ba (* autoload.cc (NtCreateFile): Add.)

Diff:
---
 winsup/cygwin/syscalls.cc | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
index 7a8e5d4fd..2bea79768 100644
--- a/winsup/cygwin/syscalls.cc
+++ b/winsup/cygwin/syscalls.cc
@@ -1547,6 +1547,13 @@ open (const char *unix_path, int flags, ...)
 	  fh = fh_file;
 	}
 
+      /* Reserve an fdtable entry here, before calling open_with_arch() below.
+         Otherwise there's a tiny chance of hitting OPEN_MAX further on which
+         could create a new file without any way for Cygwin to refer to it. */
+      cygheap_fdnew fd;
+      if (fd < 0)
+        __leave;		/* errno already set */
+
       if (fh->dev () == FH_PROCESSFD && fh->pc.follow_fd_symlink ())
 	{
 	  /* Reopen file by descriptor */
@@ -1573,14 +1580,6 @@ open (const char *unix_path, int flags, ...)
 	try_to_bin (fh->pc, fh->get_handle (), DELETE,
 		    FILE_OPEN_FOR_BACKUP_INTENT);
 
-      cygheap_fdnew fd;
-
-      if (fd < 0)
-	{
-	  fh->close();
-	  __leave;		/* errno already set */
-	}
-
       fd = fh;
       if (fd <= 2)
 	set_std_handle (fd);