[newlib-cygwin] Cygwin: fork: fix a potential hang in fork

[Corinna Vinschen]

https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=717c36c0a4e387a73146c1181b7b89f091176e85

commit 717c36c0a4e387a73146c1181b7b89f091176e85
Author: Corinna Vinschen <corinna@vinschen.de>
Date:   Mon Aug 29 12:25:24 2022 +0200

    Cygwin: fork: fix a potential hang in fork
    
    while debugging a problem introduced in commit
    63b503916d42 ("Cygwin: tls_pathbuf: Use Windows heap")
    a hang in fork was encountered using the original implementation
    of tls_pathbuf:
    
    Using tmp_pathbuf inside the code block guarded by __malloc_trylock
    may call malloc from tmp_pathbuf::w_get and thus trying to lock an
    exclusive SRW lock recursively, which results in a deadlock.
    
    Allocate a small SECURITY_ATTRIBUTES block on the stack rather than
    allocating a 64K tmp_pathbuf.  This avoids the potential malloc call.
    
    Drop the __malloc_trylock call entirely.  There must not be a malloc
    call inside the frok::parent block guarded by __malloc_lock, and
    just trying to lock is too dangerous inside fork while other threads
    might actually chage the content of the heap.  Additionally, add a
    comment frowning on malloc usage inside tis code block.
    
    Fixes: 44a79a6eca3d ("Cygwin: convert malloc lock to SRWLOCK")
    Signed-off-by: Corinna Vinschen <corinna@vinschen.de>

Diff:
---
 winsup/cygwin/fork.cc                    | 8 +++++---
 winsup/cygwin/local_includes/cygmalloc.h | 1 -
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/winsup/cygwin/fork.cc b/winsup/cygwin/fork.cc
index e4931a286..e553c015a 100644
--- a/winsup/cygwin/fork.cc
+++ b/winsup/cygwin/fork.cc
@@ -296,7 +296,10 @@ frok::parent (volatile char * volatile stack_here)
   si.lpReserved2 = (LPBYTE) &ch;
   si.cbReserved2 = sizeof (ch);
 
-  bool locked = __malloc_trylock ();
+  /* NEVER, EVER, call a function which in turn calls malloc&friends while this
+     malloc lock is active! */
+  __malloc_lock ();
+  bool locked = true;
 
   /* Remove impersonation */
   cygheap->user.deimpersonate ();
@@ -308,8 +311,7 @@ frok::parent (volatile char * volatile stack_here)
 
   ch.silentfail (!*with_forkables); /* fail silently without forkables */
 
-  tmp_pathbuf tp;
-  PSECURITY_ATTRIBUTES sa = (PSECURITY_ATTRIBUTES) tp.w_get ();
+  PSECURITY_ATTRIBUTES sa = (PSECURITY_ATTRIBUTES) alloca (1024);
   if (!sec_user_nih (sa, cygheap->user.saved_sid (),
 		     well_known_authenticated_users_sid,
 		     PROCESS_QUERY_LIMITED_INFORMATION))
diff --git a/winsup/cygwin/local_includes/cygmalloc.h b/winsup/cygwin/local_includes/cygmalloc.h
index 73a2f060b..5e1fe8154 100644
--- a/winsup/cygwin/local_includes/cygmalloc.h
+++ b/winsup/cygwin/local_includes/cygmalloc.h
@@ -32,7 +32,6 @@ extern "C" void __set_ENOMEM ();
 #elif defined (__INSIDE_CYGWIN__)
 
 # define __malloc_lock() AcquireSRWLockExclusive (&mallock)
-# define __malloc_trylock() TryAcquireSRWLockExclusive (&mallock)
 # define __malloc_unlock() ReleaseSRWLockExclusive (&mallock)
 extern SRWLOCK NO_COPY mallock;
 void malloc_init ();