[PATCH] Cygwin: ptsname_r: always return an error number on failure

[Ken Brown]

On 1/20/2021 1:00 PM, Ken Brown via Cygwin-patches wrote:
> Following Linux, return ENOTTY on a bad file descriptor and also set
> errno to ENOTTY.
> 
> Previously 0 was returned and errno was set to EBADF.  Returning 0
> violates the requirement in
> https://man7.org/linux/man-pages/man3/ptsname_r.3.html that an error
> number should be returned on failure.  (That man page doesn't specify
> setting errno.)
> 
> Addresses: https://lists.gnu.org/archive/html/bug-gnulib/2021-01/msg00245.html
> ---
>   winsup/cygwin/release/3.2.0 | 3 +++
>   winsup/cygwin/syscalls.cc   | 5 ++++-
>   2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/winsup/cygwin/release/3.2.0 b/winsup/cygwin/release/3.2.0
> index 43725cec2..f748a9bc8 100644
> --- a/winsup/cygwin/release/3.2.0
> +++ b/winsup/cygwin/release/3.2.0
> @@ -52,3 +52,6 @@ Bug Fixes
>   - Fix the errno when a path contains .. and the prefix exists but is
>     not a directory.
>     Addresses: https://lists.gnu.org/archive/html/bug-gnulib/2021-01/msg00214.html
> +
> +- Fix the return value when ptsname_r(3) is called with a bad file descriptor
> +  Addresses: https://lists.gnu.org/archive/html/bug-gnulib/2021-01/msg00245.html
> diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
> index 4742c6653..18d9e3f88 100644
> --- a/winsup/cygwin/syscalls.cc
> +++ b/winsup/cygwin/syscalls.cc
> @@ -3364,7 +3364,10 @@ ptsname_r (int fd, char *buf, size_t buflen)
>   
>     cygheap_fdget cfd (fd);
>     if (cfd < 0)
> -    return 0;
> +    {
> +      set_errno (ENOTTY);
> +      return ENOTTY;
> +    }
>     return cfd->ptsname_r (buf, buflen);
>   }
>   
> 

I'm not really convinced we should blindly follow Linux here, when EBADF would 
seem to make more sense.  See

   https://lists.gnu.org/archive/html/bug-gnulib/2021-01/msg00264.html

Corinna, what's your preference?

Ken