Problems with the (new) implementation of AF_UNIX datagram sockets

[Corinna Vinschen]

Hi Ken,

On Apr 14 12:15, Ken Brown wrote:
> Hi Corinna,
> 
> This is a follow-up to
> 
>   https://cygwin.com/pipermail/cygwin/2021-April/248284.html
> 
> I don't know if you've been following that thread, but two serious problems
> with datagram sockets (on the topic/af_unix branch) have shown up.
> 
> 1. Writing will block until a connection to the peer's pipe can be made.  In
> particular, if there are two consecutive writes with the same peer, the
> second one will block until the peer reads the first message.  This happens
> because the peer's pipe is not available for the second connection until the
> peer disconnects the first connection.  This is currently done in recvmsg,
> and I don't see a straightforward way to do it anywhere else.

I'm a bit puzzeled.  The idea for datagrams was to call open/send/close
in each invocation of sendmsg.  Therefore the pipe should become
available as soon as the other peer has sent it's data block.  The time
a sendmsg has to wait for the pipe being available should be quite short!

> 2. There's no way for select to test whether a datagram socket is ready for
> writing.  That's because we can't know whether a connection to a
> hypothetical peer's pipe will be possible.  According to Stevens, the issue
> *should* be whether there's space in the socket's send buffer.  But our
> sockets don't have a send buffer until they connect to a pipe.

Even then, there's no guarantee a send will succeed, given that
select/send are not running atomically.  However, we *could* for a start
always return success from select for this scenario.  If we have a
nonblocking socket, it should fail opening the pipe and return EGAIN,
which is perfectly fine.  If we have a blocking socket, it could block
on send, which is perfectly valid, too, because of the non-atomicity.

Or am I missing something?

> I think the solution to both problems is for Cygwin to maintain a send
> buffer for datagram sockets.  Does that seem right, or do you have another
> idea?

In theory the send buffer should be a shared buffer between all peers,
so this could be constructed as a shared ring buffer, accessible from
af_unix_shmem_t.  But then again, this introduces a security problem,
so that's not a good idea.  So, process-local buffers.

But you also have the problem how to empty the buffer.  Do you start a
new thread which checks if the pipe is getting available and if so,
sends the buffer content?  In which process?  And what do you do if
there's still data in the send buffer when the process exits?  This is
annoyingly complicated and error-prone.

Another idea might be to implement send/recv on a DGRAM socket a bit
like accept.  Rather than creating a single_instance socket, we create a
max_instance socket as for STREAM socket listeners.  The server side
accepts the connection at recv and immediately opens another pipe
instance, so we always have at least one dangling instance for the next
peer.


Corinna


P.S.: Idle musings...

The only other implementation of AF_UNIX sockets using named pipes on
Windows I know of (U/WIN) implements the gory details as part of their
priviledged server process, i. e., their equivalent of cygserver.  The
difference is that the entire system is based on this server process, so
the U/WIN processes don't run at all if that service isn't running,
quite unlike Cygwin.  Requiring a server running just to allow AF_UNIX
sockets to work seems a bit off for us...

Having said that, maybe the idea to implement AF_UNIX sockets as named
pipes is... outdated?  Roughly 90% of our users are running a W10
version supporting AF_UNIX sockets natively (albeit missing native
SOCK_DGRAM support).  Perhaps it's time to switch...?