Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

James Johnston JamesJ@motionview3d.com
Mon Dec 23 19:15:00 GMT 2013

(apologies to the list; I accidentally sent this reply directly to Anton and
so his reply that he just sent will appear out of order on the list

Hi Anton,

Thanks for the response.  Comments below...

> MSDN does not list pipe handles as waitable with WaitForMultipleObjectsEx
> (only a user comment at the end does).

That is true.  One would rightfully assume that it's not supported.  I have
assumed that myself in the past when working on asynchronous code for pipes
for a completely unrelated application.

However, their own .NET Framework 4.0 does this, so it must be at least
supported on some undocumented level.  This is the stock version of the
framework available on Windows 8; the older version that does not have the
wait, NETFX 2.0, is not installed by default.  And of course, the older
version doesn't have any of the new features available in NETFX 4.x - of
which there are many.  Sadly, "fixing" the code in the .NET Framework to use
a "supported" asynchronous mechanism is not realistic.  Nobody except
Microsoft can patch the .NET Framework, unfortunately... (it is not
(L)GPL!!)  The reference code is just for debugging and learning; you can't
use it to modify anything.

Here is the reference source code:


Check line 192 for a call to the WaitForAvailableConsoleInput function.  I
was unable to find the source code for that function, which is an internal
C/C++ function (this is a mixed-mode assembly).  However, I was able to
determine that function calls PeekNamedPipe followed by
WaitForMultipleObjectsEx using the very handy freeware API Monitor from
rohitab.com.  Using this tool I also learned the exact parameters that they
pass to each of those functions, and then distill it into a C program that I
sent in my original message that reproduces the issue on my machine.
This WaitForAvailableConsoleInput function is new to .NET Framework 4.0, and
according to the comment, was added in order to prevent the ReadFile call
from blocking in order to support aborting a thread.  I think both of us can
probably suggest better ways of handling this that uses a supported
mechanism.  But this is how they did it. :(

As mentioned in the original message, this pattern of PeekNamedPipe &
WaitForMultipleObjectsEx doesn't cause any problems except with Cygwin for
reasons that are currently unknown to me.

> If you trace execution without
> PeekNamedPipe(), you'll see that WaitForMultipleObjectsEx() returns
> immediately, and that it is ReadFile() that blocks and waits for the input
> (which will be supplied after 3 seconds).

Right; this is very strange and contrary to what one would naturally expect
if WaitForMultipleObjectsEx was supported on named pipes (which it
apparently is at least on some undocumented level, if their own .NET
Framework 4.0 is doing it).  Intuitively, peeking something shouldn't affect
any synchronization.  Yet it does...  (Perhaps waiting on a pipe is not
"fully" supported and Cygwin found an edge case!)

> --quote--
> Is it possible to wait for data to be available on a non-GUI-based console
> input? The issue is that WaitFor* methods do not support PIPE handles and
> therefore STDIN is not supported if the process input is fed from another
> process, e.g. using the pipe | functionality of cmd.
> The only way I know to query a pipe for data without actually reading the
> data is to use PeekNamedPipe(), but then you lose the ability to wait on a
> timeout effectively, because it has no timeout so you would have to
> implement a manual loop instead.
> --quote--

Indeed; there is no clean solution.  All the ideas that come to mind have

 * WaitForMultipleObjectsEx: not officially supported, as previously noted.
 * Overlapped I/O: perfect supported solution except requires the pipe to be
initialized to support that, which probably wasn't done.
 * PeekNamedPipe in a busy loop as suggested in the quote: (1) high CPU
utilization, (2) and/or high latency on the pipe, due to unnecessary
sleeping, (3) your code stays active in memory and can't be paged out to
disk, (4) all the above kills power efficiency...

I've used overlapped I/O myself in the past on an unrelated application, but
I had full control over the creation of the pipes and full control over the
reading/writing of the pipes - not the case here.

So I can see why they went for WaitForMultipleObjectsEx method... the other
methods have significant performance and/or functional limitations, and they
probably found out from a group within Microsoft that it's ok to wait on a
pipe anyway.

Best regards,

James Johnston

Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

More information about the Cygwin mailing list