cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/30/2021 1:00 PM, Corinna Vinschen wrote:
> On Aug 30 17:53, Corinna Vinschen wrote:
>> On Aug 30 16:05, Corinna Vinschen wrote:
>>> On Aug 30 09:36, Ken Brown wrote:
>>>> BTW, when I was working on the pipe approach to AF_UNIX sockets
>>>> (topic/af_unix branch), I had occasion to step through
>>>> select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota -
>>>> fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote
>>>> peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi
>>>> is suggesting now.
>>>
>>> If that's working reliable these days (keeping fingers crossed for W7),
>>> it's ok if we use that.  We may want to check if the above observation
>>> in terms on WriteQuotaAvailable on a pipe with pending read is still an
>>> issue.
>>
>> Ok, I wrote a small testcase.  It creates a named pipe, reads from the
>> pipe, then, later, writes to the pipe.  Interlaced with these calls, it
>> calls NtQueryInformationFile(FilePipeLocalInformation) on the write side
>> of the pipe.  Kind of like this:
>>
>>    CreatePipe
>>    NtQueryInformationFile
>>    ReadFile
>>    NtQueryInformationFile
>>    WriteFile
>>    NtQueryInformationFile
>>
>> Here's the result:
>>
>> Before ReadFile:
>>
>>    InboundQuota: 65536
>>    ReadDataAvailable: 0
>>    OutboundQuota: 65536
>>    WriteQuotaAvailable: 65536
>>
>> While ReadFile is running:
>>
>>    InboundQuota: 65536
>>    ReadDataAvailable: 0
>>    OutboundQuota: 65536
>>    WriteQuotaAvailable: 65494	!!!
>>
>> After WriteFile and ReadFile succeeded:
>>
>>    InboundQuota: 65536
>>    ReadDataAvailable: 0
>>    OutboundQuota: 65536
>>    WriteQuotaAvailable: 65536
>>
>> That means, while a reader on the reader side is waiting for data, the
>> WriteQuotaAvailable on the write side is decremented by the amount of
>> data requested by the reader (42 bytes in my case), just as outlined in that
>> mail from 2004.  And this is on W10 now.
>>
>> What to do with this information?  TBD.
> 
> Ok, let's discuss this.  I added more code to my testcase and here's
> what I see.  I dropped all data from the output which doesn't change.
> 
> What I'm trying to get a grip on are the dependencies here.
> 
> After creating the pipe:
> 
>    read side: ReadDataAvailable: 0
>    write side: WriteQuotaAvailable: 65536
> 
> After writing 20 bytes...
> 
>    read side: ReadDataAvailable: 20
>    write side: WriteQuotaAvailable: 65516
> 
> After writing 40 more bytes...
> 
>    read side: ReadDataAvailable: 60
>    write side: WriteQuotaAvailable: 65476
> 
> After reading 42 bytes...
> 
>    read side: ReadDataAvailable: 18
>    write side: WriteQuotaAvailable: 65518
> 
> After writing 20 bytes...
> 
>    read side: ReadDataAvailable: 38
>    write side: WriteQuotaAvailable: 65498
> 
> *While* reading 42 bytes with an empty buffer...
> 
>    read side: ReadDataAvailable: 0
>    write side: WriteQuotaAvailable: 65494
> 
> Another important fun fact:  Assuming the read and write buffer sizes
> are differently specified.  I called CreateNamedPipe with an outbuffer
> size of 32K and an inbuffer size of 64K:
> 
> After creating the pipe:
> 
>    read side:
>      InboundQuota: 65536
>      ReadDataAvailable: 0
>      OutboundQuota: 32768
>      WriteQuotaAvailable: 32768
>    write side:
>      InboundQuota: 65536
>      ReadDataAvailable: 0
>      OutboundQuota: 32768
>      WriteQuotaAvailable: 65536	!!!
> 
> This last data point shows that:
> 
> - InboundQuota and OutboundQuota are always constant values and
>    do not depend on the side the information has been queried on.
>    That certainly makes sense.
> 
> - WriteQuotaAvailable does not depend on the OutboundQuota, but on
>    the InboundQuota, and very likely on the InboundQuota of the read
>    side.  The OutboundQuota *probably* only makes sense when using
>    named pipes with remote clients, which we never do anyway.
> 
> The preceeding output shows that ReadDataAvailable on the read side and
> WriteQuotaAvailable on the write side are connected.  If we write 20
> bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
> decremented by 20.
> 
> So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
> 
> Except when a ReadFile is pending on the read side.  It's as if the
> running ReadFile already reserved write quota.  So the write side
> WriteQuotaAvailable is the number of bytes we can write without blocking,
> after all pending ReadFiles have been satisfied.
> 
> Unfortunately that doesn't really make sense when looked at it from the
> user space.
> 
> What that means in the first place is that WriteQuotaAvailable on the
> write side is unreliable.  What we really need is InboundQuota -
> read.ReadDataAvailable.  The problem with that is that the write side
> usually has no access to the read side of the pipe.

For the purposes of select.cc:pipe_data_available, we only need to know whether 
InboundQuota - read.ReadDataAvailable is positive.  If WriteQuotaAvailable is 
positive, then we're OK, even though its precise value might be too small.  But 
if WriteQuotaAvailable == 0, we don't know whether the buffer is actually full 
or there's a pending ReadFile.  It's only in this case that we need access to 
the read side.

What if we reverse the roles of the read and write sides of the pipe, so that 
the write side is the server and the read side is the client.  We can then try 
to use ImpersonateNamedPipeClient to get information about the read side when 
WriteQuotaAvailable == 0.  If we succeed, then we can determine whether or not 
there's space in the buffer.  If we fail, we simply report, possibly 
incorrectly, that there is space.  This is no worse than the current situation 
(on the master branch), in which we use OutboundQuota - ReadDataAvailable, which 
is always positive.

Ken