[Fwd: 1.5.11-1: sftp performance problem]

Corinna Vinschen vinschen@redhat.com
Tue Sep 14 09:09:00 GMT 2004

On Sep 13 14:09, Bob Byrnes wrote:
> On Sep 12,  4:42pm, Corinna Vinschen wrote:
> -- Subject: Re: [Fwd: 1.5.11-1: sftp performance problem]
> This all works most of the time.  But it interacts badly with the
> POSIX atomic write requirements related to PIPE_BUF.  In particular,
> select should say that a pipe is not writable when there is < PIPE_BUF
> space available (and our implementation does this).

I reread the description of CreateNamedPipe in MSDN and now I'm wondering
if exactly that, trying to mimic POSIX atomic writes, is the culprit of
the problem.  MSDN states:

  "Whenever a pipe write operation occurs, the system first tries to charge
   the memory against the pipe write quota. If the remaining pipe write
   quota is enough to fulfill the request, the write operation completes
   immediately. If the remaining pipe write quota is too small to fulfill
   the request, the system will try to expand the buffers to accommodate
   the data using nonpaged pool reserved for the process. The write operation
   will block until the data is read from the pipe so that the additional
   buffer quota can be released. Therefore, if your specified buffer size
   is too small, the system will grow the buffer as needed, but the downside
   is that the operation will block."

I'm not sure if my interpretation is correct, but I'd guess that the OS
would also try to force a flush, as soon as the write buffer had to be
expanded beyond the write quota.  Our implementation of select *prevents*
the write buffer to be expanded...  Do you understand what I mean?

> > |                                         I guess this means that local
> > | pipes always do buffering as described in the previous paragraph, and
> > | this can't be disabled using FILE_FLAG_WRITE_THROUGH.
> > 
> > Did you try that?
> I haven't yet, but I will.  Disabling buffering would fix the problem.

Just another random idea.  What if the pipe isn't called \\.\xxx but
instead \\${hostname}\xxx ?  Perhaps (but not likely) the pipe is then
treated as remote.

> Or if we could somehow control the buffering parameters (the high-water

Hmm, there's just this DefaultTimeout value which seems to be somewhat

> mark or the timer), that would also probably be sufficient.  In particular,
> setting the high-water mark to reserve PIPE_BUF bytes would be perfect.

The problem is that the buffer sizes given to CreateNamedPipe are just
*advisory* to the system.

> > Dunno if that's a *better* idea, but would it be reasonable to try changing
> > pipes to use overlapped I/O?
> Maybe, but that seems complicated.  I'm hoping for something simpler.

Well, it's not *that* complicated.  But when reading MSDN, I'm entertaining
some doubt that it would really help for pipes.


Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          mailto:cygwin@cygwin.com
Red Hat, Inc.

More information about the Cygwin-patches mailing list