Tee and file redirections are very slow to write anything.
Hamish McIntyre-Bhatty
hamishmb@live.co.uk
Thu Feb 25 09:21:30 GMT 2021
On 25/02/2021 07:50, Brian Inglis wrote:
> On 2021-02-24 16:50, Brian Inglis wrote:
>> On 2021-02-24 15:41, Duncan Roe wrote:
>>> On Wed, Feb 24, 2021 at 04:58:24PM -0500, Eliot Moss wrote:
>>>> On 2/24/2021 3:48 PM, ASSI wrote:
>>>>> Hamish McIntyre-Bhatty via Cygwin writes:
>>>>>> I found recently when trying to save output from a script for later
>>>>>> inspection that "tee" and file redirections seem to have massive
>>>>>> delays when run in Cygwin - usually nothing is written to file or
>>>>>> stdout until after the command has finished - not very helpful.
>>>>>
>>>>> You will want to switch from fully buffered to line-buffered or even
>>>>> unbuffered output.
>>
>>>> And this does not have to do with Cygwin. The same happens on Linux.
>>>> The default is that terminal I/O is unbuffered while other stream are
>>>> buffered. Pipes come under "other streams". One can make
>>>> programmatic
>>>> changes to get around this, but most programs won't override the
>>>> default behavior on their own ...
>>
>>> The (Linux) default is that terminal I/O is *line* buffered
>>>
>>> The man page for tee doesn't show an option to change buffering,
>>> while that for
>>> grep does.
>>
>> I believe the default for both Cygwin and Linux is 64KB pipe buffer,
>> so if you want to see smaller chunks as they are generated, you need
>> to add some utility that may allow you to change that e.g.
>>
>> $ tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq
>>
>> but read the disclaimers on the stdbuf and grep man pages, which is
>> why it is not done more, especially under Cygwin where Windows adds
>> its own performance penalties.
>> Some utilities may use read(2/3p), write(2/3p), or mmap(3) if they
>> can and don't care about text or lines, for more efficient access to
>> disk files, rather than buffered stream I/O functions.
>
> From what I have been able to find, Cygwin <stdio.h> BUFSIZ is only
> 1K, compared to Linux 8K, and Cygwin internal 64K, and that is used in
> many places in coreutils like tee, which will slow everything down by
> a factor of at least 8 plus increased overhead.
>
> Suggest <stdio.h> BUFSIZ be bumped to at least Linux value of 8K, if
> not 64K.
So from the discussion above, I've not sure I fully understand why the
behaviour is different on Cygwin to Linux for me, especially if Linux's
buffer size is sometimes bigger.
Perhaps:
#1: It depends on the program being run and how it sets its buffers up?
#2: stdbuf -o L enables line buffering and should fix my problem?
I'm now thinking it may have been a Python script that was behaving this
way - I might go and double check in case this isn't what I think it is.
Hamish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x18F1759B3457223F.asc
Type: application/pgp-keys
Size: 3131 bytes
Desc: not available
URL: <https://cygwin.com/pipermail/cygwin/attachments/20210225/150017db/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://cygwin.com/pipermail/cygwin/attachments/20210225/150017db/attachment-0001.sig>
More information about the Cygwin
mailing list