This is the mail archive of the
mailing list for the Cygwin project.
Re: gawk Regression: CR characters are not stripped on Windows
- From: Eric Blake <eblake at redhat dot com>
- To: cygwin at cygwin dot com, bug-gawk at gnu dot org, Eli Zaretskii <eliz at gnu dot org>
- Date: Tue, 27 Feb 2018 10:56:50 -0600
- Subject: Re: gawk Regression: CR characters are not stripped on Windows
- Authentication-results: sourceware.org; auth=none
- References: <CAGHpTB+bfbts=fOBSQPN7c-NDh8FTXR+EauhDhiVrqbgawcYoA@mail.gmail.com>
[urrgh - Cygwin's list policy in supplying reply-to makes it difficult
On 02/27/2018 01:22 AM, Orgad Shaneh wrote:
Cross-posting per Eli Zaretskii's request.
CR characters used to be automatically stripped on Windows (MSYS2 and
Cygwin environments). This is broken in 4.2.0.
You should not think of Cygwin as a Windows environment, but as a
Linux-alike environment. gawk on Linux does not automatically strip
CRs, therefore gawk on Cygwin should not automatically strip CRs.
What MSYS2 does is different, and that environment is entitled to use
patches to make interoperability with native windows program nicer, at
the expense of being less like Linux.
Furthermore, the change in Cygwin predates the gawk 4.2.0 release, and
was intentionally made in a coordinated release in Feb 2017 alongside
sed and grep:
following on from discussions about bash after ShellShock:
Changing gawk back to automatically strip CRs on Cygwin would be a
As Eli said, this change was deliberate. But this has several drawbacks.
1. The gawk info page states that:
Under MS-Windows, 'gawk' (and many other text programs) silently
translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on
and on Feb 8 the following section was added:
Recent versions of Cygwin open all files in binary mode. This means
that you should use 'RS = "\r?\n"' in order to be able to handle
standard MS-Windows text files with carriage-return plus line-feed line
Or mount your Windows text files under a text mount in Cygwin (so that
such files already have \r stripped), or add steps to your pipelines to
strip CR before handing the data to gawk.
This breaks compatibility between different gawk versions. What were
the reasons for this change in cygwin, and why was it pushed upstream?
See the discussion in Feb 2017 for rationale, but the executive summary
is that Cygwin attempts to emulate Linux, silent corruption of binary
files was deemed worse than manually having to explicitly strip CR when
dealing with Windows text output.
2. Git and other tools automatically convert text files to CRLF on
Not Cygwin git. The problems you are encountering are more likely to
happen when you mix and match tools from disparate environments, rather
than when you use all tools from the same source.
This means that any awk script that runs on both platforms
must use RS = "\r?\n".
or strip the CR in any other means. But the same is true of any script
that must run on both Windows and Linux.
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
Problem reports: http://cygwin.com/problems.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple