Re: Cygwin fails to utilize Unicode replacement character

[Steven Penny <svnpenn at gmail dot com>]

On Tue, 4 Sep 2018 20:41:48, Thomas Wolff wrote:
> No idea what you consider dangerous. Anyway, we obviously agree that 
> hardly any available console font supports the REPLACEMENT CHARACTER. 
> You had previously suggested code that might work (using CreateFont(0, 
> 0, ....)). Maybe you can sort out with Corinna how to get that work 
> inside cygwin. Otherwise, my opinion:
> - *working* fallback from FFFD to 2592: good

i am fine with this, but i think corinna feels it is too much code for not
enough benefit - thats her decision.

> - fix FFFD: not good, because the .notdef glyph is not an appropriate 
> indication of illegal encoding (like broken UTF-8 bytes)

not sure what you even mean by this - FFFD doesnt need fixing - Windows just
need to adopt some fonts with proper unicode support. we are dealing with their
lack of doing that.

> the .notdef glyph is not an appropriate indication of illegal encoding (like
> broken UTF-8 bytes)

true, but neither is U+2592. as far as i know U+2592 is not defined officially
anywhere as being a representation of anything other than "MEDIUM SHADE".
Corinna originally added it in 2009:

http://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git&a=commitdiff&h=161211d

with no justification of why it was chosen that i can tell. similarly, mintty
actually changed from U+FFFD to U+2592 in 2009:

http://github.com/mintty/mintty/commit/90c11d3

with actually a good reason, which was to avoid ambiguity with fonts that didnt
have U+FFFD. but again, no reason why U+2592 was chosen. i personally see both
sides of the argument but i tend to land of the side of any standards if they
exist. Here is the standard for U+FFFD:

http://unicode.org/charts/nameslist/n_FFF0.html

> - revert to 2592: OK

if we were to use something other than U+FFFD, I would propose U+25A1, as it is
also defined by Unicode:

   25A1	 □ 	White Square
   •	may be used to represent a missing ideograph

http://unicode.org/charts/nameslist/n_25A0.html

and it has better support than U+FFFD:

   yes:
   - Consolas
   - Courier New
   - DejaVu Sans Mono
   - MS Gothic
   - NSimSun

   no:
   - Lucida Console
   - SimSun-ExtB


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple