This is the mail archive of the cygwin-talk mailing list for the cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

internationalized text processing


Never posted here, but somewhat off-topic for the general Cygwin list...

I've been frustrated at the lack of guidance I've found online regarding 
how to build an internationalized text-based Unix application, in particular 
with reference to handling different character sets. For example, "man"
recognizes the character set specified in locale settings and uses Unicode 
punctuation appropriately. OTOH, the cygutils "ascii -e" utility does not 
recognize that my LANG specifies UTF-8 and outputs garbage for the 
extended half. Should this be considered a bug?

Closer to a real application, how should a program that reads, processes, 
and outputs text detect the character set so that it can process the text 
correctly? In the absence of a BOM in the input file, should the program 
blindly presume the locale-specified character set or analyze the file to 
guess it? And what advice is there for guessing? And, once detected, what 
takes precedence for output: the input file's character set or the locale 
setting?

I presume these issues are either settled or being debated somewhere.

--Ken Nellis

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]