Perl Unidecode modules - which to use (if not Text::Unidecode)?
L A Walsh
cygwin@tlinx.org
Mon Apr 5 09:26:38 GMT 2021
On 2021/04/04 14:26, Joel Rees via Cygwin wrote:
>
>> 1. What perl Unicode modules should I consider, if not Text::Unidecode?
>> The present need
>> is to be able to convert those few "foreign" characters (like
>> ÇĆĈĊçĉċĜĞĠĢĝģğġËÌÍÎÏÒÓÔÕ)
>> that are basically ASCII with accent marks to their closest ASCII
>> equivalents, but I'd
>> like to do more with Unicode in the future, without going down any
>> dead-ends as far as
>> being able to run under cygwin is concerned.
>>
>>
>
> "Stripping those few foreign accent characters" is probably not really what
> you want to do.
>
----
Why not? You don't know his use case and you are misinterpreting his
example as random garbage.
Those aren't a random foreign encoding -- those are C's G's then E, I O
with accent variations that he may want to collapse for purposes of storing
in a text storage and retrieval (search) application. They are all well
formed/well-coded UTF-8 characters -- they are not some 8-bit encoding
that was remangled during a no-recoding display of them in a UTF-8
context.
I didn't know about Text::Unidecode -- but it specifically to create
Latinized alternatives to foreign characters. That was another hint
that it wasn't a random mistake. The manpage for it says:
It often happens that you have non-Roman text data in Unicode,
but you
can't display it -- usually because you're trying to show it to a
user
via an application that doesn't support Unicode, or because the fonts
you need aren't accessible. You could represent the Unicode
characters
as "???????" or "\15BA\15A0\1610...", but that's nearly useless
to the
user who actually wants to read what the text says.
An example was like:
tperl
use utf8;
use Text::Unidecode;
my $name="\x{5317}\x{4EB0}";
printf "name, %s == %s\n", $name, unidecode($name);
'
name, 北亰 == Bei Jing
It's not just about removing accents but getting an English
like translation based on the foreign text.
All of the characters he used as example were well coded utf-8
characters --
> Those "accent characters" are misinterpreted foreign encoding (likely not
> to be Unicode) characters. Simply "stripping" the "accent characters" will
> basically convert them to truly meaningless junk. I suppose the meaningless
> junk can then be interpreted by the reader as "used to be a be a foreign
> word here", but why bother contributing further to information entropy?
>
> 2. I see some talk of Internationalization in Chapter 2 of "Setting up
>
>> Cygwin", but
>> cannot see anything relating to perl modules, and I don't see any easy way
>> to search many
>> months of the mailing list for a keyword... is there any information I
>> should know about?
>>
>
>
> Have you read the perldoc on internationalization?
> --
> Problem reports: https://cygwin.com/problems.html
> FAQ: https://cygwin.com/faq/
> Documentation: https://cygwin.com/docs.html
> Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
>
>
More information about the Cygwin
mailing list