bugs in CP1258 converter (1)

[Bruno Haible <haible at ilog dot fr>]

Hi,

CP1258 is an encoding designed for Vietnamese. However, the glibc iconv
converter for this encoding cannot convert even the simplest Vietnamese
text, here: the greeting taken from the Emacs HELLO file.

$ printf 'Vietnamese (Ti\xe1\xba\xbfng Vi\xe1\xbb\x87t)\tCh\xc3\xa0o b\xe1\xba\xa1n\n' > viet-greeting.txt
$ iconv -f UTF-8 -t CP1258 < viet-greeting.txt > /dev/null
iconv: illegal input sequence at position 14

CP1258 is particular among the 8-bit encodings: it contains five combining
characters
   U+0300  COMBINING GRAVE ACCENT
   U+0301  COMBINING ACUTE ACCENT
   U+0303  COMBINING TILDE
   U+0309  COMBINING HOOK ABOVE
   U+0323  COMBINING DOT BELOW

In the CP1258 encoding, the combining characters follow the base character,
like in Unicode. But CP1258 also contains some precomposed characters. For
example, the "e" in "Viet" above is

   U+1EC7  LATIN SMALL LETTER E WITH CIRCUMFLEX AND DOT BELOW

which in CP1258 is represented as

   U+00EA  LATIN SMALL LETTER E WITH CIRCUMFLEX
   U+0323  COMBINING DOT BELOW

But note that CP1258 does *not* have the combining CIRCUMFLEX.

For this reason, the Unicode to CP1258 converter must partially decompose
some precomposed characters.

And the CP1258 to Unicode converter must produce composed, not decomposed
output, because Normalisation Form C is the standard for text interchange.
So this conversion direction must combine U+00EA U+0323 to a single Unicode
character. This also achieves round-trip compatibility for
Unicode -> CP1258 -> Unicode conversion.

The only way to implement this CP1258 to Unicode converter is as a stateful
converter. (If it were a stateless encoding, it could not convert 0xEA 0xF2
to U+1EC7 but lone 0xEA to U+00EA.)

But it turns out that the converter structure in iconv/skeleton.c is not
prepared for stateful encodings whose EMIT_SHIFT_TO_INIT macro outputs
something in the FROM_DIRECTION, only for those where output occurs in the
TO_DIRECTION.

So here are two patches:
  (2) A fix to iconv/skeleton.c to cope with such encodings.
  (3) The CP1258 converter and associated tests.

A similar patch for CP1255 will follow later.

Bruno