This is the mail archive of the cygwin mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Please support CP932. (I have problem using subversion with SJIS)


On Jan 23 14:50, Corinna Vinschen wrote:
> On Jan 23 14:49, Nayuta Taga wrote:
> > In short, '~' (U+007E TILDE) turns into U+203E (OVERLINE) when
> > LANG=ja_JP.SJIS.
> > 
> > Then I looked into cygwin and subversion again.
> > (1) cygwin1.dll converts L"foo~" (UCS-2) to "foo~" (CP932).
> > (2) Because subversion's internally uses UTF-8,
> >     "foo~" (CP932) should be converted to "foo~" (UTF-8).
> > (3) It uses iconv to convert from *SJIS* to UTF-8,
> >     because nl_langinfo(CODESET) returns "SJIS" when LANG=ja_JP.SJIS.
> > (4) The final string is "foo\xe2\x80\xbe".
> >     (e2 80 be is UTF-8 representation of U+203E)
> 
> SJIS is the charset name for the Windows codepage 932.  The multibyte to
> widechar conversion (and vice versa) for SJIS even uses the Windows
> conversion functions under the hood.  And the character 0x7e in SJIS is
> identical to the Unicode character U+00fe.
> 
> So, why does iconv turn U+007e into U+203E?
> 
> This sounds like a bug in iconv, not in Cygwin.  Your patch just adds an
> additional charset name CP932 for the exact same charset SJIS.  What
> this does is just cancel the recognition of the charset in iconv.  That
> sounds like a hack, rather than a solution.

Ouch.  I understand now.  Standard SJIS is *really* different from
Microsoft CP932 in two code points:

  CP932 0x5c == U+005E
  SJIS  0x5c == U+00A5

  CP932 0x7e == U+007E
  SJIS  0x7e == U+203E

Bummer.  Actually the problem is SJIS, not CP932.  One of the basic
ideas in Cygwin is that every character set has at least an intact
ASCII code range.

Hmm.

Would it be a valid help for your case if Cygwin's SJIS conversion would
convert 0x5c to U+00A5 and 0x7e to 203E, so that the SJIS conversion
would be really correct *and* bijective?  To me this sounds like the
better solution than adding a CP932 charset identifier.


Cor

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]