ABI incompatibility between libffi and LLVM-generated callees

[Mark H Weaver <mhw at netris dot org>]

Hello all,

I'm a developer of Guile 2.0, which uses libffi, and we've received
multiple bug reports of test failures on OS X related to libffi.  We
recently discovered the root cause of these failures.

Functions compiled using LLVM for the x86_64 architecture assume that
callers will sign-extend integer arguments less than 64-bits that are
passed in registers.  Indeed, both GCC and LLVM generate code that does
this sign-extension in callers, so it seems that their assumption is
reasonable.

The only problem is that libffi does *not* sign-extend integer arguments
passed in registers on x86_64.  Instead it zero-extends them, even if
they are signed.  This works when calling GCC-compiled code, because GCC
performs the sign-extension in the callee as well as the caller, but
LLVM-compiled code is more strict.

Having read section 3.2.3 ("Parameter Passing") of the SysV x86_64 ABI,
which is admittedly somewhat vague on this issue, it is far from clear
to me that the LLVM behavior is a bug.  It seems to me that callers
should sign-extend to be on the safe side.

I believe this is a case of GCC following Postel's Law of being
permissive in what one accepts, whereas LLVM is more strict in its
requirements.

However you might choose to interpret the ABI requirements, the fact is
that LLVM-compiled libraries are widely deployed in Mac OS X, and
libffi is currently unable to properly call such libraries properly.

For more details on this, see:

  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13342#26
  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13342#32

Comment #26 shows the difference in assembly code generated by GCC and
LLVM for functions that accept int8_t arguments.

Also see the recent report of libffi test failures on OS X:

  http://sourceware.org/ml/libffi-discuss/2012/msg00162.html

I suggest that libffi should be changed to sign-extend.
What do you think?

    Regards,
      Mark