Re: Improving libm-test.inc structure and maintenance

["Joseph S. Myers" <joseph at codesourcery dot com>]

On Sun, 5 May 2013, Ondrej Bilka wrote:

> Why do auto-libm-test-out  have to be checked in? If outputs be computed 
> automaticaly we can save them in test directory. I could write utility
> that caches mpc results on disk and returns them.

Generating expected results would dominate the time required for a 
complete testsuite run if it wasn't checked in; build/test directories are 
generally expected to be transient.  Also, I don't want to assume the user 
testing glibc has an MPC version recent enough to contain mpc_log10, but 
do want to use mpc_log10 when generating the expected results rather than 
hacking around its absence.  Nor do I want MPFR or MPC bugs to cause 
different people to get different glibc test results because their 
expected results were computed differently.

> One possibility would be use dryrun format which also could allow
> automate bug report of inaccurate results. I could combine this with 
> program that tries random values and looks for inaccuracies. 

I think testing random values (cf. also Jakub's code to do such testing 
for fma), probably with appropriate biases to include reasonable 
proportions of large inputs, small inputs, etc. rather than being uniform 
across the exponent range, is best as a separate project that could test 
more libm implementations than just glibc.  See the last point under "libm 
testsuite" at 
<http://sourceware.org/glibc/wiki/Development_Todo/Master>.  Humans then 
need to review results to identify if bugs found are genuinely new before 
reporting them.

I should perhaps add what I think the goal is for how libm functions 
behave, and that such random tests should test:

Each function with a floating-point result (real or complex) should behave 
as if it computes an infinite-precision result that is within a few ulp 
(in both real and complex parts, for functions with complex results) of 
the mathematically correct value (interpreted together with ISO C or POSIX 
semantics for the function in question) of the function at the exact value 
passed as the input.  Exceptions should be raised, and errno set, 
appropriately for this value (provided that there is no requirement to set 
errno for complex functions) and in accordance with ISO C / POSIX 
semantics, and it should then be rounded according to the current rounding 
direction to the result that is returned to the user.

Functions should behave as if the infinite-precision result computed is 
zero, infinity or NaN if and only if that is the mathematically correct 
infinite-precision result.  They should behave as if the 
infinite-precision result computed always has the same sign as the 
mathematically correct result.

If the mathematical result has magnitude well below half the least 
subnormal magnitude, the returned value should be either zero or the least 
subnormal (in each case, with the correct sign), according to the current 
rounding direction and with the underflow exception raised.  If the 
mathematical result is more than a few ulp above the overflow threshold 
for the current rounding direction, similarly the value returned should be 
the appropriate overflow value for the current rounding direction, with 
the overflow exception raised.

Where the mathematical result underflows and is not exactly representable 
as a floating-point value, the underflow exception should be raised (so 
there may be spurious underflow exceptions in cases where the underflowing 
result is exact, but not missing underflow exceptions in cases where it is 
inexact).

All the above applies to both real and complex parts, for complex 
functions.

In addition to the above, for fully-specified functions such as fma, sqrt 
and rint, the result and corresponding exceptions should be exactly 
according to the fully-specified semantics of the function; there should 
be no spurious or missing exceptions, including "inexact" exceptions.  
This includes detection of tininess for underflow exceptions in the same 
way as for other operations with binary floating-point results on the 
architecture in question, in accordance with the requirements of IEEE 
754-2008.  (There are a few existing fma tests using 
UNDERFLOW_EXCEPTION_BEFORE_ROUNDING to verify such architecture-specific 
requirements for fma.)

-- 
Joseph S. Myers
joseph@codesourcery.com