Re: to XFAIL or not to XFAIL

[DJ Delorie <dj at delorie dot com>]

> If we use FAIL both for new failures and for tests which are known to
> fail, then it is difficult to use the testsuite as a regression
> testsuite.
> 
> If we use XFAIL for tests which are known to fail, then we distinguish
> new failures from old failures.  This makes it possible to use the
> testsuite as both a regression testsuite and a real testsuite.  To
> check the real bugs, look at both FAIL and XFAIL results.

If we had a third case, say KFAIL, for "known to fail but ignore it
anyway," I'd be happy to go along with your policy.  Using XFAIL for
both "ignore until later" and "impossible to fix on this target" makes
it difficult to assess the overall quality of the software.

As for the regression problem, one simple solution is to run the
testsuite before and after your changes, and diff the outputs.  Or,
compare your results with some similar platform your changes don't
affect.