Add FAILs? Was: RFC: Testsuite patches, a few comments and questions...

[Fernando Nasser <fnasser at cygnus dot com>]

Andrew Cagney wrote:
> 
> If the 5.0 branch suddenly gained a heap more failures, I've this fear
> that people would try to fix the branch when a proper fix on the trunk
> would be easier in the long term.
> 
BTW, I figured that we were not testing bitfields properly (the tests we have test the compiler, but
not the gdb handling of them) and that they are broken in all platforms that I tested (Solaris,
Linux).  I have the tests that properly test these things but I thought it was the tradition to fix
the bug first and them add the tests as regression tests, together with the fix.

I don't like this approach for two reasons:

1) Someone else can see the FAILs and volunteer some time to fix it before you get a chance to (for
instance, I still could not get around to it).

2) You lose the opportunity to check on which platforms it is broken and which ones it is not.  This
can give you some hints: endianess, debug info format, ptrace x proc, etc.

3) Something is broken there and people my try to use it without knowing it.  But there is another
solution for this part (see below). 

> -- The following are more general comments:
> 
> With regard to XFAIL.  My understanding is that it indicates something
> that can't be fixed (technical limitation) rather than won't be fixed
> (not enough time :-( :-).  The test maintainers will probably correct me
> :-).  I think people checking in testsuite fixes / extensions should be
> encouraged, even/especially when those fixes increase the number of
> testsuite failures.
> 

I couldn't agree more.  I offered to create a new classification to KNOWN bugs on the testsuite, but
it was not accepted.  I was told that when we finished fixing the FAILs we would then go and start
looking into the XFAILs.  I don't think so.  Many XFAILs are legitimate so it is a big problem to
separate what really needs attention.

It is arguable that, if we have a very good bug tracking system in place and if we are very careful
to cross list the known bugs to the tests that have been improperly (in my opinion) XFAILed.  This
has not been done yet, and IMHO this XFAIL issue is a mess.  I have found and posted several times
things that are legitimate FAILs and that are marked as XFAILs.

I really would like to listen to peoples opinions on this and to take some action.  But I would need
us to agree on something first.



-- 
Fernando Nasser
Cygnus Solutions (a Red Hat company)    E-Mail:  fnasser@cygnus.com