Re: [Q] Number of unit tests in Mauve and of assertions inClasspath?

[Sascha Brawer <brawer at dandelis dot ch>]

Markus Gälli <markus.gaelli@iam.unibe.ch> wrote on Thu, 11 Mar 2004 09:
19:30 +0100:

>> roughly 145 [Mauve tests] are currently failing.
>I guess this 145 are not independent, meaning, if you fixed one, many
>others would be fixed too. Right?

In some cases, there exist multiple tests for the same functionality. For
example the tests for solving quadratic and cubic equations, where the
testee is called with a bunch of somewhat random test equations. You'll
have the same situation with any tests that are feeding random input data
into a function and check whether the result is as expected. Any failures
would be sort of mutually dependent: As soon as you fix any of these
failures, the other failures are likely to disappear immediately (because
it's usually the same code that is failing). Stated in more abstract
terms, the "fixing order" is only a partial, not a total ordering on tests.

>I am just writing a paper stating that it makes sense to sort failing
>tests by size of covered methods beginning with the smallest.

Interesting... Why would this be an improvement -- because you assume
that smaller methods get invoked by larger ones, and the smaller methods
would be the specific point of failure? In that case, couldn't you just
sort the failed tests based on a (post-order) traversal of the observed
call graph? Or, this may be easier to obtain, based on the total number
of instructions/bytecodes executed by the testee and its callees? Well, I
guess the reason for sorting by method size will be stated in your paper.
Would you mind posting a link to your paper on this list when you've
published it? Thanks in advance!

>My motivation for asking here, was that if you had assertions (which I 
>surely understand that you don't have), more of the failing
>tests would fail at the most specific assertion, making my sorting a
>bit more useless.

This seems like a reasonable assumption, but I wouldn't know how to
falsify it.

>But hey, you just gave me a wonderful argument: In some big and
>important projects, assertions are even not an option!

Well, I guess we will eventually use assertions in Classpath. There's no
law saying we must not use 1.4 language features. The reason why we
haven't been using assertions in the past is the lack of support by free
Java compilers. Which would be easy enough to fix (a quick-and-dirty fix
would be just to ignore the assert statement).

>> It is worth noting that a significant number of subtests fail when
>> you run them on a Sun JDK.  Sometimes the testcase is at fault, and
>> sometimes the fault lies in the JDK.
>>> - How many run green for Classpath?
>> What do you mean by "green"?
>Running without failure.

Some other test harnesses allow to express which tests are known to fail
in what environment (XFAIL). IMHO, that would be quite useful for Mauve,
especially because of the failures in the Sun reference implementation.
Nobody has done this yet, but anyone is welcome to contribute an
implementation. :-)

>[...]
>- Does Mauve sort failing unit tests?
>- If not, how could it be done?

Mauve simply takes a large text file that lists the names of test
classes, loads the respective class (which must implement the
gnu.testlet.Testlet interface) and invokes its "test" method. So the
"sorting" is just the ordering in the text file. If you sorted this list
in any particular way, the output (= failing tests) would be sorted in
the same order.

>- Do you know any (as responsive :-) java open source community with
>some big project and which
>uses both, JDK 1.4 (thus theoretically assertions) and unit tests?

I guess most JUnit users would fall into that category, but I
unfortunately don't know any specific examples.

Best regards,

-- Sascha

Sascha Brawer, brawer@dandelis.ch, http://www.dandelis.ch/people/brawer/