Re: Mauve Code Coverage

[Stephen Crawley <crawley at dstc dot edu dot au>]

Yan,

I'm being the "Devil's Advocate" here ... bear with me.  Basically,
I think the idea is a good one, if we can pull it off.

> > I think this is a good idea in principle.  But a better idea would be
> > for you to run the coverage tool on Mauve/Classpath/some VM and post the
> > results.  If you could set something up to do this automatically, that
> > would be even better!
> 
> Not sure, it is much more rewarding for developpers (here, test writers) to 
> run the code coverage themselves than letting a quality engineer (here, 
> Koalog) do it for them.

I take your point that it >>could<< be more convenient for testcase
developers to run the coverage tests themselves.  But that assumes that
the "barrier to entry" for using the tool isn't too high.

1)  I assume that Koalog will want to limit the their offer of free licenses
to "testcase developers".  Have you considered how many you will provide,
and your criteria for deciding who to give them to?  Will there be much
paperwork?

2)  How easy will it be to use the coverage tool?  In particular, is it
much work to install the tool?  Is it much work to set up the coverage
tool for a particular project?  [Could the setup files be CVS'd?]

3)  Does the tool cope with calls (and callbacks) from Java and native
code?  A lot of this happens in a typical Classpath-based VM.

> > Another point is that coverage testing will only really help if people
> > volunteer to write good test cases to fill in the gaps.
> 
> That's the point. See above.

I think you missed >>my<< point.  That is that we are desperately short 
of people writing testcases at the moment.  Take a look at the rate of
Mauve checkins as per the Changelog.  A coverage tool won't change this.
(This doesn't mean we shouldn't use a coverage tool ... but lets not
get our expectations too high!)    

> > Finally, coverage testing won't tell us about test cases that are
> > incorrect; e.g. ones that test for implementation specific behavior, or
> > ones that don't work against various Sun JDK's.  In other words, Mauve's
> > limitations are not just restricted to coverage.
> 
> In a usual projet, you check that all the test pass under several environments 
> (OS and JDK). In your case, you would also check that the coverage results 
> are the same under these environments.
> Computing the code coverage is indeed a simple but powerful way to test the 
> tests, which is exactly what you want to do here.

Am I missing something here?  Can a Classpath cleanroom developer really do a
coverage analysis of running Mauve testcases against a Sun JDK?  Doesn't the
developer need to use Sun source code to do this?

-- Steve