Re: Sending test results from build robot?

[Hans-Peter Nilsson <hp at bitrange dot com>]

On Mon, 19 Dec 2005, Jan-Benedict Glaw wrote:
> Would it be of some help (at least to document that at least some VAX
> targets are still alive) to regluarly submit testsuite results? At a
> maximum, it could bulid one complete run in about 30min, but I guess
> you don't want to have a report ~50 times per day :->

If you trig your script on commits, this will happen a few times
a day.  You don't have to poll the CVS server, you can do this
for emails to the binutils-cvs list.  I do this for my
autotester and it works fine, but I should mention that I have a
delay for 15m until the actual cvs update (actually re-checkout
over existing checked-out tree; never update in src) begins,
which works fine to avoid getting an unfinished set of commits.

(It doesn't auto-report as I still want to inspect the results
before reporting and I wouldn't want an external
semi-catastrophic failure to cause it to mailbomb the list.  I
might be over-cautious.  It's been running fine for a while, but
as I don't have control over the machines where it runs, a
mistaken change in installed packages can cause it to spuriously
fail.)

> Apart from the fact that I monitor changes, how often should I send
> reports?

Most useful would IMHO be to only send reports when there's a
failure.

Dejagnu has the nice property (though limited: you can't tweak
e.g. the number or identity of unexpected errors for exit != 0;
you have to edit the test-cases) that you get a nonzero return
code from "make check" only when there are errors or FAILs
present, so this makes report-on-errors easy to implement.

The binutils testsuite had no known FAILs until H.J.:s recent
commit.

brgds, H-P