Re: [PATCH 0/N] test-suite improvement - PASS/FAIL: initial patch

[Siddhesh Poyarekar <siddhesh at redhat dot com>]

On Sun, 16 Sep 2012 11:46:57 -0400 (EDT), Tomas wrote:
> I have reworked the patch, so now it creates .test-result files to
> each test, where is stores result of single test-case in text form
> and than collects it. What do you think?

If you want a predictable way to find out if a test passed or failed,
then you can simply parse the stderr of the `make -k check` result.
There will only be three types of content in it:

1) Warnings, either from compilation or from the test case itself
2) Compiler errors
3) The Error line signifying which test failed.

AFAICT, the only exceptions to the above are:

1) The abi-check result does not *quite* follow the above rule. It
    prints out the error message on stdout directly instead of a .out
    file.
2) The various standards tests don't have a direct PASS/FAIL
   equivalent.

You can use this information to get the kind of summary you want:

* All 'Error' lines translate to FAIL tests. You can parse that line to
  find out which test failed, possibly even printing out the content of
  the relevant .out file.

* For abi-check, you need to make a modification to make it follow the
  same flow as other tests and then use the same logic for it.

* For the standards check, it may make sense to save expected or
  thresholds for the conformance so that a PASS/FAIL can be determined.
  I am not very well versed with this part of the tests, so someone
  else may need to confirm/deny this.

Regards,
Siddhesh