Re: [2.20] [4/6] Enumerate tests with special rules in tests-special variable

[Brooks Moses <bmoses at google dot com>]

On 01/09/2014 06:13 PM, Joseph S. Myers wrote:
> For the tests following the default makefile rules, $(tests) (and
> $(xtests)) provides such an enumeration.  Others, however, are added
> directly as dependencies of the "tests" and "xtests" makefile
> targets.  This patch changes the makefiles to put them in variables
> tests-special and xtests-special, with appropriate dependencies on the
> tests listed there then being added centrally.

Having such an enumeration strikes me as a generally-useful thing in any 
case, so I am in favor of this.

How would this interact with the idea you mentioned in an earlier patch in 
the series of breaking up tests that have an "examine the output" step into 
multiple tests?  Do we end up having multiple lines in the enumeration for them?

> Some questions arising from this patch:
>
> * Should we define a rule that Makeconfig is always included by
>    subdirectory makefiles immediately after they define subdir, whether
>    or not they actually need an inclusion of Makeconfig, so they do not
>    need to worry about whether conditionals on variables from
>    Makeconfig can be used in a particular part of the file?  (If we do,
>    then Rules would require that Makeconfig has already been included,
>    rather than including it.  And various makefiles wouldn't need dummy
>    "all:" targets because that in Makeconfig would always suffice.)

I have no opinion here.

> * If a test can only be run conditionally, the additions to the tests
>    variable, or to dependencies of the tests target / the tests-special
>    variable (before / after this patch), are conditional in the
>    makefiles, but whether the makefile targets that say how to run such
>    a test are also conditional varies.  This patch doesn't change such
>    conditionals, but should we define a rule for whether such things
>    are conditional (probably saying that the makefile rules aren't,
>    just the additions to the relevant variable)?

And minimal opinion here, but making things consistent seems good, and 
making the rules non-conditional seems like the simplest choice.

> * Should tests that don't generate output files be changed (in a
>    separate patch or patches) to do so?

What would the output files contain?

It's always nice, when running tests by hand, to get a "Success" output 
rather than just an exit and have doubt about whether it actually ran anything.

If there are compile errors, do they go in the output file, or...?

If there are runtime errors like a segfault, will they get captured in the 
output file?  If so, I think that's a strong argument for always having an 
output file.

Also, if there is always an output file with a consistent name, then that 
will make it easier to copy it back from a target when we get to the point 
of implementing such a thing.

So, I'm generally in favor of this idea.

- Brooks