This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [2.20] [2/6] Generate .test-result files for tests with special rules
- From: "Carlos O'Donell" <carlos at redhat dot com>
- To: "Joseph S. Myers" <joseph at codesourcery dot com>, libc-alpha at sourceware dot org
- Date: Thu, 13 Feb 2014 23:43:08 -0500
- Subject: Re: [2.20] [2/6] Generate .test-result files for tests with special rules
- Authentication-results: sourceware.org; auth=none
- References: <Pine dot LNX dot 4 dot 64 dot 1401100208000 dot 9412 at digraph dot polyomino dot org dot uk> <Pine dot LNX dot 4 dot 64 dot 1401100211060 dot 9412 at digraph dot polyomino dot org dot uk>
On 01/09/2014 09:11 PM, Joseph S. Myers wrote:
> This patch extends the generation of PASS and FAIL status in
> .test-result files for individual tests to cover tests with their own
> custom makefile rules.
>
> In general it's just adding $(evaluate-test) calls to all such rules.
> Where a rule had multiple commands, they need linking in the form
>
> command1 && command2; $(evaluate-test)
>
> so that $(evaluate-test) still gets run even if the first command
> fails and so the second isn't run at all.
>
> Note that the tests the makefiles expect to fail (posix/annexc and
> conformtest) currently get FAIL listed in the .test-result file,
> rather than XFAIL; patch 3 will introduce a better XFAIL mechanism.
>
> There are a few stylistic questions here:
>
> * Should $(evaluate-test) go on the same line as the previous part of
> the command if it fits, or always on a separate line (or on a
> separate line unless the whole thing fits on a single line)? (Patch
> 1 uses a single line; this patch generally puts it on the same line
> as the previous part of the command if it fits.)
Always on a separate line, though separate line unless the whole thing
fits is also OK.
> * Where && is used between commands, should it go at the start of a
> line (normal GNU style, and used in this patch) or at the end of a
> line (used in various places in existing makefiles), and if at the
> start, should the continuation line be indented?
GNU style.
> * Should all tests that generate a .out file and then further examine
> it (e.g. comparing with an expected file) have their makefile rules
> split (in a separate patch) into part that generates the file and
> part that examines its contents (the two parts each having their own
> test results)? This would reduce the need for use of && in test
> commands.
I think effectively splitting them into two tests is best.
> * Should all the tests generating output files not named
> <something>.out be changed (in a separate patch) to use the .out
> naming convention?
Yes.
> Tested x86_64.
>
> 2014-01-10 Joseph Myers <joseph@codesourcery.com>
>
> * Makefile ($(objpfx)c++-types-check.out): Use $(evaluate-test).
> ($(objpfx)check-local-headers.out): Likewise.
> ($(objpfx)begin-end-check.out): Likewise.
> * Makerules (check-abi-%): Likewise.
> * catgets/Makefile ($(objpfx)test1.cat): Likewise.
> ($(objpfx)test2.cat): Likewise.
> ($(objpfx)de/libc.cat): Likewise.
> ($(objpfx)test-gencat.out): Likewise.
> * conform/Makefile ($(objpfx)run-conformtest.out): Likewise.
> * elf/Makefile ($(objpfx)order.out): Likewise.
> ($(objpfx)noload-mem): Likewise.
> ($(objpfx)tst-pathopt.out): Likewise.
> ($(objpfx)tst-rtld-load-self.out): Likewise.
> ($(objpfx)tst-array1.out): Likewise.
> ($(objpfx)tst-array1-static.out): Likewise.
> ($(objpfx)tst-array2.out): Likewise.
> ($(objpfx)tst-array3.out): Likewise.
> ($(objpfx)tst-array4.out): Likewise.
> ($(objpfx)tst-array5.out): Likewise.
> ($(objpfx)tst-array5-static.out): Likewise.
> ($(objpfx)check-textrel.out:): Likewise.
> ($(objpfx)check-execstack.out): Likewise.
> ($(objpfx)check-localplt.out): Likewise.
> ($(objpfx)order2.out): Likewise.
> ($(objpfx)tst-leaks1-mem): Likewise.
> ($(objpfx)tst-leaks1-static-mem): Likewise.
> ($(objpfx)tst-initorder.out): Likewise.
> ($(objpfx)tst-initorder2.out): Likewise.
> ($(objpfx)tst-unused-dep.out): Likewise.
> * grp/Makefile ($(objpfx)tst_fgetgrent.out): Likewise.
> * iconv/Makefile (test-iconvconfig): Likewise.
> * iconvdata/Makefile ($(objpfx)mtrace-tst-loading): Likewise.
> ($(objpfx)iconv-test.out): Likewise.
> ($(objpfx)tst-tables.out): Likewise.
> * intl/Makefile ($(objpfx)mtrace-tst-gettext): Likewise.
> ($(objpfx)tst-gettext.out): Likewise.
> ($(objpfx)tst-translit.out): Likewise.
> ($(objpfx)tst-gettext2.out): Likewise.
> ($(objpfx)tst-gettext4.out): Likewise.
> ($(objpfx)tst-gettext6.out): Likewise.
> * io/Makefile ($(objpfx)ftwtest.out): Likewise.
> * libio/Makefile ($(objpfx)test-freopen.out): Likewise.
> ($(objpfx)tst-fopenloc.check): Likewise.
> * malloc/Makefile ($(objpfx)tst-mtrace.out): Likewise.
> * misc/Makefile ($(objpfx)tst-error1-mem): Likewise.
> * posix/Makefile ($(objpfx)globtest.out): Likewise.
> ($(objpfx)wordexp-tst.out): Likewise.
> ($(objpfx)annexc.out): Likewise.
> ($(objpfx)tst-fnmatch-mem): Likewise.
> ($(objpfx)bug-regex2-mem): Likewise.
> ($(objpfx)bug-regex14-mem): Likewise.
> ($(objpfx)bug-regex21-mem): Likewise.
> ($(objpfx)bug-regex31-mem): Likewise.
> ($(objpfx)tst-vfork3-mem): Likewise.
> ($(objpfx)tst-rxspencer-mem): Likewise.
> ($(objpfx)tst-pcre-mem): Likewise.
> ($(objpfx)tst-boost-mem): Likewise.
> ($(objpfx)tst-getconf.out): Likewise.
> ($(objpfx)bug-ga2-mem): Likewise.
> ($(objpfx)bug-glob2-mem): Likewise.
> * resolv/Makefile ($(objpfx)mtrace-tst-leaks): Likewise.
> ($(objpfx)mtrace-tst-leaks2): Likewise.
> * stdio-common/Makefile ($(objpfx)tst-unbputc.out): Likewise.
> ($(objpfx)tst-printf.out): Likewise.
> ($(objpfx)tst-setvbuf1.out): Likewise.
> * stdlib/Makefile ($(objpfx)isomac.out): Likewise.
> ($(objpfx)tst-fmtmsg.out): Likewise.
> * string/Makefile ($(objpfx)tst-svc.out): Likewise.
> * sysdeps/x86/Makefile ($(objpfx)tst-xmmymm.out): Likewise.
>
> localedata:
> 2014-01-10 Joseph Myers <joseph@codesourcery.com>
>
> * Makefile ($(objpfx)sort-test.out): Use $(evaluate-test).
> ($(objpfx)tst-fmon.out): Likewise.
> ($(objpfx)tst-numeric.out): Likewise.
> ($(objpfx)tst-locale.out): Likewise.
> ($(objpfx)tst-rpmatch.out): Likewise.
> ($(objpfx)tst-trans.out): Likewise.
> ($(objpfx)tst-mbswcs.out): Likewise.
> ($(objpfx)tst-ctype.out): Likewise.
> ($(objpfx)tst-wctype.out): Likewise.
> ($(objpfx)tst-langinfo.out): Likewise.
> ($(objpfx)mtrace-tst-leaks): Likewise.
> nptl:
> 2014-01-10 Joseph Myers <joseph@codesourcery.com>
>
> * nptl/Makefile ($(objpfx)tst-stack3-mem): Use $(evaluate-test).
> ($(objpfx)tst-tls6.out): Likewise.
> ($(objpfx)tst-cleanup0.out): Likewise.
> ($(objpfx)tst-cancel-wrappers.out): Likewise.
> ($(objpfx)tst-oddstacklimit.out): Likewise.
>
> nptl_db:
> 2014-01-10 Joseph Myers <joseph@codesourcery.com>
>
> * nptl_db/Makefile ($(objpfx)db-symbols.out): Use
> $(evaluate-test).
This looks good to me and is pretty mechanical.
Cheers,
Carlos.