Re: Splitting up perf test compilation and data collection

[Doug Evans <dje at google dot com>]

On Sat, Sep 21, 2013 at 8:37 PM, Yao Qi <yao@codesourcery.com> wrote:
> On 09/21/2013 01:02 AM, Doug Evans wrote:
>>>
>>> For example,
>>> >We want to be able to run the perf tests in parallel, but we don't want
>>> >test data polluted because, for example, several copies of gcc were also
>>> >running compiling other tests or other tests were running.
>>
>> [pulled out as a separate thread]
>>
>
> Doug,
> I don't think running perf tests in parallel is a good idea :).  The
> perf data of a test is less useful because of the interference from the
> execution of another test.

On the same machine, no (modulo for development purposes to, for
example, regression test the test harness).  But running them in
parallel on separate machines ... yes.

The other issue that comes into play for us is that it's easier if we
build the binaries on one set of machines and run the perf tests on
others.  If we can't use upstream's test framework so be it.  At this
point I'm just looking to see if there's enough common ground.

>> Another thought is to require a convention that perf .exp files are
>> written like so:
>>
>> if [doing_perf_compilation] {
>>      ...
>> }
>>
>> if [doing_perf_testing] {
>>      ...
>> }
>
>
> AFAICT, this is only useful to the test cases in which compilation takes
> some time, such as gdb.perf/solib.exp.  It is not very common, is it?

Building binaries big enough to collect representative data for the
kinds of programs gdb needs to handle could take awhile.

But that was not the reason I suggested that as a possibility.
It was suggested as another way to split up the compilation vs data
collection steps without having to use separate test harnesses.  The
above handles, I think, your use case and ours.  It just requires a
particular style for writing the .exp file.  [For completeness sake,
another possibility is to allow for having separate files: one for
compilation and one for data collection, not my first choice though.]
At this point I don't have a strong opinion on the implementation, the
higher order bit for this piece of the puzzle is to allow for separate
compilation and data collection without having to have separate
harnesses for each.  If someone has a strong opinion on how this piece
has to be implemented, that still supports separate compilation and
data collection, we would certainly try using it.

> I have one test case that doesn't have compilation step, and let GDB to load
> a pre-compiled binary.  The pre-compiled binary is a huge c++ program with
> debug info, and is used to test GDB speed on handling symbol and debug info.
> It is impossible to compile the huge program in each time of running the
> test case.

We have huge programs too ... :-)

The above suggestion can handle both: If one has a precompiled binary
the compilation step is empty or non-existent.  But, to repeat, this
particular thread is, for me, about finding an agreeable way to
separate the compilation step from the data-collection step, allowing
sites to do arbitrary things in between.