This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH] Support separate benchmark outputs
- From: Siddhesh Poyarekar <siddhesh dot poyarekar at gmail dot com>
- To: Ondřej Bílka <neleai at seznam dot cz>
- Cc: Siddhesh Poyarekar <siddhesh at redhat dot com>, GNU C Library <libc-alpha at sourceware dot org>
- Date: Wed, 17 Apr 2013 13:05:33 +0530
- Subject: Re: [PATCH] Support separate benchmark outputs
- References: <20130416122544 dot GH3063 at spoyarek dot pnq dot redhat dot com> <20130416132838 dot GA29626 at domone dot kolej dot mff dot cuni dot cz> <20130416140355 dot GI3063 at spoyarek dot pnq dot redhat dot com> <20130416155032 dot GA30216 at domone dot kolej dot mff dot cuni dot cz> <CAAHN_R1U_8_OMuzVKJ2q2w8i_XWYrx0mYmvHodBLxmNLiznSkQ at mail dot gmail dot com> <20130416185720 dot GA790 at domone dot kolej dot mff dot cuni dot cz> <CAAHN_R0AVLCRDf=Qp1269U3xxUN9LFh_+32mbjyvOVfqSC-rsw at mail dot gmail dot com> <20130417061507 dot GA5120 at domone dot kolej dot mff dot cuni dot cz>
On 17 April 2013 11:45, OndÅej BÃlka <neleai@seznam.cz> wrote:
> And how you describe conditions of your benchmark. Conditions under
> which we test should be carefully selected to get as much information as
> possible.
We assume ideal conditions, i.e. how the function would behave if it
were relatively uncontested at the processor and cache level. That is
why I believe we take into consideration the best time and not the
average. We test various defined variations (alignment, sizes) to see
how the functions behave and scale, again in these ideal conditions.
>> Our goal is to provide acceptable average case performance,
>
> A code that I commented is following:
>
>> + for (i = 0; i < 32; ++i)
>> + {
>> + HP_TIMING_NOW (start);
>> + CALL (impl, dst, src, len);
>> + HP_TIMING_NOW (stop);
>> + HP_TIMING_BEST (best_time, start, stop);
>> + }
>
> It does not provide average case performance. It computes best case performance.
> This favours implementation with complex control flow that are in 5% of
> cases where conditions are rigth faster but not so in 95% of bulk data.
> Please compute average.
I disagree for reasons I explained above. In any case I'd like to
separate the plain copying over of the tests from any changes that
could be made to these files. So if you have any ideas that you want
to implement here, you could do them once these tests are in.
> How will we choose these specific loads?
> You must deal with conflicts. How when user submits regression of new
> implementation in firefox but all our benchmarks show that new
> implementation is faster. Who is true?
It depends on the call profile of firefox. Once you have a call
profile, you should be able to figure out the reason why there is a
regression in performance despite showing improvements in *all*
microbenchmarks. One possibility is that we have misunderstood the
pattern of allocations that applications usually use, which would
eventually deepen our understanding of the general problem of
copying/comparing/writing/reading data. The other case (more common,
in my experience) would be that there is something wrong with the way
firefox uses the functions.
I have a strong preference for keeping these microbenchmark tests for
reasons I have explained in earlier emails. System profiling is a
good supplement but being able to define tests specifically for the
code to maximize its coverage is still very valuable.
Siddhesh
--
http://siddhesh.in