Re: [PATCH] randomize benchtests

[OndÅej BÃlka <neleai at seznam dot cz>]

On Mon, Apr 22, 2013 at 05:44:14PM +0530, Siddhesh Poyarekar wrote:
> On 22 April 2013 17:30, OndÅej BÃlka <neleai@seznam.cz> wrote:
> >    uint64_t total = 0, max = 0, min = 0x7fffffffffffffff;
> >    struct timespec start, end;
> > -
> > +  int seed = 42;
> 
> You get the same sequence all the time.  Use getpid() as seed or even time().
>
And this is precisely as intended. With that randomization you would get
different results each time and we want this to be predictable. 
> > +      clock_gettime (CLOCK_PROCESS_CPUTIME_ID, &start);
> > +      for (k = 0; k < iters; k++)
> > +        {
> > +         i = rand_r (&seed)%NUM_SAMPLES;
> > +         BENCH_FUNC(i);
> > +        }
> > +      clock_gettime (CLOCK_PROCESS_CPUTIME_ID, &end);
> 
> This is wrong.  The interval also has the time taken to call rand_r.
> 
This is not wrong. You are interested only on differences between
implementations and adding same time from rand_r calls does not change
that. 

To get timing of function  can add measurement of only rand_r and subtract it from these.
> > -  double d_total_s = total * 1e-9;
> > -  double d_iters = iters;
> >    double d_total_i = (double)ITER * NUM_SAMPLES;
> > -  printf (FUNCNAME ": ITERS:%g: TOTAL:%gs, MAX:%gns, MIN:%gns, %g iter/s\n",
> > -         d_total_i, d_total_s, max / d_iters, min / d_iters,
> > -         d_total_i / d_total_s);
> > +  printf (FUNCNAME ": AVERAGE:%gns\tMIN:%gns,\tMAX:%gns\n",
> > +               d_total / d_total_i, min / iters, max / iters);
> 
> If you have total iterations and iters/sec, you can compute average
> directly.  You've gotten rid of two pieces of core information to add
> one piece of derived information.  I am not very picky about what the
> output looks like though.
Well expected value is most important statistical property. Yes, you can
take iters/sec and mentally convert them to average time. But I thougth
that purpose of computers is simplify tasks that can be performed
mechanicaly.

Total iterations are useless implementation detail and you should print
only usefull data.

> 
> Also, if you wait for the HP_TIMING patch to be reviewed and checked
> in, you'll get cycles as a unit instead of nanoseconds.
> 
> Siddhesh
> --
> http://siddhesh.in

-- 

Of course it doesn't work. We've performed a software upgrade.