Re: Kill the SCM_NIMPs

[Mikael Djurfeldt <mdj at mdj dot nada dot kth dot se>]

Maciej Stachowiak <mstachow@alum.mit.edu> writes:

> Yes, given Mikael's comments this would be a good time of applying our
> policy of benchmarking before and after potentially performance-affecting
> changes.

Well, my last message was meant to say that in this case there is no
reason to expect performance penalties, but, OK, I won't protest... :)


This is a suggestion for a good way of benchmarking a large change:

1. Check out two working directories: guile-core.old and
   guile-core.new

2. Implement your changes in guile-core.new.

3. Run `cvs update' in both directories.

4. Configure both directories with --disable-shared

5. Compile and install a static guile of each kind: guile.old, guile.new
   and make sure that both work well.

6. Write a suitable benchmark (or reuse an old one) in the testsuite
   from CVS module guile-modules.

   * Beware of the difference between normal and debugging evaluator,
     and between user, system, and GC time.  (A conservative collector
     can add big random noise to the GC time depending on the contents
     on the stack.  A problem is that one sometimes want to include GC
     time.  Sometimes GC times actually behave predictably.)

   * It is a good idea to use the median method to filter out
     reasonable values.  This has the good property that the result
     actually is a real measured value, not contaminated by extremes.

   * It is extremely important that the two binaries exist at the same
     time in the directory tree and that they are run under similar
     conditions on the operating system.  In order to get good results
     (if they are repeatable with good precision, this is a good sign)
     it may be necessary to reboot your system, and maybe not start
     certain processes.

   * Beware of your own way of selecting results.  If you have bad
     repeatability and run a few times, you can show anything by
     selecting the appropriate results!  Ideally, you should not
     select at all.  Sometimes, one may suspect that a results isn't
     representative, though.  In such cases, run several times to see
     what is representative!

7. If the decision is taken that the benefits of the change wins over
   the penalties, then commit the change!  BUT, for large changes it is
   a good idea to create pre- and post-tags so that the changes is
   easy to isolate afterwards and revert, if needed:

     cd .../guile-core.old
     cvs update
     cvs tag mdj_pre_hubhub
     cd .../guile-core.new
     <commit things using e.g. pcl-cvs>
     cvs tag mdj_post_hubhub

(And, probably, I've just repeated what Jim has already written in his
 benchmark package (didn't have a look before posting this...).)

/mdj