This is the mail archive of the
xsl-list@mulberrytech.com
mailing list .
Re: Latest XSLTMark benchmark
- To: xsl-list at lists dot mulberrytech dot com
- Subject: Re: [xsl] Latest XSLTMark benchmark
- From: David_N_Bertoni at lotus dot com
- Date: Fri, 30 Mar 2001 12:25:37 -0500
- Reply-To: xsl-list at lists dot mulberrytech dot com
Mike Kay wrote:
> A new set of results for the XSLTMark benchmark has been published at
> http://www.datapower.com/XSLTMark/res_2001_03_28.html#ChartOverall
>
> The benchmarking methodology is to compile the stylesheet, and then run
it a
> large number of times over the same source document, averaging the
results.
>
> I have taken a look at some of the drivers for different XSLT processors.
>
> As far as I can see the drivers for Oracle, MSXML, and Sun XSLTC parse
and
> load the source document once, and then transform it many times from the
> same in-memory tree, while the drivers for xt, Saxon, and Xalan parse the
> source document and build the tree each time round the loop.
>
> Since parsing and building the tree account for a significant proportion
of
> the transformation time, this would significantly skew the results.
>
> Are my suspicions correct?
That was my conclusion as well. I also examined their Xalan-C++ driver,
which is poorly written, leaks memory, and uses a suboptimal source tree
for the transformation. It took 15 minutes for me to re-write their
driver, and the results are significantly improved. I only wish someone
had bothered to post a request to the Xalan list for a review of their
driver, or at least some notice that they were doing this.
> (Even if this is corrected, the methodology is flawed, because a
processor
> could legitimately cache the results of parsing the document in memory,
> unknown to the driver. Ideally the source file should be trivially
modified
> between each iteration to prevent this).
Companies which post benchmarks ought to be more responsible.
Dave Bertoni
IBM Corporation (Xalan-C++ committer)
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list