Extreme slowdown due to malloc?

[Achim Gratz]

ASSI writes:
>> I have a Cygwin malloc speedup patch that *might* help the m-t part.
>> I'll prepare and submit that to cygwin-patches shortly.
>
> Well, if you want to test it with the new ZStandard, give it a spin…
> I'll check how far I can strip that test down so you can use the Cygwin
> source tree for testing.

OK, it's actually pretty simple, do this inside a checkout of
newlib-cygwin:

$ find newlib winsup texinfo -type f > flist
$ zstd --train-cover --ultra -22 -T0 -vv --filelist=flist -o dict-cover

On Linux, it reads in all the files in about two seconds, while it takes
quite a while longer on Cygwin.  But the real bummer is that
constructing the partial suffix arrays (which is single-threaded) will
seemingly take forever, while it's done much faster on Linux.  You can
pare down the number of files like that:

$ shuf -n 320 flist > slist

and then use that shorter file instead.  I get this:

*** Linux E3-1225v3 4C/4T 3.2/3.6GHz
|------+----------+-------+----------+---------+--------+--------+------------|
|    n |     user |   sys |    total |    wall |   util | serial | pagefaults |
|------+----------+-------+----------+---------+--------+--------+------------|
|  100 |  116.092 | 0.187 |  116.279 |   30.82 | 377.3% |   2.0% |          0 |
|  200 |  145.481 | 0.135 |  145.616 |   38.65 | 376.8% |   2.1% |          0 |
|  400 |  288.341 | 0.414 |  288.755 |   77.84 | 371.0% |   2.6% |          0 |
|  800 |  517.288 | 0.623 |  517.911 |  138.93 | 372.8% |   2.4% |          0 |
| 1600 | 1229.348 | 1.752 | 1231.100 |  333.37 | 369.3% |   2.8% |          0 |
| 3200 | 2508.250 | 3.632 | 2511.882 |  678.96 | 370.0% |   2.7% |          0 |
| 6400 | 4380.693 | 5.352 | 4386.045 | 1176.43 | 372.8% |   2.4% |          0 |
|------+----------+-------+----------+---------+--------+--------+------------|

*** Cygwin E3-1276v3 4C/8T 3.6/4.0GHz 
|------+----------+--------+----------+---------+--------+--------+------------|
|    n |     user |    sys |    total |    wall |   util | serial | pagefaults |
|------+----------+--------+----------+---------+--------+--------+------------|
|  100 |  141.906 |  0.796 |  142.702 |   20.53 | 695.1% |   2.2% |     327860 |
|  200 |  198.140 |  1.328 |  199.468 |   29.39 | 678.7% |   2.6% |     452870 |
|  400 |  425.749 |  2.328 |  428.077 |   66.03 | 648.3% |   3.3% |     752357 |
|  800 |  822.250 |  3.499 |  825.749 |  150.42 | 549.0% |   6.5% |    1277198 |
| 1600 | 1773.578 |  8.483 | 1782.061 |  383.42 | 464.8% |  10.3% |    3011298 |
| 3200 | 4322.281 | 15.890 | 4338.171 | 1292.92 | 335.5% |  19.8% |    5746903 |
| 6400 | 8499.750 | 29.437 | 8529.187 | 3275.66 | 260.4% |  29.6% |   10543919 |
|------+----------+--------+----------+---------+--------+--------+------------|

So even with smaller number of files (where the serial portion of the
code is not dominating yet) you see that the faster machine expends more
cycles already.  Looking at the differences there is a strong indication
for those pagefaults to constitute the main portion of that extra time.
The last column is the time per pagefault in µs assuming that the extra
time was all spent there.  This is obviously not quite correct, as that
number should roughly be constant if that assumption holds, but it's
close enough to uphold the original hypothesis.

*** Linux vs. Cygwin
|------+--------------+--------------+----------------+--------------+------------+------|
|    n | Linux  total | Linux scaled | Cygwin   total | Cygwin-Linux | pagefaults | t/pf |
|------+--------------+--------------+----------------+--------------+------------+------|
|  100 |      116.279 |      104.651 |        142.702 |       38.051 |     327860 | 116. |
|  200 |      145.616 |      131.054 |        199.468 |       68.414 |     452870 | 151. |
|  400 |      288.755 |      259.880 |        428.077 |      168.197 |     752357 | 224. |
|  800 |      517.911 |      466.120 |        825.749 |      359.629 |    1277198 | 282. |
| 1600 |     1231.100 |     1107.990 |       1782.061 |      674.071 |    3011298 | 224. |
| 3200 |     2511.882 |     2260.694 |       4338.171 |     2077.477 |    5746903 | 361. |
| 6400 |     4386.045 |     3947.441 |       8529.187 |     4581.746 |   10543919 | 435. |
|------+--------------+--------------+----------------+--------------+------------+------|



Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada