Extreme slowdown due to malloc?

Mark Geisert mark@maxrnd.com
Tue Dec 22 04:37:14 GMT 2020


Hi Achim,

Achim Gratz wrote:
> I've been experimenting a bit with ZStandard dictionaries.  The
> dictionary builder is probably not the most optimized piece of software

Is this what leads you to suspect malloc?  Really heavy use of malloc?

> and if you feed it large amounts of data it needs quite a lot of
> cycles.  So I thought I run some of this on Cygwin since that machine is
> faster and has more threads than my Linux box.  Unfortunately that plan
> shattered due to extreme slowness of the first (single-threaded) part of
> the dictionary builder that sets up the partial suffix array.
> 
> |------+---------------+---------------|
> |      | E3-1225v3     | E3-1276v3     |
> |      | 4C/4T         | 4C/8T         |
> |      | 3.2/3.6GHz    | 3.6/4.0GHz    |
> |------+---------------+---------------|
> |  100 | 00:14 /   55s | 00:23 /  126s |
> |  200 | 00:39 /  145s | 01:10 /  241s |
> |  400 | 01:12 /  266s | 01:25 /  322s |
> |  800 | 02:06 /  466s | 11:12 / 1245s |
> | 1600 | 03:57 /  872s | > 2hr         |
> | 3200 | 08:03 / 1756s | n/a           |
> | 6400 | 16:17 / 3581s | n/a           |
> |------+---------------+---------------|
> 
> The obvious difference is that I/O takes a lot longer on Cygwin (roughly
> a minute for reading all the data) and that I have an insane amount of
> page faults on Windows (as reported by time) vs. none on Linux.

How much RAM does the Windows machine have?  Do you have a paging file?  Is it 
fixed size or "let Windows manage"?  How big is it?

> While doing that I also noticed that top shows the program taking 100%
> CPU in the multithreaded portion of the program, while it should show
> close to 800% at that time.  I'm not sure if that information just isn't
> available on Windows or if procps-ng needs to look someplace else for
> that to be shown as expected.

No offense, but are you sure it's actually running multi-threaded on Windows?

I have a Cygwin malloc speedup patch that *might* help the m-t part.  I'll prepare 
and submit that to cygwin-patches shortly.
Cheers,

..mark


More information about the Cygwin-apps mailing list