Extreme slowdown due to malloc?
Mark Geisert
mark@maxrnd.com
Tue Dec 22 04:37:14 GMT 2020
Hi Achim,
Achim Gratz wrote:
> I've been experimenting a bit with ZStandard dictionaries. The
> dictionary builder is probably not the most optimized piece of software
Is this what leads you to suspect malloc? Really heavy use of malloc?
> and if you feed it large amounts of data it needs quite a lot of
> cycles. So I thought I run some of this on Cygwin since that machine is
> faster and has more threads than my Linux box. Unfortunately that plan
> shattered due to extreme slowness of the first (single-threaded) part of
> the dictionary builder that sets up the partial suffix array.
>
> |------+---------------+---------------|
> | | E3-1225v3 | E3-1276v3 |
> | | 4C/4T | 4C/8T |
> | | 3.2/3.6GHz | 3.6/4.0GHz |
> |------+---------------+---------------|
> | 100 | 00:14 / 55s | 00:23 / 126s |
> | 200 | 00:39 / 145s | 01:10 / 241s |
> | 400 | 01:12 / 266s | 01:25 / 322s |
> | 800 | 02:06 / 466s | 11:12 / 1245s |
> | 1600 | 03:57 / 872s | > 2hr |
> | 3200 | 08:03 / 1756s | n/a |
> | 6400 | 16:17 / 3581s | n/a |
> |------+---------------+---------------|
>
> The obvious difference is that I/O takes a lot longer on Cygwin (roughly
> a minute for reading all the data) and that I have an insane amount of
> page faults on Windows (as reported by time) vs. none on Linux.
How much RAM does the Windows machine have? Do you have a paging file? Is it
fixed size or "let Windows manage"? How big is it?
> While doing that I also noticed that top shows the program taking 100%
> CPU in the multithreaded portion of the program, while it should show
> close to 800% at that time. I'm not sure if that information just isn't
> available on Windows or if procps-ng needs to look someplace else for
> that to be shown as expected.
No offense, but are you sure it's actually running multi-threaded on Windows?
I have a Cygwin malloc speedup patch that *might* help the m-t part. I'll prepare
and submit that to cygwin-patches shortly.
Cheers,
..mark
More information about the Cygwin-apps
mailing list