[Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage

["tom at atoptech dot com" <sourceware-bugzilla at sourceware dot org>]

We recently compiled and ran our application on Centos 5 which has glibc 2.5. We
found many of our benchmarks and regression runs resulted in 50-100% increase in
memory usage. We eventually pin-pointed the problem to malloc and after
examining the 2.5 malloc source code we found that malloc now adjusts the mmap
threshold dynamically. For our application, the dynamic adjustment does not
work. The end result is more fragmentation and many large malloc requests
results in more and more memory allocated.

Our application is Place-And-Route for backend chip design. It is very common
for an innovation of our application to run for several days using up to 32GB of
memory (depending on the size of the chip, i.e., multi-million gates). During an
 invocation of our application many different algorithms are executed with
different memory usage patterns (many very large memory allocations and frees.) 

We relied on the old malloc behavior to mmap these requests, because this tended
to reduce the fragmentation.

My guess is that this malloc change is probably just hitting people in userland,
because the Linux distributions use older versions of glibc. Again Centos 5, is
using 2.5.

I found some discussion of this change via google. And there was some
questioning as to whether this change was valid. Again, in our application the
memory usage pattern varies greatly and unless you have a good statistical model
of the memory usage pattern, it is hight unlikely to dynamically adjust the mmap
threshold and get it right.

Because we have the ability to turn off the dynamic threshold (mallopt), we have
a work around. But I thought it was best to give feedback on how this changes
was impacting our application.

Tom Geocaris
Atoptech

-- 
           Summary: malloc dynamic mmap threshold causes 50%-100% increase
                    in memory usage
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper at redhat dot com
        ReportedBy: tom at atoptech dot com
                CC: glibc-bugs at sources dot redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=11044

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.