[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

["dlstevens at us dot ibm dot com" <sourceware-bugzilla at sourceware dot org>]

------- Additional Comments From dlstevens at us dot ibm dot com  2006-03-15 21:21 -------
Ryan, sounds good. Here's the full background on this, as far as I can
remember, in case the context helps:

I started out writing some LTP tests for IPv6 functions and I wanted to test
a memory allocation failure in a function that allocates the memory for its
returned structures. The idea was to set the soft limit in the test to whatever
was already allocated, and call the function under test, check its return for
the proper allocation failure error, raise the soft limit and move on.

When I couldn't provoke the error, I figured I was setting the soft limit
incorrectly, so I tried setting the soft limit and then allocating page-size
chunks (with malloc) to try to figure out why my soft-limit calculation was
off, and by how much. In those runs, I didn't get a NULL return at all, which
was unexpected, and the program would die with a segmentation violation at
least some of the time. (that test is long gone, or I'd append it-- I thought
this was easy to reproduce :-), so didn't in the original).

I looked at the malloc() code and discovered it appeared to expect a "-1"
return on _brk() failure, but strace() of the running program showed the Linux
kernel was returning the old brk value in that case (and looking at the Linux
kernel code verified that). Only later did I discover that glibc is remapping
the system call value, though I could tell if it was doing that correctly in
all cases-- so that may be a red herring, maybe not. I appeared to be getting
the same sbrk() value when it failed, but the memory after it was not usable
(referencing it caused a segmentation violation). So, I suspect that sbrk/brk
isn't returning the right thing for malloc() to detect the allocation failure,
which resulted in malloc() trying to use the invalid chunk of memory at the
sbrk() return value and SEGV-ing within glibc at that point.

The whole thing is complicated in malloc() by it not doing sbrk()/brk() quite
as often as I expected, which is what the business about the mmap comment
refers to. I looked at the code at the time, but don't remember enough about it
to be helpful, though you may already know more about it than I ever did. :-)

Bottom line for the bug report, as I said before, was that I was never able to
get a (graceful) failed memory allocation by setting the soft limit, and at
least some of those attempts resulted in SEGV's within glibc, which I wouldn't
expect to happen if everything is working correctly. :-)

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.