[Bug libc/214] sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

["rsa at us dot ibm dot com" <sourceware-bugzilla at sourceware dot org>]

------- Additional Comments From rsa at us dot ibm dot com  2006-03-06 21:37 -------
I think I misunderstood the problem a it with my previous entry.

On Powerpc32 we have the following assembler code in brk.S

...

        DO_CALL(SYS_ify(brk)) /* new brk in r3 */
        lwz     r6,8(r1) /* saved old brk */

...

        cmplw   r6,r3 /* old brk vs. new brk */
        addi    r1,r1,16
        mtlr    r0
        li      r3,0 /* zero is the return value for less-than-or-equal-to */
        blelr+
        li      r3,ENOMEM /* else ENOMEM */

Where r6 holds 'oldbrk' and r3 holds the 'curbrk'.  The blerl checks to see if
r6 is less than or equal to r3.  If so it returns 0, meaning '0' is returned in
the normal success case as well as the case where the brk is not moved.

Changing the code to:

        if (__brk (oldbrk + increment) != 0)
              return (void *) -1;
        return oldbrk;

Would certainly be wrong on powerpc32 and powerpc64.

I'm beginning to think that the problem is related to the soft-limit not being
set properly.

GLIBC interpets sbrk() as __brk(curbrk + increment) and sends a brk syscall to
the kernel.  The Linux kernel, in mm/mmap.c:sys_brk subtracts the start of the
process memory area from the proposed brk() is supposed to return the oldbrk if
the rlimit would be exceeded.  It should not increment curbrk in this case.

We can see from my example that calling sbrk() after exceeding the soft-limit,
that we set with setrlimit(), doesn't prevent the kernel from issuing the brk().

Only when the brk() exceeds a page boundary does the kernel complain.

I will investigate setrlimit() in GLIBC and in the kernel.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.