stack overflow bug in ofstream::operator<<

Corinna Vinschen
Thu Jun 30 11:15:00 GMT 2005

On Jun 28 13:59, Christopher Faylor wrote:
> On Tue, Jun 28, 2005 at 06:41:09PM +0100, Dave Korn wrote:
> >Well, I was proposing testing how much space was available on the
> >stack, so that we never alloca past the bottom of it, and hence don't
> >*need* any kind of signal handler.
> If we are going to check how much stack space is available, it might be
> nice to make a __alloca_which_returns_NULL which returns NULL when there
> isn't enough space available.  That could be used for other things.

I'm not sure if we can substitute alloca by something which is able
to return NULL.  The gcc implementation of alloca is designed so that
it doesn't return but jumps back to the caller, and the caller itself
gets the address for the new stack area by calling

  lea 12(%esp), %eax
  movl %eax, -> target pointer variable

So alloca is designed in a way which disallows to return NULL at all.

However, I think I found a way to test if enough space is available on the
stack.  ld stores the size of the maximum available stack in an absolut
symbol called __size_of_stack_reserve__.  This is the default stack size
for the main thread and threads started with no explicit stacksize.  So,
what I did is to create the following inline function:

inline bool
check_alloca (size_t size)
  extern unsigned long _size_of_stack_reserve__;
  register char *_curstack __asm__ ("%esp");
  register size_t stacksize = pthread_self ()->attr.stacksize
                              ?: (size_t) &_size_of_stack_reserve__;
  return (size_t) ((_curstack - (_tlsbase - stacksize)) / 65536) * 65536 > size;

and then something similar to what you proposed in writev:

fhandler_base::writev ([...])

  char *buf;
  bool allocaed;

  if ((allocaed = check_alloca (tot)))
    buf = (char *) alloca (tot);
    buf = (char *) malloc (tot);


    ssize_t ret = write (buf, tot);
  if (!allocaed)
    free (buf);
  return ret;

Does that sound feasable?


Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader
Red Hat, Inc.

More information about the Cygwin-developers mailing list