[1.7] Weird error "grep: writing output: Cannot allocate memory"

[Dave Korn <dave dot korn dot cygwin at googlemail dot com>]

    Hi[ppo],

  I've been seeing these odd error messages scattered in amongst my grep
output occasionally of late, and decided to spend some time tracking them
down.  It now seems to me that something very strange is going on with
WriteConsoleW, as called from fhandler_console::write_normal().  This is
possibly the same underlying bug that might explain why even a non-cygwin grep
might show the problems documented in the earlier thread[*], although it's
also possible that there were two separate issues cropping up there, of which
this is one and the other was some kind of regex backtracking/recursion memory
explosion.

  The attached STC reproduces the problem, and apparently:

gcc-3 -mno-cygwin stc.c -o stc -W -Wall -g -O0 -o stc-mingw.exe

gcc-4 stc.c -o stc -W -Wall -g -O0

.. it is cygwin independent, since both executables produce similar results:

$ ./stc
alloc: 0
hConsole 0x1f
size 72d2: rv 0, last error 8 out ffffffff
size 7252: rv 0, last error 8 out ffffffff
size 71d2: rv 0, last error 8 out ffffffff
size 7152: rv 0, last error 8 out ffffffff
size 70d2: rv 0, last error 8 out ffffffff
size 7052: rv 0, last error 8 out ffffffff
size 6fd2: rv 0, last error 8 out ffffffff
size 6f52: rv 0, last error 8 out ffffffff
size 6ed2: rv 0, last error 8 out ffffffff
size 6e52: rv 0, last error 8 out ffffffff
size 6dd2: rv 0, last error 8 out ffffffff
size 6d52: rv 0, last error 8 out ffffffff
size 6cd2: rv 0, last error 8 out ffffffff
size 6c52: rv 0, last error 8 out ffffffff
size 6bd2: rv 0, last error 8 out ffffffff
size 6b52: rv 0, last error 8 out ffffffff
size 6ad2: rv 0, last error 8 out ffffffff
size 6a52: rv 0, last error 8 out ffffffff
size 69d2: rv 0, last error 8 out ffffffff
size 6952: rv 0, last error 8 out ffffffff
size 68d2: rv 0, last error 8 out ffffffff
size 6852: rv 0, last error 8 out ffffffff
 [ snip pages of 'A' ]
size 67d2: rv 1, last error 8 out 67d2

F:\cygwin-1.7\tmp\console>stc-mingw.exe
alloc: 0
hConsole 00000007
size 72d2: rv 0, last error 8 out ffffffff
size 7252: rv 0, last error 8 out ffffffff
size 71d2: rv 0, last error 8 out ffffffff
size 7152: rv 0, last error 8 out ffffffff
size 70d2: rv 0, last error 8 out ffffffff
size 7052: rv 0, last error 8 out ffffffff
size 6fd2: rv 0, last error 8 out ffffffff
size 6f52: rv 0, last error 8 out ffffffff
size 6ed2: rv 0, last error 8 out ffffffff
size 6e52: rv 0, last error 8 out ffffffff
size 6dd2: rv 0, last error 8 out ffffffff
size 6d52: rv 0, last error 8 out ffffffff
size 6cd2: rv 0, last error 8 out ffffffff
size 6c52: rv 0, last error 8 out ffffffff
size 6bd2: rv 0, last error 8 out ffffffff
size 6b52: rv 0, last error 8 out ffffffff
size 6ad2: rv 0, last error 8 out ffffffff
size 6a52: rv 0, last error 8 out ffffffff
size 69d2: rv 0, last error 8 out ffffffff
size 6952: rv 0, last error 8 out ffffffff
size 68d2: rv 0, last error 8 out ffffffff
size 6852: rv 0, last error 8 out ffffffff
 [ likewise omit 'A's ]
size 67d2: rv 1, last error 8 out 67d2

  I've read the msdn page for WriteConsole, and it says you could expect an
out of memory error if the size is > 64k.  I don't understand what is going on
here, but it's causing the problems reported by grep for me, at any rate.

  Can anyone see something wrong with the usage of WriteConsole in this context?

    cheers,
      DaveK
-- 
[*] - http://www.cygwin.com/ml/cygwin/2009-04/msg00371.html

#include <Windows.h>
#include <stdio.h>

#define BADSIZE 0x72d2

static short buffer[BADSIZE + 1];

int main (int argc, const char **argv)
{
  BOOL rv;
  HANDLE hConsole = 0;
  DWORD outsize = ~0;
  DWORD badsize = BADSIZE;

  (void) argc;
  (void) argv;

  short *bufptr = buffer;
  while (bufptr - buffer < BADSIZE)
  {
    *bufptr++ = 0x41;
  }
  *bufptr++ = 0;

  rv = AllocConsole ();
  fprintf (stderr, "alloc: %d\n", rv);
  hConsole = GetStdHandle (STD_OUTPUT_HANDLE);
  fprintf (stderr, "hConsole %p\n", hConsole);
  rv = WriteConsoleW (hConsole, &buffer[0], badsize, &outsize, NULL);
  fprintf (stderr, "size %lx: rv %d, last error %ld out %lx\n", badsize, rv, GetLastError (), outsize);

  while (!rv && badsize >= 0x1000)
  {
    badsize -= 0x80;
    rv = WriteConsoleW (hConsole, &buffer[0], badsize, &outsize, NULL);
    fprintf (stderr, "size %lx: rv %d, last error %ld out %lx\n", badsize, rv, GetLastError (), outsize);
  }

  return 0;
}


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/