wprintf/vfprintf.c vs. large precision: allocates far too much memory

[Jim Meyering <jim at meyering dot net>]

Looking at code for http://bugzilla.redhat.com/238406, I spotted
an unrelated problem: the potential overflows I mentioned there.
Here's code demonstrating the problem:

Here's a test program:
-------------------------------
#include <stdio.h>
#include <wchar.h>
int
main ()
{
  if (sizeof (size_t) != 4)
    return 1;

  wprintf (L"foo:%1.350000000s:bar\n", "x");
  return 0;
}

Compile and run it under valgrind on a system with 4-byte size_t and
enough virtual memory, and you'll get something like this:

  $ gcc -std=gnu99 -Wall -O wprintf-bug.c && valgrind ./a.out
  ...
  ==25864== Warning: set address range perms: large range 1400000128 (undefined)
  ==25864== Warning: set address range perms: large range 326258336 (noaccess)
  foo:==25864==
  ==25864== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 13 from 1)
  ==25864== malloc/free: in use at exit: 0 bytes in 0 blocks.
  ==25864== malloc/free: 2 allocs, 1 frees, 1,400,000,128 bytes allocated.

Notice that it allocates well over 1GB of memory.
Also, the output is just "foo:",
while it should output "foo:x:bar\n".

--------------------------------------------------

Note that you can increase the precision to induce integer overflow,
too, making vfprinf.c call alloca(0), with len = SIZE_MAX / 4.  I was
surprised to see that resulted in no "obvious" malfunction, like an
array overrun.  Investigating, I found the cause: the subsequent use of
__mbsrtowc (mbsrtowcs_l.c) which I expected to write beyond the end of
the zero-length buffer, has exactly the same buffer-overflowing expression:

        data.__outbufend = data.__outbuf + len * sizeof (wchar_t);

So, if len is too large, this expression overflows in the same way
as those in vfprinf.c, and (seemingly by chance!) the converter
operates only on the smaller than expected amount of memory that
was allocated:

	    if (__libc_use_alloca (len * sizeof (wchar_t)))		      \
	      string = (CHAR_T *) alloca (len * sizeof (wchar_t));	      \
	    else if ((string = (CHAR_T *) malloc (len * sizeof (wchar_t)))    \
		     == NULL)						      \

Here's an untested patch to protect against an inordinately large
precision.  However, if the string itself has length SIZE_MAX / 4
or greater, the expressions still overflow.

diff --git a/stdio-common/vfprintf.c b/stdio-common/vfprintf.c
index 20c07ce..1e2d928 100644
--- a/stdio-common/vfprintf.c
+++ b/stdio-common/vfprintf.c
@@ -1026,7 +1026,9 @@ vfprintf (FILE *s, const CHAR_T *format, va_list ap)
 	    const char *mbs = (const char *) string;			      \
 	    mbstate_t mbstate;						      \
 									      \
-	    len = prec != -1 ? (size_t) prec : strlen (mbs);		      \
+	    len = strlen (mbs);						      \
+	    if (prec != -1)						      \
+	      len = (size_t) prec;					      \
 									      \
 	    /* Allocate dynamically an array which definitely is long	      \
 	       enough for the wide character version.  */		      \