Bug in linux-glibc1 vsnprintf

[Frans van Dorsselaer <dorssel@rulhm1.LeidenUniv.nl>]

Hello,

I think I found a bug in vsnprintf().

bash$ gcc -v
Reading specs from /usr/lib/gcc-lib/i486-unknown-linux-gnulibc1/2.7.2.3/specs
gcc version 2.7.2.3
bash$

using libc.so.5.4.33
This is linux-glibc1.

Problem:
  int vsnprintf(char *str, size_t n, const char *format, va_list ap);

  My 'man' page (16 September 1995) says:

  "RETURN VALUE
     The  return  value is the number of characters stored, not
     including the terminating null.  If this value  equals  n,
     then there was not enough space in str for all the output.
     You should try again with a bigger output string.
   <...>
   CONFORMING TO
     These are GNU extensions."

  ANSI-C C9X draft (21-11-1997) says:

  "7.13.6.11 The vsnprintf function
   <...>
   Returns

   3 The vsnprintf function returns the number of characters that would have
     been written had n been sufficiently large, not counting the
     terminating null character.  Thus, the null-terminated output has been
     completely written if and only if the returned value is less than n."

  Besides the fact that vsnprintf() has to change in the future, because
  these two rules clash, I found that the current vsnprintf() implementation
  complies with neither rule.  In fact, the return values for different
  parameters are inconsistent.

  When n is not large enough, _sometimes_ the return value is -1,
  but _sometimes_ it is the value according to the ANSI-C C9X draft.

  I don't know of _any_ specs that say the return value can be negative.
  I think the vsnprintf() specs (both GNU _and_ ANSI) indicate that the return value
  cannot be negative.



Example:

-----------------------------
#include <stdio.h>
#include <stdarg.h>


int wrapper(char *str, size_t n, const char *format, ...)
{
	va_list ap;
	int r;

	va_start(ap, format);
	r = vsnprintf(str, n, format, ap);
	va_end(ap);

	return r;
}


int main(void)
{
	char s[10];
	int r;
	int i;

	for (i = 1 ; i <= 10 ; ++i) {
		r = wrapper(s, i, "12345");
		printf("vsnprintf(s, %i, ""12345"") returned %d\n", i, r);
	}

	for (i = 1 ; i <= 10 ; ++i) {
		r = wrapper(s, i, "123");
		printf("vsnprintf(s, %i, ""123"") returned %d\n", i, r);
	}

	return 0;
}
-------------------------------

Ouput:

bash$ ./vsnprintf_bug
vsnprintf(s, 1, 12345) returned -1
vsnprintf(s, 2, 12345) returned -1
vsnprintf(s, 3, 12345) returned -1
vsnprintf(s, 4, 12345) returned -1
vsnprintf(s, 5, 12345) returned -1
vsnprintf(s, 6, 12345) returned 5
vsnprintf(s, 7, 12345) returned 5
vsnprintf(s, 8, 12345) returned 5
vsnprintf(s, 9, 12345) returned 5
vsnprintf(s, 10, 12345) returned 5
vsnprintf(s, 1, 123) returned 3          <-- This one is weird ...
vsnprintf(s, 2, 123) returned -1
vsnprintf(s, 3, 123) returned -1
vsnprintf(s, 4, 123) returned 3
vsnprintf(s, 5, 123) returned 3
vsnprintf(s, 6, 123) returned 3
vsnprintf(s, 7, 123) returned 3
vsnprintf(s, 8, 123) returned 3
vsnprintf(s, 9, 123) returned 3
vsnprintf(s, 10, 123) returned 3
bash$

Notes:

  The program consists of a simple wrapper function (actually an
  implementation of snprintf()) in order to make the call to vsnprintf()
  with a va_list.

  The format string contains no %'s, so no actual formatting is done.

  In the case 5 characters make up the format string, the return value is
  always -1 for all n <= 5.  At least it is consistent.

  In the case 3 characters make up the format string, the return value is
  sometimes -1, and sometimes 3 for n <= 3.  This is not
  even consistent.

  Similar problems occur if the format string _does_ contain %'s and extra
  args are passed accordingly.

  I observed wrong return values only when n is not large enough.


Regards,

--
Frans E. van Dorsselaer
<dorssel@MolPhys.LeidenUniv.nl>


-- 
H.J. Lu (hjl@gnu.org)