[libiberty and gdb] floatformat_is_valid

[Daniel Jacobowitz <drow at mvista dot com>]

GDB's "struct value" has a union in it.  This union, named "aligner", has a
long double member as well as several others; its goal is to be a cheap way
to ensure that we can load all sorts of values straight from the value
structure, without alignment problems.  Unfortunately, this means that when
debugging GDB we tend to feed random bytes to printf ("%.35Lg").  That's not
kosher, and on i386 I've found a whole bunch of numbers which can crash it -
the gist is that GMP shifts normalized numbers over until it hits a one bit,
and if the explicit integer bit isn't set it walks right out of the number.

This could show up in other places too.  Anywhere you print floating-point
data from the inferior, really.

So here's a patch which checks for the problem.  I added
floatformat_is_valid to libiberty/floatformat.c.  Is this OK?
[Do I need approval?  Floatformat.c is in libiberty, but marked as part of
GDB.]

Oh, this patch misses some copyright year updates.  I'll get them before
checking in.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

2003-09-15  Daniel Jacobowitz  <drow@mvista.com>

	* floatformat.h (floatformat_is_valid): Add prototype.

2003-09-15  Daniel Jacobowitz  <drow@mvista.com>

	* floatformat.c (floatformat_is_valid): New function.
	(get_field, put_field): Correct comments.

2003-09-15  Daniel Jacobowitz  <drow@mvista.com>

	* values.c (unpack_double): Call floatformat_is_valid.

Index: gdb/values.c
===================================================================
RCS file: /cvs/src/src/gdb/values.c,v
retrieving revision 1.54
diff -u -p -r1.54 values.c
--- gdb/values.c	14 Sep 2003 16:32:14 -0000	1.54
+++ gdb/values.c	15 Sep 2003 14:30:58 -0000
@@ -739,7 +739,14 @@ unpack_double (struct type *type, const 
 	 also not defined either.  Oops!
 
          Hopefully someone will add both the missing floatformat
-         definitions and floatformat_is_invalid() function.  */
+         definitions and the new cases for floatformat_is_valid ().  */
+
+      if (!floatformat_is_valid (floatformat_from_type (type), valaddr))
+	{
+	  *invp = 1;
+	  return 0.0;
+	}
+
       return extract_typed_floating (valaddr, type);
     }
   else if (nosign)
Index: libiberty/floatformat.c
===================================================================
RCS file: /cvs/src/src/libiberty/floatformat.c,v
retrieving revision 1.7
diff -u -p -r1.7 floatformat.c
--- libiberty/floatformat.c	15 Apr 2003 21:29:34 -0000	1.7
+++ libiberty/floatformat.c	15 Sep 2003 14:30:58 -0000
@@ -149,7 +149,7 @@ static unsigned long get_field PARAMS ((
 					unsigned int,
 					unsigned int));
 
-/* Extract a field which starts at START and is LEN bytes long.  DATA and
+/* Extract a field which starts at START and is LEN bits long.  DATA and
    TOTAL_LEN are the thing we are extracting it from, in byteorder ORDER.  */
 static unsigned long
 get_field (data, order, total_len, start, len)
@@ -273,7 +273,7 @@ static void put_field PARAMS ((unsigned 
 			       unsigned int,
 			       unsigned long));
 
-/* Set a field which starts at START and is LEN bytes long.  DATA and
+/* Set a field which starts at START and is LEN bits long.  DATA and
    TOTAL_LEN are the thing we are extracting it from, in byteorder ORDER.  */
 static void
 put_field (data, order, total_len, start, len, stuff_to_put)
@@ -402,6 +402,39 @@ floatformat_from_double (fmt, from, to)
       mant_off += mant_bits;
       mant_bits_left -= mant_bits;
     }
+}
+
+/* Return non-zero iff the data at FROM is a valid number in format FMT.  */
+
+int
+floatformat_is_valid (fmt, from)
+     const struct floatformat *fmt;
+     char *from;
+{
+  if (fmt == &floatformat_i387_ext)
+    {
+      /* In the i387 double-extended format, if the exponent is all
+	 ones, then the integer bit must be set.  If the exponent
+	 is neither 0 nor ~0, the intbit must also be set.  Only
+	 if the exponent is zero can it be zero, and then it must
+	 be zero.  */
+      unsigned int exponent, int_bit;
+      unsigned char *ufrom = (unsigned char *) from;
+
+      exponent = get_field (ufrom, fmt->byteorder, fmt->totalsize,
+			    fmt->exp_start, fmt->exp_len);
+      int_bit = get_field (ufrom, fmt->byteorder, fmt->totalsize,
+			   fmt->man_start, 1);
+
+      if ((exponent == 0) != (int_bit == 0))
+	return 0;
+      else
+	return 1;
+    }
+
+  /* Other formats with invalid representations should be added
+     here.  */
+  return 1;
 }
 
 
Index: include/floatformat.h
===================================================================
RCS file: /cvs/src/src/include/floatformat.h,v
retrieving revision 1.5
diff -u -p -r1.5 floatformat.h
--- include/floatformat.h	6 Mar 2002 06:29:41 -0000	1.5
+++ include/floatformat.h	15 Sep 2003 14:30:58 -0000
@@ -118,4 +118,9 @@ extern void
 floatformat_from_double PARAMS ((const struct floatformat *,
 				 double *, char *));
 
+/* Return non-zero iff the data at FROM is a valid number in format FMT.  */
+
+extern int
+floatformat_is_valid PARAMS ((const struct floatformat *fmt, char *from));
+
 #endif	/* defined (FLOATFORMAT_H) */