A generic reader for Guile?

[Mikael Djurfeldt <mdj@thalamus.nada.kth.se>]

I had planned to surprise you guys with a Christmas present this year.
But I was so lazy during Christmas so that I only got half-finished.
Now I realize that it will be too long time before I can finish it, so
I'll post the idea in case someone else feels like working on it.

Here's what I did:

I ran flex for a simple specification file.  From this output I
deduced what was fixed code from a template and what was variable
(dependent upon the specification file).  It turned out that
everything that was variable consisted either of C preprocessor
macro definitions or tables of numbers.

So I thought: Why not turn the macros into variables and write Scheme
level accessors for them, and turn the tables into uniform vectors?
Then it is possible to install the tables from the Scheme level.

Look what we have!  Ta ta!

An extremely efficient, dynamically configurable generic lexer!

In order to generate proper numbers for the tables I wrote a Scheme
program which converted a Scheme spec into a suitable flex input file,
ran flex and extracted the numbers from the flex output.

It worked!

Then I did the same for bison, but I never had time to finish this
stage.

I had planned to write the rules corresponding to Guile's syntax and
then surprise you all.  I think that the resulting parser would be
faster than Guile's own.  8-)

Now, *this* was a hack, but I think the idea of a dynamically
configurable table-based parser is great: you write the time critical
inner loops in C and crunch the rules from the grammar into tables.

Wouldn't this be great as a basis for the multi-language support in
Guile?

Wouldn't YOU, dear reader, find it just great to spend your summer
holiday reading the table-based parser section in Aho's dragon book,
studying the tricks used in flex and bison to achieve optimum
performance and implement this idea?  :-)

It may even be possible to port the template code directly from
flex/bison and port the C code for generating the tables to Scheme.

Maybe we should divide it into two projects?  One scanner (flex) and
one parser (bison).