Re: using the PLT scheme lexer

[Michael Vanier <mvanier@bbb.caltech.edu>]

Playing around with the example lexer, it appears as if the lexer will
always return the first matching token, even if that token is the prefix to
a longer token which is also a valid match.  For instance, if you have two
token categories, one of which matches "-" (e.g. a symbol) and one of which
matches "-42" (an integer), the lexer matches "-" instead of "-42".  This
is different from standard lex/flex behavior.  Is this a bug?

Mike

> Date: Thu, 13 Dec 2001 14:12:01 -0700 (MST)
> From: Scott Owens <sowens@cs.utah.edu>
> 
> In general, the return value of the lexer is not restricted.  The token
> structure is provided for interoperation with a parser, but you are free
> to return whatever values you prefer.
> 
> Since the lexer generator is a new tool to v200, I would appreciate any
> suggestions/feedback on you experience using it.
> 
> -Scott Owens
> 
> On Thu, 13 Dec 2001, Michael Vanier wrote:
> 
> > 
> > OK, now that I've found the lexer, here's a simple question.
> > 
> > I can run the lexer to return a single value of type <struct:token>, but I
> > can't figure out how to extract the fields from the struct.  As far as I
> > can tell (and I'm a total newbie with the module system), the token struct
> > type is defined in collects/parser-tools/private-lex/token.ss.  I can't
> > import it directly because it already gets imported when I import the lex
> > library.  However, the token accessor functions do not appear to be
> > exported.  Does that mean that tokens are an opaque data type?  If I can't
> > get the value of the token, the lex module is not usable by me.  I don't
> > want or need to use the yacc module.  Sorry for my utter cluelessness.
> > 
> > Mike
> > 
> > 
> 
>