DaniWeb IT Discussion Community - View Single Post

Lexer- Tokenizer problem

Aug 28th, 2007

hi, i use the flex tool {http://www.gnu.org/software/flex/manual/} to generate a tokenizer ,but i have the following problem {it has to do with the way that flex tokenizes the input::

FILE : flex.l

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
%{
		#define WEB 0
		#define SPACE 1
		#define STRING 2	
%}
 
string_component [0-9a-zA-Z \t\.!#$%^&()*@_]
 
%%
 
"daniweb"		              {return WEB;}
[ \t\n]			{return SPACE;}
{string_component}+	{return STRING;}
 
%%
 
#include <iostream>
 
using namespace std;
 
int main()
{	
	cout<<yylex()<<endl;
	cout<<yylex()<<endl;
 
	return 0;
}
 
int yywrap(void){return 1;}
%{
		#define WEB 0
		#define SPACE 1
		#define STRING 2	
%}

string_component [0-9a-zA-Z \t\.!#$%^&()*@_]

%%

"daniweb"		              {return WEB;}
[ \t\n]			{return SPACE;}
{string_component}+	{return STRING;}

%%

#include <iostream>
			
using namespace std;
		
int main()
{	
	cout<<yylex()<<endl;
	cout<<yylex()<<endl;

	return 0;
}

int yywrap(void){return 1;}

Example file:

Help with Code Tags

C++ Syntax (Toggle Plain Text)

test_string daniweb

What i want is to have the above string tokenized as
STRING SPACE WEB
instead flex recognizes it as STRING, because it tries to match the longest input....

How can i fix this problem?
all ideas are welcomed....

PS:: to compile:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
flex flex.l
g++ lex.yy.c
./a.out <example
flex flex.l
g++ lex.yy.c
./a.out <example