Ad:

C++ Discussion Thread
Unsolved
Views: 2190

Page 1
2
Next

You are currently viewing page 1 of this multi-page discussion thread

Aug 28th, 2007

Lexer- Tokenizer problem

Expand Post »

hi, i use the flex tool {http://www.gnu.org/software/flex/manual/} to generate a tokenizer ,but i have the following problem {it has to do with the way that flex tokenizes the input::

FILE : flex.l

C++ Syntax (Toggle Plain Text)
%{
		#define WEB 0
		#define SPACE 1
		#define STRING 2	
%}
 
string_component [0-9a-zA-Z \t\.!#$%^&()*@_]
 
%%
 
"daniweb"		              {return WEB;}
[ \t\n]			{return SPACE;}
{string_component}+	{return STRING;}
 
%%
 
#include <iostream>
 
using namespace std;
 
int main()
{	
	cout<<yylex()<<endl;
	cout<<yylex()<<endl;
 
	return 0;
}
 
int yywrap(void){return 1;}
%{
		#define WEB 0
		#define SPACE 1
		#define STRING 2	
%}

string_component [0-9a-zA-Z \t\.!#$%^&()*@_]

%%

"daniweb"		              {return WEB;}
[ \t\n]			{return SPACE;}
{string_component}+	{return STRING;}

%%

#include <iostream>
			
using namespace std;
		
int main()
{	
	cout<<yylex()<<endl;
	cout<<yylex()<<endl;

	return 0;
}

int yywrap(void){return 1;}

Example file:

C++ Syntax (Toggle Plain Text)

test_string daniweb

What i want is to have the above string tokenized as
STRING SPACE WEB
instead flex recognizes it as STRING, because it tries to match the longest input....

How can i fix this problem?
all ideas are welcomed....

PS:: to compile:

C++ Syntax (Toggle Plain Text)
flex flex.l
g++ lex.yy.c
./a.out <example
flex flex.l
g++ lex.yy.c
./a.out <example

Similar Threads

StringTokenizer problem in Java
simple program tokenizer problem in Java

n.aggel

Reputation Points: 23

Solved Threads: 12

Posting Whiz in Training

Offline

202 posts
since Nov 2006

Permalink

Aug 28th, 2007

Re: Lexer- Tokenizer problem

Your string component matches spaces, and now you're complaining that you don't want to match spaces.

You can't have it both ways.

Salem

Team Colleague

Reputation Points: 5862

Solved Threads: 950

Posting Sage

Offline

7,164 posts
since Dec 2005

Permalink

Aug 29th, 2007

Re: Lexer- Tokenizer problem

Click to Expand / Collapse Quote originally posted by Salem ...

Your string component matches spaces, and now you're complaining that you don't want to match spaces.

You can't have it both ways.

Thank you for answering {apparently, few people have read the post...}

Yes you are rigth, it seems that i can't have it both ways... but from where i stand i want to use flex in order to do the following:::

Recognize some specif keywords {in the simplified example i provided the keyword was "daniweb"} and recognize everything else as a string...any ideas on how can i do that?

PS: maybe start conditions could help me solve the problem?{ i havven't understand them so well...}
PS2:in the beggining i thought it wouldn't be that difficult, but i was wrong...

n.aggel

Reputation Points: 23

Solved Threads: 12

Posting Whiz in Training

Offline

202 posts
since Nov 2006

Permalink

Aug 29th, 2007

Re: Lexer- Tokenizer problem

What is this Flex? some kinda regular expression library or something. Do you even need it or can your problem be simplified?

Last edited by iamthwee; Aug 29th, 2007 at 2:42 pm.

iamthwee

Featured Poster

Reputation Points: 1536

Solved Threads: 431

Posting Expert

Offline

5,865 posts
since Aug 2005

Permalink

Aug 29th, 2007

Re: Lexer- Tokenizer problem

Click to Expand / Collapse Quote originally posted by iamthwee ...

What is this Flex? some kinda regular expression library or something. Do you even need it or can your problem be simplified?

Flex

Flex (The Fast Lexical Analyzer)
Flex is a fast lexical analyser generator. It is a tool for generating programs that perform pattern-matching on text. Flex is a non-GNU free implementation of the well known Lex program.

http://www.gnu.org/software/flex/
http://flex.sourceforge.net/

n.aggel

Reputation Points: 23

Solved Threads: 12

Posting Whiz in Training

Offline

202 posts
since Nov 2006

Permalink

Aug 29th, 2007

Re: Lexer- Tokenizer problem

Um ok, please explain this:

string_component [0-9a-zA-Z \t\.!#$%^&()*@_]

and what you think it does?

Last edited by iamthwee; Aug 29th, 2007 at 3:11 pm.

iamthwee

Featured Poster

Reputation Points: 1536

Solved Threads: 431

Posting Expert

Offline

5,865 posts
since Aug 2005

Permalink

Aug 29th, 2007

Re: Lexer- Tokenizer problem

There's a way to set precedence of regex's in flex. I don't remember the exact syntax, but you should put it before your catchall regex that you have defined there.

Last edited by nedrocks; Aug 29th, 2007 at 5:40 pm.

nedrocks

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

1 posts
since Aug 2007

Permalink

Aug 30th, 2007

Re: Lexer- Tokenizer problem

Click to Expand / Collapse Quote originally posted by nedrocks ...

There's a way to set precedence of regex's in flex. I don't remember the exact syntax, but you should put it before your catchall regex that you have defined there.

i haven't seen what you mention in the manual...

unfortunately i haven't found the solution...i worked around my problem by changing the grammar {i.e. bison file}, and finally i gave the project... Now when i find the time i will try to find a solution using starting conditions

n.aggel

Reputation Points: 23

Solved Threads: 12

Posting Whiz in Training

Offline

202 posts
since Nov 2006

Permalink

Aug 30th, 2007

Re: Lexer- Tokenizer problem

First you gotta know what your regular expressions are doing.

To me string_component [0-9a-zA-Z \t\.!#$%^&()*@_] and the example you have given is contradictory, like salem mentioned.

iamthwee

Featured Poster

Reputation Points: 1536

Solved Threads: 431

Posting Expert

Offline

5,865 posts
since Aug 2005

Permalink

Aug 30th, 2007

Re: Lexer- Tokenizer problem

using boost.spirit may be much easier: http://www.boost.org/libs/spirit/doc/quick_start.html

cpp Syntax (Toggle Plain Text)
#include <boost/spirit/core.hpp>
#include <iostream>
#include <string>
#include <vector>
#include <algorithm>
#include <boost/assign.hpp>
using namespace std ;
using namespace boost ;
using namespace boost::spirit ;
using namespace boost::assign ;
 
struct parse_it
{ 
  void operator() ( const string& str ) const
  {
    vector<string> tokens ;
    const char* cstr = str.c_str() ;
    size_t n = 0 ;
    while( n < str.size() )
      n += parse( cstr + n,
                  (+space_p) [  push_back_a( tokens, "SPACE" ) ] |
                  str_p("daniweb") [ push_back_a( tokens, "WEB" ) ] |
                  str_p("lexer") [ push_back_a( tokens, "LEX" ) ] |
                  str_p("tokenizer") [ push_back_a( tokens, "TOK" ) ] |
                  (+~space_p) [ push_back_a( tokens, "STRING" ) ]
                ).length ;
    cout << '\n' << "parsed: " << str << "\ntokens: " ;      
    copy( tokens.begin(), tokens.end(), 
               ostream_iterator<string>(cout," ") ) ;
    cout << '\n' ;      
  }
};
int main()
{
  vector<string> test_cases = list_of
                ( "test daniweb lexer xyz tokenizer lexer" )
                ( "daniweblexer tokenizerlexer abcd lexerlexer" )
                ( "daniwebtest lexerdaniweblexertest tokenizerxxx" ) ;
  for_each( test_cases.begin(), test_cases.end(), parse_it() ) ;
}
/**
>g++ -Wall -std=c++98 -I/usr/local/include keyword.cpp && ./a.out
 
parsed: test daniweb lexer xyz tokenizer lexer
tokens: STRING SPACE WEB SPACE LEX SPACE STRING SPACE TOK SPACE LEX
 
parsed: daniweblexer tokenizerlexer abcd lexerlexer
tokens: WEB LEX SPACE TOK LEX SPACE STRING SPACE LEX LEX
 
parsed: daniwebtest lexerdaniweblexertest tokenizerxxx
tokens: WEB STRING SPACE LEX WEB LEX STRING SPACE TOK STRING
*/
#include <boost/spirit/core.hpp>
#include <iostream>
#include <string>
#include <vector>
#include <algorithm>
#include <boost/assign.hpp>
using namespace std ;
using namespace boost ;
using namespace boost::spirit ;
using namespace boost::assign ;

struct parse_it
{ 
  void operator() ( const string& str ) const
  {
    vector<string> tokens ;
    const char* cstr = str.c_str() ;
    size_t n = 0 ;
    while( n < str.size() )
      n += parse( cstr + n,
                  (+space_p) [  push_back_a( tokens, "SPACE" ) ] |
                  str_p("daniweb") [ push_back_a( tokens, "WEB" ) ] |
                  str_p("lexer") [ push_back_a( tokens, "LEX" ) ] |
                  str_p("tokenizer") [ push_back_a( tokens, "TOK" ) ] |
                  (+~space_p) [ push_back_a( tokens, "STRING" ) ]
                ).length ;
    cout << '\n' << "parsed: " << str << "\ntokens: " ;      
    copy( tokens.begin(), tokens.end(), 
               ostream_iterator<string>(cout," ") ) ;
    cout << '\n' ;      
  }
};
int main()
{
  vector<string> test_cases = list_of
                ( "test daniweb lexer xyz tokenizer lexer" )
                ( "daniweblexer tokenizerlexer abcd lexerlexer" )
                ( "daniwebtest lexerdaniweblexertest tokenizerxxx" ) ;
  for_each( test_cases.begin(), test_cases.end(), parse_it() ) ;
}
/**
>g++ -Wall -std=c++98 -I/usr/local/include keyword.cpp && ./a.out

parsed: test daniweb lexer xyz tokenizer lexer
tokens: STRING SPACE WEB SPACE LEX SPACE STRING SPACE TOK SPACE LEX

parsed: daniweblexer tokenizerlexer abcd lexerlexer
tokens: WEB LEX SPACE TOK LEX SPACE STRING SPACE LEX LEX

parsed: daniwebtest lexerdaniweblexertest tokenizerxxx
tokens: WEB STRING SPACE LEX WEB LEX STRING SPACE TOK STRING
*/

Last edited by vijayan121; Aug 30th, 2007 at 2:55 pm.

vijayan121

Reputation Points: 1159

Solved Threads: 285

Posting Virtuoso

Offline

1,606 posts
since Dec 2006

Page 1
2
Next

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in C++ Forum Timeline: please help

Next Thread in C++ Forum Timeline: Optomizing for Pentium Pro

DaniWeb Message

Cancel Changes

User Name
Password