954,523 Members — Technology Publication meets Social Media
Username:
Password:
Lost login information?
Have something to say? Contribute New Article Reply to this Article

Haskell lexer help

Hello everyone, I've been doing a project for class in Haskell. Basically I've completed my program except some parts in my lexer function. I have functions marked undefined where I haven't completed it.

lexer :: String -> Pos -> [Token]
lexer str pos = 
     case str of
       x:xs | isSpace x -> lexer xs (updatePosChar pos x)
       x:xs | isIdChar x -> identifierOrKeyword [x] xs pos
       x:xs | isDigit x ->  identifierOrKeyword [x] xs pos
       x:xs | x == '\'' ->  result1 : result2 where
			result1 = TokPos pos (reserved [x])
			result2 = lexer xs newPos
			newPos = updatePosChar pos x
       x:xs | isSep1 [x] -> TokPos pos (reserved [x]) : 
                            lexer xs (incSourceCol pos 1)
       x:y:xs | isComment [x, y] -> undefined -- FIXME
       x:y:xs | isOp2 [x, y] -> undefined -- FIXME
       x:xs | isOp1 [x] -> TokPos pos (reserved [x]) :
			lexer xs (incSourceCol pos 1)
       x:_ -> error (show pos ++ ": Lexer error: unknown character " ++ [x]) 
       [] -> []
     where
       identifierOrKeyword :: String -> String -> Pos -> [Token]
       identifierOrKeyword ident str pos = 
           case str of 
             x:xs | isIdChar x || isDigit x -> identifierOrKeyword (ident ++ [x]) xs pos
             xs -> let newp = updatePosString pos ident
                       newtok = if isKeyword ident
				then reserved ident
				else TokIdent ident 
                   in TokPos pos newtok : lexer xs newp


The line I need help on is:

x:y:xs | isComment [x, y] -> undefined -- FIXME

I have tried numerous things, but reverted to calling it undefined so I could work on the rest of the function.
You can see how I do the other cases such as isOp1 [x], etc.. I know how these work when I am only expecing a single character or predictable string (like "Bool", "True", etc...) but when I need to recognize a comment such as "//here is a comment" I need to be able to make the program understand if // then all characters before /n is part of that comment. That is what y is for. But i can't figure out the syntax to make it work.

If you need additional explanation to help me, post it. Also, please ignore any possible formatting error, the copy&paste didn't go over very well :D
Thanks

ckwolfe
Newbie Poster
10 posts since Apr 2011
Reputation Points: 10
Solved Threads: 0
 

You need to make a helper function, analogous to identifierOrKeyword, that scans past the first newline in the string. The span or break functions might come in handy.

Rashakil Fol
Super Senior Demiposter
Team Colleague
2,658 posts since Jun 2005
Reputation Points: 1,135
Solved Threads: 177
 

This article has been dead for over three months

Post: Markdown Syntax: Formatting Help
You