Tokenizers

[Dani]

I'm feeling a little dumb when it comes to the editor that we are using, and I was wondering if anyone knew of any good online tutorials for an introduction to tokenizers.

[pritaeas]

Not a tutorial, but I know PEAR has a tokenizer.

[Dani]

Yeah, not what I need. I'm trying to make all these changes to the tokenizer used by the editor and it's confusing me because I have no experience with how a tokenizer works. :(

[Airshow]

Dani,

I looked at the code behind the editor a couple of months ago and have to agree, it ain't the easiest to follow. I felt it needed to be taken apart and put back together in a more organised fashion.

If you want to take a step back from the actual code and remind yourself of the concepts and teminology in this area, then Wikipedia's entry on Lexical Analysis seems pretty good (with the usual provisos for Wikis). Some of the links may also prove fruitful.