I recently took a course on assembler/compiler construction. In it we covered parsing algorithms such as LL(n), LR(n), LALR(n), and SLR(n).
I understand how these parsing algorithms can be used to determine if an input string follows a context free grammar (CFG). At some point I also understood how to tokenize the string using the grammar, however my assignment code is not commented (in hindsight that was a mistake) and I do not understand it anymore.
Now I am in need of a parser to convert lines of code into useful tokens and cannot for the life of me remember how to use a parsing algorithm and a context free grammar to do lexical analysis on a string to extract tokens.
How can I take a generic CFG (with accompanying deterministic finite automaton) and an input string to generate an array of tokens?