Member Avatar

Write A program that reads a statement of c++ langauge and prints a table of tokens and classes (keyword, identifier

Member Avatar

What have you got so far?

Member Avatar

this question from compiler design .
we need to know the keywords, identifiers, reserved words...etc in a written statement in programming language like c++ and type them in table

Member Avatar

Sorry but I need to clarify because of the differences between your 1st and 2nd post. Are you supposed to write some code or just get the list of keywords?

Edited by Stuugie
Member Avatar

I need c++ source code to identify the keywords in any statement typed in the program
like :
for (int i = 0 ; i < 10 ; i++)
for , int : keywords
i : identifier
< , + : oprator

Member Avatar

OK, so what you actually need is a lexical analyzer, also called a tokenizer from it's main mode of operation, which is to break a stream of characters into tokens. There are several discussions of this topic on DaniWeb, if you do a search on 'tokenizer'. You might want to check out this post, for example, as well as the StateMachine class in my own Suntiger Algol compiler.

Edited by Schol-R-LEA
Member Avatar 
import java.util.ArrayList;
import java.util.regex.*;



public class Lexer {
    public static enum TokenType {
        // Token types cannot have underscores
        NUMBER("-?[0-9]+"), BINARYOP("[*|/|+|-]"), WHITESPACE("[ \t\f\r\n]+");

        public final String pattern;

        private TokenType(String pattern) {
            this.pattern = pattern;
        }
    }

    public static class Token {
        public TokenType type;
        public String data;

        public Token(TokenType type, String data) {
            this.type = type;
            this.data = data;
        }

        @Override
        public String toString() {
            return String.format("(%s %s)", type.name(), data);
        }
    }

    public static ArrayList<Token> lex(String input) {
        // The tokens to return
        ArrayList<Token> tokens = new ArrayList<Token>();

        // Lexer logic begins here
        StringBuffer tokenPatternsBuffer = new StringBuffer();
        for (TokenType tokenType : TokenType.values())
            tokenPatternsBuffer.append(String.format("|(?<%s>%s)", tokenType.name(), tokenType.pattern));
        Pattern tokenPatterns = Pattern.compile(new String(tokenPatternsBuffer.substring(1)));

        // Begin matching tokens
        Matcher matcher = tokenPatterns.matcher(input);

        while (matcher.find()) {

            if (matcher.group(TokenType.NUMBER.name())!= null) {
                 tokens.add(new Token(TokenType.NUMBER, matcher.group(TokenType.NUMBER.name())));
                continue;
            }
           else if (matcher.group(TokenType.BINARYOP.name()) != null) {
                tokens.add(new Token(TokenType.BINARYOP, matcher.group(TokenType.BINARYOP.name())));
                continue;
            }
           else if (matcher.group(TokenType.WHITESPACE.name()) != null)
                continue;
        }

        return tokens;
    }

    public static void main(String[] args) {
        String input = "11 + 22 - 33";

        // Create tokens and print them
        ArrayList<Token> tokens = lex(input);
        for (int i = 0 ; i < tokens.size(); i++)
            System.out.println(tokens.get(i));
    }
}
Member Avatar

I have an error in th following part and I can't solve th problem with this error , so if you can do it, help me.

while (matcher.find()) {

            if (matcher.group(TokenType.NUMBER.name())!= null) {
                 tokens.add(new Token(TokenType.NUMBER, matcher.group(TokenType.NUMBER.name())));
                continue;
            }
           else if (matcher.group(TokenType.BINARYOP.name()) != null) {
                tokens.add(new Token(TokenType.BINARYOP, matcher.group(TokenType.BINARYOP.name())));
                continue;
            }
           else if (matcher.group(TokenType.WHITESPACE.name()) != null)
                continue;
        }
Member Avatar

cannot recognize the method group(String)

Member Avatar

cannot recognize the method group(String)

Does matcher have such a method?