Help me

[Sara Masri]

Write A program that reads a statement of c++ langauge and prints a table of tokens and classes (keyword, identifier

[Stuugie]

What have you got so far?

[Sara Masri]

this question from compiler design .
we need to know the keywords, identifiers, reserved words...etc in a written statement in programming language like c++ and type them in table

[Stuugie]

Sorry but I need to clarify because of the differences between your 1st and 2nd post. Are you supposed to write some code or just get the list of keywords?

[Ancient Dragon]

google for c++ keywords and you will find this article

[Sara Masri]

I need c++ source code to identify the keywords in any statement typed in the program
like :
for (int i = 0 ; i < 10 ; i++)
for , int : keywords
i : identifier
< , + : oprator

[Schol-R-LEA]

OK, so what you actually need is a lexical analyzer, also called a tokenizer from it's main mode of operation, which is to break a stream of characters into tokens. There are several discussions of this topic on DaniWeb, if you do a search on 'tokenizer'. You might want to check out this post, for example, as well as the StateMachine class in my own Suntiger Algol compiler.

[Sara Masri]

import java.util.ArrayList;
import java.util.regex.*;



public class Lexer {
    public static enum TokenType {
        // Token types cannot have underscores
        NUMBER("-?[0-9]+"), BINARYOP("[*|/|+|-]"), WHITESPACE("[ \t\f\r\n]+");

        public final String pattern;

        private TokenType(String pattern) {
            this.pattern = pattern;
        }
    }

    public static class Token {
        public TokenType type;
        public String data;

        public Token(TokenType type, String data) {
            this.type = type;
            this.data = data;
        }

        @Override
        public String toString() {
            return String.format("(%s %s)", type.name(), data);
        }
    }

    public static ArrayList<Token> lex(String input) {
        // The tokens to return
        ArrayList<Token> tokens = new ArrayList<Token>();

        // Lexer logic begins here
        StringBuffer tokenPatternsBuffer = new StringBuffer();
        for (TokenType tokenType : TokenType.values())
            tokenPatternsBuffer.append(String.format("|(?<%s>%s)", tokenType.name(), tokenType.pattern));
        Pattern tokenPatterns = Pattern.compile(new String(tokenPatternsBuffer.substring(1)));

        // Begin matching tokens
        Matcher matcher = tokenPatterns.matcher(input);

        while (matcher.find()) {

            if (matcher.group(TokenType.NUMBER.name())!= null) {
                 tokens.add(new Token(TokenType.NUMBER, matcher.group(TokenType.NUMBER.name())));
                continue;
            }
           else if (matcher.group(TokenType.BINARYOP.name()) != null) {
                tokens.add(new Token(TokenType.BINARYOP, matcher.group(TokenType.BINARYOP.name())));
                continue;
            }
           else if (matcher.group(TokenType.WHITESPACE.name()) != null)
                continue;
        }

        return tokens;
    }

    public static void main(String[] args) {
        String input = "11 + 22 - 33";

        // Create tokens and print them
        ArrayList<Token> tokens = lex(input);
        for (int i = 0 ; i < tokens.size(); i++)
            System.out.println(tokens.get(i));
    }
}

[Sara Masri]

I have an error in th following part and I can't solve th problem with this error , so if you can do it, help me.

while (matcher.find()) {

            if (matcher.group(TokenType.NUMBER.name())!= null) {
                 tokens.add(new Token(TokenType.NUMBER, matcher.group(TokenType.NUMBER.name())));
                continue;
            }
           else if (matcher.group(TokenType.BINARYOP.name()) != null) {
                tokens.add(new Token(TokenType.BINARYOP, matcher.group(TokenType.BINARYOP.name())));
                continue;
            }
           else if (matcher.group(TokenType.WHITESPACE.name()) != null)
                continue;
        }

[Ancient Dragon]

what is the error?

[Sara Masri]

cannot recognize the method group(String)

[Ancient Dragon]

cannot recognize the method group(String)

Does matcher have such a method?