Write A program that reads a statement of c++ langauge and prints a table of tokens and classes (keyword, identifier

Recommended Answers

What have you got so far?

Jump to Post

Sorry but I need to clarify because of the differences between your 1st and 2nd post. Are you supposed to write some code or just get the list of keywords?

Jump to Post

OK, so what you actually need is a lexical analyzer, also called a tokenizer from it's main mode of operation, which is to break a stream of characters into tokens. There are several …

Jump to Post

All 11 Replies

What have you got so far?

this question from compiler design .
we need to know the keywords, identifiers, reserved words...etc in a written statement in programming language like c++ and type them in table

Sorry but I need to clarify because of the differences between your 1st and 2nd post. Are you supposed to write some code or just get the list of keywords?

google for c++ keywords and you will find this article

I need c++ source code to identify the keywords in any statement typed in the program
like :
for (int i = 0 ; i < 10 ; i++)
for , int : keywords
i : identifier
< , + : oprator

OK, so what you actually need is a lexical analyzer, also called a tokenizer from it's main mode of operation, which is to break a stream of characters into tokens. There are several discussions of this topic on DaniWeb, if you do a search on 'tokenizer'. You might want to check out this post, for example, as well as the StateMachine class in my own Suntiger Algol compiler.

import java.util.ArrayList;
import java.util.regex.*;



public class Lexer {
    public static enum TokenType {
        // Token types cannot have underscores
        NUMBER("-?[0-9]+"), BINARYOP("[*|/|+|-]"), WHITESPACE("[ \t\f\r\n]+");

        public final String pattern;

        private TokenType(String pattern) {
            this.pattern = pattern;
        }
    }

    public static class Token {
        public TokenType type;
        public String data;

        public Token(TokenType type, String data) {
            this.type = type;
            this.data = data;
        }

        @Override
        public String toString() {
            return String.format("(%s %s)", type.name(), data);
        }
    }

    public static ArrayList<Token> lex(String input) {
        // The tokens to return
        ArrayList<Token> tokens = new ArrayList<Token>();

        // Lexer logic begins here
        StringBuffer tokenPatternsBuffer = new StringBuffer();
        for (TokenType tokenType : TokenType.values())
            tokenPatternsBuffer.append(String.format("|(?<%s>%s)", tokenType.name(), tokenType.pattern));
        Pattern tokenPatterns = Pattern.compile(new String(tokenPatternsBuffer.substring(1)));

        // Begin matching tokens
        Matcher matcher = tokenPatterns.matcher(input);

        while (matcher.find()) {

            if (matcher.group(TokenType.NUMBER.name())!= null) {
                 tokens.add(new Token(TokenType.NUMBER, matcher.group(TokenType.NUMBER.name())));
                continue;
            }
           else if (matcher.group(TokenType.BINARYOP.name()) != null) {
                tokens.add(new Token(TokenType.BINARYOP, matcher.group(TokenType.BINARYOP.name())));
                continue;
            }
           else if (matcher.group(TokenType.WHITESPACE.name()) != null)
                continue;
        }

        return tokens;
    }

    public static void main(String[] args) {
        String input = "11 + 22 - 33";

        // Create tokens and print them
        ArrayList<Token> tokens = lex(input);
        for (int i = 0 ; i < tokens.size(); i++)
            System.out.println(tokens.get(i));
    }
}

I have an error in th following part and I can't solve th problem with this error , so if you can do it, help me.

while (matcher.find()) {

            if (matcher.group(TokenType.NUMBER.name())!= null) {
                 tokens.add(new Token(TokenType.NUMBER, matcher.group(TokenType.NUMBER.name())));
                continue;
            }
           else if (matcher.group(TokenType.BINARYOP.name()) != null) {
                tokens.add(new Token(TokenType.BINARYOP, matcher.group(TokenType.BINARYOP.name())));
                continue;
            }
           else if (matcher.group(TokenType.WHITESPACE.name()) != null)
                continue;
        }

what is the error?

cannot recognize the method group(String)

cannot recognize the method group(String)

Does matcher have such a method?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.