C Tokens defined in textbook, but...

[aeinstein]

Hi All,

Just starting my online C programming class and my textbook states "In a computer language, a token is the smallest unit of the language that has a unique meaning. Thus, the reserved words, programmer-created identifiers, and all special mathematical symbols, such as + and -, are considered tokens of the C language."

Since the text previously states that the three type of C identifiers consist of keywords/reserved words, standard identifiers and programmer-created identifiers, then I'm presuming that both keywords and standard identifiers are also tokens, correct? And that punctuation marks ("()", "{}", ";" etc.), even though they represent attributes of the language, that they are considered to be part of the grammatical structure and are outside the definition of tokens, per se?

Regards,
Paul

[nezachem]

> are outside the definition of tokens
No. The Standard defines token as either of:
keyword
identifier
constant
string-literal
punctuator
and punctuator in turn is either of
[ ] ( ) { } . ->
++ -- & * + - ~ !
/ % << >> < > <= >= == != ^ | && ||
? : ; ...
= *= /= %= += -= <<= >>= &= ^= |=
, # ##
<: :> <% %> %: %:%:

[aeinstein]

TYVM for your reply nezachem. So the text is wrong in regards to mathematical operators being considered as tokens?

Paul

[nezachem]

Umm... they are tokens.

[Adak]

All operators (and thus all mathematics operators), are tokens. Your compiler help should have a list of them.

[aeinstein]

Thanks for your replies nezachem and Adak, very helpful and very much appreciated.

Paul

PS: My apologies nezachem, I totally missed the quoted part of your reply, which is why I asked for clarification.