3.2. Token Attributes

Some tokens can represent many different lexemes. Examples are:

• variable and function names
• integer constants
• string constants
• character constants

• lexeme fruit might be represented by token TOK_IDENTIFIER.
• lexeme 533 might be represented by token TOK_INTCONST.
• lexeme 'y' might be represented by token TOK_CHARCONST.

Token attributes

---

During the parsing stage, the compiler will only be concerned with tokens. Any integer constant, for example, is treated like any other.

But during later processing, it will certainly be important just which constant was written.

To deal with that, a token that can have many associated lexemes has an attribute, which can be the lexeme if you like. During semantic processing, the compiler examines the token attributes.

An attribute is not always a lexeme. For example, the attribute for a TOK_INTCONST token might be an integer, telling the number that was written.