Tokens, patterns, and lexemes in Compiler Design - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a token in compiler design?

A token is a basic unit of meaning in source code, identified by the compiler during lexical analysis. It represents a category like a keyword, identifier, operator, or symbol.

beginner

Define lexeme with an example.

A lexeme is the actual sequence of characters in the source code that matches a pattern for a token. For example, in the code int x = 5;, int is a lexeme representing the token 'keyword'.

intermediate

What is a pattern in the context of tokens?

A pattern is a rule or description (often a regular expression) that defines the set of lexemes that form a particular token. For example, the pattern for an identifier might be letters followed by letters or digits.

intermediate

How do tokens, patterns, and lexemes relate to each other?

Patterns describe the rules for tokens. Lexemes are the actual text in the source code that match these patterns. Tokens are the categories assigned to these lexemes during lexical analysis.

beginner

Why is lexical analysis important in compiling source code?

Lexical analysis breaks source code into tokens by matching lexemes to patterns. This simplifies parsing and helps detect errors early, making the compilation process efficient.

Which of the following best describes a lexeme?

A The category or type of a token

B The actual sequence of characters in source code matching a token

C A rule describing a token

D A compiler optimization technique

A lexeme is the exact sequence of characters in the source code that matches a token's pattern.

What does a pattern define in lexical analysis?

A The rule that describes which lexemes form a token

B The syntax of the programming language

C The meaning of a token

D The output of the compiler

A pattern defines the rule or regular expression that lexemes must match to be recognized as a token.

Which of these is NOT a token category?

A Keyword

B Identifier

C Syntax tree

D Operator

A syntax tree is a data structure used later in compilation, not a token category.

During lexical analysis, what is the main task?

A Parsing source code into statements

B Optimizing the compiled program

C Converting tokens into machine code

D Breaking source code into tokens by matching lexemes to patterns

Lexical analysis breaks source code into tokens by matching lexemes to their patterns.

If the source code contains the word while, what is the token and lexeme respectively?

A Token: keyword, Lexeme: while

B Token: identifier, Lexeme: while

C Token: operator, Lexeme: while

D Token: keyword, Lexeme: loop

The word 'while' is a lexeme that matches the pattern for the token category 'keyword'.

Explain in your own words the difference between a token, a pattern, and a lexeme.

Think about how a dictionary word (lexeme) fits a definition (pattern) and belongs to a part of speech (token).

Why does a compiler need to identify tokens using patterns and lexemes before parsing?

Consider how breaking a sentence into words helps understand its meaning.