Implementing a lexical analyzer in Compiler Design - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of a lexical analyzer in a compiler?

A lexical analyzer reads the source code and breaks it into tokens, which are meaningful sequences like keywords, identifiers, and symbols. It simplifies the input for the next compiler stage.

beginner

Define a token in the context of lexical analysis.

A token is a sequence of characters from the source code that represents a basic unit like a keyword, identifier, operator, or punctuation, recognized by the lexical analyzer.

intermediate

What role do regular expressions play in implementing a lexical analyzer?

Regular expressions describe patterns for tokens. The lexical analyzer uses these patterns to recognize and extract tokens from the source code.

intermediate

Explain the difference between a lexical error and a syntax error.

A lexical error occurs when the lexical analyzer finds an invalid sequence of characters that doesn't match any token pattern. A syntax error happens later when the parser finds that the token sequence doesn't follow the language grammar.

beginner

Why is it important for a lexical analyzer to remove whitespace and comments?

Whitespace and comments do not affect the program's meaning, so the lexical analyzer removes them to simplify the token stream and make parsing easier.

What does a lexical analyzer output after processing source code?

A A sequence of tokens

B Machine code

C Syntax tree

D Error messages only

The lexical analyzer outputs tokens, which are the basic units for the parser to analyze.

Which tool is commonly used to define token patterns for a lexical analyzer?

A Regular expressions

B Context-free grammar

C Assembly language

D Linker scripts

Regular expressions describe the patterns that tokens must match.

What should a lexical analyzer do when it encounters an unknown character sequence?

A Pass it to the parser

B Ignore it silently

C Convert it to a token anyway

D Report a lexical error

Unknown sequences that don't match any token pattern cause lexical errors.

Which of the following is NOT typically a token type?

A Identifier

B Whitespace

C Keyword

D Operator

Whitespace is usually ignored and not treated as a token.

Why is lexical analysis considered the first phase of a compiler?

A It optimizes the program

B It generates executable code

C It prepares the source code by breaking it into tokens for parsing

D It links libraries

Lexical analysis is the first step to convert raw code into tokens for the parser.

Describe the main steps involved in implementing a lexical analyzer.

Think about how the analyzer reads input and decides what each piece means.

Explain how a lexical analyzer helps the overall compilation process.

Consider the role of tokens and error detection.

Practice

1. What is the main job of a lexical analyzer in compiler design?

easy

A. Break input text into meaningful tokens

B. Execute the program code

C. Optimize the compiled code

D. Generate machine code

Solution

1. Step 1: Understand the role of lexical analyzer

   A lexical analyzer reads the input text and divides it into tokens like keywords, identifiers, and symbols.

2. Step 2: Differentiate from other compiler phases

   It does not execute code, optimize, or generate machine code; those are later phases.

3. Final Answer:

   Break input text into meaningful tokens -> Option A

4. Quick Check:

   Lexical analyzer = tokenization [OK]

Hint: Lexical analyzer splits text into tokens [OK]

Common Mistakes:

• Confusing lexical analysis with code execution
• Thinking it optimizes or generates machine code
• Assuming it handles syntax errors

2. Which of the following is a correct way to define a token pattern in a lexical analyzer?

easy

A. token = (123)+

B. token = [a-zA-Z_][a-zA-Z0-9_]*

C. token = {print}

D. token = 123abc

Solution

1. Step 1: Recognize token pattern syntax

   Token patterns use regular expressions; [a-zA-Z_][a-zA-Z0-9_]* matches identifiers starting with letter or underscore.

2. Step 2: Check other options for correctness

   token = (123)+ uses invalid syntax for token pattern; C uses braces incorrectly; D is not a valid pattern.

3. Final Answer:

   token = [a-zA-Z_][a-zA-Z0-9_]* -> Option B

4. Quick Check:

   Valid regex pattern = token = [a-zA-Z_][a-zA-Z0-9_]* [OK]

Hint: Token patterns use regex-like syntax starting with letters or underscore [OK]

Common Mistakes:

• Using invalid characters in token patterns
• Confusing literal strings with patterns
• Ignoring start character rules

3. Given the input string int x = 10;, what tokens will a lexical analyzer produce?

medium

A. ["int", "x=10;"]

B. ["int x", "= 10;"]

C. ["int", "x", "=", "10", ";"]

D. ["int", "x", "=", "1", "0", ";"]

Solution

1. Step 1: Identify tokens in the input

   The lexical analyzer splits by language rules: keyword 'int', identifier 'x', operator '=', number '10', and symbol ';'.

2. Step 2: Compare with options

   ["int", "x", "=", "10", ";"] correctly lists all tokens separately. Others combine tokens incorrectly or split numbers wrongly.

3. Final Answer:

   ["int", "x", "=", "10", ";"] -> Option C

4. Quick Check:

   Tokens = separate meaningful units [OK]

Hint: Tokens separate keywords, identifiers, operators, numbers, symbols [OK]

Common Mistakes:

• Combining multiple tokens into one
• Splitting numbers into digits
• Ignoring punctuation as tokens

4. Identify the error in this lexical analyzer rule: NUMBER = [0-9]+(.[0-9]+)?

medium

A. Plus (+) should be replaced with star (*)

B. Parentheses should be removed

C. Square brackets are not allowed

D. Dot (.) should be escaped as \.

Solution

1. Step 1: Analyze the regex pattern

   The dot (.) in regex means any character; to match a literal dot, it must be escaped as \.

2. Step 2: Check other parts of the pattern

   Plus (+) is correct for one or more digits; square brackets and parentheses are valid here.

3. Final Answer:

   Dot (.) should be escaped as \. -> Option D

4. Quick Check:

   Escape dot for literal period [OK]

Hint: Escape dot in regex to match literal '.' [OK]

Common Mistakes:

• Not escaping dot causing wrong matches
• Changing quantifiers unnecessarily
• Removing valid grouping parentheses

5. How can a lexical analyzer handle comments in source code effectively?

hard

A. By recognizing comment patterns and skipping them

B. By converting comments into tokens

C. By treating comments as syntax errors

D. By storing comments as identifiers

Solution

1. Step 1: Understand comment handling

   Comments are ignored by the compiler; the lexical analyzer must detect comment start and end patterns and skip their content.

2. Step 2: Evaluate other options

   Converting comments into tokens or treating them as errors is incorrect; storing as identifiers is wrong.

3. Final Answer:

   By recognizing comment patterns and skipping them -> Option A

4. Quick Check:

   Skip comments during tokenization [OK]

Hint: Detect and skip comments, don't tokenize them [OK]

Common Mistakes:

• Tokenizing comments as code
• Raising errors on comments
• Confusing comments with identifiers