Implementing a lexical analyzer in Compiler Design - Mini Project: Build & Apply

Implementing a lexical analyzer

📖 Scenario: You are building a simple lexical analyzer for a tiny programming language. This analyzer will read a line of code and identify tokens such as keywords, identifiers, and numbers.

🎯 Goal: Create a basic lexical analyzer that can recognize keywords, identifiers, and numbers from a given input string.

📋 What You'll Learn

Create a list of keywords for the language

Set up a sample input string representing a line of code

Write logic to split the input into tokens and classify each token

Output the tokens with their types

💡 Why This Matters

🌍 Real World

Lexical analyzers are the first step in translating programming languages. They break down code into meaningful pieces called tokens.

💼 Career

Understanding lexical analysis is essential for compiler developers, language designers, and anyone working on tools that process code.

1

DATA SETUP: Define keywords and input string

Create a list called keywords with these exact strings: 'if', 'else', 'while', 'return'. Then create a string variable called code_line with the exact value 'if count return 10'.

Compiler Design

# Create the list of keywords and the input string
# Your code here

Hint

Use square brackets to create the list and quotes for strings.

2

CONFIGURATION: Prepare to split the input into tokens

Create a variable called tokens that splits the string code_line into parts separated by spaces using the split() method.

Compiler Design

keywords = ['if', 'else', 'while', 'return']
code_line = 'if count return 10'
# Split the code_line into tokens
# Your code here

Hint

Use the split() method without arguments to split by spaces.

3

CORE LOGIC: Classify each token

Create an empty dictionary called token_types. Then use a for loop with variable token to go through each item in tokens. Inside the loop, use if and elif statements to classify each token as follows: if token is in keywords, set token_types[token] to 'keyword'; else if token consists only of digits (use token.isdigit()), set token_types[token] to 'number'; otherwise, set token_types[token] to 'identifier'.

Compiler Design

keywords = ['if', 'else', 'while', 'return']
code_line = 'if count return 10'
tokens = code_line.split()
# Create token_types dictionary and classify tokens
# Your code here

Hint

Use token.isdigit() to check if the token is a number.

4

COMPLETION: Finalize the lexical analyzer output

Create a list called result that contains strings combining each token and its type from token_types in the format "token: type". Use a list comprehension with token iterating over tokens.

Compiler Design

keywords = ['if', 'else', 'while', 'return']
code_line = 'if count return 10'
tokens = code_line.split()
token_types = {}
for token in tokens:
    if token in keywords:
        token_types[token] = 'keyword'
    elif token.isdigit():
        token_types[token] = 'number'
    else:
        token_types[token] = 'identifier'
# Create result list with formatted token and type strings
# Your code here

Hint

Use a list comprehension with an f-string to format each token and its type.

Practice

1. What is the main job of a lexical analyzer in compiler design?

easy

A. Break input text into meaningful tokens

B. Execute the program code

C. Optimize the compiled code

D. Generate machine code

Solution

1. Step 1: Understand the role of lexical analyzer

   A lexical analyzer reads the input text and divides it into tokens like keywords, identifiers, and symbols.

2. Step 2: Differentiate from other compiler phases

   It does not execute code, optimize, or generate machine code; those are later phases.

3. Final Answer:

   Break input text into meaningful tokens -> Option A

4. Quick Check:

   Lexical analyzer = tokenization [OK]

Hint: Lexical analyzer splits text into tokens [OK]

Common Mistakes:

• Confusing lexical analysis with code execution
• Thinking it optimizes or generates machine code
• Assuming it handles syntax errors

2. Which of the following is a correct way to define a token pattern in a lexical analyzer?

easy

A. token = (123)+

B. token = [a-zA-Z_][a-zA-Z0-9_]*

C. token = {print}

D. token = 123abc

Solution

1. Step 1: Recognize token pattern syntax

   Token patterns use regular expressions; [a-zA-Z_][a-zA-Z0-9_]* matches identifiers starting with letter or underscore.

2. Step 2: Check other options for correctness

   token = (123)+ uses invalid syntax for token pattern; C uses braces incorrectly; D is not a valid pattern.

3. Final Answer:

   token = [a-zA-Z_][a-zA-Z0-9_]* -> Option B

4. Quick Check:

   Valid regex pattern = token = [a-zA-Z_][a-zA-Z0-9_]* [OK]

Hint: Token patterns use regex-like syntax starting with letters or underscore [OK]

Common Mistakes:

• Using invalid characters in token patterns
• Confusing literal strings with patterns
• Ignoring start character rules

3. Given the input string int x = 10;, what tokens will a lexical analyzer produce?

medium

A. ["int", "x=10;"]

B. ["int x", "= 10;"]

C. ["int", "x", "=", "10", ";"]

D. ["int", "x", "=", "1", "0", ";"]

Solution

1. Step 1: Identify tokens in the input

   The lexical analyzer splits by language rules: keyword 'int', identifier 'x', operator '=', number '10', and symbol ';'.

2. Step 2: Compare with options

   ["int", "x", "=", "10", ";"] correctly lists all tokens separately. Others combine tokens incorrectly or split numbers wrongly.

3. Final Answer:

   ["int", "x", "=", "10", ";"] -> Option C

4. Quick Check:

   Tokens = separate meaningful units [OK]

Hint: Tokens separate keywords, identifiers, operators, numbers, symbols [OK]

Common Mistakes:

• Combining multiple tokens into one
• Splitting numbers into digits
• Ignoring punctuation as tokens

4. Identify the error in this lexical analyzer rule: NUMBER = [0-9]+(.[0-9]+)?

medium

A. Plus (+) should be replaced with star (*)

B. Parentheses should be removed

C. Square brackets are not allowed

D. Dot (.) should be escaped as \.

Solution

1. Step 1: Analyze the regex pattern

   The dot (.) in regex means any character; to match a literal dot, it must be escaped as \.

2. Step 2: Check other parts of the pattern

   Plus (+) is correct for one or more digits; square brackets and parentheses are valid here.

3. Final Answer:

   Dot (.) should be escaped as \. -> Option D

4. Quick Check:

   Escape dot for literal period [OK]

Hint: Escape dot in regex to match literal '.' [OK]

Common Mistakes:

• Not escaping dot causing wrong matches
• Changing quantifiers unnecessarily
• Removing valid grouping parentheses

5. How can a lexical analyzer handle comments in source code effectively?

hard

A. By recognizing comment patterns and skipping them

B. By converting comments into tokens

C. By treating comments as syntax errors

D. By storing comments as identifiers

Solution

1. Step 1: Understand comment handling

   Comments are ignored by the compiler; the lexical analyzer must detect comment start and end patterns and skip their content.

2. Step 2: Evaluate other options

   Converting comments into tokens or treating them as errors is incorrect; storing as identifiers is wrong.

3. Final Answer:

   By recognizing comment patterns and skipping them -> Option A

4. Quick Check:

   Skip comments during tokenization [OK]

Hint: Detect and skip comments, don't tokenize them [OK]

Common Mistakes:

• Tokenizing comments as code
• Raising errors on comments
• Confusing comments with identifiers