Implementing a lexical analyzer in Compiler Design - Mini Project: Build & Apply

Implementing a lexical analyzer

📖 Scenario: You are building a simple lexical analyzer for a tiny programming language. This analyzer will read a line of code and identify tokens such as keywords, identifiers, and numbers.

🎯 Goal: Create a basic lexical analyzer that can recognize keywords, identifiers, and numbers from a given input string.

📋 What You'll Learn

Create a list of keywords for the language

Set up a sample input string representing a line of code

Write logic to split the input into tokens and classify each token

Output the tokens with their types

💡 Why This Matters

🌍 Real World

Lexical analyzers are the first step in translating programming languages. They break down code into meaningful pieces called tokens.

💼 Career

Understanding lexical analysis is essential for compiler developers, language designers, and anyone working on tools that process code.

1

DATA SETUP: Define keywords and input string

Create a list called keywords with these exact strings: 'if', 'else', 'while', 'return'. Then create a string variable called code_line with the exact value 'if count return 10'.

Compiler Design

# Create the list of keywords and the input string
# Your code here

Need a hint?

Use square brackets to create the list and quotes for strings.

2

CONFIGURATION: Prepare to split the input into tokens

Create a variable called tokens that splits the string code_line into parts separated by spaces using the split() method.

Compiler Design

keywords = ['if', 'else', 'while', 'return']
code_line = 'if count return 10'
# Split the code_line into tokens
# Your code here

Need a hint?

Use the split() method without arguments to split by spaces.

3

CORE LOGIC: Classify each token

Create an empty dictionary called token_types. Then use a for loop with variable token to go through each item in tokens. Inside the loop, use if and elif statements to classify each token as follows: if token is in keywords, set token_types[token] to 'keyword'; else if token consists only of digits (use token.isdigit()), set token_types[token] to 'number'; otherwise, set token_types[token] to 'identifier'.

Compiler Design

keywords = ['if', 'else', 'while', 'return']
code_line = 'if count return 10'
tokens = code_line.split()
# Create token_types dictionary and classify tokens
# Your code here

Need a hint?

Use token.isdigit() to check if the token is a number.

4

COMPLETION: Finalize the lexical analyzer output

Create a list called result that contains strings combining each token and its type from token_types in the format "token: type". Use a list comprehension with token iterating over tokens.

Compiler Design

keywords = ['if', 'else', 'while', 'return']
code_line = 'if count return 10'
tokens = code_line.split()
token_types = {}
for token in tokens:
    if token in keywords:
        token_types[token] = 'keyword'
    elif token.isdigit():
        token_types[token] = 'number'
    else:
        token_types[token] = 'identifier'
# Create result list with formatted token and type strings
# Your code here

Need a hint?

Use a list comprehension with an f-string to format each token and its type.