Tokens, patterns, and lexemes in Compiler Design - Mini Project: Build & Apply

Understanding Tokens, Patterns, and Lexemes

📖 Scenario: You are learning how a simple compiler breaks down source code into meaningful pieces. This process involves identifying tokens, patterns, and lexemes from a line of code.

🎯 Goal: Build a simple representation of tokens, patterns, and lexemes from a given line of code to understand how compilers analyze source code.

📋 What You'll Learn

Create a list of code words as the data set

Define a pattern to identify keywords

Extract tokens by matching code words to the pattern

Complete the final list showing tokens with their lexemes

💡 Why This Matters

🌍 Real World

Compilers and interpreters use tokens, patterns, and lexemes to understand and translate programming code.

💼 Career

Understanding these concepts is essential for roles in compiler development, programming language design, and software tooling.

1

Create the code words list

Create a list called code_words containing these exact strings: 'int', 'main', '(', ')', '{', 'return', '0', ';', '}'.

Compiler Design

# Create the list code_words with the exact code words
# Your code here

Need a hint?

Use square brackets to create a list and include all code words as strings separated by commas.

2

Define the keyword pattern

Create a list called keywords containing these exact strings: 'int' and 'return'.

Compiler Design

code_words = ['int', 'main', '(', ')', '{', 'return', '0', ';', '}']
# Create the list keywords with 'int' and 'return'
# Your code here

Need a hint?

Keywords are special reserved words in programming. List them exactly as given.

3

Extract tokens with their types

Create a list called tokens that contains tuples for each word in code_words. Each tuple should have the word and its type: if the word is in keywords, type is 'keyword', if the word is a parenthesis or brace 'symbol', if the word is a number 'number', otherwise 'identifier'. Use a for loop with word as the iterator variable.

Compiler Design

code_words = ['int', 'main', '(', ')', '{', 'return', '0', ';', '}']
keywords = ['int', 'return']
# Create tokens list with tuples (word, type) using a for loop
# Your code here

Need a hint?

Check each word against keywords, symbols, and numbers to assign the correct token type.

4

Complete the lexemes list

Create a list called lexemes that contains only the first element (the word) from each tuple in tokens using a list comprehension.

Compiler Design

code_words = ['int', 'main', '(', ')', '{', 'return', '0', ';', '}']
keywords = ['int', 'return']
tokens = []
symbols = ['(', ')', '{', '}', ';']
for word in code_words:
    if word in keywords:
        tokens.append((word, 'keyword'))
    elif word in symbols:
        tokens.append((word, 'symbol'))
    elif word.isdigit():
        tokens.append((word, 'number'))
    else:
        tokens.append((word, 'identifier'))
# Create lexemes list with only the words from tokens
# Your code here

Need a hint?

Use a list comprehension to extract the first item from each tuple in tokens.