Lex/Flex tool overview in Compiler Design - Mini Project: Build & Apply

Lex/Flex Tool Overview

📖 Scenario: You are learning about how computers read and understand text. Lex and Flex are tools that help break text into pieces called tokens. This is useful when making programs that read languages or commands.

🎯 Goal: Build a simple Lex/Flex file that recognizes a few words and numbers. This will show how Lex/Flex works to identify parts of text.

📋 What You'll Learn

Create a Lex/Flex file with basic token definitions

Add rules to recognize keywords and numbers

Include a section to print recognized tokens

Complete the Lex/Flex file structure correctly

💡 Why This Matters

🌍 Real World

Lex/Flex tools are used to create programs that read and understand text, such as compilers and interpreters.

💼 Career

Understanding Lex/Flex helps in jobs related to programming language development, software tools, and text processing.

1

Create the Lex/Flex file header

Start by writing the first section of the Lex/Flex file. Create a section enclosed in %{ and %} where you include the stdio.h header. This section sets up the environment for your Lex/Flex program.

Compiler Design

%{
// Include standard input-output header
// Your code here
%}

Hint

Use %{ and %} to start and end the header section. Inside, write #include <stdio.h>.

2

Add definitions for tokens

Add a definitions section after the header. Define two tokens: NUMBER as a pattern matching one or more digits ([0-9]+), and KEYWORD as the word if. Write these definitions using %option or simple pattern names.

Compiler Design

%{
#include 
%}

// Define tokens for NUMBER and KEYWORD
// Your code here

Hint

Write NUMBER [0-9]+ and KEYWORD if after the header section.

3

Write rules to recognize tokens

Create the rules section by writing patterns and actions. For the if keyword, write a rule that matches if and prints Keyword: if. For numbers, write a rule that matches digits and prints Number: followed by the matched text. Use printf and yytext to print matched text.

Compiler Design

%{
#include 
%}

NUMBER [0-9]+
KEYWORD if

%%
// Add rules to print recognized tokens
// Your code here
%%

Hint

Use %% to start and end the rules section. Write rules for if and {NUMBER} patterns with printf.

4

Add the main function to run the lexer

Add the final section with a main function. Inside main, call yylex() to start the lexer. This completes the Lex/Flex file so it can be compiled and run.

Compiler Design

%{
#include 
%}

NUMBER [0-9]+
KEYWORD if

%%
if      { printf("Keyword: if\n"); }
{NUMBER} { printf("Number: %s\n", yytext); }
%%

// Add main function to run lexer
// Your code here

Hint

Write int main() with yylex(); inside to run the lexer.

Practice

1. What is the primary purpose of the Lex/Flex tool in compiler design?

easy

A. To optimize the compiled program

B. To generate machine code from source code

C. To break input text into meaningful tokens

D. To manage memory allocation during execution

Solution

1. Step 1: Understand Lex/Flex role

   Lex/Flex is a tool used to scan input text and split it into tokens based on patterns.

2. Step 2: Differentiate from other compiler parts

   Generating machine code and optimization are done by other compiler stages, not Lex/Flex.

3. Final Answer:

   To break input text into meaningful tokens -> Option C

4. Quick Check:

   Lex/Flex = Tokenizer [OK]

Hint: Lex/Flex splits text into tokens, not code generation [OK]

Common Mistakes:

• Confusing Lex/Flex with code generator
• Thinking Lex/Flex optimizes code
• Assuming Lex/Flex manages memory

2. Which of the following is the correct way to define a pattern in a Lex/Flex file?

easy

A. action { pattern }

B. action : pattern

C. pattern -> action

D. pattern { action }

Solution

1. Step 1: Recall Lex/Flex syntax

   In Lex/Flex, patterns are written first, followed by actions enclosed in braces.

2. Step 2: Verify option formats

   Only pattern { action } matches the correct syntax: pattern { action }

3. Final Answer:

   pattern { action } -> Option D

4. Quick Check:

   Pattern before action with braces [OK]

Hint: Pattern comes first, then action in braces [OK]

Common Mistakes:

• Reversing pattern and action order
• Using colons or arrows incorrectly
• Placing action outside braces

3. Given the Lex/Flex rule:
[0-9]+ { printf("Number: %s\n", yytext); }
What will be printed when the input is 123abc?

medium

A. Number: 123abc

B. Number: 123

C. Number: abc

D. No output

Solution

1. Step 1: Understand the pattern

   The pattern [0-9]+ matches one or more digits only, so it matches '123' at the start.

2. Step 2: Analyze input processing

   Lex/Flex matches '123' and executes the action printing 'Number: 123'. The 'abc' part does not match this pattern.

3. Final Answer:

   Number: 123 -> Option B

4. Quick Check:

   Digits matched only = Number: 123 [OK]

Hint: Pattern matches digits only, stops before letters [OK]

Common Mistakes:

• Assuming whole input matches pattern
• Including letters in digit pattern
• Expecting no output for partial matches

4. Identify the error in this Lex/Flex rule:
[a-z]+ printf("Word: %s\n", yytext);

medium

A. Missing braces around the action

B. Pattern syntax is incorrect

C. Missing semicolon after pattern

D. yytext is undefined

Solution

1. Step 1: Check rule syntax

   Lex/Flex requires actions to be enclosed in braces { } after the pattern.

2. Step 2: Identify missing braces

   The given rule lacks braces around the printf action, causing a syntax error.

3. Final Answer:

   Missing braces around the action -> Option A

4. Quick Check:

   Actions need braces { } [OK]

Hint: Always put actions inside braces { } [OK]

Common Mistakes:

• Omitting braces for single actions
• Confusing pattern syntax with action syntax
• Assuming semicolon needed after pattern

5. How can you modify a Lex/Flex rule to ignore whitespace characters (spaces, tabs) while scanning input?

hard

A. [ \t\n]+ { /* do nothing to skip whitespace */ }

B. [ \t\n]+ { printf("Whitespace found\n"); }

C. [a-z]+ { /* skip whitespace here */ }

D. Whitespace cannot be ignored in Lex/Flex

Solution

1. Step 1: Understand whitespace handling

   To ignore whitespace, define a pattern matching spaces, tabs, newlines with an empty action.

2. Step 2: Check options for correct ignoring

   [ \t\n]+ { /* do nothing to skip whitespace */ } matches whitespace characters and has an empty action, effectively skipping them.

3. Final Answer:

   [ \t\n]+ { /* do nothing to skip whitespace */ } -> Option A

4. Quick Check:

   Empty action skips matched whitespace [OK]

Hint: Use empty action for whitespace pattern to skip [OK]

Common Mistakes:

• Printing message instead of skipping
• Using wrong pattern for whitespace
• Believing whitespace cannot be ignored