Lex/Flex tool overview in Compiler Design - Step-by-Step Execution

Concept Flow - Lex/Flex tool overview

Write Lex/Flex rules

↓

Run Lex/Flex tool

↓

Generate C source code

↓

Compile generated code

↓

Use lexer in program

↓

Input text scanned

↓

Tokens produced for parser

The Lex/Flex tool reads rules, generates C code for a lexer, which is compiled and used to scan input text into tokens.

Execution Sample

Compiler Design

%{
#include <stdio.h>
%}
%%
[0-9]+  { printf("NUMBER\n"); }
%%

This Lex/Flex code matches numbers and prints 'NUMBER' when found.

Analysis Table

Step	Action	Input	Matched Pattern	Output/Result
1	Start scanning	123abc	None yet	No output
2	Match digits	123abc	[0-9]+	Print 'NUMBER'
3	Advance input	abc	No match	No output
4	End scanning		No more input	Stop

💡 Input fully scanned; no more tokens to produce.

State Tracker

Variable	Start	After Step 2	After Step 3	Final
Input Pointer	0 (start)	3 (after '123')	6 (after 'abc')	6 (end)
Current Token	None	NUMBER	None	None

Key Insights - 2 Insights

Why does the lexer print 'NUMBER' only after matching digits?

Because the pattern [0-9]+ matches one or more digits, as shown in execution_table step 2 where input '123' matches and triggers the action.

What happens when input characters do not match any pattern?

The lexer skips or stops scanning as in step 3, where 'abc' does not match the digit pattern, so no output is produced.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what pattern is matched?

A [A-Z]+

B [a-z]+

C [0-9]+

D No pattern matched

💡 Hint

Check the 'Matched Pattern' column at step 2 in execution_table.

At which step does the input pointer reach the end of input?

A Step 4

B Step 2

C Step 3

D Step 1

💡 Hint

Look at the 'Input Pointer' variable in variable_tracker and the 'Action' in execution_table.

If the input was '456xyz', what would be the output at step 2?

A Print 'WORD'

B Print 'NUMBER'

C No output

D Error

💡 Hint

The pattern [0-9]+ matches digits at the start, as shown in execution_table step 2.

Concept Snapshot

Lex/Flex reads pattern-action rules.
Generates C code for a lexer.
Lexer scans input text.
Matches patterns produce tokens.
Tokens used by parser in compiler.

Full Transcript

Lex/Flex is a tool used in compiler design to create lexical analyzers. The process starts by writing rules that define patterns and actions. Running Lex/Flex generates C source code for a lexer. This code is compiled and used in a program to scan input text. The lexer reads the input, matches patterns like numbers or words, and produces tokens. These tokens are then passed to the parser for further processing. For example, a rule matching digits prints 'NUMBER' when digits are found. The lexer moves through the input, matching patterns step-by-step until all input is processed.

Practice

1. What is the primary purpose of the Lex/Flex tool in compiler design?

easy

A. To optimize the compiled program

B. To generate machine code from source code

C. To break input text into meaningful tokens

D. To manage memory allocation during execution

Solution

1. Step 1: Understand Lex/Flex role

   Lex/Flex is a tool used to scan input text and split it into tokens based on patterns.

2. Step 2: Differentiate from other compiler parts

   Generating machine code and optimization are done by other compiler stages, not Lex/Flex.

3. Final Answer:

   To break input text into meaningful tokens -> Option C

4. Quick Check:

   Lex/Flex = Tokenizer [OK]

Hint: Lex/Flex splits text into tokens, not code generation [OK]

Common Mistakes:

• Confusing Lex/Flex with code generator
• Thinking Lex/Flex optimizes code
• Assuming Lex/Flex manages memory

2. Which of the following is the correct way to define a pattern in a Lex/Flex file?

easy

A. action { pattern }

B. action : pattern

C. pattern -> action

D. pattern { action }

Solution

1. Step 1: Recall Lex/Flex syntax

   In Lex/Flex, patterns are written first, followed by actions enclosed in braces.

2. Step 2: Verify option formats

   Only pattern { action } matches the correct syntax: pattern { action }

3. Final Answer:

   pattern { action } -> Option D

4. Quick Check:

   Pattern before action with braces [OK]

Hint: Pattern comes first, then action in braces [OK]

Common Mistakes:

• Reversing pattern and action order
• Using colons or arrows incorrectly
• Placing action outside braces

3. Given the Lex/Flex rule:
[0-9]+ { printf("Number: %s\n", yytext); }
What will be printed when the input is 123abc?

medium

A. Number: 123abc

B. Number: 123

C. Number: abc

D. No output

Solution

1. Step 1: Understand the pattern

   The pattern [0-9]+ matches one or more digits only, so it matches '123' at the start.

2. Step 2: Analyze input processing

   Lex/Flex matches '123' and executes the action printing 'Number: 123'. The 'abc' part does not match this pattern.

3. Final Answer:

   Number: 123 -> Option B

4. Quick Check:

   Digits matched only = Number: 123 [OK]

Hint: Pattern matches digits only, stops before letters [OK]

Common Mistakes:

• Assuming whole input matches pattern
• Including letters in digit pattern
• Expecting no output for partial matches

4. Identify the error in this Lex/Flex rule:
[a-z]+ printf("Word: %s\n", yytext);

medium

A. Missing braces around the action

B. Pattern syntax is incorrect

C. Missing semicolon after pattern

D. yytext is undefined

Solution

1. Step 1: Check rule syntax

   Lex/Flex requires actions to be enclosed in braces { } after the pattern.

2. Step 2: Identify missing braces

   The given rule lacks braces around the printf action, causing a syntax error.

3. Final Answer:

   Missing braces around the action -> Option A

4. Quick Check:

   Actions need braces { } [OK]

Hint: Always put actions inside braces { } [OK]

Common Mistakes:

• Omitting braces for single actions
• Confusing pattern syntax with action syntax
• Assuming semicolon needed after pattern

5. How can you modify a Lex/Flex rule to ignore whitespace characters (spaces, tabs) while scanning input?

hard

A. [ \t\n]+ { /* do nothing to skip whitespace */ }

B. [ \t\n]+ { printf("Whitespace found\n"); }

C. [a-z]+ { /* skip whitespace here */ }

D. Whitespace cannot be ignored in Lex/Flex

Solution

1. Step 1: Understand whitespace handling

   To ignore whitespace, define a pattern matching spaces, tabs, newlines with an empty action.

2. Step 2: Check options for correct ignoring

   [ \t\n]+ { /* do nothing to skip whitespace */ } matches whitespace characters and has an empty action, effectively skipping them.

3. Final Answer:

   [ \t\n]+ { /* do nothing to skip whitespace */ } -> Option A

4. Quick Check:

   Empty action skips matched whitespace [OK]

Hint: Use empty action for whitespace pattern to skip [OK]

Common Mistakes:

• Printing message instead of skipping
• Using wrong pattern for whitespace
• Believing whitespace cannot be ignored