Implementing a lexical analyzer in Compiler Design - Step-by-Step Execution

Concept Flow - Implementing a lexical analyzer

Start Input Source Code

↓

Read Next Character

↓

Is Character Whitespace?

Yes→Ignore and Read Next

No↓

Is Character Part of Token?

No→Error or Skip

Yes↓

Build Token String

↓

Is Token Complete?

No→Read Next Character

Yes↓

Classify Token Type

↓

Output Token

↓

More Characters?

Yes→Read Next Character

No↓

End

The lexical analyzer reads characters one by one, groups them into tokens, classifies each token, and outputs tokens until the input ends.

Execution Sample

Compiler Design

input = "int x = 10;"
pos = 0
while pos < len(input):
  ch = input[pos]
  if ch.isalpha():
    token = ch
    pos += 1

This code reads characters from input, detects if a character is a letter, and starts building a token.

Analysis Table

Step	pos	ch	Condition	Action	Token Built	Output Token
1	0	i	ch.isalpha()=true	Start token with 'i'	i
2	1	n	ch.isalpha()=true	Add 'n' to token	in
3	2	t	ch.isalpha()=true	Add 't' to token	int
4	3		ch.isalpha()=false	Token complete	int	int (keyword)
5	4	x	ch.isalpha()=true	Start token with 'x'	x
6	5		ch.isalpha()=false	Token complete	x	x (identifier)
7	6	=	ch.isalpha()=false	Single char token	=	=
8	7		Whitespace	Ignore
9	8	1	ch.isdigit()=true	Start token with '1'	1
10	9	0	ch.isdigit()=true	Add '0' to token	10
11	10	;	ch.isdigit()=false	Token complete	10	10 (number)
12	11	;	ch.isalpha()=false	Single char token	;	;
13	12		pos >= len(input)	End of input

💡 pos reaches input length 12, no more characters to read

State Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	After 6	After 7	After 8	After 9	After 10	After 11	Final
pos	0	1	2	3	4	5	6	7	8	9	10	11	12
ch		i	n	t		x		=		1	0	;
token		i	in	int	int	x	x	=		1	10	10

Key Insights - 3 Insights

Why does the token 'int' complete when a space character is read?

Because the space is not part of an identifier, the lexical analyzer recognizes the token is complete at step 4 in the execution_table.

How does the analyzer handle single character tokens like '=' and ';'?

At steps 7 and 12, the analyzer treats these characters as complete tokens immediately since they are operators or punctuation.

Why is whitespace ignored and not output as a token?

Whitespace is used only to separate tokens and is skipped as shown at steps 4, 6, 8, and 13 in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table at step 3. What is the token built so far?

A i

B in

C int

D n

💡 Hint

Check the 'Token Built' column at step 3 in the execution_table.

At which step does the lexical analyzer output the token 'x'?

A Step 5

B Step 6

C Step 7

D Step 4

💡 Hint

Look for 'Output Token' column showing 'x (identifier)' in the execution_table.

If the input had no spaces, how would the token completion change?

A Tokens complete only on non-alphanumeric characters

B Tokens would never complete

C Tokens complete on whitespace only

D Tokens complete after fixed length

💡 Hint

Refer to the 'Condition' and 'Action' columns in the execution_table to see when tokens complete.

Concept Snapshot

Lexical Analyzer Basics:
- Reads input character by character
- Skips whitespace
- Groups characters into tokens
- Recognizes token types (keywords, identifiers, numbers, symbols)
- Outputs tokens sequentially
- Stops at end of input

Full Transcript

A lexical analyzer reads source code one character at a time. It skips spaces and groups letters or digits into tokens like keywords, identifiers, or numbers. When it encounters a character that can't be part of the current token, it finishes that token and outputs it. Single characters like '=' or ';' are tokens by themselves. This process repeats until all input is read. The execution table shows each step with the current character, token being built, and when tokens are output. Variables like position and token string change as the analyzer moves through the input.