What tokens and context windows mean in AI for Everyone - Step-by-Step Execution

Concept Flow - What tokens and context windows mean

Input Text

↓

Split into Tokens

↓

Tokens Enter Context Window

↓

Model Processes Tokens

↓

Output Generated

Text is broken into tokens, which fit into a limited context window that the AI uses to understand and generate responses.

Execution Sample

AI for Everyone

Input: "Hello world!"
Tokens: ["Hello", " world", "!"]
Context window size: 4 tokens
Process tokens in window
Generate output

This example shows how input text is split into tokens, placed into a context window, then processed to produce output.

Analysis Table

Step	Action	Tokens in Context Window	Explanation
1	Receive input text	[]	No tokens yet, just raw text.
2	Split text into tokens	["Hello", " world", "!"]	Text split into 3 tokens.
3	Load tokens into context window	["Hello", " world", "!"]	All tokens fit since window size is 4.
4	Model processes tokens	["Hello", " world", "!"]	Model reads tokens to understand input.
5	Generate output	["Hello", " world", "!"]	Model uses tokens to create response.
6	End	[]	Processing complete.

💡 All tokens processed within context window size; output generated.

State Tracker

Variable	Start	After Step 2	After Step 3	After Step 5	Final
Input Text	"Hello world!"	"Hello world!"	"Hello world!"	"Hello world!"	"Hello world!"
Tokens	[]	["Hello", " world", "!"]	["Hello", " world", "!"]	["Hello", " world", "!"]	["Hello", " world", "!"]
Context Window	[]	[]	["Hello", " world", "!"]	["Hello", " world", "!"]	[]

Key Insights - 3 Insights

Why do we split text into tokens instead of using whole words?

Tokens can be smaller than words, like parts of words or punctuation, allowing the model to handle many languages and variations efficiently, as shown in step 2 of the execution_table.

What happens if the input has more tokens than the context window size?

The model can only see tokens up to the context window size at once, so older tokens may be dropped or truncated, limiting how much past text the model remembers, which is implied by the context window limit in the concept_flow.

Does the context window change during processing?

No, the context window size is fixed; tokens enter it together for processing, as shown in step 3 and 4 where all tokens fit inside the window.

Visual Quiz - 3 Questions

Look at the execution_table at step 2. How many tokens does the input text split into?

A 3 tokens

B 2 tokens

C 4 tokens

D 1 token

💡 Hint

Check the 'Tokens in Context Window' column at step 2 in the execution_table.

At which step does the model start processing the tokens?

A Step 5

B Step 3

C Step 4

D Step 2

💡 Hint

Look for the step where the action is 'Model processes tokens' in the execution_table.

If the context window size was 2 tokens instead of 4, what would happen at step 3?

A All 3 tokens still load

B Only first 2 tokens load into the context window

C No tokens load

D Tokens load one by one

💡 Hint

Context window size limits how many tokens fit at once, see concept_flow and execution_table step 3.

Concept Snapshot

Tokens are small pieces of text like words or parts of words.
A context window is the fixed number of tokens the AI can look at once.
Text is split into tokens, then loaded into the context window.
The AI processes tokens in the window to understand and respond.
If input is too long, tokens outside the window are not seen.
Context window size limits how much text the AI remembers at once.

Full Transcript

This concept explains how AI models handle text by breaking it into tokens, which are small pieces like words or parts of words. These tokens fit into a limited context window, which is the number of tokens the model can see and process at one time. The process starts with receiving input text, splitting it into tokens, loading those tokens into the context window, processing them to understand the input, and then generating output. The context window size limits how many tokens the model can consider at once, so if the input is longer than this size, some tokens may be ignored. This helps the AI manage memory and focus on recent or relevant parts of the text.