Context window and token limits in Prompt Engineering / GenAI - Full Explanation

Introduction

Imagine trying to have a long conversation but only being able to remember a few sentences at a time. This is the challenge that AI language models face with their context window and token limits. Understanding these limits helps us know how much information the AI can handle at once.

Explanation

Context Window

The context window is the amount of text the AI model can look at and understand in one go. It includes both the input you give and the AI's own responses. If the conversation or text is longer than this window, the AI might forget earlier parts.

The context window sets the maximum text length the AI can process at once.

Tokens

Tokens are small pieces of text, like words or parts of words, that the AI uses to read and write. Instead of counting letters or words, the AI counts tokens to measure text length. Different words can be one or more tokens.

Tokens are the units the AI counts to manage text length.

Token Limits

Token limits define the maximum number of tokens the AI can handle in its context window. If you exceed this limit, the AI will only consider the most recent tokens within the limit, dropping older ones. This affects how much history the AI remembers.

Token limits restrict how much text the AI can keep in memory during a conversation.

Impact on Conversations

Because of token limits, very long conversations or documents may lose earlier details as the AI forgets older tokens. This means the AI might not recall all previous information, affecting the quality of responses in long chats.

Token limits can cause the AI to forget earlier parts of long conversations.

Real World Analogy

Imagine you have a small whiteboard to write notes during a meeting. You can only fit a limited number of notes on it. When the board is full, you erase the oldest notes to make space for new ones. This way, you only remember the most recent points.

Context Window → The size of the whiteboard where notes are written

Tokens → Each individual note or bullet point written on the whiteboard

Token Limits → The maximum number of notes the whiteboard can hold before erasing old ones

Impact on Conversations → Forgetting older notes when the whiteboard is full, so only recent points are visible

Diagram

Diagram

┌───────────────────────────────┐
│        Context Window          │
│ ┌───────────────┐             │
│ │ Token 1       │             │
│ │ Token 2       │             │
│ │ Token 3       │             │
│ │ ...           │             │
│ │ Token N       │             │
│ └───────────────┘             │
│ (Max tokens allowed)           │
└───────────────────────────────┘

Older tokens → dropped when limit exceeded → only recent tokens kept

This diagram shows the context window as a container holding tokens up to a maximum limit, with older tokens removed when the limit is exceeded.

Key Facts

Context Window → The maximum amount of text the AI model can process at once.

Token → A small piece of text, like a word or part of a word, used by AI to count text length.

Token Limit → The maximum number of tokens the AI can handle in its context window.

Token Overflow → When the input exceeds the token limit, causing older tokens to be dropped.

Memory Loss in AI → The AI forgetting earlier parts of a conversation due to token limits.

Common Confusions

Thinking tokens are the same as words.

Thinking tokens are the same as words. Tokens can be whole words or parts of words; some words split into multiple tokens.

Believing the AI remembers everything from the start of a conversation.

Believing the AI remembers everything from the start of a conversation. The AI only remembers tokens within its context window; older parts are forgotten if limits are exceeded.

Assuming token limits only apply to user input.

Assuming token limits only apply to user input. Token limits include both user input and AI-generated responses combined.

Summary

The context window limits how much text the AI can process at once, measured in tokens.

Tokens are small text pieces that the AI counts to manage input and output length.

When token limits are exceeded, the AI forgets older text, affecting long conversations.

Practice

1. What does the context window in a language model refer to?

easy

A. The speed at which the model generates text

B. The maximum amount of text the model can process at once

C. The number of layers in the model

D. The size of the model's vocabulary

Solution

1. Step 1: Understand the term 'context window'

   The context window is the chunk of text the model reads at one time.

2. Step 2: Relate to model processing limits

   The model cannot process more text than this window size at once.

3. Final Answer:

   The maximum amount of text the model can process at once -> Option B

4. Quick Check:

   Context window = max text processed [OK]

Hint: Context window means max text input size [OK]

Common Mistakes:

• Confusing context window with model layers
• Thinking it relates to speed
• Mixing it with vocabulary size

2. Which of the following is the correct way to check if input text fits within a model's token limit in Python?

easy

A. if len(tokenizer.encode(text)) <= token_limit:

B. if len(text) <= token_limit:

C. if len(text.split()) <= token_limit:

D. if text.length <= token_limit:

Solution

1. Step 1: Understand token counting

   Tokens are pieces of text, not just characters or words, so we must use the tokenizer.

2. Step 2: Use tokenizer to encode text

   Using tokenizer.encode(text) gives the token list; its length is token count.

3. Final Answer:

   if len(tokenizer.encode(text)) <= token_limit: -> Option A

4. Quick Check:

   Use tokenizer.encode() to count tokens [OK]

Hint: Use tokenizer.encode() to count tokens, not len(text) [OK]

Common Mistakes:

• Counting characters instead of tokens
• Counting words by splitting text
• Using incorrect syntax like text.length

3. Given a model with a token limit of 10, what will be the output of this Python code snippet?

text = "Hello world! This is AI."
tokens = tokenizer.encode(text)
print(len(tokens) <= 10)

medium

A. Error: tokenizer not defined

B. False

C. True

D. 10

Solution

1. Step 1: Check for defined variables

   The code uses tokenizer.encode(text), but tokenizer is not defined or imported.

2. Step 2: Trace execution

   Execution stops at tokens = tokenizer.encode(text) with NameError: name 'tokenizer' is not defined. No output is printed.

3. Final Answer:

   Error: tokenizer not defined -> Option A

4. Quick Check:

   Undefined tokenizer causes NameError [OK]

Hint: Check for undefined variables like tokenizer [OK]

Common Mistakes:

• Assuming tokens equal words
• Ignoring tokenizer definition
• Confusing output with token count

4. You have a model with a 50-token limit. This code throws an error. What is the likely cause?

input_text = "A very long text..."  # over 100 tokens
tokens = tokenizer.encode(input_text)
if len(tokens) > 50:
    model.generate(tokens)

medium

A. The input tokens exceed the model's token limit

B. The tokenizer.encode() function is missing parentheses

C. The if condition should be len(tokens) < 50

D. The model.generate() function cannot accept tokens directly

Solution

1. Step 1: Trace code execution flow

   Input exceeds 100 tokens, so len(tokens) > 50 is True and model.generate(tokens) executes.

2. Step 2: Check model.generate() input type

   Usually, model.generate() expects input_ids as a tensor, not raw token list from encode(), causing TypeError.

3. Final Answer:

   The model.generate() function cannot accept tokens directly -> Option D

4. Quick Check:

   model.generate() needs tensor input_ids, not list [OK]

Hint: model.generate() expects text, not token list [OK]

Common Mistakes:

• Assuming generate accepts tokens directly
• Ignoring correct token limit check
• Misreading if condition logic

5. You want to send a long document to a language model with a 1000-token limit. Which approach best ensures the model processes the entire document without errors?

hard

A. Only send the first 100 tokens to reduce load

B. Send the whole document at once and hope the model truncates it correctly

C. Split the document into chunks of 1000 tokens or less and process each separately

D. Increase the model's token limit by changing its architecture

Solution

1. Step 1: Understand token limit constraints

   The model cannot process more than 1000 tokens at once, so input must fit this limit.

2. Step 2: Choose a method to handle long text

   Splitting the document into chunks under 1000 tokens ensures all parts are processed without errors.

3. Step 3: Evaluate other options

   Sending all at once risks truncation; sending only 100 tokens loses data; changing architecture is not feasible.

4. Final Answer:

   Split the document into chunks of 1000 tokens or less and process each separately -> Option C

5. Quick Check:

   Chunking long text fits token limits [OK]

Hint: Split long text into token-sized chunks [OK]

Common Mistakes:

• Sending too long text at once
• Ignoring most of the document
• Thinking token limit can be changed easily