Caching strategies for LLMs in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Caching strategies for LLMs

This pipeline shows how caching helps large language models (LLMs) work faster by saving and reusing parts of their work instead of repeating it.

Data Flow - 7 Stages

1Input Text

1 prompt string→User provides a text prompt to the LLM→1 prompt string

"What is the weather today?"

↓

2Tokenization

1 prompt string→Convert text into tokens (small pieces)→1 prompt token list (e.g., 6 tokens)

["What", "is", "the", "weather", "today", "?"]

↓

3Cache Lookup

1 prompt token list→Check if tokens or partial results are already saved in cache→Cache hit or miss with cached token embeddings or empty

Cache hit for tokens ["What", "is"]

↓

4Embedding Computation

Tokens not in cache (e.g., 4 tokens)→Compute token embeddings for new tokens→Embedding vectors for new tokens (e.g., 4 vectors)

Computed embeddings for ["the", "weather", "today", "?"]

↓

5Cache Update

New token embeddings→Save new embeddings into cache for future reuse→Updated cache with new embeddings

Cache now stores embeddings for ["What", "is", "the", "weather", "today", "?"]

↓

6Model Inference

Full token embeddings (cached + new)→Run LLM layers to generate output tokens→Output token probabilities

Model predicts next word probabilities

↓

7Output Generation

Output token probabilities→Convert probabilities to text tokens and join→Generated text string

"It is sunny today."

Training Trace - Epoch by Epoch

Loss
2.5 |****
2.0 |*** 
1.5 |**  
1.0 |*   
0.5 |    
     +----
      1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	2.3	0.15	Initial training with high loss and low accuracy
2	1.8	0.30	Loss decreased, accuracy improved as model learns
3	1.4	0.45	Continued improvement in loss and accuracy
4	1.1	0.60	Model converging, caching helps speed training
5	0.9	0.70	Stable decrease in loss, accuracy rising steadily

Prediction Trace - 6 Layers

Layer 1: Tokenization

Layer 2: Cache Lookup

Layer 3: Embedding Computation

Layer 4: Cache Update

Layer 5: Model Inference

Layer 6: Output Generation

Model Quiz - 3 Questions

What is the main benefit of using cache in LLMs?

A Makes the model forget old data

B Speeds up processing by reusing previous computations

C Increases the size of the model

D Changes the model architecture

Key Insight

Caching in LLMs saves time by storing and reusing token embeddings. This reduces repeated work during both training and prediction, making the model faster without changing its accuracy.

Practice

1. What is the main purpose of caching in large language models (LLMs)?

easy

A. To save previous answers and avoid repeating work

B. To increase the size of the model

C. To change the model's training data

D. To make the model forget old information

Solution

1. Step 1: Understand caching concept

   Caching stores previous results so the system can reuse them instead of recalculating.

2. Step 2: Apply to LLMs context

   In LLMs, caching saves time and resources by reusing answers for repeated inputs.

3. Final Answer:

   To save previous answers and avoid repeating work -> Option A

4. Quick Check:

   Caching = Save and reuse answers [OK]

Hint: Caching means saving past answers to reuse them [OK]

Common Mistakes:

• Thinking caching changes model size
• Confusing caching with training data updates
• Believing caching deletes old info

2. Which Python tool is commonly used for simple caching in LLM applications?

easy

A. os.listdir

B. functools.lru_cache

C. math.sqrt

D. random.shuffle

Solution

1. Step 1: Identify caching tools in Python

   functools.lru_cache is a built-in decorator for caching function results.

2. Step 2: Check other options

   random.shuffle shuffles lists, math.sqrt calculates square roots, os.listdir lists files; none cache results.

3. Final Answer:

   functools.lru_cache -> Option B

4. Quick Check:

   Python caching tool = lru_cache [OK]

Hint: lru_cache is Python's simple caching decorator [OK]

Common Mistakes:

• Choosing random.shuffle as caching
• Confusing math functions with caching
• Picking file system functions

3. Given this Python code using a dictionary cache for LLM responses, what will be printed?

cache = {}
def get_response(input_text):
    if input_text in cache:
        return cache[input_text]
    response = f"Answer for {input_text}"
    cache[input_text] = response
    return response

print(get_response('hello'))
print(get_response('hello'))

medium

A. None\nAnswer for hello

B. Answer for hello\nNone

C. Error: KeyError

D. Answer for hello\nAnswer for hello

Solution

1. Step 1: Analyze first call get_response('hello')

   Cache is empty, so it creates 'Answer for hello', stores it, and returns it.

2. Step 2: Analyze second call get_response('hello')

   Input is in cache, so it returns cached 'Answer for hello' without recomputing.

3. Final Answer:

   Answer for hello Answer for hello -> Option D

4. Quick Check:

   Cache hit returns saved answer [OK]

Hint: Cache returns saved answer on repeated input [OK]

Common Mistakes:

• Assuming second call returns None
• Expecting error on repeated key
• Thinking cache clears automatically

4. This code tries to cache LLM outputs but has a bug. What is the error?

cache = {}
def get_response(input_text):
    if input_text in cache:
        return cache[input_text]
    response = f"Answer for {input_text}"
    cache = {input_text: response}
    return response

print(get_response('test'))
print(get_response('test'))

medium

A. Cache is reset each call, losing previous entries

B. generate_answer function is undefined

C. Syntax error in dictionary assignment

D. Infinite recursion in get_response

Solution

1. Step 1: Check cache update line

   cache = {input_text: response} replaces whole cache dict, losing old data.

2. Step 2: Understand effect on repeated calls

   Each call resets cache, so repeated inputs are not cached properly.

3. Final Answer:

   Cache is reset each call, losing previous entries -> Option A

4. Quick Check:

   Cache replaced, not updated [OK]

Hint: Use cache[key] = value to update, not assign new dict [OK]

Common Mistakes:

• Thinking generate_answer is missing
• Assuming syntax error in dict
• Believing recursion happens

5. You want to cache partial results of LLM calls to speed up responses when inputs share common prefixes. Which caching strategy best fits this need?

hard

A. Use random sampling to cache some inputs

B. Cache only full input strings as dictionary keys

C. Use a trie (prefix tree) to store cached outputs by input prefixes

D. Clear cache after every call to save memory

Solution

1. Step 1: Understand prefix sharing in inputs

   Inputs sharing prefixes can reuse partial results if cached by prefix.

2. Step 2: Identify suitable data structure

   A trie (prefix tree) efficiently stores and retrieves data by prefixes, ideal for this case.

3. Final Answer:

   Use a trie (prefix tree) to store cached outputs by input prefixes -> Option C

4. Quick Check:

   Prefix caching = trie structure [OK]

Hint: Trie caches shared prefixes efficiently [OK]

Common Mistakes:

• Caching only full inputs misses prefix reuse
• Random caching is inefficient
• Clearing cache wastes saved data