Caching strategies for LLMs in Prompt Engineering / GenAI - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of caching in Large Language Models (LLMs)?

Caching in LLMs is used to store previous computations or outputs to speed up future requests, reducing response time and saving computational resources.

intermediate

Explain token-level caching in LLMs.

Token-level caching saves the hidden states or outputs for each token generated so that when generating the next token, the model can reuse these cached states instead of recomputing from scratch.

intermediate

What is the difference between short-term and long-term caching in LLMs?

Short-term caching stores recent computations during a single session or request to speed up immediate next steps, while long-term caching saves outputs or embeddings across sessions to reuse for repeated queries or similar inputs.

beginner

How does caching help reduce latency in LLM applications?

By reusing previously computed results, caching avoids repeating expensive calculations, which lowers the time the model takes to respond, thus reducing latency for users.

advanced

Name a challenge when implementing caching strategies for LLMs.

One challenge is managing cache invalidation, ensuring that cached data stays relevant and accurate when inputs or model parameters change.

What does token-level caching store in LLMs?

A Hidden states of tokens generated

B Raw input text

C Final output only

D Model weights

Token-level caching stores the hidden states of tokens to reuse during next token generation.

Which caching type is used to speed up repeated queries across sessions?

A Short-term caching

B Long-term caching

C Token-level caching

D No caching

Long-term caching saves outputs or embeddings across sessions for repeated queries.

Why is cache invalidation important in LLM caching?

A To keep cached data accurate and relevant

B To increase cache size

C To speed up training

D To reduce model size

Cache invalidation ensures that outdated or incorrect cached data is removed or updated.

Caching in LLMs primarily helps to:

A Increase model size

B Add more training data

C Reduce response time

D Change model architecture

Caching reduces response time by reusing previous computations.

Which of the following is NOT a benefit of caching in LLMs?

A Lower latency

B Reduced computation cost

C Faster response for repeated inputs

D Improved model accuracy

Caching improves speed and cost but does not directly improve model accuracy.

Describe how token-level caching works in Large Language Models and why it is useful.

Think about how the model generates text one token at a time.

Explain the challenges involved in managing cache invalidation for LLM caching strategies.

Consider what happens if the model or input changes but the cache is not refreshed.

Practice

1. What is the main purpose of caching in large language models (LLMs)?

easy

A. To save previous answers and avoid repeating work

B. To increase the size of the model

C. To change the model's training data

D. To make the model forget old information

Solution

1. Step 1: Understand caching concept

   Caching stores previous results so the system can reuse them instead of recalculating.

2. Step 2: Apply to LLMs context

   In LLMs, caching saves time and resources by reusing answers for repeated inputs.

3. Final Answer:

   To save previous answers and avoid repeating work -> Option A

4. Quick Check:

   Caching = Save and reuse answers [OK]

Hint: Caching means saving past answers to reuse them [OK]

Common Mistakes:

• Thinking caching changes model size
• Confusing caching with training data updates
• Believing caching deletes old info

2. Which Python tool is commonly used for simple caching in LLM applications?

easy

A. os.listdir

B. functools.lru_cache

C. math.sqrt

D. random.shuffle

Solution

1. Step 1: Identify caching tools in Python

   functools.lru_cache is a built-in decorator for caching function results.

2. Step 2: Check other options

   random.shuffle shuffles lists, math.sqrt calculates square roots, os.listdir lists files; none cache results.

3. Final Answer:

   functools.lru_cache -> Option B

4. Quick Check:

   Python caching tool = lru_cache [OK]

Hint: lru_cache is Python's simple caching decorator [OK]

Common Mistakes:

• Choosing random.shuffle as caching
• Confusing math functions with caching
• Picking file system functions

3. Given this Python code using a dictionary cache for LLM responses, what will be printed?

cache = {}
def get_response(input_text):
    if input_text in cache:
        return cache[input_text]
    response = f"Answer for {input_text}"
    cache[input_text] = response
    return response

print(get_response('hello'))
print(get_response('hello'))

medium

A. None\nAnswer for hello

B. Answer for hello\nNone

C. Error: KeyError

D. Answer for hello\nAnswer for hello

Solution

1. Step 1: Analyze first call get_response('hello')

   Cache is empty, so it creates 'Answer for hello', stores it, and returns it.

2. Step 2: Analyze second call get_response('hello')

   Input is in cache, so it returns cached 'Answer for hello' without recomputing.

3. Final Answer:

   Answer for hello Answer for hello -> Option D

4. Quick Check:

   Cache hit returns saved answer [OK]

Hint: Cache returns saved answer on repeated input [OK]

Common Mistakes:

• Assuming second call returns None
• Expecting error on repeated key
• Thinking cache clears automatically

4. This code tries to cache LLM outputs but has a bug. What is the error?

cache = {}
def get_response(input_text):
    if input_text in cache:
        return cache[input_text]
    response = f"Answer for {input_text}"
    cache = {input_text: response}
    return response

print(get_response('test'))
print(get_response('test'))

medium

A. Cache is reset each call, losing previous entries

B. generate_answer function is undefined

C. Syntax error in dictionary assignment

D. Infinite recursion in get_response

Solution

1. Step 1: Check cache update line

   cache = {input_text: response} replaces whole cache dict, losing old data.

2. Step 2: Understand effect on repeated calls

   Each call resets cache, so repeated inputs are not cached properly.

3. Final Answer:

   Cache is reset each call, losing previous entries -> Option A

4. Quick Check:

   Cache replaced, not updated [OK]

Hint: Use cache[key] = value to update, not assign new dict [OK]

Common Mistakes:

• Thinking generate_answer is missing
• Assuming syntax error in dict
• Believing recursion happens

5. You want to cache partial results of LLM calls to speed up responses when inputs share common prefixes. Which caching strategy best fits this need?

hard

A. Use random sampling to cache some inputs

B. Cache only full input strings as dictionary keys

C. Use a trie (prefix tree) to store cached outputs by input prefixes

D. Clear cache after every call to save memory

Solution

1. Step 1: Understand prefix sharing in inputs

   Inputs sharing prefixes can reuse partial results if cached by prefix.

2. Step 2: Identify suitable data structure

   A trie (prefix tree) efficiently stores and retrieves data by prefixes, ideal for this case.

3. Final Answer:

   Use a trie (prefix tree) to store cached outputs by input prefixes -> Option C

4. Quick Check:

   Prefix caching = trie structure [OK]

Hint: Trie caches shared prefixes efficiently [OK]

Common Mistakes:

• Caching only full inputs misses prefix reuse
• Random caching is inefficient
• Clearing cache wastes saved data