Caching strategies for LLMs in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Cache Hit in LLMs

In the context of caching strategies for large language models (LLMs), what does a cache hit mean?

A The requested output is found in the cache, so the model can reuse it without recomputing.

B The model fails to find any relevant data in the cache and must query an external database.

C The model generates a new response from scratch without using cached data.

D The cache is full and cannot store any more outputs.

💡 Hint

Think about what happens when the model can reuse previous results quickly.

A cache hit means the model finds the requested output already stored in the cache, so it can return it immediately without extra computation.

❓ Model Choice

intermediate

Choosing a Cache Type for LLMs

Which caching strategy is best suited for storing frequently requested LLM outputs to reduce latency?

A Least Recently Used (LRU) cache, which evicts the least recently accessed outputs when full.

B Write-back cache, where outputs are stored only in cache and written back later to main storage.

C Write-through cache, where every output is immediately written to the main storage and cache.

D Random replacement cache, which evicts random outputs when the cache is full.

💡 Hint

Think about which method keeps the most useful outputs available.

LRU cache keeps the most recently used outputs, which are more likely to be requested again, reducing latency effectively.

❓ Metrics

advanced

Evaluating Cache Effectiveness

Given an LLM caching system, which metric best measures how often the cache successfully provides outputs without recomputation?

A Throughput, the number of outputs generated per second.

B Latency, the time taken to generate outputs from scratch.

C Cache miss rate, the percentage of requests not found in cache.

D Cache hit rate, the percentage of requests found in cache.

💡 Hint

This metric shows how often the cache helps avoid extra work.

Cache hit rate measures the fraction of requests served directly from cache, indicating cache effectiveness.

🔧 Debug

advanced

Identifying Cache Invalidation Issue

An LLM caching system returns outdated responses after the underlying model is updated. What is the most likely cause?

A The model is overfitting to cached outputs.

B Cache hit rate is too low, causing frequent recomputations.

C Cache invalidation is not properly implemented, so old outputs remain in cache.

D The cache size is too large, causing slow lookups.

💡 Hint

Think about what happens when cached data does not reflect model changes.

Without proper cache invalidation, the system keeps serving outdated outputs even after model updates.

❓ Predict Output

expert

Output of LRU Cache Simulation Code

What is the output of this Python code simulating an LRU cache for LLM outputs?

Prompt Engineering / GenAI

from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity):
        self.cache = OrderedDict()
        self.capacity = capacity

    def get(self, key):
        if key not in self.cache:
            return -1
        self.cache.move_to_end(key)
        return self.cache[key]

    def put(self, key, value):
        if key in self.cache:
            self.cache.move_to_end(key)
        self.cache[key] = value
        if len(self.cache) > self.capacity:
            self.cache.popitem(last=False)

cache = LRUCache(2)
cache.put('a', 'output1')
cache.put('b', 'output2')
print(cache.get('a'))
cache.put('c', 'output3')
print(cache.get('b'))
print(cache.get('c'))

A

-1
output2
output3

B

output1
-1
output3

C

output1
output2
output3

D

-1
-1
output3

💡 Hint

Remember that the cache capacity is 2 and least recently used items get removed.

Initially, 'a' and 'b' are cached. Accessing 'a' moves it to recent. Adding 'c' evicts 'b'. So 'b' returns -1, 'a' and 'c' return their outputs.