Caching strategies for LLMs in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Caching strategies for LLMs

This pipeline shows how caching helps large language models (LLMs) work faster by saving and reusing parts of their work instead of repeating it.

Data Flow - 7 Stages

1Input Text

1 prompt string→User provides a text prompt to the LLM→1 prompt string

"What is the weather today?"

↓

2Tokenization

1 prompt string→Convert text into tokens (small pieces)→1 prompt token list (e.g., 6 tokens)

["What", "is", "the", "weather", "today", "?"]

↓

3Cache Lookup

1 prompt token list→Check if tokens or partial results are already saved in cache→Cache hit or miss with cached token embeddings or empty

Cache hit for tokens ["What", "is"]

↓

4Embedding Computation

Tokens not in cache (e.g., 4 tokens)→Compute token embeddings for new tokens→Embedding vectors for new tokens (e.g., 4 vectors)

Computed embeddings for ["the", "weather", "today", "?"]

↓

5Cache Update

New token embeddings→Save new embeddings into cache for future reuse→Updated cache with new embeddings

Cache now stores embeddings for ["What", "is", "the", "weather", "today", "?"]

↓

6Model Inference

Full token embeddings (cached + new)→Run LLM layers to generate output tokens→Output token probabilities

Model predicts next word probabilities

↓

7Output Generation

Output token probabilities→Convert probabilities to text tokens and join→Generated text string

"It is sunny today."

Training Trace - Epoch by Epoch

Loss
2.5 |****
2.0 |*** 
1.5 |**  
1.0 |*   
0.5 |    
     +----
      1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	2.3	0.15	Initial training with high loss and low accuracy
2	1.8	0.30	Loss decreased, accuracy improved as model learns
3	1.4	0.45	Continued improvement in loss and accuracy
4	1.1	0.60	Model converging, caching helps speed training
5	0.9	0.70	Stable decrease in loss, accuracy rising steadily

Prediction Trace - 6 Layers

Layer 1: Tokenization

Layer 2: Cache Lookup

Layer 3: Embedding Computation

Layer 4: Cache Update

Layer 5: Model Inference

Layer 6: Output Generation

Model Quiz - 3 Questions

What is the main benefit of using cache in LLMs?

A Makes the model forget old data

B Speeds up processing by reusing previous computations

C Increases the size of the model

D Changes the model architecture

Key Insight

Caching in LLMs saves time by storing and reusing token embeddings. This reduces repeated work during both training and prediction, making the model faster without changing its accuracy.