Bird
Raised Fist0

LLM Serving Infrastructure - Batching, KV Cache, Speculative Decoding

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Batching with KV Cache vs Speculative Decoding

In a high-throughput LLM serving system with a 100ms latency SLA, would you use batching with KV cache or speculative decoding? Justify your choice.

📊
Comparison
DimensionBatching with KV CacheSpeculative Decoding
ThroughputHighMedium
LatencyMediumLow
Resource EfficiencyMediumLow
ComplexityHighMedium
ScalabilityHighMedium
✅
Verdict
Default to batching with KV cache for high-throughput systems with consistent request patterns and latency SLAs above 50ms. Switch to speculative decoding when latency is critical and request patterns are variable.
💡
Key Insight
The choice between batching with KV cache and speculative decoding hinges on balancing throughput and latency against request pattern consistency.
⚠️
Interview Trap
❌ Wrong: Always use batching with KV cache for better throughput.
Ignores latency requirements and request pattern variability.
✅ Consider both throughput and latency requirements, along with request pattern consistency, to choose the appropriate method.
Batching with KV Cache
"Maximizes throughput by reusing computation across requests."
Weaknesses
  • Increased latency due to batch formation time.
  • Complexity in managing KV cache consistency.
  • Less effective when request patterns are highly variable.
  • Requires significant memory for storing KV pairs.
Ideal When
  • n > 10K requests per second
  • latency SLA > 50ms
  • consistent request patterns
  • sufficient memory resources available
OpenAI uses batching with KV cache to efficiently serve GPT-3 requests, optimizing for high throughput.
Speculative Decoding
"Reduces latency by preemptively generating multiple potential outputs."
Weaknesses
  • Higher computational cost due to speculative paths.
  • Increased complexity in managing speculative branches.
  • Potential for wasted computation if speculative paths are incorrect.
  • Requires careful tuning to balance speculation and accuracy.
Ideal When
  • latency SLA < 50ms
  • variable request patterns
  • compute resources are abundant
  • n < 5K requests per second
Google uses speculative decoding in its search autocomplete to provide instant results with minimal latency.
🐍
Batching with KV Cache - Example
▼ Show
Python
def serve_with_batching_kv_cache(requests):
    batch = form_batch(requests)
    kv_cache = initialize_kv_cache(batch)
    responses = model_inference(batch, kv_cache)
    return responses
🐍
Speculative Decoding - Example
▼ Show
Python
def serve_with_speculative_decoding(request):
    speculative_paths = generate_speculative_paths(request)
    best_path = select_best_path(speculative_paths)
    response = model_inference(best_path)
    return response
✅
Use Batching with KV Cache When
  • •n > 10K requests per second
  • •latency SLA > 50ms
  • •consistent request patterns
  • •memory resources are sufficient
✅
Use Speculative Decoding When
  • •latency SLA < 50ms
  • •variable request patterns
  • •compute resources are abundant
  • •n < 5K requests per second
↳
Nuances
  • Batching with KV cache can lead to higher latency if batch formation is slow.
  • Speculative decoding can waste resources if speculative paths are frequently incorrect.
  • In environments with highly variable request patterns, speculative decoding may outperform batching.
  • Batching requires careful management of KV cache to avoid stale data issues.

Practice

(1/5)
1. In a news feed system designed for personalization at scale, which component is primarily responsible for filtering content based on user preferences?
easy
A. Content Delivery Network
B. Data Warehouse
C. Recommendation Engine
D. Load Balancer

Solution

  1. Step 1: Identify the role of each component

    The Recommendation Engine is responsible for filtering and suggesting content based on user preferences.

  2. Step 2: Eliminate wrong options

    A: Content Delivery Network is for content distribution, not personalization. C: Load Balancer manages traffic, not content filtering. D: Data Warehouse stores data, not directly involved in real-time personalization.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Recommendation Engine filters content for users. [OK]
Hint: Recommendation Engine filters content for users. [OK]
Common Mistakes:
2. In an ad auction system, if the ads are not being displayed as expected, which failure mode is most likely?
medium
A. Data leakage from training to test set.
B. Latency in fetching user data.
C. Incorrect bid values being used.
D. Overfitting due to too many features.

Solution

  1. Step 1: Identify the symptom

    If ads are not displayed as expected, latency in fetching user data can cause delays, leading to missed ad opportunities.

  2. Step 2: Eliminate wrong options

    • Incorrect bid values: Affects ranking, not display.
    • Data leakage: Affects model accuracy, not display.
    • Overfitting: Affects model performance, not display timing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Latency affects ad display timing. [OK]
Hint: Latency affects ad display timing. [OK]
Common Mistakes:
3. In a hate speech detection system, should you prioritize using a deep learning model over a simpler model like logistic regression when computational resources are limited?
medium
A. Yes, deep learning models always perform better.
B. Yes, because deep learning models require less feature engineering.
C. No, simpler models are more interpretable and efficient.
D. No, because simpler models cannot handle large datasets.

Solution

  1. Step 1: Evaluate resource constraints

    When computational resources are limited, simpler models like logistic regression are more efficient and interpretable.

  2. Step 2: Eliminate wrong options

    Deep learning models may perform better but require more resources, and simpler models can handle large datasets with proper techniques.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Limited resources = Simpler models [OK]
Hint: Limited resources = Simpler models [OK]
Common Mistakes:
4. When designing a session-based recommendation system, why might you choose a Transformer model over an RNN?
medium
A. RNNs require less data preprocessing.
B. Transformers are faster for small datasets.
C. RNNs are more interpretable.
D. Transformers handle long-range dependencies better.

Solution

  1. Step 1: Compare model capabilities

    Transformers are known for effectively capturing long-range dependencies due to their attention mechanism.

  2. Step 2: Eliminate incorrect reasons

    Transformers are not necessarily faster for small datasets; RNNs are not more interpretable or require less preprocessing.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Transformers for long-range dependencies. [OK]
Hint: Transformers for long-range dependencies. [OK]
Common Mistakes:
5. As the user base scales 10x, which component of a DLRM-based CTR prediction system is most likely to become a bottleneck?
hard
A. Data preprocessing pipeline
B. Training data storage capacity
C. Model inference latency
D. Embedding table size

Solution

  1. Step 1: Consider scaling effects on components.

    Embedding tables grow with user and feature diversity.

  2. Step 2: Eliminate wrong options.

    Preprocessing (A) and inference (B) scale with compute resources. Storage (D) is less critical than real-time constraints.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Embedding tables grow with scale. [OK]
Hint: Embedding tables grow with scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard