Bird
Raised Fist0

LLM Serving Infrastructure - Batching, KV Cache, Speculative Decoding

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
LLM Serving Infrastructure - Batching, KV Cache, Speculative Decoding
Optimize LLM serving with batching, KV caching, and speculative decoding for high throughput and low latency.
system-designseniorFreq: very_highGenAI Systems
Prerequisites:Transformer ArchitectureTokenization
🎯
Problem Framing
Business Objective

Maximize user engagement by providing fast and accurate language model responses.

ML Objective

Minimize latency and maximize throughput for LLM inference while maintaining response accuracy.

Constraints
  • P99 latency < 100ms
  • Throughput > 1000 QPS
  • Memory usage must fit within available hardware limits
  • Compliance with data privacy regulations
  • Scalability to handle peak traffic
Clarifying Questions
  • What is the expected peak QPS during traffic spikes?
  • Are there specific latency requirements for different user segments?
  • What hardware resources are available for deployment?
  • How often do model updates occur?
  • What are the data privacy and compliance requirements?
Out of Scope
✗ Model training and fine-tuning
✗ Frontend UI design
✗ Non-LLM related backend services
📊
Offline Metrics
Latency distribution (P50, P90, P99) to ensure response times meet SLA
Throughput (QPS) to measure system capacity
📈
Online Metrics
User engagement rate (e.g., session length, return visits)
System uptime and availability
⚖️
Trade-offs

Improving latency might reduce throughput if not managed properly due to resource constraints. Balancing caching strategies can help mitigate this.

💼
Business Metric

User engagement rate

🗄️
Data Sources
  • User interaction logs (real-time)
  • Pre-trained model weights (static)
  • System performance logs (real-time)
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions to adjust model serving strategies.

Data Volume

Approximately 10 TB/month of interaction logs

Freshness

Real-time data processing for interaction logs; model weights updated monthly

⚠️
Data Challenges
  • Handling large volumes of real-time data
  • Ensuring data privacy and compliance
  • Managing data drift due to changing user behavior
  • Cold start for new users with no interaction history
🔧
Key Features
embedding
User interaction history
Captures user preferences and improves response relevance.
categorical
Session context
Provides context for current user session, enhancing response accuracy.
numerical
Time of day
Adjusts model behavior based on typical user activity patterns.
categorical
Device type
Optimizes responses for different device capabilities.
numerical
Network latency
Helps in adjusting response strategies based on network conditions.
⚡
Feature Engineering Notes
Real-time vs Batch

User interaction history and session context require real-time computation, while device type and time of day can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_session_context(user_id, session_data):
    # Example feature computation
    context = {'recent_interactions': get_recent_interactions(user_id),
               'session_length': len(session_data)}
    return context
🤖
Model Selection
Baseline

Rule-based system using predefined templates

Proposed Model

Transformer-based LLM with KV caching and speculative decoding

Transformer-based LLMs provide state-of-the-art language understanding and generation capabilities. KV caching reduces redundant computations, and speculative decoding improves latency.
✗
Alternatives Rejected
RNN-based models
Insufficient for capturing long-range dependencies and slower inference times.
BERT-based models
Primarily designed for understanding tasks, not generation, and lacks efficient decoding strategies.
🏋️
Training Strategy

Offline training with periodic updates to incorporate new data and improve model accuracy.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
ThroughputTarget > 1000 QPS
ArchitectureSingle-stage with batching and KV caching to optimize for latency and throughput.
CachingCache KV pairs with a TTL of 5 minutes to balance freshness and resource usage. Invalidate cache on model updates.
FallbackUse rule-based responses or cached results when the model is unavailable.
📡
Data Drift Signals
Change in user interaction patterns
Shift in session context distributions
Network latency variations
📉
Model Drift Signals
Increase in response time
Decrease in user engagement metrics
Shift in prediction score distributions
🔔
Alerting & Retraining
Retraining Trigger

Significant drop in user engagement rate or increase in latency beyond SLA.

Alerting Strategy

On-call engineer paged for latency spikes or engagement drops; logs analyzed for root cause; auto-recovery scripts for minor issues.

🎯
Interview Angles
  • How to optimize LLM serving for low latency and high throughput?
  • What are the trade-offs between caching strategies and model freshness?
  • How to handle data and model drift in LLM serving?
⚠️
Common Mistakes
Ignoring the impact of caching on model freshness
Underestimating the complexity of speculative decoding
Failing to consider data privacy and compliance
Overlooking the need for real-time feature computation
📌
Key Decisions
  • Choosing the right model architecture for low latency
  • Implementing effective caching strategies
  • Balancing real-time and batch feature computations

Practice

(1/5)
1. For evaluating an LLM's ability to generate coherent text, which metric is most appropriate?
easy
A. Precision
B. Accuracy
C. BLEU score
D. Recall

Solution

  1. Step 1: Identify metrics for text generation

    BLEU score measures text generation quality by comparing n-grams.

  2. Step 2: Eliminate wrong options

    Accuracy, precision, and recall are more suited for classification tasks, not text coherence.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    BLEU for text coherence, not classification. [OK]
Hint: BLEU for text coherence, not classification. [OK]
Common Mistakes:
2. When deciding between a full factorial design and a fractional factorial design for an A/B test, which is more appropriate when the number of factors is large and why?
medium
A. Full factorial design, because it tests all possible combinations.
B. Fractional factorial design, because it is easier to interpret.
C. Fractional factorial design, because it reduces the number of tests needed.
D. Full factorial design, because it provides more accurate results.

Solution

  1. Step 1: Consider the number of factors.

    With a large number of factors, testing all combinations becomes impractical.

  2. Step 2: Eliminate wrong options.

    Full factorial is impractical for large factors, and fractional factorial is not necessarily easier to interpret.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Large factors need fewer tests. [OK]
Hint: Large factors need fewer tests. [OK]
Common Mistakes:
3. In an anomaly detection system using an Autoencoder, what is a likely failure mode if the model is overfitting?
medium
A. The model will detect too many false positives.
B. The model will have a high reconstruction error on the training data.
C. The model will fail to detect any anomalies.
D. The model will require more labeled data to function correctly.

Solution

  1. Step 1: Understand overfitting in Autoencoders.

    Overfitting occurs when the model learns the noise in the training data, leading to poor generalization.

  2. Step 2: Eliminate wrong options.

    The model will fail to detect any anomalies. is incorrect because the model will still detect anomalies, but with poor accuracy. The model will have a high reconstruction error on the training data. is incorrect because overfitting typically results in low reconstruction error on training data. The model will require more labeled data to function correctly. is incorrect because labeled data is not required for Autoencoders.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Overfitting = false positives increase [OK]
Hint: Overfitting = false positives increase [OK]
Common Mistakes:
4. In an ML monitoring platform, when should you prioritize retraining the model over just alerting the team?
medium
A. When there's a significant drift detected in input data
B. When the model's accuracy drops slightly
C. When the model's predictions are consistent but incorrect
D. When the alerting system triggers multiple false positives

Solution

  1. Step 1: Identify scenarios for retraining

    Retraining is crucial when there's a significant drift in input data, indicating the model may no longer be valid.

  2. Step 2: Eliminate wrong options

    Minor accuracy drops or consistent incorrect predictions may not require retraining. False positives in alerts suggest tuning the alert system, not retraining.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Significant drift = Retrain [OK]
Hint: Significant drift = Retrain [OK]
Common Mistakes:
5. When choosing between FAISS and HNSW for a vector search task, which is more suitable for handling dynamic datasets with frequent updates?
medium
A. HNSW, because it provides better precision.
B. FAISS, because it supports dynamic updates efficiently.
C. FAISS, because it has better indexing speed.
D. HNSW, because it allows for efficient updates and deletions.

Solution

  1. Step 1: Evaluate update capabilities.

    HNSW supports dynamic updates efficiently.

  2. Step 2: Eliminate wrong options.

    FAISS is not optimized for frequent updates; indexing speed and precision are not the primary concerns here.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    HNSW handles dynamic updates well. [OK]
Hint: HNSW handles dynamic updates well. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard