Bird
Raised Fist0

The Latency Spike Mystery

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Latency Spike Mystery
Started 3 days ago, persisting since then.

An LLM serving infrastructure handling 500K requests per minute for a conversational AI platform.

Symptom
P99 latency increased from 90ms to 250ms, and speculative decoding success rate dropped from 85% to 60%.
📡
Signals
latency
P99 latency for LLM responses
Increased from 90ms to 250ms
metric
Speculative decoding success rate
Dropped from 85% to 60%
log
Error logs from KV cache
Increased cache misses by 40%
distribution
Batch size distribution
Shifted from average 10 to 5
code
Recent deployment changes
New batching logic deployed 3 days ago
🔬
Hypotheses
Batching logic is causing smaller batch sizes, increasing latency.
Supporting Signals
Batch size distributionRecent deployment changes
Verify: Check batch size logs before and after deployment.
df['batch_size'].plot(kind='hist', bins=20)
KV cache misses are causing increased latency.
Supporting Signals
Error logs from KV cache
Verify: Analyze cache hit/miss ratio over time.
df['cache_miss_ratio'].plot()
🎯
Root Cause
New batching logic inadvertently reduced batch sizes, increasing processing overhead per request.
Why It Happened

A recent deployment changed the batching algorithm to optimize for throughput but failed to account for latency impact.

🔧
Fix Strategy
ImmediateRollback to previous batching logic.
Short-termAdjust batching logic to maintain optimal batch sizes for latency.
Long-termImplement adaptive batching that dynamically adjusts based on current load and latency metrics.
🐍
Python Fix
Python
def adjust_batching_logic(current_load, latency):
    if latency > 100:
        return max(10, current_load // 2)
    return min(20, current_load)
✗
Common Wrong Diagnoses
▼ Show
KV cache configuration issue
Why candidates think this: Increased cache misses seem directly related to latency.
Why wrong: Cache misses were a symptom of smaller batch sizes, not the root cause.
Network latency increase
Why candidates think this: P99 latency spike suggests network issues.
Why wrong: Network metrics were stable; the issue was internal processing.
📊
Monitoring to Add
  • Real-time monitoring of batch sizes
  • Alert on sudden changes in cache miss rates
  • Latency threshold alerts for P95 and P99
📋
Process Change

Require latency impact analysis for all deployment changes.

🏗️
Architectural Change

Introduce a feedback loop in the batching system to adaptively adjust batch sizes based on real-time latency metrics.

💡
Key Insight
Batch size changes can drastically affect latency, especially in high-throughput systems.

Practice

(1/5)
1. An agent system fails to adapt to new tasks despite having a learning component. What is the most likely failure mode?
medium
A. Inadequate tool use integration
B. Insufficient memory capacity
C. Overfitting to initial tasks
D. Poor initial planning strategy

Solution

  1. Step 1: Identify symptoms

    The system fails to adapt, indicating a learning issue.

  2. Step 2: Analyze potential causes

    Overfitting to initial tasks prevents generalization to new tasks, while other options relate to different issues.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Adaptation failure = Overfitting issue [OK]
Hint: Adaptation failure often means overfitting. [OK]
Common Mistakes:
2. In a delivery forecasting system, when should you prefer using a simple linear regression model over a complex deep learning model?
medium
A. When the dataset is extremely large and complex
B. When interpretability and simplicity are more important than accuracy
C. When the data has non-linear patterns
D. When the model needs to capture intricate temporal dependencies

Solution

  1. Step 1: Consider the trade-offs between model complexity and interpretability

    Simple models are easier to interpret and explain.

  2. Step 2: Eliminate wrong options

    Complex models are needed for large, complex datasets (A), non-linear patterns (C), and intricate dependencies (D).

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Choose simplicity for interpretability. [OK]
Hint: Choose simplicity for interpretability. [OK]
Common Mistakes:
3. In a search ranking system, when should you prefer a learning-to-rank model over a traditional BM25 algorithm?
medium
A. When you have a small dataset
B. When interpretability is crucial
C. When computational resources are limited
D. When you have rich user interaction data

Solution

  1. Step 1: Assess the data and resource context

    Learning-to-rank models excel with rich interaction data.

  2. Step 2: Evaluate the constraints

    BM25 is better for small datasets, interpretability, and limited resources.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Use learning-to-rank with rich interaction data. [OK]
Hint: Use learning-to-rank with rich interaction data. [OK]
Common Mistakes:
4. A Two-Tower Model for Retrieval is underperforming. What is a likely failure mode if the user and item embeddings are not aligned?
medium
A. The similarity scores between users and items are inaccurate.
B. The model fails to generalize to unseen items.
C. The model's training time increases significantly.
D. The model overfits to the training data.

Solution

  1. Step 1: Identify the impact of misaligned embeddings

    Misaligned embeddings lead to inaccurate similarity scores, affecting retrieval performance.

  2. Step 2: Eliminate wrong options

    The model overfits to the training data. is unrelated to embedding alignment. The model fails to generalize to unseen items. is more about item diversity. The model's training time increases significantly. is not directly caused by embedding misalignment.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Misaligned embeddings cause inaccurate scores. [OK]
Hint: Misaligned embeddings cause inaccurate scores. [OK]
Common Mistakes:
5. As the number of users and ads increases tenfold in an ad auction system, which component is most likely to fail first?
hard
A. User Data Storage
B. Ad Ranking Module
C. Bid Management System
D. Network Bandwidth

Solution

  1. Step 1: Consider scale impact

    With a tenfold increase in users and ads, the User Data Storage is likely to face capacity issues first due to the sheer volume of data.

  2. Step 2: Eliminate wrong options

    • Ad Ranking Module: Computationally intensive but not storage-bound.
    • Bid Management System: Handles bids, not directly affected by user volume.
    • Network Bandwidth: Affects data transfer, but storage is a more immediate bottleneck.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Storage fails first with data volume increase. [OK]
Hint: Storage fails first with data volume increase. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard