Bird
Raised Fist0

LLM Serving Infrastructure - Batching, KV Cache, Speculative Decoding

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🎯
Design an LLM Serving Infrastructure with Batching, KV Cache, and Speculative Decoding
45 min · senior
senior

You are tasked with designing a serving infrastructure for a large language model (LLM) that supports 10 million daily active users. The system must handle peak loads of 100,000 QPS with a P99 latency SLA of 200ms. Your team consists of 5 engineers, and you have 4 months to deliver a production-ready system. The infrastructure should efficiently utilize batching, KV caching, and speculative decoding to optimize performance.

Given Info
  • The LLM model size is 175 billion parameters.
  • The system must support both synchronous and asynchronous requests.
  • Speculative decoding should reduce latency without significantly increasing resource usage.
  • The infrastructure should be scalable to accommodate future growth.
  • You have access to a distributed GPU cluster for inference.
📋
Guided Steps
1/5
Step 1
Batching Strategy
How will you implement batching to optimize throughput while meeting latency requirements?
💡 Hint
Consider the trade-off between batch size and latency, especially during peak loads.
Key Decision
Dynamic batching because it allows adjusting batch sizes based on current load to optimize throughput without breaching latency SLA.
Interviewer Expectation
Mid: Proposes fixed-size batching. Senior: Implements dynamic batching with adaptive algorithms to adjust batch sizes in real-time based on load conditions.
⚠️Common Wrong Answer
Using a fixed batch size, which can lead to either underutilization of resources or exceeding latency SLAs during peak times.
Step 2
KV Cache Utilization
How will you leverage KV caching to improve inference speed?
💡 Hint
Think about how caching can reduce redundant computations and what data should be cached.
Key Decision
Implement KV caching for repeated queries because it reduces redundant computation and speeds up response times.
Interviewer Expectation
Mid: Suggests basic caching of recent queries. Senior: Designs a sophisticated KV cache strategy that includes cache invalidation policies and considers cache hit rates.
⚠️Common Wrong Answer
Not implementing a cache invalidation strategy, leading to stale data and incorrect responses.
Step 3
Speculative Decoding
How will you implement speculative decoding to reduce perceived latency?
💡 Hint
Consider the impact of speculative decoding on resource usage and how to manage it.
Key Decision
Use speculative decoding with a fallback mechanism because it can reduce perceived latency without significantly increasing resource usage.
Interviewer Expectation
Mid: Implements speculative decoding without fallback. Senior: Designs a fallback mechanism to handle incorrect speculations efficiently.
⚠️Common Wrong Answer
Over-relying on speculative decoding, which can lead to resource exhaustion and increased costs.
Step 4
Scalability and Resource Management
How will you ensure the system is scalable and efficiently manages resources?
💡 Hint
Think about how to handle peak loads and future growth without over-provisioning.
Key Decision
Auto-scaling with resource monitoring because it ensures efficient resource usage and scalability during peak loads.
Interviewer Expectation
Mid: Proposes manual scaling. Senior: Designs an auto-scaling strategy with real-time monitoring and predictive scaling based on usage patterns.
⚠️Common Wrong Answer
Relying solely on manual scaling, which can lead to resource wastage or unavailability during peak times.
Step 5
API Design and Data Flow
Design the API endpoints and data flow for the LLM serving infrastructure.
💡 Hint
Consider how to structure the API to support both synchronous and asynchronous requests efficiently.
Key Decision
Design RESTful API with WebSocket support for asynchronous requests because it provides flexibility and efficiency in handling different request types.
Interviewer Expectation
Mid: Designs basic RESTful API. Senior: Incorporates WebSocket for asynchronous requests and optimizes data flow for minimal latency.
⚠️Common Wrong Answer
Designing only a RESTful API without considering asynchronous request handling, leading to inefficiencies in real-time applications.
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Dynamic batching strategy that adapts to load conditions.
  • Effective KV caching with cache invalidation policies.
  • Speculative decoding with a fallback mechanism.
  • Auto-scaling strategy with real-time resource monitoring.
Good to Have - 30%
  • Sophisticated cache management considering hit rates.
  • Predictive scaling based on usage patterns.
  • WebSocket support for asynchronous requests.
  • Optimized data flow for minimal latency.
Bonus - 10%
  • Advanced adaptive algorithms for dynamic batching.
  • Innovative speculative decoding techniques that minimize resource usage.
  • Comprehensive monitoring and alerting system for resource management.
✅
Model Answer
▼ Reveal

The ideal design involves a dynamic batching strategy to optimize throughput while meeting latency requirements, effective KV caching to reduce redundant computations, and speculative decoding with a fallback mechanism to minimize perceived latency. The system should be scalable with auto-scaling and resource monitoring, and the API should support both synchronous and asynchronous requests efficiently.

Key Decisions
  • Dynamic batching because it adapts to varying load conditions, optimizing resource utilization.
  • KV caching with invalidation policies to ensure data freshness and reduce redundant computations.
  • Speculative decoding with a fallback mechanism to handle incorrect speculations efficiently.
  • Auto-scaling with real-time monitoring to manage resources effectively during peak loads.
Anti-patterns
Using fixed batch sizes that do not adapt to load changes.
Implementing caching without considering cache invalidation.
Relying too heavily on speculative decoding, leading to resource exhaustion.
Designing an API that does not efficiently handle asynchronous requests.

Practice

(1/5)
1. For a vision-language model designed to generate image captions, which metric should be prioritized to evaluate the quality of the generated captions?
easy
A. Accuracy
B. BLEU score
C. Precision
D. Recall

Solution

  1. Step 1: Understand the task requirements

    Image captioning requires evaluating the quality of generated text against reference captions.

  2. Step 2: Choose the appropriate metric

    BLEU score is specifically designed to evaluate the quality of text generation tasks like translation and captioning.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    BLEU evaluates text generation quality. [OK]
Hint: BLEU evaluates text generation quality. [OK]
Common Mistakes:
2. For a RAG system focused on improving user satisfaction, which metric should be prioritized?
easy
A. Precision
B. Recall
C. User Engagement Rate
D. F1 Score

Solution

  1. Step 1: Align metrics with business objectives.

    User satisfaction is best measured by user engagement, reflecting how users interact with the system.

  2. Step 2: Eliminate wrong options.

    Precision, recall, and F1 score are technical metrics that don't directly measure user satisfaction.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    User satisfaction = engagement. [OK]
Hint: User satisfaction = engagement. [OK]
Common Mistakes:
3. When choosing between behavioral signals and device fingerprinting for bot detection, which is more suitable for detecting sophisticated bots and why?
medium
A. Behavioral signals, because they are harder for bots to mimic.
B. Device fingerprinting, because it provides a unique identifier.
C. Behavioral signals, because they are easier to implement.
D. Device fingerprinting, because it is less prone to false positives.

Solution

  1. Step 1: Consider sophistication of bots.

    Sophisticated bots can mimic device fingerprints but struggle with human-like behavior.

  2. Step 2: Eliminate wrong options.

    Device fingerprinting is easier to spoof; implementation ease is irrelevant to detection capability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sophisticated bots struggle with human behavior [OK]
Hint: Sophisticated bots struggle with human behavior [OK]
Common Mistakes:
4. In a visual search system, what is a likely failure mode if the image embedding model is not properly trained?
medium
A. Increased latency in search results
B. Inaccurate retrieval results
C. Higher storage costs
D. Increased server downtime

Solution

  1. Step 1: Identify the impact of poor embeddings

    Poorly trained embeddings lead to inaccurate retrieval because the vector representations don't capture image semantics well.

  2. Step 2: Eliminate wrong options

    Latency, storage, and downtime are not directly affected by embedding quality.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Poor embeddings = poor retrieval. [OK]
Hint: Poor embeddings = poor retrieval. [OK]
Common Mistakes:
5. As the scale of data increases in a semantic search system using Bi-Encoders, which component is likely to fail first and why?
hard
A. Network bandwidth, due to increased data transfer
B. Document storage, due to increased storage requirements
C. Query processing, due to increased latency
D. Embedding computation, due to increased computational load

Solution

  1. Step 1: Analyze the impact of data scale on components

    As data scales, the computational load for embedding computation increases significantly.

  2. Step 2: Eliminate wrong options

    Query processing and network bandwidth are affected but not as critically as embedding computation. Document storage is a concern but not a failure point.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Scale = embedding load [check]
Hint: Scale = embedding load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard