Bird
Raised Fist0

Design a Real-Time ML Inference System - Model Serving, Shadow Mode, Rollback

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Real-Time ML Inference System - Model Serving, Shadow Mode, Rollback
Design a scalable, low-latency ML inference platform with shadow testing and rollback capabilities
hardAI and ML System DesignFreq: medium
Prerequisites:System Design FundamentalsCAP TheoremDistributed SystemsML Model Serving Concepts
⚙️
Functional Requirements
  • Serve ML model predictions in real-time with <100ms P99 latency
  • Support shadow mode to run new models on live traffic without affecting production results
  • Enable instant rollback to previous model versions on failure detection
  • Provide model versioning and metadata management
  • Log inference requests and responses for auditing and debugging
  • Support A/B testing and gradual rollout of new models
📊
Non-Functional Requirements
  • Handle 2 billion daily active users generating inference requests
  • Achieve 99.99% availability for inference API
  • Support 100ms P99 end-to-end latency including network and compute
  • Scale to 925K inference requests per second at peak
  • Ensure data consistency for model metadata and rollback operations
❓
Clarifying Questions to Ask
  • What is the average and peak QPS expected for inference requests?
  • Are models stateless or do they require session affinity?
  • What is the rollback granularity - per model, per version, or per request?
  • How long should shadow mode run before promoting a model?
  • What are the SLAs for model metadata consistency?
  • Is inference done on CPU, GPU, or specialized hardware?
Out of Scope
✗ Training or retraining ML models
✗ Feature engineering pipelines
✗ Offline batch inference
✗ User interface or visualization dashboards
📌
Assumptions
  • 2 billion DAU
  • 40 inference requests per user per day
  • 100 bytes average request payload
  • Inference response size ~200 bytes
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 requests/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M requests/sec
read qpsAssuming 1.5× read amplification for metadata and logging: 2.78M × 1.5 ≈ 4.17M reads/sec
💾
Storage Estimate
daily storage(925,926 × (100 + 200)) bytes × 86400 ≈ 25.9 TB/day
retention days7
total storage with replication25.9 TB × 7 days × 3 replicas ≈ 544 TB
🖥️
Server Estimate
model serving nodesAssuming 10K RPS per GPU node: 2.78M / 10K ≈ 278 GPU nodes
metadata storage nodesUsing Cassandra with 50K writes/sec per node: 2.78M / 50K ≈ 56 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Client Apps   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  API Gateway   |
          +-------+--------+
                  |
        +---------+----------+
        |                    |
        v                    v
 +------+-----+        +-----+-------+
 | Model Serving|       | Shadow Mode |
 |   Cluster   |       |  Cluster    |
 +------+-----+        +-----+-------+
        |                    |
        v                    v
 +------+--------------------+-------+
 |       Feature Store / Metadata DB    |
 +------+--------------------+-------+
        |                    |
        v                    v
 +------+-----+        +-----+-------+
 |  Logging   |        | Rollback    |
 |  Service   |        | Controller  |
 +------------+        +-------------+
🧩
Components
API Gateway
Entry point for all inference requests, routes to serving or shadow clusters
Why thisEnvoy proxy supports high throughput, low latency routing with TLS termination and observability
Why notNginx lacks native gRPC support and dynamic routing needed for shadow mode
Model Serving Cluster
Hosts production ML models, performs real-time inference with low latency
Why thisTensorFlow Serving supports model versioning, batching, and GPU acceleration for <100ms P99 latency
Why notCustom REST microservices add operational complexity and lack optimized batching
Shadow Mode Cluster
Runs new model versions on live traffic in parallel without affecting production outputs
Why thisSeparate TensorFlow Serving cluster isolates shadow inference load and enables safe testing
Why notFeature flags or in-process shadowing risk impacting production latency and stability
Feature Store / Metadata DB
Stores model metadata, feature vectors, and inference logs with strong consistency
Why thisApache Cassandra provides linear scalability and tunable consistency for 2.78M writes/sec
Why notPostgreSQL requires manual sharding and cannot handle write QPS at this scale
Logging Service
Ingests and stores inference request and response logs for auditing and debugging
Why thisKafka supports high-throughput, durable, ordered log ingestion with consumer groups for processing
Why notRabbitMQ lacks horizontal scalability and throughput for 2.78M messages/sec
Rollback Controller
Manages model version rollback and triggers traffic switch on failure detection
Why thisKubernetes operators enable declarative version control and automated rollback with minimal downtime
Why notManual rollback scripts are error-prone and slow at scale
🔄
Data Flow
1
Client Apps → API Gateway
Send inference request
Requests include model version or default to production
2
API Gateway → Model Serving Cluster
Route request for production inference
Ensures <100ms P99 latency path
3
API Gateway → Shadow Mode Cluster
Duplicate request asynchronously for shadow inference
Shadow results do not affect client response
4
Model Serving Cluster → Feature Store / Metadata DB
Log inference metadata and results
Strong consistency ensures accurate auditing
5
Shadow Mode Cluster → Logging Service
Send shadow inference logs via Kafka
Enables offline analysis and validation
6
Rollback Controller → Model Serving Cluster
Trigger rollback to previous model version
Automated rollback minimizes downtime

Practice

(1/5)
1. In the adaptive streaming request flow, at step 4 the CDN edge server checks the client's current bandwidth estimate. What does the CDN do if the bandwidth is below the threshold for HD quality?
easy
A. It immediately switches the stream to the lowest bitrate available without buffering
B. It pauses the stream and waits for the client to buffer more data
C. It serves the next video chunk at a lower bitrate from its cache if available
D. It requests the origin server to send a new manifest with lower bitrate options

Solution

  1. Step 1: Understand CDN adaptive streaming role at step 4

    The CDN edge server uses bandwidth info to decide streaming quality dynamically.

  2. Step 2: Analyze adaptive streaming behavior on low bandwidth

    Switching immediately to lowest bitrate (It immediately switches the stream to the lowest bitrate available without buffering) can cause playback stalls. Requesting new manifest (It requests the origin server to send a new manifest with lower bitrate options) is unnecessary at CDN edge. Serving lower bitrate chunk from cache (It serves the next video chunk at a lower bitrate from its cache if available) requires prior caching. The CDN typically pauses streaming to allow client buffering (It pauses the stream and waits for the client to buffer more data) to avoid playback interruption.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    CDN pauses streaming to buffer on low bandwidth [check]
Hint: Pause to buffer before switching bitrate [check]
Common Mistakes:
2. In a message delivery system, at Step 4 the Delivery Worker reads the message offset from Kafka and attempts to deliver it to the consumer. What happens if the consumer's acknowledgment is not received within the configured timeout?
easy
A. The Delivery Worker retries delivery without committing the offset
B. The Delivery Worker marks the message as delivered and commits the offset
C. The Delivery Worker discards the message and moves to the next offset
D. The Delivery Worker pauses all deliveries until manual intervention

Solution

  1. Step 1: Understand Kafka offset commit semantics

    Kafka commits offsets only after successful acknowledgment to avoid message loss.

  2. Step 2: Delivery Worker behavior on missing ack

    If no ack is received, the worker retries delivery to ensure at-least-once semantics; it does not commit offset or discard messages.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Retries without committing offset -> ensures message redelivery [check]
Hint: No ack -> retry delivery, no offset commit [OK]
Common Mistakes:
3. A company wants to decouple rapid feature deployments from the risk of impacting the entire system. Which architectural approach best supports this requirement?
easy
A. A microservices architecture with independently deployable services
B. A layered monolith with a shared database and common deployment
C. A monolithic architecture with modular codebase and feature flags
D. A serverless architecture with tightly coupled functions

Solution

  1. Step 1: Identify deployment isolation needs

    Decoupling feature deployments requires independent service lifecycles, which microservices provide.

  2. Step 2: Evaluate alternatives

    Monoliths (A and C) share deployment units, risking system-wide impact. Serverless (D) can be tightly coupled if functions share state or deployment.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Microservices enable independent deploys and reduce blast radius [check]
Hint: Independent deploys need microservices, not just modular code [check]
Common Mistakes:
4. A web application experiences increasing CPU load on a single database server, causing query latency spikes. Which scaling approach best addresses this bottleneck without changing the application code?
easy
A. Distribute the database load by sharding data across multiple servers (horizontal scaling)
B. Add more CPU cores and RAM to the existing database server (vertical scaling)
C. Use a caching layer in front of the database to reduce read load
D. Switch from SQL to NoSQL database to improve write throughput

Solution

  1. Step 1: Identify the bottleneck as CPU and memory on a single DB server

    Vertical scaling adds resources to the existing server to handle more load without architectural changes.

  2. Step 2: Evaluate horizontal scaling and other options

    Horizontal scaling requires application changes for sharding; caching reduces reads but not CPU-bound writes; switching DB types is a different trade-off.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Vertical scaling adds resources to one server without code changes [check]
Hint: Vertical scaling adds power to one machine [check]
Common Mistakes:
5. For internal service communication requiring bidirectional streaming with mobile clients that have intermittent connectivity, which approach is best?
medium
A. Use gRPC bidirectional streaming directly for real-time updates
B. Use gRPC unary calls with client-side polling and exponential backoff
C. Use WebSockets with JSON payloads for flexible streaming
D. Use REST long polling to simulate streaming behavior

Solution

  1. Step 1: Analyze client connectivity constraints

    Mobile clients often have intermittent connectivity and poor support for HTTP/2 streaming.

  2. Step 2: Evaluate streaming options

    Direct gRPC streaming (A) is unreliable on mobile. WebSockets (C) add complexity and JSON is less efficient. REST long polling (D) is less efficient and higher latency. Unary calls with polling and backoff (B) handle intermittent connectivity gracefully.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Mobile intermittent -> unary calls + polling with backoff [check]
Hint: Mobile clients -> unary calls + polling, not streaming [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard