Bird
Raised Fist0

Design a Real-Time ML Inference System - Model Serving, Shadow Mode, Rollback

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Real-Time ML Inference System - Model Serving, Shadow Mode, Rollback
45 min · senior
senior

You are building a real-time ML inference system for a 50 million daily active user (DAU) startup. The system must serve predictions with P99 latency under 200ms, support encrypted data in transit and at rest, enable shadow mode for safe model rollout without impacting production traffic, and allow quick rollback to previous model versions in case of failures or regressions. The system must handle 10,000 inference requests per second at peak and integrate with existing microservices.

Constraints
  • End-to-end encryption for data in transit and at rest
  • P99 latency under 200ms for inference responses
  • Support shadow mode to run new models in parallel without affecting production outputs
  • Ability to rollback to previous model versions within seconds
  • Handle 10,000 inference requests per second at peak
  • Models can be updated frequently (multiple times per day)
📋
Guided Steps
1/5
Step 1
Design the Core Model Serving Architecture
How will you architect the real-time inference serving layer to meet 10,000 RPS with P99 < 200ms latency?
💡 Hint
Consider horizontally scalable serving frameworks with low-latency RPC and model caching.
Key Decision
Use TensorFlow Serving with gRPC endpoints behind an Envoy proxy for load balancing and TLS termination to achieve low-latency, horizontally scalable inference serving.
Interviewer Expectation
Mid-level: Names TensorFlow Serving and gRPC. Senior: Explains Envoy for TLS termination and load balancing, and how model caching reduces cold start latency.
⚠️Common Wrong Answer
Using REST over HTTP/1.1 without TLS termination - leads to higher latency and no encryption, failing P99 SLA.
Step 2
Implement Shadow Mode for Safe Model Rollout
How will you design shadow mode to run new models in parallel without impacting production outputs?
💡 Hint
Think about duplicating requests and isolating shadow predictions from production responses.
Key Decision
Use Kafka with topic partitioning keyed by request ID to duplicate inference requests to both production and shadow model serving clusters asynchronously, ensuring shadow predictions do not affect live traffic.
Interviewer Expectation
Mid-level: Mentions Kafka for asynchronous duplication. Senior: Explains partition key choice for ordering and exactly-once semantics, and how shadow results are logged for offline analysis.
⚠️Common Wrong Answer
Direct synchronous calls to shadow model in the request path - doubles latency and risks impacting production SLA.
Step 3
Design Model Versioning and Rollback Mechanism
How will you manage model versions and enable quick rollback within seconds?
💡 Hint
Consider a centralized model registry and atomic version switches in serving infrastructure.
Key Decision
Use MLflow Model Registry integrated with TensorFlow Serving’s model versioning API to atomically switch active model versions, enabling rollback by updating the serving config and triggering a hot reload.
Interviewer Expectation
Mid-level: Names MLflow for model registry. Senior: Explains atomic version switch via TensorFlow Serving APIs and how rollback latency is minimized by preloading models.
⚠️Common Wrong Answer
Redeploying entire serving cluster for rollback - causes downtime and violates rollback latency SLA.
Step 4
Ensure Data Encryption and Security Compliance
How will you ensure end-to-end encryption for inference requests and model data at rest?
💡 Hint
Look at TLS for transport and encryption at rest with key management.
Key Decision
Use Envoy proxy for TLS termination on inference requests and encrypt model artifacts at rest using AWS KMS-managed keys in S3 buckets.
Interviewer Expectation
Mid-level: Mentions TLS and encryption at rest. Senior: Details Envoy’s TLS termination, mutual TLS for service-to-service auth, and KMS key rotation policies.
⚠️Common Wrong Answer
Relying on application-layer encryption only - risks data exposure in transit or at rest.
Step 5
Design APIs for Inference, Shadow Mode Control, and Rollback
Sketch the 3 core API endpoints for inference requests, toggling shadow mode, and rolling back models.
💡 Hint
Think about REST or gRPC endpoints with authentication and versioning.
Key Decision
Expose gRPC endpoints: Predict(request), SetShadowMode(enabled), and RollbackModel(version) secured with OAuth2 tokens and versioned via protobuf schemas.
Interviewer Expectation
Mid-level: Defines basic endpoints. Senior: Explains API versioning, authentication, and idempotency for rollback calls.
⚠️Common Wrong Answer
Using unauthenticated HTTP endpoints without versioning - leads to security risks and breaking changes.
🔌
API Design Challenge
Sketch 3 core API endpoints for the ML inference system that cover: sending inference requests, enabling shadow mode, and rolling back models.
Hints
  • How does the client send inference requests and receive predictions?
  • How can shadow mode be toggled or configured via API?
  • What API allows rollback to a previous model version safely?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of TensorFlow Serving with gRPC for scalable low-latency inference
  • Kafka-based asynchronous shadow mode request duplication
  • Atomic model versioning and rollback via MLflow and TensorFlow Serving APIs
  • End-to-end encryption with Envoy TLS termination and KMS-managed keys
Good to Have - 30%
  • Preloading models to reduce cold start latency
  • Mutual TLS for service-to-service authentication
  • Logging shadow predictions for offline analysis
  • API versioning and OAuth2 authentication
Bonus - 10%
  • Use of Envoy for load balancing and observability
  • Idempotent rollback API design
  • Partition key design in Kafka for ordering guarantees
Anti-patterns to Avoid
Synchronous shadow mode calls in request path - doubles latency and risks violating SLA by increasing response time
Redeploying serving cluster for rollback - causes downtime and slow recovery, violating rollback latency requirements
Using HTTP/1.1 REST without TLS - leads to high latency and security vulnerabilities due to lack of encryption
Storing models without encryption - risks data breach and non-compliance with security standards
Polling-based shadow mode toggling - causes unnecessary load and delays, impacting system responsiveness
No API versioning - leads to breaking changes and client incompatibility, causing system instability
✅
Model Answer
▼ Reveal

The system uses TensorFlow Serving with gRPC behind Envoy for encrypted, low-latency inference. Kafka duplicates requests asynchronously for shadow mode, while MLflow Model Registry enables atomic model versioning and rollback. APIs are secured with OAuth2 and versioned via protobuf.

Key Decisions
  • Use TensorFlow Serving with gRPC endpoints behind Envoy proxy to achieve sub-200ms P99 latency and horizontal scalability
  • Implement shadow mode by duplicating inference requests asynchronously via Kafka with request ID partitioning to avoid impacting production latency
  • Manage model versions and enable rollback using MLflow Model Registry integrated with TensorFlow Serving’s atomic version switch API, allowing hot reloads without downtime
  • Ensure end-to-end encryption with Envoy TLS termination for inference requests and encrypt model artifacts at rest using AWS KMS-managed keys
  • Design gRPC APIs with OAuth2 authentication and protobuf versioning for secure, backward-compatible inference, shadow mode toggling, and rollback operations
  • TensorFlow Serving with gRPC for low-latency, scalable model serving
  • Envoy proxy for TLS termination, load balancing, and mutual TLS authentication
  • Kafka with request ID partitioning for asynchronous shadow mode request duplication
  • MLflow Model Registry integrated with TensorFlow Serving for atomic model versioning and rollback
  • AWS S3 with KMS-managed encryption keys for secure model artifact storage
  • gRPC APIs with OAuth2 authentication and protobuf versioning for inference, shadow control, and rollback
  • Preloading models in serving nodes to minimize cold start and rollback latency
  • Logging shadow mode predictions asynchronously for offline validation without impacting production

Practice

(1/5)
1. A CI/CD system must ensure that build artifacts are immutable and versioned for traceability. Which component is primarily responsible for this requirement?
easy
A. The artifact registry, which stores and versions build outputs
B. The rollback manager, which tracks previous deployment states
C. The test runner, which validates artifact correctness
D. The deployment orchestrator, which manages rollout strategies

Solution

  1. Step 1: Identify artifact immutability responsibility

    The artifact registry stores build outputs as immutable, versioned entities to ensure traceability and reproducibility.

  2. Step 2: Eliminate other components

    The deployment orchestrator manages rollout but does not store artifacts; test runner validates but does not version; rollback manager tracks deployments but not artifacts.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Artifact registry ensures immutable, versioned storage [check]
Hint: Immutable artifacts live in the registry, not deployer [check]
Common Mistakes:
2. A web crawler must avoid overloading a single domain by spacing requests appropriately. Which component is primarily responsible for enforcing this constraint?
easy
A. Politeness Policy Manager, which schedules requests respecting domain-specific delays
B. Scheduler, which balances load across multiple crawler instances
C. Duplicate Detection module, which filters repeated URLs to reduce load
D. URL Frontier, which stores URLs to be crawled next without timing control

Solution

  1. Step 1: Identify the role of Politeness Policy Manager

    This component enforces delays between requests to the same domain to prevent overloading servers.

  2. Step 2: Eliminate other options

    Duplicate Detection filters URLs but does not control timing; Scheduler balances load but does not enforce domain-specific delays; URL Frontier queues URLs but lacks timing logic.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Politeness Policy controls request timing per domain [check]
Hint: Politeness Policy controls request timing per domain [OK]
Common Mistakes:
3. In a distributed cache using consistent hashing, at Step 4 the Cache Node receives a key lookup request. What happens if the node does not have the key locally?
easy
A. The node returns a cache miss immediately without forwarding
B. The node uses the consistent hash ring to forward the request to the correct node
C. The node queries a centralized metadata service for key location
D. The node broadcasts the request to all other cache nodes

Solution

  1. Step 1: Trace key lookup at Cache Node

    If the node lacks the key, it must find the correct node responsible for that key using the consistent hash ring.

  2. Step 2: Forwarding via consistent hash ring

    The node consults the hash ring to identify the node owning the key and forwards the request there; centralized metadata (The node queries a centralized metadata service for key location) adds latency and single point of failure; broadcasting (The node broadcasts the request to all other cache nodes) is inefficient.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Consistent hash ring lookup enables correct node forwarding [check]
Hint: Cache miss triggers consistent hash ring forwarding [check]
Common Mistakes:
4. If the origin server becomes unreachable, what exactly breaks in a CDN using origin pull with TTL caching?
medium
A. Edge nodes continue serving stale cached content indefinitely without errors
B. Edge nodes serve stale content until TTL expires, then return errors for new requests
C. Edge nodes immediately return errors for all requests without cached content
D. Edge nodes switch to push mode automatically to fetch content from other origins

Solution

  1. Step 1: Understand origin pull with TTL caching failure

    When origin is unreachable, edge nodes serve cached content until TTL expires.

  2. Step 2: Identify behavior after TTL expiry

    After TTL expires, edge nodes cannot refresh content and must return errors for new requests.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Origin down + expired TTL -> edge nodes return errors [check]
Hint: Expired TTL + origin down -> errors returned [check]
Common Mistakes:
5. At 10x scale, which component in a distributed unique ID generation system is most likely to fail first and why?
hard
A. The centralized timestamp authority, because it becomes a bottleneck for clock synchronization
B. The network links between data centers, due to increased latency causing clock skew and ID collisions
C. The ID generator nodes, because sequence number overflow occurs under high request rates within the same millisecond
D. The persistent storage for sequence numbers, due to write amplification and I/O saturation

Solution

  1. Step 1: Analyze bottlenecks at 10x request rate

    High request rates increase IDs generated per millisecond, stressing sequence number capacity.

  2. Step 2: Sequence number overflow causes ID collisions or generator stalls

    Sequence number bits are limited; overflow leads to duplicates or blocking until next timestamp.

  3. Step 3: Other components less likely to fail first

    Timestamp authority is decentralized or loosely synchronized; network latency affects ordering but not immediate failure; persistent storage is often avoided for sequence state.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Sequence number overflow -> first failure under 10x scale [check]
Hint: Sequence number bits limit -> overflow first at high scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard