Bird
Raised Fist0

Design a Real-Time ML Inference System - Model Serving, Shadow Mode, Rollback

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
API Gateway

Receives client request

Accepts incoming request and forwards to Model Serving Cluster

Protocol:WebSocket
BeforeClient request is pending at API Gateway
AfterRequest forwarded to Model Serving Cluster via WebSocket
Model Serving Cluster

Performs real-time inference

Processes request using current production model and returns result

Protocol:gRPC
BeforeRequest received from API Gateway, no inference done
AfterInference result computed and sent back to API Gateway
Logging Service

Logs inference request and result

Receives inference data asynchronously for auditing and analysis

Protocol:Kafka
BeforeInference result generated but not logged
AfterInference data durably stored in Logging Service
Feature Store / Metadata DB

Provides feature data for inference

Serves feature vectors and metadata to Model Serving Cluster

Protocol:Internal API
BeforeModel Serving Cluster needs feature data
AfterFeature data retrieved and used for inference
Rollback Controller

Monitors model performance

Analyzes logs and triggers rollback if degradation detected

Protocol:Internal API
BeforeModel performance metrics not evaluated
AfterRollback triggered or not based on analysis
Shadow Mode Cluster

Runs parallel inference in shadow mode

Processes same request asynchronously without affecting output

Protocol:Kafka
BeforeRequest processed only by Model Serving Cluster
AfterShadow inference results logged for comparison
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│ API Gateway   │
└───────┬───────┘
        │ WebSocket
        ▼
┌───────────────┐
│ Model Serving │
│ Cluster       │
└───────┬───────┘
        │ gRPC
        ▼
┌───────────────┐
│ Logging       │
│ Service       │
└───────┬───────┘
        │ Kafka
        ▼
┌───────────────┐
│ Feature Store │
│ / Metadata DB │
└───────┬───────┘
        │
        │
        ▼
┌───────────────┐
│ Rollback      │
│ Controller    │
└───────┬───────┘
        │
        │
        ▼
┌───────────────┐
│ Shadow Mode   │
│ Cluster       │
└───────────────┘
Design Rationale

This architecture supports real-time ML inference with a Model Serving Cluster handling live requests via the API Gateway. The Shadow Mode Cluster runs parallel inference asynchronously to validate new models without impacting production results. The Logging Service captures all inference data for auditing and performance analysis. Feature Store provides necessary feature vectors to the serving cluster. The Rollback Controller continuously monitors logged metrics and can trigger rollbacks if model degradation is detected. This design ensures low-latency inference with safety nets for testing and rollback, balancing real-time responsiveness with reliability and observability.

Primary Focus

✓

Practice

(1/5)
1. A web crawler must avoid overloading a single domain by spacing requests appropriately. Which component is primarily responsible for enforcing this constraint?
easy
A. Politeness Policy Manager, which schedules requests respecting domain-specific delays
B. Scheduler, which balances load across multiple crawler instances
C. Duplicate Detection module, which filters repeated URLs to reduce load
D. URL Frontier, which stores URLs to be crawled next without timing control

Solution

  1. Step 1: Identify the role of Politeness Policy Manager

    This component enforces delays between requests to the same domain to prevent overloading servers.

  2. Step 2: Eliminate other options

    Duplicate Detection filters URLs but does not control timing; Scheduler balances load but does not enforce domain-specific delays; URL Frontier queues URLs but lacks timing logic.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Politeness Policy controls request timing per domain [check]
Hint: Politeness Policy controls request timing per domain [OK]
Common Mistakes:
2. In a distributed tracing system, at step 4 the Trace Collector receives a span with a missing parent ID. What is the most likely state of the trace at this point?
easy
A. The trace is complete and ready for visualization
B. The span is treated as a root span, starting a new trace segment
C. The span is discarded because it cannot be linked to any trace
D. The span is buffered indefinitely until the parent span arrives

Solution

  1. Step 1: Understand how tracing handles missing parent IDs

    Spans without parent IDs are treated as root spans, starting new trace trees.

  2. Step 2: Eliminate other options

    Discarding spans loses data unnecessarily; buffering indefinitely risks memory leaks; trace completeness requires all spans, so it's not complete yet.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Missing parent ID -> new root span starts trace segment [check]
Hint: Missing parent ID means new root span [check]
Common Mistakes:
3. In a relational database transaction flow, at Step 4 the transaction manager checks the lock table before committing. What happens if the lock is held by another transaction at this step?
easy
A. The transaction manager immediately aborts the transaction without retry
B. The transaction manager waits (blocks) until the lock is released or a timeout occurs
C. The transaction manager skips the lock check and commits anyway
D. The transaction manager escalates the lock to a global lock manager

Solution

  1. Step 1: Understand lock table role at Step 4

    The transaction manager uses the lock table to ensure no conflicting writes before commit.

  2. Step 2: Behavior on lock contention

    If the lock is held, the transaction manager blocks to maintain serializability, waiting or timing out rather than aborting immediately or skipping checks.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Lock contention causes blocking until release or timeout [check]
Hint: Lock held -> transaction blocks, no skipping or immediate abort [check]
Common Mistakes:
4. For a high-traffic e-commerce site needing to invalidate product pages immediately after price updates, which cache invalidation strategy is best and why?
medium
A. Use surrogate keys to invalidate all related product pages atomically
B. Use client-side cache control headers to force revalidation
C. Use short TTL to ensure freshness but accept higher cache misses
D. Use long TTL with manual purge to minimize cache misses

Solution

  1. Step 1: Identify need for immediate, atomic invalidation

    Price updates require instant invalidation of all related pages to avoid stale data.

  2. Step 2: Evaluate strategies

    Long TTL with manual purge delays invalidation; short TTL causes high cache misses; client-side headers don't guarantee server cache invalidation; surrogate keys enable atomic invalidation of related content.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Surrogate keys enable atomic, immediate invalidation of related cache [check]
Hint: Surrogate keys invalidate related content atomically [check]
Common Mistakes:
5. For real-time query processing with low latency, which architecture is better suited for serving search results, and why?
medium
A. Use a distributed file system to store raw crawled pages and query them directly
B. Use a batch indexing system that updates the index once per day for consistency
C. Use an incremental indexing system with near real-time updates to the inverted index
D. Use a centralized relational database to store all document metadata and perform queries

Solution

  1. Step 1: Compare batch vs incremental indexing

    Batch indexing has high latency, unsuitable for real-time queries. Incremental indexing supports near real-time updates.

  2. Step 2: Evaluate storage/query approaches

    Querying raw pages or centralized DBs is too slow and unscalable for low latency search serving.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Incremental indexing enables low-latency query serving [check]
Hint: Incremental indexing enables near real-time search [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard