Bird
Raised Fist0

Design a Data Infrastructure for ML - Feature Store, Training Pipeline, Registry

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Data Infrastructure for ML - Feature Store, Training Pipeline, Registry
Design a scalable, consistent, and low-latency data infrastructure to support ML feature storage, training pipelines, and model registry at 2B DAU scale
hardAI and ML System DesignFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed DatabasesStream ProcessingML Pipelines
⚙️
Functional Requirements
  • Store and serve ML features with low latency for online inference
  • Ingest and process raw data streams into feature values for training and serving
  • Support batch and streaming training pipelines with consistent feature snapshots
  • Maintain a model registry with versioning, metadata, and lineage tracking
  • Provide APIs for feature retrieval, training data export, and model lifecycle management
📊
Non-Functional Requirements
  • Support 2 billion daily active users generating 40 feature updates per user per day
  • Achieve 99.99% availability for feature serving APIs with <100ms P99 latency
  • Ensure strong consistency for training data snapshots to avoid training-serving skew
  • Data retention of 90 days for feature historical data with replication for durability
  • Handle peak write throughput of up to 1.5 million writes per second
❓
Clarifying Questions to Ask
  • What is the expected feature update frequency per user or entity?
  • Are features primarily numerical, categorical, or mixed types?
  • Is the feature store required to support both online (low latency) and offline (batch) access?
  • What consistency guarantees are required between training and serving data?
  • What is the expected retention period for feature data and model metadata?
  • Are there any compliance or security requirements for data storage?
Out of Scope
✗ Designing ML model architectures or training algorithms
✗ Feature engineering logic and transformations beyond data ingestion
✗ User-facing UI or visualization tools for ML monitoring
✗ Infrastructure for model deployment and serving beyond registry metadata
📌
Assumptions
  • 2 billion DAU
  • 40 feature updates per user per day
  • Average feature update size 100 bytes
  • Retention period 90 days
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 1.5× peak factor → 925,926 × 1.5 ≈ 1,388,889 writes/sec
read qpsAssuming 10 reads per write for online serving → 9,259,260 reads/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024^3) ≈ 7.45 TB/day
retention storage gb7.45 TB × 90 days ≈ 670.5 TB
with replication gbAssuming 3x replication → 670.5 TB × 3 ≈ 2.01 PB
🖥️
Server Estimate
feature store nodesAssuming each node handles 10K writes/sec → 1,388,889 / 10,000 ≈ 139 nodes
stream processing nodesAssuming each node processes 500 MB/s → 7.45 TB/day ≈ 86 MB/s → 1-2 nodes suffice
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Data Sources  |
          +-------+--------+
                  | Raw data streams
                  v
          +----------------+
          | Kafka Cluster  |
          +-------+--------+
                  | Stream ingestion
                  v
          +----------------+     +----------------+
          | Stream Process |     | Batch Pipeline |
          | (Apache Flink) |     | (Apache Spark) |
          +-------+--------+     +-------+--------+
                  |                      |
                  | Feature updates      | Training datasets
                  v                      v
          +----------------+     +----------------+
          | Feature Store  |     | Model Registry |
          | (Cassandra +   |     | (PostgreSQL +  |
          |  Redis Cache)  |     |  Elasticsearch)|
          +-------+--------+     +-------+--------+
                  |                      |
                  | Feature serving APIs | Model metadata APIs
                  v                      v
          +----------------+     +----------------+
          | Online Inference|     | Model Training |
          | Services       |     | Orchestration  |
          +----------------+     +----------------+
🧩
Components
Kafka Cluster
Ingest raw data streams and provide durable, ordered event logs for downstream processing
Why thisKafka supports high-throughput, partitioned, replicated logs with exactly-once semantics for stream processing
Why notRabbitMQ lacks partitioning and throughput at 1M+ writes/sec scale; Kinesis is AWS-specific and less flexible
Stream Processing (Apache Flink)
Process streaming data in real-time to compute feature values and update feature store
Why thisFlink provides low-latency, stateful stream processing with exactly-once guarantees and event-time semantics
Why notSpark Streaming has higher latency and micro-batch model unsuitable for sub-second feature updates
Feature Store (Cassandra + Redis Cache)
Store feature values with high write throughput and serve low-latency reads for online inference
Why thisCassandra scales linearly for writes with tunable consistency; Redis cache provides sub-millisecond reads with TTL for freshness
Why notPostgreSQL cannot handle 1.4M writes/sec without complex sharding; DynamoDB is costly at this scale
Batch Pipeline (Apache Spark)
Generate consistent training datasets by joining historical feature data and labels for offline training
Why thisSpark supports large-scale batch processing with fault tolerance and integration with data lakes
Why notPresto is optimized for ad-hoc queries, not complex batch ETL; Flink batch mode is less mature
Model Registry (PostgreSQL + Elasticsearch)
Store model metadata, versioning, lineage, and support search over models
Why thisPostgreSQL provides ACID transactions for metadata consistency; Elasticsearch enables fast full-text search on model attributes
Why notMongoDB lacks strong consistency guarantees; Cassandra is poor for complex queries and joins
Online Inference Services
Consume features from feature store and serve ML predictions with low latency
Why thisStateless microservices can horizontally scale and cache features locally for sub-100ms P99 latency
Why notMonolithic inference services limit scalability and increase latency under load
Model Training Orchestration
Coordinate training jobs using datasets from batch pipeline and register models in registry
Why thisKubernetes-based orchestration with Argo workflows supports scalable, fault-tolerant ML pipelines
Why notAd-hoc scripts lack reliability, scalability, and lineage tracking

Practice

(1/5)
1. A web crawler must efficiently avoid re-crawling the same URLs too frequently while ensuring freshness. Which component best handles URL scheduling and politeness constraints?
easy
A. Implement a URL Frontier with priority queues and politeness policies
B. Use a distributed key-value store to store page content for freshness checks
C. Rely on the PageRank component to prioritize URLs for crawling
D. Use a message queue to distribute URLs randomly among crawler workers

Solution

  1. Step 1: Identify URL scheduling role

    The URL Frontier manages which URLs to crawl next, enforcing politeness and freshness constraints.

  2. Step 2: Evaluate other components

    PageRank ranks pages post-crawl, not for scheduling. Distributed KV stores hold content but don't schedule. Random distribution ignores politeness.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    URL Frontier enforces crawl politeness and freshness [check]
Hint: URL Frontier schedules URLs with politeness [check]
Common Mistakes:
2. In a sliding window rate limiter, at step 3 the system records a new request timestamp. What happens internally to the stored counters at this step?
easy
A. Old timestamps outside the sliding window are removed before adding the new timestamp
B. The leaky bucket drains tokens at a fixed rate before accepting the request
C. The token bucket refills tokens based on elapsed time before decrementing
D. The fixed window counter resets to zero if the current minute changed

Solution

  1. Step 1: Understand sliding window counter mechanics

    Sliding window counters maintain timestamps of recent requests and remove those outside the window to keep an accurate count.

  2. Step 2: At step 3, old timestamps beyond the window are purged before adding the new request timestamp

    This ensures the count reflects only requests within the sliding window duration.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sliding window removes expired timestamps before counting new requests [check]
Hint: Sliding window purges old timestamps before counting new ones [check]
Common Mistakes:
3. At 10x scale, which component in the News Feed system is most likely to fail first and why?
hard
A. The database, because it cannot handle the increased write volume for posts
B. The Delivery Workers, because they cannot process the exponentially larger fan-out queues fast enough
C. The cache layer, because cache invalidation traffic grows superlinearly with user count
D. The fan-out message queue (Kafka), because partition count limits prevent scaling beyond a threshold

Solution

  1. Step 1: Analyze bottlenecks at 10x scale

    Database (A) can be sharded; cache (C) scales horizontally; Kafka (D) can add partitions but has limits.

  2. Step 2: Delivery Workers face exponential fan-out growth

    Fan-out queues grow with followers; Delivery Workers must push updates to millions more followers, causing processing lag and backlog.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Delivery Workers bottleneck due to fan-out processing load [check]
Hint: Delivery Workers bottleneck on fan-out processing at scale [check]
Common Mistakes:
4. At 10x scale, which component in the Google Search architecture is most likely to become the first bottleneck, and why?
hard
A. The distributed file system storing raw crawled pages, because storage bandwidth saturates before compute
B. The crawler's URL Frontier, because it must maintain global URL ordering and politeness state, which is hard to shard
C. The query serving layer, because caching effectiveness drops and latency-sensitive lookups increase exponentially
D. The PageRank computation cluster, because the graph size grows superlinearly, increasing memory and CPU demands drastically

Solution

  1. Step 1: Analyze scaling challenges per component

    The PageRank cluster faces superlinear growth in graph edges, causing memory and CPU bottlenecks.

  2. Step 2: Compare other components

    URL Frontier can shard with heuristics; query serving scales with caching; storage bandwidth scales better than PageRank compute.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    PageRank graph growth causes first bottleneck at 10x scale [check]
Hint: PageRank graph grows superlinearly -> first bottleneck [check]
Common Mistakes:
5. At 10x scale, which component in a sliding window rate limiter is most likely to fail first and why?
hard
A. The timestamp storage, because memory usage grows linearly with request volume and causes exhaustion
B. The fixed window counter, because it cannot handle bursty traffic at scale
C. The token bucket refill scheduler, because it cannot keep up with token replenishment at high rates
D. The leaky bucket drain mechanism, because it becomes a bottleneck under bursty load

Solution

  1. Step 1: Analyze sliding window memory scaling

    Sliding window counters store timestamps for each request, so memory grows linearly with request volume.

  2. Step 2: At 10x scale, timestamp storage exhausts memory first, causing failures before other components become bottlenecks

    Fixed window counters use constant memory, token/leaky buckets have fixed state sizes.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sliding window timestamp storage grows with traffic -> memory exhaustion [check]
Hint: Sliding window memory grows with requests, fixed window constant [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard