Bird
Raised Fist0

Design a Data Infrastructure for ML - Feature Store, Training Pipeline, Registry

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/7
Kafka Cluster

Ingest feature event

New raw data event published to Kafka topic for features

Protocol:Kafka Protocol
BeforeNo new feature event in Kafka topic
AfterFeature event durably stored in Kafka topic partition
Stream Processing (Apache Flink)

Consume and transform event

Flink reads event, computes real-time features, writes to Feature Store

Protocol:Internal RPC
BeforeFeature event available in Kafka but not processed
AfterReal-time features computed and ready to store
Feature Store (Cassandra + Redis Cache)

Store features

Persist features in Cassandra and cache hot features in Redis

Protocol:Cassandra CQL + Redis GET/SET
BeforeNo new features stored for this event
AfterFeatures persisted in Cassandra and cached in Redis
Batch Pipeline (Apache Spark)

Run batch feature computation

Spark reads raw data and computes aggregated features for training

Protocol:Batch Data Access
BeforeBatch features outdated, no recent computation
AfterBatch features updated and stored in Feature Store
Model Training Orchestration

Trigger training job

Orchestrator triggers model training using latest batch features

Protocol:Job Scheduling API
BeforeNo training job running for latest features
AfterTraining job started with updated features
Model Registry (PostgreSQL + Elasticsearch)

Register new model

Store trained model metadata and index for search

Protocol:SQL + Elasticsearch API
BeforeModel registry missing latest model info
AfterNew model metadata stored and indexed
Online Inference Services

Serve model and features

Inference service loads model and fetches features from cache

Protocol:Redis GET + Model Load API
BeforeInference service running old model and stale features
AfterInference service serving latest model and fresh features
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│       Kafka Cluster            │
│  (Durable event queue for ML) │
└───────────────┬───────────────┘
                │ Kafka Protocol
                ▼
┌───────────────────────────────┐
│ Stream Processing (Flink)     │
│ (Real-time feature extraction)│
└───────────────┬───────────────┘
                │ Internal RPC
                ▼
┌───────────────────────────────┐
│ Feature Store                 │
│ (Cassandra + Redis Cache)    │
└───────────────┬───────────────┘
                │ Batch Data
                ▼
┌───────────────────────────────┐
│ Batch Pipeline (Spark)        │
│ (Offline feature computation) │
└───────────────┬───────────────┘
                │ Training Data
                ▼
┌───────────────────────────────┐
│ Model Training Orchestration  │
│ (Triggers model training jobs)│
└───────────────┬───────────────┘
                │ Model Metadata
                ▼
┌───────────────────────────────┐
│ Model Registry                │
│ (PostgreSQL + Elasticsearch) │
└───────────────┬───────────────┘
                │ Model Info
                ▼
┌───────────────────────────────┐
│ Online Inference Services     │
│ (Serve models & features)    │
└───────────────────────────────┘
Design Rationale

This architecture illustrates a data infrastructure for ML workflows, starting with feature events ingested into a Kafka cluster. Stream Processing with Apache Flink consumes these events to compute real-time features, which are stored in a Feature Store backed by Cassandra and Redis cache for low-latency access. Concurrently, a Batch Pipeline using Apache Spark computes aggregated features offline for training. The Model Training Orchestration component triggers training jobs using these batch features. Once trained, models are registered in a Model Registry built on PostgreSQL and Elasticsearch for metadata storage and search. Finally, Online Inference Services serve the latest models and features for real-time predictions, fetching cached features for low latency. This vertical flow ensures durability, scalability, and separation of concerns between real-time and batch processing.

Primary Focus

✓

Practice

(1/5)
1. A system must enforce a maximum number of API calls per user per minute to prevent abuse. Which component in an API Gateway architecture is best suited to handle this requirement?
easy
A. The API Gateway applies rate limiting before routing requests to backend services
B. The backend microservices implement rate limiting individually to distribute load
C. A centralized database tracks request counts and enforces limits synchronously
D. Clients are responsible for self-throttling their request rates

Solution

  1. Step 1: Identify rate limiting responsibility

    The API Gateway is the entry point and can efficiently enforce rate limits before requests reach backend services, preventing overload.

  2. Step 2: Eliminate other options

    The backend microservices implement rate limiting individually to distribute load spreads rate limiting, risking inconsistent enforcement and backend overload. A centralized database tracks request counts and enforces limits synchronously adds latency and complexity by synchronous DB checks. Clients are responsible for self-throttling their request rates trusts clients, which is insecure.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    API Gateway enforces rate limits upfront to protect backend [check]
Hint: Rate limiting belongs at the gateway entry point [check]
Common Mistakes:
2. A public API receives unpredictable bursts of traffic that can overwhelm backend services. Which component is best suited to protect the system from overload?
easy
A. A distributed cache like Redis to store frequent requests for faster responses
B. A rate limiter that enforces request quotas per client IP or API key
C. A load balancer that evenly distributes requests across servers
D. A message queue to buffer incoming requests before processing

Solution

  1. Step 1: Identify the protection mechanism against traffic bursts

    Distributed cache (A) improves response time but doesn't limit request rate; load balancer (C) distributes load but can't prevent overload from bursts; message queue (D) buffers but can be overwhelmed if bursts are too large.

  2. Step 2: Recognize rate limiter role

    Rate limiter (B) enforces quotas per client, directly preventing backend overload by rejecting or delaying excess requests.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Rate limiter enforces quotas to prevent overload [check]
Hint: Rate limiter controls request bursts to protect backend [check]
Common Mistakes:
3. For a system supporting multiple client types (web, mobile, IoT), which architecture best balances maintainability and client-specific optimization?
medium
A. Single API Gateway forwarding requests directly to microservices with client logic inside services
B. Separate BFF layers per client type behind a common API Gateway
C. One monolithic backend serving all clients with feature flags for client differences
D. Client apps call microservices directly, bypassing API Gateway and BFF

Solution

  1. Step 1: Evaluate client-specific optimization needs

    Separate BFF layers per client type allow tailored APIs, improving UX and reducing client complexity.

  2. Step 2: Assess maintainability

    Separate BFF layers per client type behind a common API Gateway isolates client-specific logic in BFFs, keeping microservices generic and maintainable. Single API Gateway forwarding requests directly to microservices with client logic inside services mixes client logic into services, increasing coupling. One monolithic backend serving all clients with feature flags for client differences's monolith reduces flexibility and scalability. Client apps call microservices directly, bypassing API Gateway and BFF breaks encapsulation and complicates security.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Separate BFFs isolate client logic, improving maintainability and optimization [check]
Hint: Separate BFFs isolate client logic, improving maintainability [check]
Common Mistakes:
4. For a system requiring guaranteed message processing with minimal latency, which retry strategy is best suited when transient failures occur?
medium
A. Manual retries triggered by operators after failure alerts
B. Immediate retry with exponential backoff and jitter
C. Scheduled retries using a delayed queue with fixed intervals
D. Infinite immediate retries until success or manual intervention

Solution

  1. Step 1: Analyze retry strategies for latency and reliability

    Scheduled retries with delayed queues (B) balance retry timing and system load, reducing hot loops.

  2. Step 2: Evaluate other options

    Immediate retries (A) risk overload; manual retries (C) add latency; infinite retries (D) cause resource exhaustion.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Delayed queue retries balance latency and failure handling [check]
Hint: Delayed retries avoid overload, unlike immediate or manual retries [OK]
Common Mistakes:
5. If the Kafka cluster used for buffering document update events becomes unavailable, what is the immediate failure mode in the full-text search indexing pipeline?
medium
A. The system automatically switches to a backup message queue with identical ordering guarantees
B. Document update ingestion blocks, causing the entire indexing pipeline to stall and new updates to be lost if not retried
C. The indexing workers continue processing stale events from local cache, causing outdated search results
D. The inverted index becomes corrupted due to missing update events

Solution

  1. Step 1: Understand Kafka unavailability impact

    Without Kafka, ingestion of new document updates cannot proceed, causing blocking or stalling.

  2. Step 2: Evaluate options

    Indexing workers do not have local caches of events (A) and cannot continue processing. Automatic failover to a backup queue with identical ordering (C) is a designed mitigation. Index corruption (D) does not happen immediately; missing events cause stale results but not corruption.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Failover to backup queue prevents pipeline stall [check]
Hint: Kafka outage stalls ingestion unless backup queue with ordering [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard