Bird
Raised Fist0

Design a Data Infrastructure for ML - Feature Store, Training Pipeline, Registry

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Data Infrastructure for ML - Feature Store, Training Pipeline, Registry
45 min · senior
senior

You are building a data infrastructure to support ML workflows for a startup with 50 million daily active users. The system must enable real-time and batch feature ingestion, low-latency feature retrieval for online inference, scalable training pipelines, and a centralized model registry with versioning and metadata. The infrastructure must handle high throughput, ensure data encryption at rest and in transit, and meet a P99 feature retrieval latency under 200ms.

Constraints
  • Data encryption at rest and in transit is mandatory due to regulatory compliance.
  • Support both offline batch and online real-time feature ingestion and retrieval.
  • P99 latency for online feature retrieval must be under 200ms at 50M DAU scale.
  • Training pipelines must be scalable and support incremental retraining.
  • Model registry must support versioning, metadata storage, and rollback capabilities.
📋
Guided Steps
1/5
Step 1
Designing the Feature Store Data Model and Storage
How would you design the feature store data model and choose storage technologies to support both batch and real-time feature ingestion and retrieval at scale?
💡 Hint
Consider storage optimized for time-series and key-value lookups with strong consistency and low latency.
Key Decision
Use Apache Hudi on top of Amazon S3 for offline batch feature storage to enable incremental updates and snapshot isolation, combined with Apache Cassandra for online feature store to provide low-latency key-value lookups with tunable consistency.
Interviewer Expectation
Mid-level: Names Cassandra and S3. Senior: Explains why Hudi enables incremental batch updates and snapshot isolation, and why Cassandra's LSM-tree and partition key design supports low-latency online retrieval at scale.
⚠️Common Wrong Answer
Using a single relational database like MySQL for both batch and online features, which fails at scale due to poor write throughput and high latency.
Step 2
Ensuring Low-Latency Online Feature Retrieval
How will you architect the online feature retrieval path to meet P99 latency under 200ms at 50M DAU scale?
💡 Hint
Think about caching layers, data partitioning, and network proximity to inference services.
Key Decision
Deploy a Redis cluster as a caching layer in front of Cassandra for hot features, with consistent hashing to shard keys and colocate Redis nodes near inference services to minimize network hops.
Interviewer Expectation
Mid-level: Mentions Redis caching. Senior: Explains consistent hashing for shard balancing, TTL eviction policies, and fallback to Cassandra on cache miss to maintain availability and latency SLAs.
⚠️Common Wrong Answer
Relying solely on Cassandra without caching, causing high tail latency and increased load on the database.
Step 3
Designing the Training Pipeline Data Flow
How would you design the training pipeline to efficiently consume features and labels, support incremental retraining, and scale with data volume?
💡 Hint
Consider data ingestion frameworks and orchestration tools that support incremental processing and fault tolerance.
Key Decision
Use Apache Spark on EMR for batch feature extraction and training data preparation, orchestrated by Apache Airflow for workflow management, and leverage Apache Kafka for streaming feature ingestion to enable incremental updates.
Interviewer Expectation
Mid-level: Names Spark and Airflow. Senior: Explains how Kafka enables incremental streaming ingestion, Spark's support for incremental batch processing with Hudi datasets, and Airflow's DAG retries and monitoring for fault tolerance.
⚠️Common Wrong Answer
Using ad-hoc scripts without orchestration or streaming, leading to brittle pipelines and inability to handle incremental updates.
Step 4
Implementing the Model Registry with Versioning and Metadata
How would you design the model registry to store model artifacts, metadata, support versioning, and enable rollback?
💡 Hint
Think about artifact storage, metadata databases, and APIs for model lifecycle management.
Key Decision
Use Amazon S3 for storing immutable model artifacts, combined with a PostgreSQL database for metadata and version tracking, exposing RESTful APIs for model registration, retrieval, and rollback operations.
Interviewer Expectation
Mid-level: Names S3 and relational DB. Senior: Explains why S3’s immutability and durability fit artifact storage, how PostgreSQL supports complex queries on metadata, and how API design enables atomic version updates and rollback.
⚠️Common Wrong Answer
Storing model binaries directly in the database, causing bloat and poor performance.
Step 5
Ensuring Security and Compliance
How will you ensure data encryption at rest and in transit, and secure access to the feature store and model registry?
💡 Hint
Consider encryption standards, key management, and authentication/authorization mechanisms.
Key Decision
Enable server-side encryption with AWS KMS for S3 and Cassandra data at rest, enforce TLS 1.2+ for all network communication, and integrate with AWS IAM and OAuth 2.0 for fine-grained access control and audit logging.
Interviewer Expectation
Mid-level: Mentions encryption and TLS. Senior: Details KMS key rotation policies, mutual TLS for service-to-service authentication, and RBAC with audit trails for compliance.
⚠️Common Wrong Answer
Relying on network perimeter security only, exposing data to insider threats and regulatory violations.
🔌
API Design Challenge
Sketch 3 core API endpoints for the feature store and model registry that clients and ML pipelines will use.
Hints
  • How does a client ingest and update features in the feature store?
  • How does an online service retrieve features for inference with low latency?
  • How does the training pipeline register and retrieve model metadata and versions?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of scalable storage technologies optimized for batch and online feature access
  • Caching layer design to meet strict P99 latency requirements
  • Streaming ingestion for incremental feature updates
  • Model registry supporting artifact storage and versioning
  • Security with encryption at rest and in transit
Good to Have - 30%
  • Workflow orchestration with retry and monitoring capabilities
  • Consistent hashing or sharding strategies for load balancing
  • Detailed API design for feature ingestion, retrieval, and model lifecycle
Bonus - 10%
  • Discussion of failure modes and recovery strategies
  • Consideration of multi-region deployment for availability and disaster recovery
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

The system uses Apache Hudi on Amazon S3 for batch feature storage enabling incremental updates and snapshot isolation, combined with Apache Cassandra for low-latency online feature retrieval. A Redis cluster caches hot features with consistent hashing and TTL eviction to meet P99 latency under 200ms at 50M DAU scale. Apache Kafka streams feature ingestion for incremental updates, while Apache Spark on EMR and Apache Airflow orchestrate scalable, fault-tolerant training pipelines. Model artifacts are stored immutably in Amazon S3, with PostgreSQL managing metadata, versioning, and rollback. Security is ensured via AWS KMS encryption, TLS 1.2+ for data in transit, and OAuth 2.0 with AWS IAM for access control and audit logging.

Key Decisions
  • Use Apache Hudi on S3 to enable incremental batch feature updates with snapshot isolation
  • Use Apache Cassandra with partition key design for low-latency online feature retrieval
  • Deploy Redis cluster with consistent hashing and TTL eviction to cache hot features and meet latency SLAs
  • Leverage Apache Kafka for streaming feature ingestion to support incremental updates
  • Use Apache Spark on EMR for scalable batch training data preparation
  • Orchestrate training pipelines with Apache Airflow to enable retries and monitoring
  • Store immutable model artifacts in Amazon S3 for durability and scalability
  • Use PostgreSQL for complex metadata queries, versioning, and rollback support
  • Enable server-side encryption with AWS KMS and enforce TLS 1.2+ for data in transit
  • Integrate AWS IAM and OAuth 2.0 for fine-grained access control and audit logging
  • Apache Hudi on S3 for incremental batch feature storage with snapshot isolation
  • Apache Cassandra for online feature store with partition key design for low-latency lookups
  • Redis cluster as caching layer with consistent hashing and TTL eviction to meet P99 latency
  • Apache Kafka for streaming feature ingestion enabling incremental updates
  • Apache Spark on EMR for scalable batch training data preparation
  • Apache Airflow for orchestrating training pipelines with retries and monitoring
  • Amazon S3 for immutable model artifact storage with high durability
  • PostgreSQL for model metadata, versioning, and rollback support
  • AWS KMS for encryption key management and server-side encryption
  • TLS 1.2+ enforced for all data in transit and OAuth 2.0 for secure access control

Practice

(1/5)
1. A globally distributed application requires strong consistency for writes but must tolerate network partitions gracefully. Which replication pattern best fits this scenario?
easy
A. Primary-replica replication with asynchronous replication to replicas
B. Multi-primary replication with asynchronous conflict resolution
C. Multi-primary replication with eventual consistency and last-write-wins
D. Primary-replica replication with synchronous writes to primary

Solution

  1. Step 1: Understand consistency requirements

    Strong consistency requires writes to be serialized and acknowledged before success.

  2. Step 2: Evaluate replication patterns

    Primary-replica with synchronous writes ensures all writes go through a single primary, preserving consistency. Multi-primary with async or eventual consistency cannot guarantee strong consistency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Primary sync writes -> strong consistency, tolerates partitions by blocking writes [check]
Hint: Strong consistency -> single primary synchronous writes [check]
Common Mistakes:
2. In the job application tracking flow, at step 5 the Application Processor updates the candidate's application status in the database. What happens if the database write at this step fails due to a transient network error?
easy
A. The Application Processor retries the write immediately, blocking the workflow until success
B. The Application Processor logs the failure and queues the update for asynchronous retry later
C. The Application Processor skips the update and marks the application as failed permanently
D. The Application Processor rolls back the entire application submission transaction

Solution

  1. Step 1: Understand failure handling at Application Processor

    Transient DB errors should not block user workflow or cause permanent failure.

  2. Step 2: Evaluate options

    Immediate retry blocking (A) hurts latency; skipping update (C) loses data; rolling back entire submission (D) is heavy-handed. Queuing for async retry (B) ensures eventual consistency without blocking.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Async retry decouples transient failures from user flow [check]
Hint: Async retry avoids blocking on transient DB errors [check]
Common Mistakes:
3. If the event queue that propagates write updates to read replicas fails, what is the immediate failure mode observed in the system?
medium
A. Writes to the primary database fail due to backpressure from the queue
B. Read replicas stop receiving updates and serve stale data indefinitely until manual intervention
C. The system automatically switches to synchronous replication to maintain consistency
D. Clients receive errors on read requests due to read replica unavailability

Solution

  1. Step 1: Identify role of event queue

    The event queue asynchronously propagates updates to read replicas.

  2. Step 2: Trace failure impact

    If the queue fails, read replicas stop receiving updates and serve stale data indefinitely.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Queue failure causes stale reads, no automatic failover [check]
Hint: Event queue failure -> stale reads until fixed manually [check]
Common Mistakes:
4. At 10x scale, which component is most likely to become the bottleneck first in the real-time driver location tracking system, and why?
hard
A. The database storing driver locations, due to write amplification from frequent updates
B. The WebSocket servers maintaining live connections, due to stateful connection limits and memory pressure
C. The surge pricing service, due to complex pricing calculations under load
D. The mobile clients, due to network bandwidth constraints

Solution

  1. Step 1: Identify stateful components in location tracking

    WebSocket servers hold persistent connections per driver and rider, consuming memory and CPU.

  2. Step 2: Analyze scaling challenges

    At 10x scale, WebSocket servers hit connection limits and memory pressure first; databases can shard and cache; surge pricing is stateless and scalable; mobile clients scale independently.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Stateful WebSocket servers bottleneck first due to connection limits [check]
Hint: Stateful WebSocket servers hit limits before stateless components [check]
Common Mistakes:
5. At 10x scale, which component in a Rendezvous Hashing-based distributed system is most likely to fail first and why?
hard
A. The client-side routing logic fails due to maintaining large node lists and computing scores
B. The central coordinator fails due to increased coordination overhead
C. The hash function computation becomes a CPU bottleneck due to increased node and key counts
D. The network bandwidth saturates because all nodes broadcast state to each other

Solution

  1. Step 1: Identify system components at scale

    Rendezvous Hashing is decentralized, so no central coordinator (The central coordinator fails due to increased coordination overhead) exists. Nodes do not broadcast state to all others (The network bandwidth saturates because all nodes broadcast state to each other) in typical implementations.

  2. Step 2: Analyze client-side routing impact

    Clients must compute scores for all nodes to pick highest; at 10x scale, this computation and memory for node lists become bottlenecks.

  3. Step 3: Consider hash computation cost

    Hash computations (The hash function computation becomes a CPU bottleneck due to increased node and key counts) are lightweight compared to maintaining large node lists and score computations.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Client routing logic scales poorly with node list size -> bottleneck [check]
Hint: Client routing logic chokes on large node lists [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard