Bird
Raised Fist0

Design a Recommendation System - Collaborative Filtering, Embedding Storage

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/7
API Gateway

Receives recommendation request

User sends request for personalized recommendations

Protocol:HTTPS
BeforeUser has not requested recommendations
AfterAPI Gateway received request and forwards to Flink
Apache Flink Stream Processor

Processes request and user event stream

Generates event messages for Kafka

Protocol:Kafka Produce
BeforeNo new events in Kafka for this user
AfterUser event messages enqueued in Kafka
Kafka Cluster

Queues user event messages

Stores events for batch and streaming jobs

Protocol:Kafka Consume
BeforeKafka has no events for this user
AfterKafka holds user events for consumption
Offline Batch Training Jobs

Consumes events and updates models

Trains embeddings and writes models to storage

Protocol:S3 + Metadata DB Write
BeforeModel Storage has previous model version
AfterModel Storage updated with new model and embeddings
Model Storage (S3 + Metadata DB)

Serves embeddings to Cassandra

Loads latest embeddings for storage

Protocol:Internal API
BeforeCassandra Embedding Store has stale embeddings
AfterCassandra Embedding Store updated with fresh embeddings
Cassandra Embedding Store

Responds to API Gateway embedding queries

Provides user/item embeddings for recommendations

Protocol:CQL Query
BeforeAPI Gateway waiting for embeddings
AfterAPI Gateway received embeddings for recommendation
Redis Cluster

Caches embeddings for fast retrieval

API Gateway queries Redis cache before Cassandra

Protocol:Redis GET
BeforeRedis cache may be cold or stale
AfterAPI Gateway served embeddings from Redis cache or fallback
🏗️
Architecture Diagram
Architecture Diagram
┌─────────────────────┐
│     API Gateway     │
└──────────┬──────────┘
           │  HTTPS Request
           ▼
┌───────────────────────────────┐
│ Apache Flink Stream Processor  │
│  - Real-time event processing  │
└──────────────┬────────────────┘
               │ Kafka Produce
               ▼
┌───────────────┐
│ Kafka Cluster │
│  - Event queue│
└───────┬───────┘
        │ Kafka Consume
        ▼
┌───────────────────────────────┐
│ Offline Batch Training Jobs    │
│  - Model training & updates    │
└──────────────┬────────────────┘
               │ Writes model to
               │ Model Storage
               ▼
┌───────────────────────────────┐
│ Model Storage (S3 + Meta DB)  │
│  - Stores trained models       │
└──────────────┬────────────────┘
               │ Loads embeddings
               ▼
┌───────────────────────────────┐
│ Cassandra Embedding Store      │
│  - Stores user/item embeddings │
└──────────────┬────────────────┘
               │ Redis GET
               ▼
┌───────────────┐
│ Redis Cluster │
│  - Cache for fast lookup       │
└───────────────┘
Design Rationale

This architecture uses an API Gateway as the entry point for user recommendation requests. The Apache Flink Stream Processor handles real-time user events and produces messages to the Kafka Cluster, which acts as a durable event queue. Offline Batch Training Jobs consume these events to periodically train and update recommendation models, storing them in Model Storage (S3 + Metadata DB). The trained embeddings are loaded into Cassandra Embedding Store for persistent storage. To reduce latency, Redis Cluster caches embeddings for quick retrieval by the API Gateway. This design balances real-time streaming with batch updates, ensuring fresh recommendations with low latency.

Primary Focus

✓

Practice

(1/5)
1. If the reconciliation service that compares debit and credit entries is temporarily down, what is the immediate failure mode observed in the digital wallet system?
medium
A. The system marks transactions as pending and delays final confirmation to users until reconciliation completes
B. New transactions are blocked until reconciliation service recovers to prevent ledger divergence
C. Transactions continue normally but inconsistencies accumulate silently until reconciliation resumes
D. Debit entries are committed immediately but credit entries are delayed, causing temporary imbalance

Solution

  1. Step 1: Understand reconciliation service role

    It ensures debit and credit entries match before confirming transactions to users.

  2. Step 2: Analyze failure impact

    Blocking all transactions (B) reduces availability unnecessarily; silent accumulation (A) risks unnoticed errors; committing debit only (D) breaks double-entry. Marking transactions pending (C) delays confirmation but preserves consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Pending status delays confirmation -> preserves ledger integrity [check]
Hint: Pending transactions delay confirmation, not silent failure [OK]
Common Mistakes:
2. For a chat system requiring low latency and fault tolerance, which architecture is best when mobile clients have intermittent connectivity?
medium
A. Use gRPC streaming for persistent connections to push messages instantly
B. Use a message queue with client-side polling and exponential backoff
C. Use WebSocket connections with server-side buffering and reconnection logic
D. Use HTTP long polling with immediate retries on failure

Solution

  1. Step 1: Evaluate client connectivity constraints

    Mobile clients have intermittent connectivity; gRPC streaming (A) requires persistent connections, which mobile networks may not support well. HTTP long polling (D) causes overhead and latency.

  2. Step 2: Compare WebSocket with server buffering vs polling

    WebSocket with server-side buffering and reconnection (C) balances low latency and fault tolerance better than client polling (B), which increases latency and load.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    WebSocket + buffering + reconnection -> best for intermittent mobile [check]
Hint: WebSocket buffers and reconnects for flaky mobile networks [check]
Common Mistakes:
3. At 10x scale, which component in a real-time chat system fails first and why?
hard
A. The database fails first due to increased query volume and locking
B. The message queue fails first due to partition key hotspots causing uneven load
C. The load balancer fails first due to inability to distribute traffic evenly
D. The chat servers fail first because WebSocket connections are stateful and cannot be resharded easily

Solution

  1. Step 1: Identify stateful bottlenecks at scale

    Chat servers maintain WebSocket connections bound to server memory, making horizontal scaling and resharding difficult.

  2. Step 2: Compare other components

    Databases (The database fails first due to increased query volume and locking) can be scaled with sharding and replicas. Load balancers (The load balancer fails first due to inability to distribute traffic evenly) scale horizontally. Message queues (The message queue fails first due to partition key hotspots causing uneven load) can rebalance partitions.

  3. Step 3: Conclude first failure point

    Chat servers hit connection limits first due to stateful nature and sticky sessions.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Stateful WebSocket servers -> scaling bottleneck [check]
Hint: Stateful WebSocket servers block scaling; stateless scale easier [check]
Common Mistakes:
4. At 10x scale, which component in the notification system is most likely to fail first and why?
hard
A. The Delivery Workers, because they cannot horizontally scale beyond a fixed number of threads per machine
B. The push notification gateway, because mobile platform rate limits throttle outgoing messages
C. The relational database storing user preferences, due to increased read/write contention
D. The presence cache, because it is stateful and cannot be sharded easily across many servers

Solution

  1. Step 1: Analyze Delivery Worker scalability

    Delivery Workers are CPU and thread-bound; increasing scale requires more workers, but thread limits and resource contention cause bottlenecks.

  2. Step 2: Compare other components

    Relational DB (A) can be sharded; push gateways (C) are rate-limited but can be scaled with multiple credentials; presence cache (D) can be sharded with consistent hashing.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Delivery Workers hit thread and CPU limits first at 10x scale [check]
Hint: Delivery Workers hit thread limits before other components [check]
Common Mistakes:
5. At 10x scale, which component of a distributed HyperLogLog-based unique user counting system is most likely to fail first, and why?
hard
A. The aggregator node, because it must merge many HyperLogLog sketches and becomes CPU-bound
B. The HyperLogLog register storage, because fixed-size registers cannot represent larger cardinalities accurately
C. The network bandwidth between servers, because merging large raw user ID sets saturates links
D. The user ID ingestion layer, because it must hash and update many HyperLogLog sketches concurrently

Solution

  1. Step 1: Analyze scaling impact on components

    Network bandwidth is low since only fixed-size sketches are transferred, not raw IDs; register size is fixed and does not fail but accuracy degrades slightly.

  2. Step 2: Identify bottleneck

    The aggregator merges many sketches concurrently, increasing CPU load and latency, making it the first to fail under 10x load.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Aggregator CPU-bound merging many sketches first bottleneck [check]
Hint: Aggregator CPU merges many sketches -> first bottleneck [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard