Bird
Raised Fist0

Design a Fraud Detection System - Real-Time Scoring, Rule Engine, Feature Store

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Fraud Detection System - Real-Time Scoring, Rule Engine, Feature Store
Design a scalable, low-latency fraud detection system with real-time scoring, dynamic rule engine, and feature store for 2B DAU
hardFinancial and Real-Time SystemsFreq: high
Prerequisites:ML Feature StoreReal-Time ScoringStream ProcessingRule Engines
⚙️
Functional Requirements
  • Ingest and process up to 2 billion user transactions daily with real-time scoring
  • Apply dynamic fraud detection rules via a rule engine that supports frequent updates without downtime
  • Maintain a feature store that stores historical and real-time features for ML models and rule evaluation
  • Provide real-time fraud scores with P99 latency under 100ms per transaction
  • Support alerting and downstream integration for flagged fraudulent transactions
  • Allow batch re-scoring and model retraining using stored features
📊
Non-Functional Requirements
  • System must handle 2 billion daily active users generating 40 transactions each on average
  • End-to-end latency for scoring must be under 100ms P99
  • System availability must be at least 99.99% (four nines)
  • Data retention for features and transactions must be at least 90 days
  • System must scale to 10x peak load without redesign
❓
Clarifying Questions to Ask
  • What is the average and peak transaction rate per user?
  • What types of fraud rules are expected? Static, dynamic, or ML-based?
  • What is the expected size and update frequency of the feature store?
  • Is scoring done synchronously in the transaction path or asynchronously?
  • What are the SLAs for fraud alerting downstream?
  • Are there regulatory or compliance constraints on data storage?
Out of Scope
✗ Design of ML model training pipelines
✗ User interface for fraud analysts
✗ Long-term archival beyond 90 days
✗ Payment gateway integration
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 transactions/messages per day
  • Average transaction message size is 100 bytes
  • Retention period for features and transactions is 90 days
⚡
QPS Calculation
write qps2B users × 40 msg/user/day ÷ 86400 sec/day ≈ 925,926 msg/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M msg/sec
read qpsAssuming 1 read per write for scoring and feature fetch: ≈ 925,926 reads/sec
💾
Storage Estimate
daily storage2B × 40 × 100 bytes = 8 TB/day raw data
retention storage8 TB/day × 90 days = 720 TB
with replication720 TB × 3 (replication factor) = 2.16 PB total storage
🖥️
Server Estimate
stream processingKafka cluster with 100 brokers to handle 3M msg/sec peak (30K msg/sec per broker max)
feature storeCassandra cluster with 200 nodes to store 2.16 PB data with 3 replicas and support 1M+ QPS reads
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          | Transaction    |
          | Producers      |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Apache Kafka   |
          | (Ingestion)    |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Stream         |
          | Processing     |
          | (Apache Flink) |
          +-------+--------+
           /             \
          v               v
 +--------+-------+  +----+---------+
 | Feature Store  |  | Rule Engine  |
 | (Apache       |  | (Drools or   |
 | Cassandra)    |  | custom engine)|
 +--------+-------+  +----+---------+
          |               |
          v               v
    +-----+---------------+-----+
    | Real-Time Scoring Service  |
    +-------------+-------------+
                  |
                  v
          +-------+--------+
          | Alerting &     |
          | Downstream     |
          +----------------+
🧩
Components
Apache Kafka
Ingests and buffers incoming transaction streams at high throughput
Why thisKafka supports partitioned logs with high throughput (millions msg/sec) and durable storage for replay
Why notRabbitMQ lacks horizontal partitioning and throughput to handle 3M msg/sec peak writes
Apache Flink
Performs real-time stream processing, feature extraction, and enrichment
Why thisFlink provides exactly-once semantics, low latency (<50ms), and stateful stream processing at scale
Why notSpark Streaming has higher latency and weaker exactly-once guarantees unsuitable for real-time scoring
Apache Cassandra
Stores historical and real-time features with high write and read throughput
Why thisCassandra's wide-column model supports time-series data with tunable consistency and linear scalability
Why notPostgreSQL requires manual sharding and cannot sustain 1M+ QPS reads/writes at this scale
Rule Engine (Drools or custom)
Evaluates dynamic fraud detection rules with low latency and supports frequent updates
Why thisDrools supports complex event processing and rule versioning with minimal downtime
Why notHardcoded rules in application code require redeployment and lack flexibility for frequent updates
Real-Time Scoring Service
Aggregates features and rule results to produce final fraud scores within 100ms P99 latency
Why thisStateless microservice architecture enables horizontal scaling and fast response times
Why notMonolithic scoring service would bottleneck and increase latency under 3M TPS load
Alerting & Downstream Integration
Sends fraud alerts to downstream systems and triggers workflows
Why thisKafka topics and webhook integrations provide reliable, scalable alert delivery
Why notDirect synchronous calls to downstream systems risk blocking and cascading failures
🔄
Data Flow
1
Transaction Producers → Apache Kafka
Ingest raw transaction events
Kafka partitions ensure parallelism and durability
2
Apache Kafka → Apache Flink
Consume transaction streams for feature extraction and enrichment
Flink maintains state for real-time feature computation
3
Apache Flink → Apache Cassandra
Write computed features and enriched data
Cassandra stores time-series features with TTL for retention
4
Apache Flink → Rule Engine
Send enriched events for rule evaluation
Rule engine applies dynamic fraud rules with low latency
5
Rule Engine → Real-Time Scoring Service
Send rule evaluation results
Scoring service aggregates features and rules to compute fraud score
6
Real-Time Scoring Service → Alerting & Downstream Integration
Send fraud alerts for flagged transactions
Alerts trigger workflows and notify downstream systems

Practice

(1/5)
1. In the request flow, at Step 4 the Delivery Worker checks the message queue's visibility timeout before processing. What happens if the visibility timeout expires before the message is acknowledged?
easy
A. The message is moved to a dead-letter queue without retry
B. The message is deleted immediately to prevent duplicates
C. The message is archived permanently without redelivery
D. The message becomes visible again and can be redelivered to another worker

Solution

  1. Step 1: Understand visibility timeout semantics

    Visibility timeout hides the message during processing; if not acknowledged, it reappears for redelivery.

  2. Step 2: Analyze options

    Immediate deletion (A) risks data loss; archiving without retry (C) prevents recovery; dead-letter queue (D) is for failed retries, not timeout expiry.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Visibility timeout expiry -> message reappears for retry [check]
Hint: Visibility timeout expiry -> message requeued for retry [check]
Common Mistakes:
2. In a distributed lock system using ZooKeeper, at step 4 the client attempts to acquire the lock by creating an ephemeral sequential node. What happens if the client crashes immediately after creating this node but before confirming lock ownership?
easy
A. ZooKeeper automatically deletes the ephemeral node, releasing the lock
B. The ephemeral node remains until manually deleted, causing a deadlock
C. The client retains the lock until a timeout expires
D. The lock is transferred to the next waiting client immediately

Solution

  1. Step 1: Understand ephemeral node semantics

    Ephemeral nodes in ZooKeeper are automatically deleted when the client session ends (e.g., crash), preventing deadlocks (A) and stale locks.

  2. Step 2: Trace lock ownership after crash

    Since the ephemeral node is deleted, the next client waiting for the lock will be notified, but the lock is not immediately transferred (D) until they acquire it explicitly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ephemeral node deletion on client crash -> lock released [check]
Hint: Ephemeral nodes vanish on client crash, freeing locks [check]
Common Mistakes:
3. If the prefix hash map component fails and becomes unavailable, what is the immediate impact on the autocomplete system's behavior?
medium
A. The system returns no results for prefix queries until the prefix hash map recovers
B. The Trie component automatically handles prefix queries without any degradation in performance
C. Autocomplete queries fall back to Elasticsearch but with increased latency and less precise prefix matching
D. User input is cached locally on the client until the prefix hash map is restored

Solution

  1. Step 1: Understand component roles

    The prefix hash map is an optimization for O(1) prefix lookups, but the Trie is the primary data structure for prefix matching.

  2. Step 2: Trace failure impact

    If the prefix hash map fails, the system falls back to the Trie for prefix queries, which still works but may have slightly higher latency.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Trie fallback ensures prefix queries continue despite prefix hash map failure [check]
Hint: Trie fallback keeps prefix search alive if prefix hash fails [check]
Common Mistakes:
4. At 10x scale, which component is most likely to fail first and why when estimating system bandwidth for a social media platform with rapidly growing DAU?
hard
A. Storage Estimator, because disk I/O cannot keep up with 10x data volume growth
B. Bandwidth Estimator, because network metrics aggregation scales linearly and hits memory limits
C. DAU Estimator, because tracking and aggregating active users in real-time becomes a bottleneck
D. QPS Estimator, because load balancer metrics become too noisy and inaccurate at high scale

Solution

  1. Step 1: Identify bottleneck at 10x DAU scale

    DAU Estimator must track millions of active users in real-time, requiring heavy aggregation and state management.

  2. Step 2: Eliminate other options

    Storage Estimator scales with disk but can be sharded; Bandwidth Estimator aggregates metrics but less stateful; QPS Estimator uses load balancer metrics which scale better.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Real-time DAU tracking state grows fastest, causing bottleneck [check]
Hint: Real-time DAU tracking state grows fastest at scale [check]
Common Mistakes:
5. At 10x traffic scale, which component in a Layer 7 load balancing architecture is most likely to become the bottleneck first, and why?
hard
A. Layer 7 load balancer CPU, due to expensive HTTP header parsing and SSL termination
B. Network switches, because increased packet forwarding saturates bandwidth
C. Backend servers, because they cannot handle increased HTTP request volume
D. DNS servers, because client-side resolution delays increase exponentially

Solution

  1. Step 1: Identify Layer 7 load balancer workload at scale

    Layer 7 load balancers perform CPU-intensive tasks like HTTP header parsing and SSL termination, which scale poorly with traffic.

  2. Step 2: Compare with other components

    Backend servers can be scaled horizontally; network switches handle packet forwarding efficiently; DNS servers are less impacted by traffic volume.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Layer 7 LB CPU bottleneck due to HTTP parsing and SSL termination [check]
Hint: Layer 7 LB CPU chokes first at high traffic scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard