Bird
Raised Fist0

Design a Fraud Detection System - Real-Time Scoring, Rule Engine, Feature Store

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Fraud Detection System - Real-Time Scoring, Rule Engine, Feature Store
45 min · senior
senior

You are building a fraud detection system for a fintech startup with 50 million daily active users (DAU). The system must score transactions in real-time with P99 latency under 200ms, support a flexible rule engine for fraud analysts to update detection logic without downtime, and maintain a feature store that aggregates user and transaction features for scoring. All data must be encrypted at rest and in transit due to regulatory compliance. The system should also support offline batch feature computation and model retraining pipelines.

Constraints
  • P99 end-to-end scoring latency < 200ms at 50M DAU (~1.5M transactions/sec peak)
  • All data encrypted at rest and in transit (TLS + AES-256)
  • Rule engine updates must be hot-reloadable without downtime
  • Feature store must support both real-time and offline feature ingestion
  • System must handle eventual consistency between feature store and scoring engine
  • Scoring results must be auditable and stored for 30 days
📋
Guided Steps
1/5
Step 1
Real-Time Scoring Architecture
How would you design the real-time scoring pipeline to meet P99 < 200ms latency at 1.5M TPS?
💡 Hint
Consider stream processing frameworks optimized for low latency and high throughput with exactly-once semantics.
Key Decision
Use Apache Flink with Kafka as the ingestion layer - Flink provides sub-100ms processing latency with event-time processing and stateful operators, Kafka partitions keyed by user_id ensure load balancing and ordering.
Interviewer Expectation
Mid: Names Kafka and Flink for streaming. Senior: Explains Kafka partition key choice for ordering and Flink state backend for fault tolerance and exactly-once semantics.
⚠️Common Wrong Answer
Using batch processing like Spark Streaming - batch micro-batches cause latency spikes above 200ms, unsuitable for real-time scoring.
Step 2
Feature Store Design
What technology and data model would you choose for the feature store to support both real-time and offline feature ingestion?
💡 Hint
Think about a storage system optimized for fast key-value lookups and scalable batch writes with strong consistency guarantees.
Key Decision
Use Apache HBase for the feature store - HBase supports low-latency random reads/writes with strong consistency, integrates with Hadoop for offline batch ingestion, and supports encryption at rest.
Interviewer Expectation
Mid: Suggests a NoSQL store like Cassandra or HBase. Senior: Explains why HBase's strong consistency and integration with Hadoop batch pipelines fit both real-time and offline ingestion, and why Cassandra's eventual consistency is a risk for scoring accuracy.
⚠️Common Wrong Answer
Using Cassandra - eventual consistency can cause stale features leading to incorrect fraud scores.
Step 3
Rule Engine Implementation
How would you implement a flexible rule engine that fraud analysts can update without downtime?
💡 Hint
Consider a rules-as-code approach with hot-reload capabilities and versioning for auditability.
Key Decision
Use Drools rule engine with rules stored in a Git-backed repository and deployed via a CI/CD pipeline supporting hot reload via REST API, enabling dynamic updates without service restarts.
Interviewer Expectation
Mid: Mentions a rule engine like Drools or custom rules. Senior: Details Git versioning for audit, CI/CD for safe deployment, and hot reload APIs to avoid downtime.
⚠️Common Wrong Answer
Hardcoding rules in application code - requires redeployment and causes downtime on updates.
Step 4
Data Encryption and Compliance
How would you ensure all data is encrypted at rest and in transit to meet compliance requirements?
💡 Hint
Look for native encryption features in storage and messaging layers plus TLS for network communication.
Key Decision
Enable TLS 1.3 for all Kafka and REST API communications; use HBase Transparent Data Encryption (TDE) with AES-256 for data at rest; integrate HashiCorp Vault for key management.
Interviewer Expectation
Mid: Mentions TLS and encryption at rest. Senior: Explains specific encryption protocols, key rotation policies, and integration with Vault for centralized secrets management.
⚠️Common Wrong Answer
Relying on application-level encryption only - risks data leakage if storage or network layers are compromised.
Step 5
Auditing and Data Retention
How would you design the system to store scoring results for 30 days with auditability?
💡 Hint
Consider append-only storage with immutable logs and queryable metadata.
Key Decision
Use Apache Kafka with log compaction enabled for scoring results storage, backed by AWS S3 for long-term retention with lifecycle policies; integrate with Elasticsearch for audit queries.
Interviewer Expectation
Mid: Suggests storing results in a database or logs. Senior: Explains Kafka log compaction for immutable append-only storage, S3 lifecycle for cost-effective retention, and Elasticsearch for fast audit queries.
⚠️Common Wrong Answer
Storing results only in relational DB - expensive at scale and slow for audit queries.
🔌
API Design Challenge
Sketch 3 core API endpoints for the fraud detection system: transaction scoring, rule management, and feature ingestion.
Hints
  • How does the client submit a transaction for real-time scoring?
  • How do fraud analysts update or query rules dynamically?
  • How are features ingested or updated in the feature store?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Real-time scoring pipeline with sub-200ms P99 latency using Kafka + Flink
  • Feature store supporting both real-time and offline ingestion with strong consistency
  • Rule engine supporting hot reload and versioning for dynamic fraud rule updates
  • End-to-end encryption at rest and in transit with proper key management
  • Auditable scoring results stored for 30 days with immutable logs
Good to Have - 30%
  • Use of Kafka partition keys to ensure ordering and load balancing
  • Integration of batch pipelines for offline feature computation
  • Centralized secrets management with Vault
  • Use of Elasticsearch for audit queries
Bonus - 10%
  • Detailed API design for scoring, rule management, and feature ingestion
  • Discussion of failure modes and fallback strategies in scoring pipeline
Anti-patterns to Avoid
Using batch processing frameworks like Spark Streaming for real-time scoring - causes latency spikes above 200ms
Choosing Cassandra for feature store despite eventual consistency - leads to stale features and inaccurate scoring
Hardcoding fraud rules in application code - causes downtime and risk during updates
Relying solely on application-level encryption without native storage or network encryption - risks data leakage if storage or network layers are compromised
✅
Model Answer
▼ Reveal

This fraud detection system uses Kafka and Apache Flink for a low-latency, exactly-once real-time scoring pipeline capable of handling 1.5M TPS with P99 latency under 200ms. Apache HBase serves as the feature store supporting both real-time and offline ingestion with strong consistency guarantees. The Drools rule engine enables flexible, hot-reloadable fraud rule updates with Git-backed versioning and CI/CD deployment. All data is encrypted at rest using HBase TDE with AES-256 and in transit via TLS 1.3, with centralized key management through HashiCorp Vault. Scoring results are stored immutably using Kafka log compaction and backed by AWS S3 lifecycle policies for 30-day retention, with Elasticsearch enabling fast audit queries.

Key Decisions
  • Use Kafka with user_id partition key to ensure ordered, scalable event streams
  • Adopt Apache Flink for sub-100ms processing latency and exactly-once semantics
  • Choose Apache HBase for strong consistency and hybrid real-time/offline feature ingestion
  • Implement Drools with Git-backed versioning and hot reload REST API for flexible rule updates
  • Enable TLS 1.3 for all network communication and AES-256 encryption at rest via HBase TDE
  • Integrate HashiCorp Vault for centralized key management and automated key rotation
  • Store scoring results in Kafka with log compaction, backed by AWS S3 lifecycle policies for cost-effective 30-day retention
  • Use Elasticsearch to provide fast, queryable audit logs for compliance
  • Kafka as ingestion layer with user_id partition key for ordered, scalable event streams
  • Apache Flink for low-latency, exactly-once real-time scoring pipeline
  • Apache HBase as feature store for strong consistency and hybrid real-time/offline ingestion
  • Drools rule engine with Git-backed versioning and hot reload for flexible rule updates
  • TLS 1.3 for all network communication and HBase TDE with AES-256 for encryption at rest
  • HashiCorp Vault for centralized key management and rotation
  • Kafka log compaction plus AWS S3 lifecycle policies for 30-day scoring result retention
  • Elasticsearch for fast audit and compliance queries on scoring data

Practice

(1/5)
1. A company wants to hide internal client IP addresses from external servers while allowing clients to access the internet. Which proxy type should be used to achieve this?
easy
A. DNS proxy, because it resolves domain names for clients
B. Reverse proxy, because it hides the backend servers from clients
C. Load balancer, because it distributes requests evenly across servers
D. Forward proxy, because it hides client IPs from external servers

Solution

  1. Step 1: Identify proxy role in hiding client IPs

    A forward proxy acts on behalf of clients, masking their IP addresses from external servers.

  2. Step 2: Eliminate other options

    Reverse proxies hide backend servers, not clients; load balancers distribute load but don't hide clients; DNS proxies only resolve names.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Forward proxy hides client IPs from external servers [check]
Hint: Forward proxy hides clients; reverse proxy hides servers [check]
Common Mistakes:
2. For detecting anomalies in high-volume metric streams with low latency, which architecture is better suited and why?
medium
A. Batch processing with hourly aggregation to reduce noise and improve accuracy
B. Polling external ML service every 10 minutes to classify anomalies asynchronously
C. Streaming anomaly detection using windowed statistical models for near real-time alerts
D. Threshold-based alerts combined with manual tuning to approximate anomalies

Solution

  1. Step 1: Analyze latency and volume constraints

    High-volume streams require low-latency detection to be actionable.

  2. Step 2: Match architecture to constraints

    Streaming anomaly detection with windowed models processes data continuously, enabling near real-time alerts, unlike batch or polling which add latency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Streaming windowed models -> low latency anomaly detection [check]
Hint: High volume + low latency -> streaming anomaly detection [check]
Common Mistakes:
3. For a high-throughput payment processing system, which approach best balances failure isolation and resource utilization?
medium
A. Implement bulkheads with separate thread pools per payment type to isolate failures
B. Use a single large thread pool shared by all payment types to maximize utilization
C. Use circuit breakers on the shared thread pool to isolate failures dynamically
D. Rely on autoscaling the shared thread pool based on load metrics

Solution

  1. Step 1: Evaluate shared thread pool risks

    A single shared pool (A) risks cascading failures; circuit breakers (C) detect failures but don't isolate resource exhaustion. Autoscaling (D) reacts late and may not prevent blast radius.

  2. Step 2: Assess bulkhead benefits

    Bulkheads (B) isolate failures per payment type, preventing one type's overload from affecting others, balancing isolation and utilization.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bulkheads isolate failures per resource partition [check]
Hint: Bulkheads isolate by partitioning resources per failure domain [check]
Common Mistakes:
4. For storing aggregated logs with fast query response and cost efficiency, which storage architecture is best when logs are mostly write-once and rarely updated?
medium
A. Use a distributed relational database with strong consistency
B. Use a NoSQL key-value store with frequent compactions
C. Use a columnar data warehouse optimized for append-only writes
D. Use a distributed file system with manual indexing

Solution

  1. Step 1: Analyze log storage requirements

    Logs are write-once, large volume, and require fast analytical queries.

  2. Step 2: Evaluate columnar data warehouse

    Columnar stores optimize for append-only writes and fast scans, reducing cost and improving query speed.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Columnar warehouse -> optimized for append-only and analytics [check]
Hint: Columnar storage excels at append-only analytics [check]
Common Mistakes:
5. At 10x scale, which component in the real-time ML inference system is most likely to fail first and why?
hard
A. Model storage backend, due to increased read latency from large model files
B. Load balancer, since it cannot efficiently distribute traffic to a rapidly scaling number of servers
C. Kafka event stream, as partition count limits throughput and causes lag under heavy update load
D. Prediction API servers, because CPU-intensive model inference cannot be parallelized easily

Solution

  1. Step 1: Analyze scaling bottlenecks

    Model storage (A) scales with caching; Kafka (C) can be scaled by adding partitions; load balancers (D) scale horizontally.

  2. Step 2: Identify CPU bottleneck in inference

    Prediction API servers (B) perform CPU/GPU-bound inference; scaling is limited by hardware and model complexity, causing first failure under 10x load.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Inference servers hit CPU/GPU limits before storage or Kafka scale [check]
Hint: Inference servers hit CPU/GPU bottleneck before other components [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard