Bird
Raised Fist0

Design a Payment System - Transaction Processing, Idempotency, Settlement

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Payment System - Transaction Processing, Idempotency, Settlement
45 min · senior
senior

You are designing a payment processing system for a fintech startup with 50 million daily active users. The system must handle high transaction throughput, ensure exactly-once transaction semantics via idempotency, and support settlement with external banking partners. The system must guarantee data encryption at rest and in transit, maintain a P99 latency under 200ms for transaction confirmation, and support offline transaction delivery for intermittent connectivity scenarios.

Constraints
  • End-to-end encryption of transaction data both at rest and in transit is mandatory due to regulatory compliance.
  • System must guarantee exactly-once transaction processing using idempotency keys to prevent double charges.
  • P99 transaction confirmation latency must be under 200ms to ensure good user experience.
  • Support offline transaction delivery and reconciliation for users with intermittent connectivity.
  • Settlement with external banking partners must be batched and reconciled daily with audit logs.
  • System must scale to handle at least 10,000 transactions per second peak load.
📋
Guided Steps
1/5
Step 1
Transaction Ingestion and Idempotency Handling
How would you design the transaction ingestion pipeline to ensure exactly-once processing with idempotency?
💡 Hint
Consider a distributed log system that supports partitioning and ordering, and a fast key-value store for idempotency checks.
Key Decision
Use Apache Kafka with transaction_id as the partition key to ensure ordering and exactly-once semantics, combined with Redis as a fast idempotency key store with TTL to prevent duplicate processing and memory bloat.
Interviewer Expectation
Mid-level: Names Kafka and Redis for ingestion and idempotency. Senior: Explains why transaction_id as Kafka partition key ensures ordering and how Redis TTL prevents stale idempotency keys causing memory bloat.
⚠️Common Wrong Answer
Using a relational database for ingestion and idempotency checks - this creates a bottleneck at high TPS and does not scale linearly.
Step 2
Transaction Processing and State Management
How will you manage transaction state transitions (e.g., pending, completed, failed) and ensure consistency under concurrent updates?
💡 Hint
Think about a distributed NoSQL database with strong consistency or lightweight transactions and how to model state transitions atomically.
Key Decision
Use Cassandra with transaction_id as partition key for linear scalability and lightweight transactions (LWT) to enforce atomic state transitions and prevent race conditions.
Interviewer Expectation
Mid-level: Mentions Cassandra for scalability. Senior: Explains why LWT is necessary to avoid lost updates and how partition key choice enables efficient range queries for transaction history.
⚠️Common Wrong Answer
Using eventual consistent stores without atomic state transitions - leads to inconsistent transaction states and user confusion.
Step 3
Settlement and Reconciliation with External Partners
How would you design the settlement process to batch transactions and reconcile with external banking partners reliably?
💡 Hint
Consider asynchronous batch processing with durable queues and audit logs for traceability.
Key Decision
Use Apache Kafka for batching settlement messages, processed by a dedicated settlement microservice that writes audit logs to Amazon S3 for immutable storage and reconciliation.
Interviewer Expectation
Mid-level: Names Kafka and S3 for batching and storage. Senior: Explains how Kafka partitions enable parallel settlement processing and how immutable S3 audit logs support compliance and dispute resolution.
⚠️Common Wrong Answer
Real-time settlement per transaction - causes high operational cost and complexity, and external partners prefer batch processing.
Step 4
Ensuring Low Latency and High Availability
What architectural choices ensure P99 latency under 200ms and high availability for transaction confirmation?
💡 Hint
Think about caching, data locality, and replication strategies.
Key Decision
Use Redis as a read-through cache for transaction status with TTL, colocated with application servers, combined with multi-AZ Cassandra clusters for fault tolerance and low read latency.
Interviewer Expectation
Mid-level: Mentions caching and replication. Senior: Explains cache invalidation strategies, trade-offs of multi-AZ replication latency, and fallback mechanisms on cache misses.
⚠️Common Wrong Answer
No caching and single AZ deployment - leads to high latency and single point of failure.
Step 5
Offline Transaction Delivery and Reconciliation
How would you support users with intermittent connectivity to ensure their transactions are eventually processed and reconciled?
💡 Hint
Consider durable local storage on client devices and eventual synchronization protocols.
Key Decision
Implement client-side durable queue with local encrypted storage and use Kafka-based backend with idempotency to safely replay transactions upon reconnection.
Interviewer Expectation
Mid-level: Mentions offline support and retries. Senior: Explains encryption on client storage, replay safety via idempotency keys, and conflict resolution during sync.
⚠️Common Wrong Answer
No offline support or relying on client retries without idempotency - leads to duplicate charges or lost transactions.
🔌
API Design Challenge
Sketch 3 core API endpoints for the payment system that cover transaction initiation, status querying, and settlement confirmation.
Hints
  • How does the client initiate a payment and provide idempotency keys?
  • How does the client query transaction status efficiently and securely?
  • How does the system confirm settlement and notify clients or downstream systems?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with transaction_id partition key for ordered ingestion and exactly-once semantics.
  • Implement idempotency with Redis TTL-based key store to prevent duplicate processing.
  • Use Cassandra with lightweight transactions for atomic state management.
  • Batch settlement via Kafka and immutable audit logs in S3 for compliance.
Good to Have - 30%
  • Redis read-through cache to meet P99 latency under 200ms.
  • Multi-AZ deployment for high availability and fault tolerance.
  • Client-side durable encrypted queue for offline transaction support.
Bonus - 10%
  • Detailed API design covering transaction initiation, status querying, and settlement confirmation.
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

This payment system design ensures exactly-once transaction processing with idempotency, low latency confirmation, secure data handling, and reliable batch settlement with external partners. It supports offline transaction capture and reconciliation for intermittent connectivity, scaling to 10,000 TPS peak load with multi-AZ high availability.

Key Decisions
  • Use Kafka with transaction_id as partition key to guarantee ordering and exactly-once ingestion semantics.
  • Employ Redis with TTL-based idempotency keys to prevent duplicate transaction processing and control memory usage.
  • Leverage Cassandra lightweight transactions (LWT) to atomically update transaction states and avoid race conditions.
  • Batch settlement messages via Kafka and store audit logs immutably in Amazon S3 for compliance and dispute resolution.
  • Implement Redis read-through cache colocated with app servers to meet P99 latency under 200ms for transaction status queries.
  • Deploy multi-AZ Cassandra clusters for fault tolerance and high availability.
  • Enable client-side encrypted durable queues to support offline transaction delivery and safe replay upon reconnection.
  • Ensure end-to-end encryption using TLS for data in transit and AES-256 for data at rest to meet regulatory requirements.
  • Apache Kafka with transaction_id as partition key ensures ordered, scalable ingestion and exactly-once processing.
  • Redis as a fast idempotency key store with TTL prevents duplicate transaction processing and memory leaks.
  • Cassandra with transaction_id partition key and lightweight transactions guarantees atomic state transitions and linear scalability.
  • Kafka-based batch settlement pipeline with audit logs stored immutably in Amazon S3 supports reliable reconciliation and compliance.
  • Redis read-through cache colocated with app servers reduces P99 latency under 200ms for transaction status queries.
  • Multi-AZ Cassandra clusters provide high availability and fault tolerance for transaction state storage.
  • Client-side encrypted durable queue enables offline transaction capture and safe replay upon reconnection.
  • End-to-end encryption using TLS for transit and AES-256 for data at rest ensures regulatory compliance and data security.

Practice

(1/5)
1. A food delivery system must efficiently match nearby drivers to new orders in real-time, minimizing latency and ensuring fairness. Which component is best suited to handle this driver assignment under high load?
easy
A. A relational database with spatial indexing to query nearest drivers synchronously
B. A simple REST API that queries all drivers and filters by distance on the client side
C. A batch job that runs every 5 minutes to assign drivers to pending orders
D. A distributed in-memory geospatial index with pub/sub notifications for driver availability

Solution

  1. Step 1: Identify driver assignment requirements

    Real-time, low-latency matching requires fast spatial queries and event-driven updates.

  2. Step 2: Evaluate component suitability

    Relational DBs with spatial indexes (A) are too slow for high write/read concurrency; batch jobs (C) introduce unacceptable delay; client-side filtering (D) is inefficient and inconsistent.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Distributed in-memory geospatial index with pub/sub -> real-time, scalable driver assignment [check]
Hint: In-memory geospatial + pub/sub for real-time driver matching [check]
Common Mistakes:
2. In the request flow for posting a new answer, at step 5 the Feed Generator consumes messages from Kafka. What happens if the Feed Generator detects a malformed message at this step?
easy
A. The Feed Generator sends the malformed message to a dead-letter queue for later inspection
B. The Feed Generator pauses consumption and retries the malformed message until fixed
C. The Feed Generator discards the message and continues processing the next message
D. The Feed Generator crashes, causing the entire feed update pipeline to stop

Solution

  1. Step 1: Understand Feed Generator error handling

    Discarding (A) loses data silently; pausing and retrying (B) blocks the pipeline indefinitely; crashing (D) causes system downtime.

  2. Step 2: Dead-letter queue usage

    Sending malformed messages to a dead-letter queue (C) isolates bad data for manual or automated inspection without blocking the main pipeline.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dead-letter queue isolates bad messages without blocking processing [check]
Hint: Dead-letter queues isolate bad messages, keep pipeline flowing [check]
Common Mistakes:
3. A distributed cache system needs to evenly distribute keys across servers and minimize data reshuffling when nodes are added or removed. Which component best handles this requirement?
easy
A. A simple modulo-based hash function assigning keys to servers by key % server_count
B. A centralized directory service mapping keys to servers explicitly
C. A consistent hashing ring with virtual nodes to balance load and reduce remapping
D. A random key assignment with periodic rebalancing of all keys

Solution

  1. Step 1: Understand key distribution and remapping

    Modulo-based hashing (A simple modulo-based hash function assigning keys to servers by key % server_count) causes massive remapping when server count changes, violating minimal reshuffling.

  2. Step 2: Evaluate consistent hashing with virtual nodes

    Consistent hashing with virtual nodes (A consistent hashing ring with virtual nodes to balance load and reduce remapping) evenly distributes keys and minimizes remapping on node changes.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Virtual nodes in consistent hashing minimize key remapping on node changes [check]
Hint: Virtual nodes smooth load, minimize reshuffle [check]
Common Mistakes:
4. For a global e-commerce platform requiring high availability and multi-region writes with eventual consistency, which key-value store choice is best and why?
medium
A. Use DynamoDB Global Tables for multi-region writes and automatic conflict resolution
B. Use Redis with master-slave replication across regions for low latency
C. Use Redis Cluster with cross-region sharding and manual conflict handling
D. Use a relational database with cross-region synchronous replication

Solution

  1. Step 1: Identify multi-region write and availability requirements

    Multi-region writes require conflict resolution and high availability without synchronous blocking.

  2. Step 2: Compare Redis and DynamoDB capabilities

    DynamoDB Global Tables support multi-region writes with built-in conflict resolution and eventual consistency; Redis lacks native multi-region write support.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DynamoDB Global Tables -> multi-region writes with conflict resolution [check]
Hint: DynamoDB Global Tables handle multi-region writes natively [check]
Common Mistakes:
5. At 10x scale, which component in a Kafka-like distributed message queue is most likely to fail first, and why?
hard
A. The Kafka brokers, because partition leaders become overloaded with write and replication traffic
B. The consumer clients, because they cannot keep up with increased message consumption rates
C. The ZooKeeper ensemble, due to coordination overhead and leader election delays under heavy load
D. The network switches, due to increased packet loss under high throughput

Solution

  1. Step 1: Identify bottlenecks at 10x scale

    Network switches (A) scale with hardware upgrades; ZooKeeper (C) handles coordination but is lightweight compared to brokers; consumers (D) can be scaled horizontally.

  2. Step 2: Kafka broker partition leader overload

    Partition leaders handle all writes and replication for their partitions; at 10x scale, leaders become hotspots causing CPU, disk, and network saturation, leading to failures.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Partition leaders overloaded -> broker failure first at scale [check]
Hint: Partition leaders handle all writes -> overload first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard