Bird
Raised Fist0

Design a Payment System - Transaction Processing, Idempotency, Settlement

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Payment System - Transaction Processing, Idempotency, Settlement
Design a scalable, reliable payment processing system with strong idempotency and settlement guarantees
hardE-Commerce and PaymentsFreq: high
Prerequisites:IdempotencyACID Transactions
⚙️
Functional Requirements
  • Process payment transactions ensuring atomicity and consistency
  • Guarantee idempotency to prevent duplicate charges on retries
  • Support multiple payment methods (credit card, wallet, bank transfer)
  • Perform asynchronous settlement and reconciliation with external banks
  • Provide real-time transaction status updates to users
  • Maintain audit logs for compliance and dispute resolution
📊
Non-Functional Requirements
  • Handle 2 billion daily active users generating 40 transactions per user per day
  • Achieve 99.99% availability with failover and disaster recovery
  • Ensure transaction processing latency under 100ms P99
  • Support 100 million concurrent transactions during peak hours
  • Guarantee exactly-once processing semantics for payments
❓
Clarifying Questions to Ask
  • What payment methods and gateways must be supported initially?
  • What is the expected peak QPS and transaction size?
  • Are transactions synchronous or can some be asynchronous?
  • What are the idempotency key generation rules and TTL?
  • What consistency guarantees are required for settlement?
  • Is eventual consistency acceptable for user-facing status updates?
Out of Scope
✗ Fraud detection and risk scoring systems
✗ User authentication and authorization
✗ Currency conversion and FX rate management
✗ Merchant onboarding and KYC processes
📌
Assumptions
  • 2 billion DAU
  • 40 transactions per user per day
  • Average transaction record size 500 bytes
  • Retention period for transaction logs: 90 days
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M writes/sec
read qpsAssuming 2 reads per write: 2 × 925,926 ≈ 1.85M reads/sec
💾
Storage Estimate
daily storage2B × 40 × 500 bytes = 40 TB/day
retention storage40 TB × 90 days = 3.6 PB
with replication3.6 PB × 3 replicas = 10.8 PB total
🖥️
Server Estimate
transaction processingAssuming 10K TPS per server for ACID DB writes, need ~278 servers
settlement workersAssuming 1K TPS per worker for batch settlement, need ~3K workers
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Client Apps   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  API Gateway   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Payment Svc   |
          +-------+--------+
          | Idempotency    |
          |  Store (Redis) |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Transaction   |
          |  DB (Cockroach)|
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Settlement Svc |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | External Banks |
          +----------------+
🧩
Components
API Gateway
Entry point for client requests, handles authentication, rate limiting
Why thisEnvoy proxy supports high throughput with built-in rate limiting and TLS termination
Why notNginx lacks native distributed rate limiting and dynamic config at scale
Payment Service
Handles transaction validation, idempotency checks, and business logic
Why thisStateless microservice in Go for low latency and easy horizontal scaling
Why notMonolithic app would not scale well under 2.78M TPS peak writes
Idempotency Store (Redis Cluster)
Stores idempotency keys with TTL to prevent duplicate transaction processing
Why thisRedis TTL natively expires stale keys without cron jobs - sub-millisecond lookup
Why notMemcached lacks persistence and TTL guarantees, risking duplicate processing
Transaction DB (CockroachDB)
Stores transaction records with ACID guarantees and distributed SQL
Why thisCockroachDB provides strong consistency, horizontal scaling, and geo-replication
Why notPostgreSQL requires complex manual sharding at 2.78M writes/sec; B-tree writes don't scale linearly
Settlement Service
Performs asynchronous batch settlement and reconciliation with external banks
Why thisKafka-based event-driven microservice enables reliable, scalable settlement workflows
Why notDirect synchronous calls to banks would increase latency and reduce throughput
External Banks
Third-party payment processors and banks for final settlement
Why thisExternal systems with their own APIs and SLAs
Why notN/A (external dependency)
🔄
Data Flow
1
Client Apps → API Gateway
Send payment request with idempotency key
Client generates unique idempotency key per transaction
2
API Gateway → Payment Service
Forward request after authentication and rate limiting
Ensures only valid requests reach backend
3
Payment Service → Idempotency Store
Check if idempotency key exists
If exists, return cached response to avoid duplicate processing
4
Payment Service → Transaction DB
Write transaction record within ACID transaction
Ensures atomicity and consistency of payment data
5
Payment Service → Idempotency Store
Store idempotency key with response and TTL
Prevents duplicate processing on retries
6
Transaction DB → Settlement Service
Emit transaction event to Kafka topic
Triggers asynchronous settlement workflow
7
Settlement Service → External Banks
Batch and send settlement requests
Handles reconciliation and final fund transfer
8
Settlement Service → Transaction DB
Update transaction status post-settlement
Ensures eventual consistency of transaction state
🗄️
transactions
PK: transaction_id
transaction_id UUID PRIMARY KEYuser_id UUIDpayment_method VARCHARamount DECIMALcurrency VARCHARstatus VARCHARcreated_at TIMESTAMPupdated_at TIMESTAMPidempotency_key VARCHAR UNIQUE
🗄️
settlements
PK: settlement_batch_id
settlement_batch_id UUID PRIMARY KEYtransaction_id UUIDsettlement_status VARCHARsettlement_date TIMESTAMPbank_reference VARCHARcreated_at TIMESTAMP
🔌
POST /payments
Request
{
  "user_id": "UUID",
  "payment_method": "string",
  "amount": "decimal",
  "currency": "string",
  "idempotency_key": "string"
}
Response
{
  "transaction_id": "UUID",
  "status": "string",
  "message": "string"
}
Creates a payment transaction ensuring idempotency using the provided key.
🔌
GET /payments/{transaction_id}
Request
{
  "transaction_id": "UUID"
}
Response
{
  "transaction_id": "UUID",
  "status": "string",
  "amount": "decimal",
  "currency": "string",
  "payment_method": "string",
  "created_at": "timestamp",
  "updated_at": "timestamp"
}
Fetches the status and details of a payment transaction.
🔌
GET /settlements/{settlement_batch_id}
Request
{
  "settlement_batch_id": "UUID"
}
Response
{
  "settlement_batch_id": "UUID",
  "settlement_status": "string",
  "settlement_date": "timestamp",
  "transactions": "list of transaction_ids"
}
Retrieves details of a settlement batch and its transactions.

Practice

(1/5)
1. A company wants to conduct a post-mortem after a major outage without blaming individuals, focusing on systemic issues and actionable improvements. Which component best supports this blameless culture approach?
easy
A. A detailed blame assignment report highlighting individual errors
B. An automated alert system that escalates to management immediately
C. A structured timeline tool capturing events and decisions neutrally
D. A punitive review board to enforce accountability

Solution

  1. Step 1: Identify blameless culture requirements

    The approach requires neutral, factual event capture without assigning blame to individuals.

  2. Step 2: Evaluate components for neutrality and actionability

    A structured timeline tool capturing events and decisions neutrally provides a structured timeline tool that records events and decisions neutrally, enabling root cause analysis without blame.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Neutral event timelines enable blameless post-mortems [check]
Hint: Blameless means neutral facts, not blame assignment [check]
Common Mistakes:
2. A system must handle millions of read requests per second with minimal latency, while writes are less frequent but must remain consistent. Which component best handles the read load without impacting write performance?
easy
A. Use a single primary SQL database with read-write locks to serialize access
B. Use a distributed cache with eventual consistency for all reads and writes
C. Deploy read replicas to serve read queries, offloading the primary database
D. Implement synchronous replication to all nodes and direct reads to any node

Solution

  1. Step 1: Identify read-heavy load handling

    Read replicas offload read queries from the primary, reducing contention and latency.

  2. Step 2: Evaluate other options

    Use a single primary SQL database with read-write locks to serialize access serializes access, causing bottlenecks; Use a distributed cache with eventual consistency for all reads and writes sacrifices consistency; Implement synchronous replication to all nodes and direct reads to any node synchronous replication increases write latency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Read replicas serve reads without blocking writes [check]
Hint: Read replicas scale reads, primary handles writes [check]
Common Mistakes:
3. At 10x scale, which component is most likely to fail first due to stateful connection limits, and why?
hard
A. The chat servers, because WebSocket connections are stateful and cannot be resharded easily
B. The message queue, because partition count limits throughput
C. The database cluster, because query volume exceeds CPU capacity
D. The load balancer, because it cannot distribute traffic evenly at high scale

Solution

  1. Step 1: Identify stateful bottleneck

    Chat servers maintain WebSocket connections bound to server memory, limiting horizontal scaling and resharding.

  2. Step 2: Compare other components

    Database (A) can be scaled with sharding; message queue (C) partitions can be increased; load balancer (D) scales horizontally.

  3. Step 3: Explain failure mechanism

    WebSocket statefulness causes connection limits per server, leading to failure under 10x load.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Stateful WebSocket limits -> chat servers fail first at scale [check]
Hint: Stateful WebSockets limit scaling; stateless scale easier [check]
Common Mistakes:
4. At 10x scale, which component in the recommendation system is most likely to fail first due to stateful connection limits, and why?
hard
A. The Kafka cluster, because partition count limits throughput and causes producer backpressure
B. The recommendation API servers, because maintaining user session state for personalized recommendations exhausts memory
C. The embedding storage database, because it cannot shard embeddings by user ID efficiently
D. The embedding service, because it holds in-memory caches that grow unbounded with user scale

Solution

  1. Step 1: Identify bottleneck at 10x scale

    Kafka partitions limit throughput; increasing partitions is complex and causes rebalancing overhead.

  2. Step 2: Evaluate other components

    Embedding DB (A) can shard by user ID; API servers (C) are typically stateless or use external session stores; embedding service caches (D) can be evicted or sharded. Kafka's partition limit causes producer backpressure and stalls event flow first.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka partition limit -> throughput bottleneck -> backpressure [check]
Hint: Kafka partitions limit throughput first at scale [check]
Common Mistakes:
5. At 10x scale, which component in an HTTP/3 deployment is most likely to become the bottleneck first and why?
hard
A. WebSocket servers because HTTP/3 does not support WebSockets natively
B. TLS handshake CPU load on servers due to QUIC's per-connection encryption overhead
C. UDP packet loss rate increases causing retransmissions and latency spikes
D. TCP connection limits because HTTP/3 uses TCP under the hood

Solution

  1. Step 1: Identify HTTP/3 transport and encryption

    HTTP/3 uses QUIC over UDP with built-in TLS encryption per connection, increasing CPU load for handshake and encryption at scale.

  2. Step 2: Eliminate other bottlenecks

    UDP packet loss is network-level, not server bottleneck; WebSockets are not natively supported by HTTP/3 but unrelated to scaling HTTP/3 itself; HTTP/3 does not use TCP.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    QUIC's per-connection TLS handshake CPU load limits HTTP/3 scale [check]
Hint: QUIC TLS handshake CPU load limits HTTP/3 scale first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard