Bird
Raised Fist0

Design a Payment Gateway like Razorpay or Stripe - Architecture, Webhook, Retry

HLDhardInterview freq: high
Choose your preparation mode4 modes available
🏗️
Design a Payment Gateway like Razorpay or Stripe - Architecture, Webhook, Retry
Design a scalable, reliable payment gateway system with webhook and retry mechanisms
hardE-Commerce and PaymentsFreq: high
Prerequisites:IdempotencyACID Transactions
⚙️
Functional Requirements
  • Process payment transactions from merchants with ACID guarantees
  • Support multiple payment methods (cards, UPI, wallets)
  • Provide webhook callbacks to merchants on payment status updates
  • Implement retry logic for webhook delivery with exponential backoff
  • Ensure idempotent transaction processing to avoid double charges
  • Allow querying transaction status and history via APIs
📊
Non-Functional Requirements
  • Handle 2 billion daily active users (DAU) generating 40 transactions per user per day
  • Achieve 99.99% availability for payment processing and webhook delivery
  • Maintain end-to-end latency under 100ms P99 for transaction processing
  • Ensure webhook delivery retries up to 24 hours with at least once semantics
  • Support horizontal scaling to 10x load without downtime
❓
Clarifying Questions to Ask
  • What payment methods and currencies must the system support initially?
  • What is the expected peak QPS and transaction size for the system?
  • What are the SLAs for webhook delivery and retry policies?
  • Should the system guarantee exactly-once or at-least-once webhook delivery?
  • Is the system expected to support refunds and chargebacks?
  • What are the data retention policies for transaction logs?
Out of Scope
✗ Fraud detection and risk scoring systems
✗ PCI DSS compliance implementation details
✗ User authentication and authorization mechanisms
✗ Merchant onboarding and KYC processes
📌
Assumptions
  • 2 billion DAU
  • 40 transactions per user per day
  • Average transaction record size 500 bytes
  • Webhook payload size 1 KB per event
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 transactions/sec
peak write qpsAssuming 3x peak factor: 925,926 × 3 ≈ 2.78M writes/sec
read qpsAssuming 2 reads per transaction: 925,926 × 2 ≈ 1.85M reads/sec
💾
Storage Estimate
daily storage2B × 40 × 500 bytes = 40 TB/day
retention days90
total storage40 TB × 90 = 3.6 PB
with replication3.6 PB × 3 (replication factor) = 10.8 PB
🖥️
Server Estimate
transaction processingAssuming 10K TPS per server: 2.78M / 10K = 278 servers
webhook deliveryAssuming 5K webhook/sec per server: 925K / 5K = 185 servers
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Merchant App  |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  API Gateway   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Payment Svc   |
          +-------+--------+
                  |
                  v
          +-------+--------+       +----------------+
          |  Transaction   |<----->|  Idempotency    |
          |  DB (Cassandra)|       |  Store (Redis)  |
          +-------+--------+       +----------------+
                  |
                  v
          +-------+--------+
          |  Event Stream  |
          |    (Kafka)     |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Webhook Worker |
          +-------+--------+
                  |
                  v
          +----------------+
          | Merchant Webhook|
          |    Endpoint    |
          +----------------+
🧩
Components
API Gateway
Entry point for merchant requests, handles authentication, rate limiting
Why thisEnvoy proxy provides high throughput, low latency, and native support for TLS termination and rate limiting
Why notNginx lacks native distributed rate limiting and dynamic service discovery at this scale
Payment Service
Core business logic for processing payments, validating requests, and initiating transactions
Why thisStateless microservice architecture using gRPC enables horizontal scaling and low latency RPC calls
Why notMonolithic app would not scale horizontally and increase deployment risk
Transaction DB (Apache Cassandra)
Stores transaction records with high write throughput and availability
Why thisCassandra supports linear write scalability, tunable consistency, and multi-region replication for 2.78M writes/sec
Why notPostgreSQL requires complex manual sharding at 2.78M writes/sec; B-tree writes don't scale linearly
Idempotency Store (Redis)
Stores transaction IDs with TTL to ensure idempotent processing of payment requests
Why thisRedis TTL natively expires stale keys without cron jobs and supports sub-millisecond lookup for idempotency checks
Why notMemcached lacks persistence and TTL granularity needed for idempotency guarantees
Event Stream (Apache Kafka)
Decouples transaction processing from webhook delivery, buffers events for retries
Why thisKafka supports high throughput (millions of events/sec), durable storage, and consumer groups for scalable webhook workers
Why notRabbitMQ does not scale as well for multi-million event throughput and has limited message retention
Webhook Worker
Consumes events from Kafka, delivers webhooks to merchants with retry and exponential backoff
Why thisStateless worker pool with Kafka consumer groups enables horizontal scaling and fault tolerance
Why notCron jobs or single-threaded retry services cannot handle 925K webhook/sec reliably
🔄
Data Flow
1
Merchant App → API Gateway
Send payment request with transaction details
API Gateway authenticates and rate limits requests
2
API Gateway → Payment Service
Forward validated payment request
gRPC call for low latency
3
Payment Service → Idempotency Store
Check if transaction ID exists to ensure idempotency
Avoids double processing of same transaction
4
Payment Service → Transaction DB
Write transaction record with ACID guarantees
Cassandra with lightweight transactions for consistency
5
Payment Service → Event Stream
Publish payment status event to Kafka topic
Decouples processing from webhook delivery
6
Webhook Worker → Event Stream
Consume payment events from Kafka
Process events for webhook delivery with retry
7
Webhook Worker → Merchant Webhook Endpoint
Deliver webhook callback with payment status
Retry with exponential backoff on failure

Practice

(1/5)
1. In a 45-minute system design answer, at step 4 you have identified key components and now start sketching data flow. What happens at this step in the component trace?
easy
A. You map how data moves between components to validate interactions
B. You choose cloud providers before understanding data flow
C. You finalize the database schema without considering data flow
D. You write API endpoint code to test data flow assumptions

Solution

  1. Step 1: Trace the role of step 4 -- sketching data flow

    This step validates how components interact and ensures data moves correctly, critical for system correctness.
  2. Step 2: Exclude premature or unrelated actions

    Finalizing schema or coding before data flow understanding risks misalignment; choosing cloud providers is premature.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Data flow mapping -> validates component interaction [check]
Hint: Map data flow before design details [check]
Common Mistakes:
2. In the TLS handshake trace, at step 4 the server sends its certificate chain to the client. What does the client do immediately after receiving this chain at step 5?
easy
A. The client closes the connection if the certificate is expired without any retries
B. The client immediately sends encrypted application data without verification
C. The client requests a new certificate from the server
D. The client verifies the certificate chain against trusted root CAs

Solution

  1. Step 1: Understand client certificate validation

    After receiving the server's certificate chain, the client validates it against its trusted root certificate authorities to ensure authenticity.
  2. Step 2: Eliminate incorrect options

    The client immediately sends encrypted application data without verification is wrong because the client must verify before sending encrypted data. The client requests a new certificate from the server is incorrect; clients do not request certificates from servers. The client closes the connection if the certificate is expired without any retries is partially true but the client may attempt other steps like checking revocation before closing.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Client verifies certificate chain before proceeding [check]
Hint: Client verifies certificate chain before encrypted data [check]
Common Mistakes:
3. In a system using Kafka for write-ahead logging to ensure strong consistency, what exactly breaks if Kafka becomes unavailable during a write operation?
medium
A. The system falls back to eventual consistency, causing stale reads temporarily
B. The client's write blocks indefinitely until Kafka recovers, preventing data loss
C. The write is accepted locally but lost permanently if the server crashes before Kafka recovers
D. The system switches to causal consistency mode automatically to maintain availability

Solution

  1. Step 1: Trace failure impact on write path

    Without Kafka, writes accepted locally but not durably logged can be lost if the server crashes before Kafka recovers.
  2. Step 2: Eliminate other options

    The system falls back to eventual consistency, causing stale reads temporarily is incorrect because fallback to eventual consistency is not automatic; The client's write blocks indefinitely until Kafka recovers, preventing data loss is wrong because writes may not block indefinitely; The system switches to causal consistency mode automatically to maintain availability is false as consistency mode does not switch automatically.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Writes accepted locally but lost if server crashes before Kafka recovers [check]
Hint: Without Kafka, local writes risk loss if server crashes [check]
Common Mistakes:
4. A team must decide between using a detailed blame-focused report or a blameless timeline with action items for their post-mortem. Which approach is best for fostering continuous improvement in a high-trust engineering culture?
medium
A. Use blameless timelines with actionable steps to encourage learning and improvement
B. Use blame-focused reports to enforce accountability and prevent future errors
C. Use only automated logs without human analysis to avoid bias
D. Skip post-mortems entirely to save time and avoid conflict

Solution

  1. Step 1: Evaluate cultural impact of blame vs blameless approaches

    Blame-focused reports create fear and reduce openness, harming continuous improvement.
  2. Step 2: Assess effectiveness of blameless timelines with action items

    They foster trust, encourage learning, and drive actionable improvements aligned with high-trust culture.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Blameless + action items -> continuous improvement [check]
Hint: Blameless + action items build trust and improve systems [check]
Common Mistakes:
5. At 10x scale, which component in a PACELC system prioritizing latency under partitions is most likely to fail first, and why?
hard
A. The global consensus service, because it requires synchronous coordination across data centers
B. The local write coordinator, because it handles all writes and cannot be sharded easily
C. The replication queue, because asynchronous replication saturates network bandwidth
D. The client-facing cache, because cache invalidation traffic grows exponentially

Solution

  1. Step 1: Identify bottleneck under 10x scale

    Local write coordinator handles all writes for its shard and is stateful, limiting horizontal scaling.
  2. Step 2: Compare other components

    Consensus service (A) is avoided under latency priority; replication queue (C) scales with network; cache (D) invalidation is less critical.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Local write coordinator statefulness -> first bottleneck at scale [check]
Hint: Stateful write coordinator limits scaling first under latency priority [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard