Bird
Raised Fist0

Idempotency in APIs - Why It Is Critical for Payment and Retry Systems

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Idempotency in APIs - Why It Is Critical for Payment and Retry Systems
Ensuring exactly-once semantics in payment APIs to prevent double charges and data inconsistency
mediumAPI Design and GatewayFreq: common
Prerequisites:IdempotencyACID Transactions
⚙️
Functional Requirements
  • Support idempotent payment API calls that prevent double charges on retries
  • Allow clients to provide an idempotency key with each request
  • Persist idempotency keys and their results atomically with payment processing
  • Support retry logic with consistent response for repeated requests with same idempotency key
  • Expose APIs to query status of payment requests by idempotency key
  • Handle concurrent requests with the same idempotency key gracefully
📊
Non-Functional Requirements
  • System must handle 2 billion daily active users (DAU)
  • Support peak write QPS of approximately 925K (derived from 2B × 40 msgs/user/day / 86400 seconds)
  • Ensure 99.99% availability for payment API endpoints
  • Achieve 100ms P99 latency for idempotency key validation and payment processing
  • Data retention for idempotency keys and results for at least 30 days
❓
Clarifying Questions to Ask
  • What is the expected maximum size and format of the idempotency key?
  • Are idempotency keys globally unique or scoped per user/account?
  • What is the maximum allowed retry window for idempotency keys?
  • Should the system support partial failures and compensating transactions?
  • Is eventual consistency acceptable or is strong consistency required for idempotency?
  • What payment methods and currencies must be supported?
Out of Scope
✗ Design of the underlying payment gateway or bank integration
✗ Fraud detection and risk analysis systems
✗ User authentication and authorization mechanisms
✗ Detailed UI/UX design for client retry handling
📌
Assumptions
  • 2 billion DAU
  • 40 payment API calls per user per day
  • 100 bytes average size per idempotency record
  • 30 days data retention for idempotency records
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor → 1,851,852 writes/sec
read qpsAssuming 1.5 reads per write (status queries) → 1,388,889 reads/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024^3) ≈ 7.45 TB/day
retention 30d storage7.45 TB × 30 = 223.5 TB
with replication 3x223.5 TB × 3 = 670.5 TB total storage
🖥️
Server Estimate
idempotency store nodesAssuming Cassandra with 100 MB/s disk throughput, 670.5 TB / 30 days ≈ 22.35 TB/day, requiring ~230 nodes
api gateway nodesAssuming each node handles 10K QPS, 1,851,852 peak write QPS / 10K = ~186 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |   Client App   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  API Gateway   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Idempotency    |
          |  Service       |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Payment Proc.  |
          |  Service       |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Cassandra DB   |
          | (Idempotency)  |
          +----------------+
🧩
Components
API Gateway
Entry point for client requests, validates idempotency key presence and format
Why thisNginx with Lua scripting for lightweight validation and routing with sub-10ms latency
Why notAWS API Gateway rejected due to vendor lock-in and limited customization at scale
Idempotency Service
Manages idempotency keys lifecycle, checks for duplicates, returns cached results
Why thisStateless microservice in Go for low latency and easy horizontal scaling
Why notMonolithic payment service rejected to avoid coupling and scaling bottlenecks
Payment Processing Service
Executes payment transactions atomically with idempotency key persistence
Why thisJava Spring Boot service with ACID transaction support integrating with payment gateway
Why notServerless functions rejected due to cold start latency and limited transaction control
Cassandra DB
Stores idempotency keys and payment results with high write throughput and replication
Why thisCassandra supports linear write scalability, tunable consistency, and 3x replication
Why notPostgreSQL rejected due to complex manual sharding and limited write scalability at 1.8M QPS
Message Queue (Kafka)
Asynchronously processes retries and eventual consistency updates
Why thisKafka provides durable, ordered, and partitioned logs for reliable retry processing
Why notRabbitMQ rejected due to lower throughput and lack of partitioned logs at scale
🔄
Data Flow
1
Client App → API Gateway
Send payment request with idempotency key
Client generates unique idempotency key per logical payment
2
API Gateway → Idempotency Service
Validate idempotency key format and check cache
Reject malformed keys early to reduce load
3
Idempotency Service → Cassandra DB
Check if idempotency key exists and fetch stored result
If exists, return cached response to client
4
Idempotency Service → Payment Processing Service
If new key, forward request for payment execution
Payment service atomically writes idempotency key and payment status
5
Payment Processing Service → Cassandra DB
Persist idempotency key and payment result atomically
Ensures exactly-once semantics even under retries
6
Payment Processing Service → Idempotency Service
Return payment result
Idempotency service caches result for future duplicate requests
7
Idempotency Service → API Gateway
Return response to client
Consistent response for retries with same idempotency key
🗄️
idempotency_records
PK: idempotency_key
idempotency_key textcreated_at timestamppayment_status textpayment_result textuser_id text
Stores idempotency keys with payment results and timestamps for lookup and expiry
🗄️
payment_transactions
PK: transaction_id
transaction_id textuser_id textamount decimalcurrency textstatus textcreated_at timestamp
Stores payment transaction details for audit and reconciliation
🔌
POST /payments
Request
{
  "idempotency_key": "string",
  "user_id": "string",
  "amount": "decimal",
  "currency": "string",
  "payment_method": "string"
}
Response
{
  "payment_status": "string",
  "transaction_id": "string",
  "message": "string"
}
Creates a payment if idempotency key is new, else returns cached result
🔌
GET /payments/status
Request
{
  "idempotency_key": "string"
}
Response
{
  "payment_status": "string",
  "transaction_id": "string",
  "message": "string"
}
Fetches payment status for given idempotency key
🔌
POST /payments/retry
Request
{
  "idempotency_key": "string"
}
Response
{
  "payment_status": "string",
  "transaction_id": "string",
  "message": "string"
}
Allows client to retry payment request safely using idempotency key

Practice

(1/5)
1. A mobile app requires tailored API responses per device type to reduce payload size and improve UX. Which component best handles this requirement in a microservices architecture?
easy
A. Use a shared database view that filters data per device type
B. Use a CDN to cache different payloads per device type
C. Let each microservice handle device-specific logic internally
D. Implement a Backend-for-Frontend (BFF) layer that adapts responses per client

Solution

  1. Step 1: Identify the role of BFF

    The BFF layer acts as a tailored API gateway that adapts responses specifically for each client type, reducing payload and improving UX.

  2. Step 2: Eliminate other options

    Use a shared database view that filters data per device type is incorrect because database views are not designed for client-specific adaptation and increase coupling. Let each microservice handle device-specific logic internally is wrong as microservices should remain generic and not handle client-specific logic. Use a CDN to cache different payloads per device type is incorrect because CDNs cache static content and cannot dynamically tailor API responses per device.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    BFF adapts APIs per client type for optimized payloads [check]
Hint: BFF adapts APIs per client type for optimized payloads [check]
Common Mistakes:
2. A web application needs to handle millions of concurrent users sending requests simultaneously. Which component is primarily responsible for managing client connections and distributing requests to backend services?
easy
A. Database server, because it stores all user data and handles queries
B. Load balancer, because it manages incoming client connections and routes requests
C. Cache server, because it speeds up data retrieval for clients
D. Message queue, because it buffers requests for asynchronous processing

Solution

  1. Step 1: Identify the role of the load balancer

    The load balancer manages client connections and distributes incoming requests evenly across backend servers to prevent overload.

  2. Step 2: Eliminate other components

    The database server handles data storage but does not manage client connections; cache servers speed up data retrieval but don't handle connections; message queues buffer asynchronous tasks but don't manage live client connections.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Load balancer routes client requests to backend servers [check]
Hint: Load balancer directs traffic, not stores or caches data [check]
Common Mistakes:
3. For a globally distributed key-value store requiring low write latency and strong consistency, which replication approach is best and why?
medium
A. Single-leader replication with leader in primary region only
B. Asynchronous replication with conflict resolution on reads
C. Synchronous replication with majority quorum writes to all regions
D. Synchronous replication with local quorum writes per region and global reconciliation

Solution

  1. Step 1: Evaluate latency and consistency trade-offs

    Global synchronous quorum to all regions (A) causes high latency; asynchronous (B) sacrifices strong consistency.

  2. Step 2: Consider local quorum with global reconciliation

    Local quorum writes reduce latency while global reconciliation maintains strong consistency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Local quorum + global reconciliation balances latency and consistency [check]
Hint: Local quorum writes reduce latency, global reconciliation ensures consistency [check]
Common Mistakes:
4. Without a clear action item tracking system after a post-mortem, what specific failure occurs in the incident resolution process?
medium
A. The root cause analysis becomes inaccurate due to missing data
B. Action items are forgotten or delayed, causing repeated incidents
C. Incident detection slows down because alerts are not configured
D. The blame culture intensifies as no one is held accountable

Solution

  1. Step 1: Identify role of action item tracking

    Tracking ensures follow-through on improvements, preventing recurrence and reducing blame.

  2. Step 2: Trace failure without tracking

    Without tracking, accountability gaps appear, blame culture can worsen as no clear ownership exists.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    No tracking -> blame culture intensifies due to lack of ownership [check]
Hint: No tracking -> blame culture grows, fixes stall [check]
Common Mistakes:
5. At 10x scale, which component of a distributed Count-Min Sketch system is most likely to fail first and why?
hard
A. The network bandwidth saturates due to large sketch state synchronization
B. The sketch arrays in memory overflow due to fixed size limits
C. The hash functions become slower, increasing latency linearly
D. The centralized coordinator bottlenecks due to serializing all update requests

Solution

  1. Step 1: Identify bottleneck in distributed updates

    Centralized coordinator serializes updates to maintain consistency, limiting throughput.

  2. Step 2: Analyze other components

    Network bandwidth and hash function speed scale better; sketch arrays have fixed size but don't overflow.

  3. Step 3: Understand scale impact

    At 10x scale, coordinator becomes overwhelmed, causing update latency and failures.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Central coordinator serialization bottlenecks updates at scale [check]
Hint: Central coordinator serializes updates, bottlenecking at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard