Bird
Raised Fist0

Idempotency in APIs - Why It Is Critical for Payment and Retry Systems

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Idempotency in APIs - Why It Is Critical for Payment and Retry Systems
45 min · senior
senior

You are designing the idempotency mechanism for a payment processing system at a startup with 50 million daily active users (DAU). The system must handle high retry rates due to network failures and client-side retries, ensuring no double charges or inconsistent states. Payments are sensitive and require encryption at rest and in transit. The system must guarantee P99 latency under 200ms for API responses and support offline retry delivery without compromising consistency.

Constraints
  • End-to-end encryption of payment data in transit and at rest is mandatory due to PCI compliance.
  • System must handle up to 1 million payment requests per second with retries.
  • P99 API latency must be under 200ms including idempotency checks.
  • Offline retry delivery must be supported without losing idempotency guarantees.
  • Idempotency keys must be scoped per user and per payment operation.
  • Storage for idempotency state must be highly available and horizontally scalable.
📋
Guided Steps
1/5
Step 1
Idempotency Key Design and Scope
How would you design the idempotency key schema and scope it to ensure uniqueness and prevent collisions?
💡 Hint
Consider user-specific and operation-specific uniqueness to avoid key reuse across different payments.
Key Decision
Use a composite idempotency key combining user_id + payment_operation_id, stored in DynamoDB with conditional writes to ensure uniqueness and atomicity.
Interviewer Expectation
Mid-level: Names DynamoDB for key-value storage. Senior: Explains composite key choice to avoid cross-user collisions and why conditional writes prevent race conditions.
⚠️Common Wrong Answer
Using a global UUID as idempotency key without user scoping, leading to potential collisions and incorrect deduplication.
Step 2
Idempotency State Storage and Consistency
Which storage system would you use to store idempotency state and why? How do you ensure consistency under concurrent retries?
💡 Hint
Look for a storage with strong conditional write support and low latency at scale.
Key Decision
Use DynamoDB with conditional PutItem operations and TTL to store idempotency state, ensuring atomic check-and-set semantics and automatic cleanup.
Interviewer Expectation
Mid-level: Mentions DynamoDB for scalability. Senior: Details conditional writes to prevent duplicate processing and TTL to avoid unbounded storage growth.
⚠️Common Wrong Answer
Using Redis without persistence or conditional writes, risking lost state and duplicate processing under failover.
Step 3
Handling Offline Retries and Delivery Guarantees
How would you design the system to handle offline retries while preserving idempotency guarantees?
💡 Hint
Consider durable message queues with exactly-once or effectively-once delivery semantics and idempotency state checks.
Key Decision
Use Apache Kafka with user_id as partition key to guarantee ordered delivery per user, combined with idempotency state checks in DynamoDB before processing each message.
Interviewer Expectation
Mid-level: Names Kafka for message durability. Senior: Explains partitioning by user_id to maintain order and how idempotency state prevents duplicate processing on retries.
⚠️Common Wrong Answer
Using RabbitMQ without partitioning, causing out-of-order retries and inconsistent payment states.
Step 4
API Endpoint Design for Idempotent Payment Processing
Sketch the 3 core API endpoints needed to support idempotent payment processing and retries.
💡 Hint
Think about endpoints for submitting payments, querying status, and acknowledging processing.
Key Decision
Design endpoints: POST /payments with idempotency-key header for submission, GET /payments/{id} for status query, and POST /payments/{id}/ack for explicit acknowledgment; backed by DynamoDB and Kafka for state and messaging.
Interviewer Expectation
Mid-level: Names endpoints and basic usage. Senior: Explains how idempotency-key header prevents duplicates, status endpoint supports client sync, and ack endpoint finalizes state to avoid retries.
⚠️Common Wrong Answer
Single endpoint without idempotency-key support, causing duplicate charges on retries.
Step 5
Encryption and Security Considerations
How do you ensure encryption and security compliance for payment data in this idempotency system?
💡 Hint
Consider encryption at rest, in transit, and access controls on storage and messaging layers.
Key Decision
Use TLS 1.3 for all API and Kafka communication, enable server-side encryption with AWS KMS for DynamoDB, and enforce IAM policies restricting access to idempotency data.
Interviewer Expectation
Mid-level: Mentions TLS and encryption. Senior: Details KMS integration, key rotation, and fine-grained IAM roles to minimize attack surface.
⚠️Common Wrong Answer
Relying only on network encryption without encrypting data at rest, risking PCI compliance violations.
🔌
API Design Challenge
Sketch 3 core API endpoints to support idempotent payment processing and retries.
Hints
  • How does the client submit a payment request with an idempotency key?
  • How does the client query the status of a previously submitted payment?
  • How does the system acknowledge successful processing to prevent duplicate retries?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of composite idempotency keys scoped per user and operation to avoid collisions.
  • Strong consistency storage with conditional writes (DynamoDB) for idempotency state.
  • Message queue with partitioning (Kafka) for ordered offline retry delivery.
  • API endpoints supporting idempotency-key header and status querying.
Good to Have - 30%
  • TTL on idempotency state to prevent storage bloat.
  • Encryption at rest and in transit with AWS KMS and TLS 1.3.
  • Fine-grained IAM policies for access control.
Bonus - 10%
  • Detailed failure mode analysis for race conditions and retry storms.
  • Explicit acknowledgment endpoint to finalize idempotency state.
Anti-patterns to Avoid
Using Redis without persistence or conditional writes, leading to lost idempotency state and duplicate payments under failover, which risks financial loss and inconsistent system state.
Global UUID idempotency keys without user scoping, causing collisions and incorrect deduplication, resulting in potential double charges or missed payments.
Message queues without partitioning (e.g., RabbitMQ default) causing out-of-order retries and inconsistent payment states, which can lead to user confusion and financial discrepancies.
Single API endpoint without idempotency-key support, resulting in duplicate charges on client retries due to inability to detect repeated requests.
✅
Model Answer
▼ Reveal

This design ensures idempotent payment processing at scale by using composite keys scoped per user and operation, leveraging DynamoDB for strong consistency and conditional writes, and Apache Kafka for ordered offline retries. Security is enforced via TLS 1.3 and AWS KMS encryption with strict IAM policies. The API design supports idempotency-key headers, status queries, and explicit acknowledgments to prevent duplicate charges and maintain consistency.

Key Decisions
  • Composite idempotency key design to avoid collisions and ensure uniqueness per user and payment operation.
  • Use of DynamoDB conditional PutItem operations to atomically check and set idempotency state, preventing race conditions.
  • TTL on DynamoDB entries to prevent unbounded storage growth.
  • Kafka partitioning by user_id to maintain message order and enable exactly-once processing semantics.
  • API design with explicit idempotency-key header and acknowledgment endpoint to finalize processing and prevent retries.
  • Enforcement of TLS 1.3 and AWS KMS encryption to meet PCI compliance requirements.
  • Fine-grained IAM policies restricting access to idempotency data and Kafka topics to authorized services only.
  • Composite idempotency key (user_id + payment_operation_id) stored in DynamoDB with conditional writes to ensure uniqueness and atomicity.
  • DynamoDB TTL feature to automatically expire idempotency state entries, preventing unbounded storage growth.
  • Apache Kafka with user_id partition key to guarantee ordered message delivery per user for offline retry handling.
  • API design includes POST /payments with idempotency-key header, GET /payments/{id} for status, and POST /payments/{id}/ack for acknowledgment.
  • TLS 1.3 enforced for all communication channels and AWS KMS used for server-side encryption of DynamoDB data.
  • IAM policies restrict access to idempotency state and Kafka topics to authorized services only.
  • Conditional PutItem operations in DynamoDB prevent race conditions during concurrent retries.
  • Kafka consumer workers keyed by user_id ensure single-threaded processing per user to maintain idempotency guarantees.

Practice

(1/5)
1. A client application needs to fetch deeply nested related data in a single request without multiple round-trips. Which API design approach best supports this requirement?
easy
A. Use GraphQL to request exactly the nested data needed in one query
B. Use WebSockets to stream nested data continuously
C. Use REST with query parameters to fetch all nested data in one endpoint
D. Use REST endpoints with multiple sequential calls to fetch nested resources

Solution

  1. Step 1: Identify nested data fetching needs

    REST typically requires multiple calls or over-fetching to get nested data, causing inefficiency.

  2. Step 2: Recognize GraphQL's single query nested fetch

    GraphQL allows clients to specify nested fields in one request, reducing round-trips and over-fetching.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GraphQL single query fetches nested data efficiently [check]
Hint: GraphQL fetches nested data in one query [check]
Common Mistakes:
2. For a large-scale web crawler, which architecture best balances crawl rate and politeness when crawling millions of domains with varying rate limits?
medium
A. Centralized Scheduler with a global queue and a single Politeness Policy enforcer
B. Distributed Scheduler with domain-based sharding and local Politeness Policy enforcement per shard
C. Multiple independent crawlers each with their own Scheduler and no shared Politeness Policy
D. Single-threaded crawler with a priority queue sorted by domain crawl time

Solution

  1. Step 1: Analyze centralized vs distributed scheduling

    Centralized Scheduler becomes a bottleneck at scale; distributed sharding allows parallelism and localized politeness enforcement.

  2. Step 2: Evaluate other options

    Independent crawlers without shared politeness risk domain overload; single-threaded crawler can't scale to millions of domains.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Distributed sharding enables scalable politeness enforcement [check]
Hint: Distributed sharding scales politeness enforcement best [OK]
Common Mistakes:
3. If the sliding window counter's timestamp cleanup process fails, what is the most likely failure observed in rate limiting?
medium
A. Requests are allowed without limit because counters reset incorrectly
B. Fixed window counters reset too frequently causing bursty traffic spikes
C. Rate limiter blocks requests indefinitely due to stale timestamps inflating counts
D. Token bucket refills tokens too slowly causing underutilization

Solution

  1. Step 1: Understand sliding window cleanup role

    Cleanup removes timestamps outside the window; failure causes stale timestamps to accumulate.

  2. Step 2: Accumulated stale timestamps inflate request counts, causing the rate limiter to block legitimate requests indefinitely

    This leads to false positives in rate limiting, denying service unnecessarily.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Failed cleanup -> stale timestamps inflate counts -> requests blocked indefinitely [check]
Hint: No cleanup means stale timestamps block requests falsely [check]
Common Mistakes:
4. For a system requiring globally unique, roughly time-ordered IDs across multiple data centers, which approach best balances latency and uniqueness guarantees?
medium
A. Use a single centralized ID generator with synchronous replication to all data centers
B. Use decentralized Snowflake-like generators with loosely synchronized clocks and node IDs
C. Generate UUID v4 randomly at each data center without coordination
D. Use database auto-increment IDs partitioned by data center and combine with data center ID

Solution

  1. Step 1: Evaluate centralized generator with sync replication

    This causes high latency and single point of failure, unsuitable for geo-distributed low-latency needs.

  2. Step 2: Assess decentralized Snowflake approach

    Decentralized generators with node IDs and loosely synchronized clocks provide low latency and uniqueness with approximate ordering.

  3. Step 3: Consider UUID v4 and DB auto-increment

    UUID v4 lacks ordering; DB auto-increment with partitioning risks collisions and coordination overhead.

  4. Final Answer:

    Option B -> Option B

  5. Quick Check:

    Decentralized Snowflake -> low latency + uniqueness + approximate ordering [check]
Hint: Decentralized with node ID and clock sync balances latency and uniqueness [check]
Common Mistakes:
5. At 10x scale, which component in a sharded CRDT distributed counter system is most likely to fail first and why?
hard
A. The shard replicas, because network partitions cause permanent state divergence without reconciliation
B. The replication layer, because increased shard count causes exponential message overhead saturating network bandwidth
C. The global aggregator node, because it becomes a bottleneck merging all shard states synchronously
D. The client-side counters, because local state grows unbounded without periodic compaction

Solution

  1. Step 1: Analyze scale impact on components

    Global aggregator (A) merges asynchronously, so less likely bottleneck. Replication overhead (C) grows but can be optimized. Client counters (D) can compact state.

  2. Step 2: Identify shard replicas failure mode

    At 10x scale, network partitions increase, causing shards to diverge permanently without reconciliation, breaking eventual consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Network partitions cause shard divergence first at scale [check]
Hint: Network partitions cause shard divergence before aggregator bottlenecks [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard