Idempotency for safe retries in HLD - System Design Guide

Problem Statement

When a client or network fails during a request, retrying the same operation can cause duplicate effects like double charges or repeated data entries. Without a way to detect and ignore repeated requests, the system can produce inconsistent or harmful results.

Solution

Idempotency ensures that repeating the same request multiple times has the same effect as doing it once. The system assigns a unique key to each request and stores the result. On retries with the same key, it returns the stored result instead of processing again, preventing duplicate side effects.

Architecture

Client

(sends req)

→API Gateway

↓

Idempotency Key

Store

This diagram shows the client sending a request with an idempotency key to the API Gateway, which checks the key store before forwarding to the service. The service processes the request and stores the result in the database. On retries, the gateway returns the stored result without reprocessing.

Trade-offs

✓ Pros

→

Prevents duplicate side effects from retries, ensuring data consistency.

→

Improves user experience by safely allowing automatic retries on failures.

→

Reduces load on backend services by avoiding repeated processing of the same request.

✗ Cons

→

Requires additional storage and management of idempotency keys and results.

→

Adds complexity to request handling logic and state management.

→

Needs careful expiration or cleanup of stored keys to avoid unbounded growth.

Use when operations have side effects that must not be repeated, especially in payment, order processing, or resource creation systems with retry-prone clients or unreliable networks.

Avoid when operations are naturally idempotent or stateless, or when retry rates are negligible and the overhead of managing keys outweighs benefits.

Real World Examples

Stripe

Stripe uses idempotency keys to ensure that payment requests retried due to network errors do not result in multiple charges.

Amazon

Amazon's order processing APIs use idempotency to prevent duplicate orders when customers retry submissions after failures.

Google Cloud Storage

Google Cloud Storage supports idempotent upload requests to avoid duplicate file writes on retries.

Alternatives

Retry with exponential backoff

Retries requests with increasing delays but does not prevent duplicate side effects if the operation is not idempotent.

Use when: Use when operations are safe to repeat or when idempotency is not feasible to implement.

Transaction-based processing

Uses database transactions to ensure atomicity but does not handle retries at the client or API level.

Use when: Use when the system can rely on strong database guarantees and retries are controlled internally.

Summary

Idempotency prevents duplicate effects from retrying the same request multiple times.

It works by storing and reusing results keyed by a unique request identifier.

This pattern is essential for operations with side effects in unreliable network environments.