Exactly-once processing challenges in HLD - System Design Guide

Problem Statement

When processing messages or events, duplicates or lost messages cause incorrect results or inconsistent state. Systems that retry operations to handle failures risk processing the same message multiple times, leading to data corruption or double charges.

Solution

Exactly-once processing ensures each message affects the system state only once, even with retries or failures. This is achieved by combining idempotent operations, deduplication, and transactional guarantees to track and commit processing atomically.

Architecture

Message Queue

(with unique

→Processing

↓

Deduplication Cache

Deduplication Cache

This diagram shows a message queue delivering uniquely identified messages to a processing component that performs idempotent operations. The state store uses transactions and deduplication cache to ensure each message updates state exactly once.

Trade-offs

✓ Pros

→

Prevents data corruption and inconsistent state caused by duplicate processing.

→

Improves reliability by safely allowing retries without side effects.

→

Supports strong correctness guarantees critical for financial and inventory systems.

✗ Cons

→

Adds complexity in tracking message IDs and managing deduplication state.

→

May increase latency due to transactional overhead and coordination.

→

Requires careful design to handle state store failures and cache eviction.

Use when processing critical data where duplicates cause severe errors, such as payments, inventory updates, or user account changes, especially at scale above thousands of messages per second.

Avoid when processing is naturally idempotent or duplicates have no harmful effects, or when system scale is low (under hundreds of messages per second) and complexity outweighs benefits.

Real World Examples

Uber

Uber uses exactly-once processing in their trip billing system to avoid double charging riders despite retries and network failures.

Stripe

Stripe ensures exactly-once payment processing to prevent duplicate charges during network retries or system crashes.

LinkedIn

LinkedIn applies exactly-once semantics in their messaging platform to avoid duplicate notifications and maintain consistent user state.

Alternatives

At-least-once processing

Processes messages one or more times, risking duplicates but simpler to implement.

Use when: When occasional duplicates are acceptable and system simplicity or throughput is prioritized.

At-most-once processing

Processes messages zero or one time, risking message loss but no duplicates.

Use when: When losing some messages is acceptable but duplicates must be avoided.

Summary

Exactly-once processing prevents duplicate effects from retries or failures in message processing.

It combines idempotent operations, deduplication, and transactional state updates to ensure correctness.

This pattern is essential for critical systems like payments but adds complexity and overhead.