Exactly-once processing challenges in HLD - Architecture Diagram

System Overview - Exactly-once processing challenges

This system ensures that each message or transaction is processed exactly once, avoiding duplicates or losses. It is critical for financial systems, order processing, and any scenario where repeated or missed processing causes errors.

Key requirements include handling retries, failures, and network issues without duplicating work or losing data.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
+-------------------+
|  Processing Service|
|  +-------------+  |
|  | Idempotency |  |
|  |  Store      |  |
|  +-------------+  |
+-------------------+
  |
  v
Message Queue <--> Idempotency Store
  |
  v
Database
  |
  v
Cache

Components

User

client

Initiates requests that require exactly-once processing

Load Balancer

load_balancer

Distributes incoming requests evenly to API Gateway instances

API Gateway

api_gateway

Receives client requests and routes them to processing services

Processing Service

service

Processes requests ensuring exactly-once semantics using idempotency checks

Idempotency Store

database

Stores unique request IDs and their processing status to prevent duplicates

Message Queue

queue

Buffers messages for asynchronous processing and retry handling

Database

database

Stores the main application data updated by processing service

Cache

cache

Speeds up read operations and reduces load on the database

Request Flow - 11 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Processing Service

Processing Service → Idempotency Store

Processing Service → Message Queue

Message Queue → Processing Service

Processing Service → Database

Processing Service → Idempotency Store

Processing Service → Cache

Processing Service → API Gateway

API Gateway → User

Failure Scenario

Component Fails:Idempotency Store

Impact:Without idempotency checks, duplicate processing may occur causing inconsistent data or double charges.

Mitigation:Use a replicated and highly available idempotency store; fallback to strict locking or reject requests until recovery.

Architecture Quiz - 3 Questions

Which component prevents processing the same request multiple times?

A Cache

B Load Balancer

C Idempotency Store

D Message Queue

Design Principle

This architecture uses an idempotency store combined with a message queue to guarantee exactly-once processing. The idempotency store prevents duplicate processing by tracking request IDs. The message queue enables reliable asynchronous processing and retries without losing messages. Updating the cache after the database ensures fast reads with consistent data. This design balances reliability, scalability, and data correctness.