Exactly-once processing challenges in HLD - System Design Exercise

Design: Exactly-once Processing System

Design focuses on the core processing pipeline ensuring exactly-once semantics. Out of scope are UI components, detailed security policies, and specific business logic of message content.

Functional Requirements

FR1: Process each message or event exactly once without duplication or loss

FR2: Support high throughput of at least 10,000 messages per second

FR3: Ensure system recovers gracefully from failures without reprocessing messages

FR4: Provide end-to-end data consistency between producers and consumers

FR5: Allow horizontal scaling to handle increased load

Non-Functional Requirements

NFR1: Maximum processing latency of 200ms per message

NFR2: Availability target of 99.9% uptime

NFR3: Support distributed environment with multiple producers and consumers

NFR4: Use industry-standard messaging and storage technologies

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Message queue or event streaming platform (e.g., Kafka, RabbitMQ)

Idempotent processing logic or deduplication store

State store or database with transactional support

Checkpointing or offset tracking mechanism

Distributed coordination service (e.g., ZooKeeper, etcd)

Design Patterns

Idempotent consumer pattern

Transactional outbox pattern

Two-phase commit or distributed transactions

Exactly-once delivery with offset commit

Event sourcing with deduplication

Reference Architecture

  +-------------+       +----------------+       +----------------+
  |  Producers  | ----> |  Message Queue | ----> |  Consumers     |
  +-------------+       +----------------+       +----------------+
                                |                        |
                                |                        v
                                |               +----------------+
                                |               | State Store /  |
                                |               | Deduplication  |
                                |               +----------------+
                                |                        |
                                |                        v
                                |               +----------------+
                                |               | Checkpointing  |
                                |               | & Offset Store |
                                |               +----------------+

Components

Producers

Any client application or service

Send messages/events to the message queue reliably

Message Queue

Apache Kafka or RabbitMQ

Durably store messages and support ordered delivery with offset tracking

Consumers

Stateless or stateful processing service

Process messages exactly once using idempotent logic and transactional updates

State Store / Deduplication Store

Distributed database with transactional support (e.g., PostgreSQL, Cassandra)

Track processed message IDs or offsets to avoid duplicates

Checkpointing & Offset Store

Kafka offset commit or external store

Record progress of processed messages to enable recovery without reprocessing

Request Flow

1. Producer sends message to the message queue with unique message ID.

2. Message queue stores message durably and assigns an offset.

3. Consumer reads message from queue and checks deduplication store for message ID.

4. If message ID not processed, consumer processes message and updates state store transactionally.

5. Consumer commits offset/checkpoint only after successful processing and state update.

6. If failure occurs, consumer restarts from last committed offset to avoid duplicates.

7. Deduplication store ensures repeated messages are ignored, achieving exactly-once processing.

Database Schema

Entities: - MessagesProcessed(message_id PRIMARY KEY, processed_timestamp) - ConsumerOffsets(consumer_id PRIMARY KEY, last_committed_offset) Relationships: - MessagesProcessed tracks unique message IDs to prevent duplicates. - ConsumerOffsets track progress per consumer group for recovery.

Scaling Discussion

Bottlenecks

State store becoming a write bottleneck due to high volume of deduplication checks

Message queue throughput limits under very high load

Checkpointing latency delaying offset commits

Network latency between distributed components affecting consistency

Solutions

Partition state store by consumer or message key to distribute load

Use scalable message queues like Kafka with partitioning and replication

Batch checkpoint commits to reduce overhead while balancing latency

Deploy components in the same data center or use faster network links

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and failure handling, and 5 minutes summarizing.

Explain the difference between at-least-once, at-most-once, and exactly-once processing

Discuss failure scenarios and how the design handles them

Describe how idempotency and transactional updates prevent duplicates

Highlight the role of checkpointing and offset management

Mention trade-offs between complexity and guarantees

Practice

1. What is the main goal of exactly-once processing in distributed systems?

easy

A. To ensure each message or transaction is processed only once without duplicates or losses

B. To process messages as fast as possible without caring about duplicates

C. To allow multiple processing attempts for fault tolerance

D. To ignore message ordering for simplicity

Solution

1. Step 1: Understand exactly-once processing definition

   Exactly-once processing means each message or transaction is handled one time only, avoiding duplicates or losses.

2. Step 2: Compare options with definition

   To ensure each message or transaction is processed only once without duplicates or losses matches the definition exactly. Other options either ignore duplicates, allow multiple processing, or ignore ordering, which are not goals of exactly-once.

3. Final Answer:

   To ensure each message or transaction is processed only once without duplicates or losses -> Option A

4. Quick Check:

   Exactly-once = no duplicates or losses [OK]

Hint: Exactly-once means no duplicates, no losses [OK]

Common Mistakes:

• Confusing exactly-once with at-least-once or at-most-once
• Ignoring message loss scenarios
• Assuming speed over correctness

2. Which of the following is a key technique to achieve exactly-once processing?

easy

A. Processing messages in any order without tracking

B. Using unique message IDs and idempotent operations

C. Ignoring message duplicates and retrying indefinitely

D. Dropping messages after first failure

Solution

1. Step 1: Identify techniques for exactly-once

   Exactly-once requires unique IDs to detect duplicates and idempotent operations to safely retry without side effects.

2. Step 2: Evaluate options

   Using unique message IDs and idempotent operations correctly lists these techniques. Options A, B, and D either ignore tracking or order, ignore duplicates, or drop messages, which break exactly-once guarantees.

3. Final Answer:

   Using unique message IDs and idempotent operations -> Option B

4. Quick Check:

   Unique IDs + idempotency = exactly-once [OK]

Hint: Unique IDs + idempotent ops ensure exactly-once [OK]

Common Mistakes:

• Assuming retries alone guarantee exactly-once
• Ignoring idempotency importance
• Not using unique identifiers

3. Consider a system where a message with ID 123 is processed twice due to a retry. The processing function is idempotent. What is the expected outcome?

medium

A. The second processing has no additional effect, preserving exactly-once semantics

B. The message is processed twice causing duplicate side effects

C. The system crashes due to duplicate message ID

D. The message is ignored completely

Solution

1. Step 1: Understand idempotent processing

   Idempotent means running the same operation multiple times has the same effect as running it once.

2. Step 2: Analyze retry with same message ID

   Since the function is idempotent, the second processing does not cause duplicate side effects, preserving exactly-once behavior.

3. Final Answer:

   The second processing has no additional effect, preserving exactly-once semantics -> Option A

4. Quick Check:

   Idempotent retry = no duplicate effect [OK]

Hint: Idempotent means retry won't duplicate effects [OK]

Common Mistakes:

• Assuming retries always cause duplicates
• Confusing idempotency with no retries
• Thinking system crashes on duplicate IDs

4. A system designed for exactly-once processing is dropping some messages after retries fail. What is the likely cause?

medium

A. Using unique IDs for messages

B. Retrying messages indefinitely

C. Implementing idempotent operations

D. Lack of transactional guarantees causing message loss

Solution

1. Step 1: Identify cause of message drops

   Dropping messages after retries usually means the system lacks transactional guarantees to ensure messages are not lost.

2. Step 2: Evaluate options

   Lack of transactional guarantees causing message loss correctly identifies this cause. Options A and C help prevent duplicates, and D would avoid drops by retrying indefinitely.

3. Final Answer:

   Lack of transactional guarantees causing message loss -> Option D

4. Quick Check:

   Missing transactions = message loss [OK]

Hint: No transactions cause message loss on failure [OK]

Common Mistakes:

• Confusing idempotency with message loss prevention
• Assuming retries always prevent drops
• Ignoring transactional role

5. You are designing a payment processing system that must guarantee exactly-once processing despite network failures. Which combination of strategies best ensures this?

hard

A. Drop duplicate payments and rely on manual reconciliation

B. Retry payments without tracking IDs and ignore failures

C. Use unique transaction IDs, idempotent payment operations, and a transactional message queue

D. Process payments asynchronously without acknowledgments

Solution

1. Step 1: Identify requirements for exactly-once in payments

   Payments must not be duplicated or lost, even with network failures.

2. Step 2: Evaluate strategies

   Unique transaction IDs detect duplicates, idempotent operations prevent repeated side effects, and transactional queues ensure atomic message delivery.

3. Step 3: Compare options

   Use unique transaction IDs, idempotent payment operations, and a transactional message queue combines all needed strategies. Options A, B, and C risk duplicates, losses, or manual errors.

4. Final Answer:

   Use unique transaction IDs, idempotent payment operations, and a transactional message queue -> Option C

5. Quick Check:

   IDs + idempotency + transactions = exactly-once [OK]

Hint: Combine IDs, idempotency, and transactions for exactly-once [OK]

Common Mistakes:

• Ignoring transactional guarantees
• Not using unique IDs
• Relying on manual fixes