Idempotent event consumers in Microservices - System Design Exercise

Design: Idempotent Event Consumer System

Design focuses on the event consumer microservice and its idempotency mechanisms. Event producers, event brokers, and downstream systems are out of scope.

Functional Requirements

FR1: Consume events from an event stream or message queue reliably

FR2: Ensure each event is processed exactly once, even if delivered multiple times

FR3: Handle duplicate events gracefully without side effects

FR4: Support concurrent processing of events for scalability

FR5: Provide monitoring and alerting for failed or stuck event processing

Non-Functional Requirements

NFR1: Must handle up to 10,000 events per second

NFR2: Event processing latency p99 under 200ms

NFR3: System availability target 99.9% uptime

NFR4: Event ordering is not guaranteed but should be preserved per event key if possible

NFR5: Use common microservices technologies and patterns

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Event broker or message queue (e.g., Kafka, RabbitMQ)

Event consumer microservice

Idempotency store or cache (e.g., Redis, database)

Processing logic and side effect handlers

Monitoring and alerting tools

Design Patterns

Idempotent consumer pattern

Deduplication using unique event IDs

Transactional outbox or event sourcing

Exactly-once processing with distributed locks

Retry and dead-letter queue handling

Reference Architecture

  +----------------+       +------------------+       +--------------------+
  | Event Producer | ----> | Event Broker     | ----> | Event Consumer      |
  +----------------+       +------------------+       +--------------------+
                                                           |          |
                                                           |          v
                                                           |    +-------------+
                                                           |    | Idempotency |
                                                           |    | Store       |
                                                           |    +-------------+
                                                           |
                                                           v
                                                  +--------------------+
                                                  | Downstream Systems  |
                                                  +--------------------+

Components

Event Broker

Apache Kafka or RabbitMQ

Reliable event delivery with at-least-once semantics

Event Consumer Microservice

Java Spring Boot or Node.js with Kafka client

Consumes events, processes business logic, ensures idempotency

Idempotency Store

Redis or relational database

Stores processed event IDs to detect duplicates and prevent reprocessing

Downstream Systems

Databases, APIs, or other microservices

Receive side effects or results of event processing

Monitoring and Alerting

Prometheus, Grafana, Alertmanager

Track consumer health, processing latency, and failures

Request Flow

1. 1. Event Producer publishes event with unique event ID to Event Broker.

2. 2. Event Consumer subscribes and receives event from Event Broker.

3. 3. Consumer checks Idempotency Store for event ID.

4. 4. If event ID exists, consumer skips processing to avoid duplicate side effects.

5. 5. If event ID does not exist, consumer processes event and applies side effects.

6. 6. After successful processing, consumer records event ID in Idempotency Store.

7. 7. Consumer acknowledges event to Event Broker to commit offset or remove message.

8. 8. Monitoring system collects metrics on processing success, latency, and errors.

Database Schema

Entities: - ProcessedEvent - event_id (PK, string): Unique identifier of the event - processed_at (timestamp): When the event was processed - status (string): Processing status (e.g., success, failed) Relationships: - No direct relationships needed; this table is used solely for idempotency checks.

Scaling Discussion

Bottlenecks

Idempotency Store becomes a hotspot due to frequent reads/writes for event ID checks

Event Consumer CPU or memory limits when processing high event throughput

Event Broker partitions or throughput limits

Network latency between consumer and idempotency store

Handling large event payloads causing slow processing

Solutions

Use a highly performant in-memory store like Redis with sharding for idempotency checks

Partition event consumption across multiple consumer instances for parallelism

Increase Event Broker partitions and tune throughput settings

Co-locate consumer and idempotency store or use caching layers to reduce latency

Optimize event payload size and processing logic; use asynchronous side effects if possible

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Explain the importance of idempotency in event-driven systems to avoid duplicate side effects

Describe how unique event IDs and an idempotency store prevent reprocessing

Discuss trade-offs between consistency, latency, and throughput

Mention failure handling with retries and dead-letter queues

Highlight monitoring to detect processing issues early

Practice

1. What is the main purpose of an idempotent event consumer in microservices?

easy

A. To generate new events based on incoming data

B. To speed up event processing by ignoring event order

C. To ensure the same event is processed only once, avoiding duplicates

D. To store all events permanently for auditing

Solution

1. Step 1: Understand event duplication problem

   In microservices, events can be delivered multiple times due to retries or network issues.

2. Step 2: Role of idempotent consumer

   An idempotent event consumer tracks processed event IDs to avoid processing the same event more than once.

3. Final Answer:

   To ensure the same event is processed only once, avoiding duplicates -> Option C

4. Quick Check:

   Idempotent consumer = avoid duplicate processing [OK]

Hint: Idempotent means safe to repeat without side effects [OK]

Common Mistakes:

• Confusing idempotency with event ordering
• Thinking it stores all events permanently
• Assuming it generates new events

2. Which of the following is a correct way to implement idempotency in an event consumer?

easy

A. Process events without checking any IDs

B. Store processed event IDs and skip duplicates

C. Ignore event payload and always acknowledge

D. Process events only if they arrive in order

Solution

1. Step 1: Identify idempotency implementation

   Idempotency requires tracking which events were already processed.

2. Step 2: Choose correct method

   Storing processed event IDs and skipping duplicates ensures no repeated processing.

3. Final Answer:

   Store processed event IDs and skip duplicates -> Option B

4. Quick Check:

   Track event IDs = idempotency [OK]

Hint: Track event IDs to skip duplicates [OK]

Common Mistakes:

• Not checking event IDs before processing
• Assuming order guarantees idempotency
• Ignoring event payload without validation

3. Consider this pseudocode for an event consumer:

processed_events = set()

def consume(event):
    if event.id in processed_events:
        return "Skipped"
    process(event)
    processed_events.add(event.id)
    return "Processed"

What will be the output if the same event with id=42 is consumed twice?

medium

A. ["Processed", "Processed"]

B. ["Skipped", "Skipped"]

C. ["Skipped", "Processed"]

D. ["Processed", "Skipped"]

Solution

1. Step 1: Analyze first event consumption

   Event with id=42 is not in processed_events initially, so it is processed and id added.

2. Step 2: Analyze second event consumption

   On second call, id=42 is in processed_events, so event is skipped.

3. Final Answer:

   ["Processed", "Skipped"] -> Option D

4. Quick Check:

   First process, then skip duplicates [OK]

Hint: First time process, next times skip [OK]

Common Mistakes:

• Assuming both events are processed
• Mixing order of outputs
• Not adding event ID after processing

4. A microservice uses an idempotent event consumer but still processes some events twice. What is the most likely cause?

medium

A. The event IDs are not unique or not stored correctly

B. The consumer processes events too slowly

C. The event payload is too large to process

D. The events arrive in the wrong order

Solution

1. Step 1: Understand idempotency failure reasons

   If events are processed twice, the system likely fails to recognize duplicates.

2. Step 2: Identify cause

   Non-unique event IDs or failure to store them properly causes duplicate processing.

3. Final Answer:

   The event IDs are not unique or not stored correctly -> Option A

4. Quick Check:

   Unique IDs + storage = no duplicates [OK]

Hint: Check event ID uniqueness and storage [OK]

Common Mistakes:

• Blaming event order for duplicates
• Assuming processing speed causes duplicates
• Ignoring event ID uniqueness

5. You design a microservice that consumes events from a message queue. To ensure idempotency, you decide to store processed event IDs in a database. Which approach best balances scalability and correctness?

hard

A. Store event IDs in a centralized database with unique constraints

B. Store event IDs in a local in-memory cache only

C. Ignore event IDs and rely on message queue retries

D. Process events multiple times and fix duplicates later

Solution

1. Step 1: Evaluate local cache approach

   Local cache is fast but not shared across instances, causing duplicates in distributed systems.

2. Step 2: Evaluate centralized DB with unique constraints

   A centralized database with unique event ID constraints ensures correctness and scales with proper design.

3. Step 3: Evaluate ignoring IDs or fixing later

   Ignoring IDs or fixing duplicates later risks data inconsistency and is not reliable.

4. Final Answer:

   Store event IDs in a centralized database with unique constraints -> Option A

5. Quick Check:

   Central DB + unique IDs = scalable correctness [OK]

Hint: Use centralized DB with unique keys for idempotency [OK]

Common Mistakes:

• Using only local cache in distributed systems
• Ignoring event IDs completely
• Accepting duplicates to fix later