Idempotent event consumers in Microservices - System Design Exercise

Design: Idempotent Event Consumer System

Design focuses on the event consumer microservice and its idempotency mechanisms. Event producers, event brokers, and downstream systems are out of scope.

Functional Requirements

FR1: Consume events from an event stream or message queue reliably

FR2: Ensure each event is processed exactly once, even if delivered multiple times

FR3: Handle duplicate events gracefully without side effects

FR4: Support concurrent processing of events for scalability

FR5: Provide monitoring and alerting for failed or stuck event processing

Non-Functional Requirements

NFR1: Must handle up to 10,000 events per second

NFR2: Event processing latency p99 under 200ms

NFR3: System availability target 99.9% uptime

NFR4: Event ordering is not guaranteed but should be preserved per event key if possible

NFR5: Use common microservices technologies and patterns

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Event broker or message queue (e.g., Kafka, RabbitMQ)

Event consumer microservice

Idempotency store or cache (e.g., Redis, database)

Processing logic and side effect handlers

Monitoring and alerting tools

Design Patterns

Idempotent consumer pattern

Deduplication using unique event IDs

Transactional outbox or event sourcing

Exactly-once processing with distributed locks

Retry and dead-letter queue handling

Reference Architecture

  +----------------+       +------------------+       +--------------------+
  | Event Producer | ----> | Event Broker     | ----> | Event Consumer      |
  +----------------+       +------------------+       +--------------------+
                                                           |          |
                                                           |          v
                                                           |    +-------------+
                                                           |    | Idempotency |
                                                           |    | Store       |
                                                           |    +-------------+
                                                           |
                                                           v
                                                  +--------------------+
                                                  | Downstream Systems  |
                                                  +--------------------+

Components

Event Broker

Apache Kafka or RabbitMQ

Reliable event delivery with at-least-once semantics

Event Consumer Microservice

Java Spring Boot or Node.js with Kafka client

Consumes events, processes business logic, ensures idempotency

Idempotency Store

Redis or relational database

Stores processed event IDs to detect duplicates and prevent reprocessing

Downstream Systems

Databases, APIs, or other microservices

Receive side effects or results of event processing

Monitoring and Alerting

Prometheus, Grafana, Alertmanager

Track consumer health, processing latency, and failures

Request Flow

1. 1. Event Producer publishes event with unique event ID to Event Broker.

2. 2. Event Consumer subscribes and receives event from Event Broker.

3. 3. Consumer checks Idempotency Store for event ID.

4. 4. If event ID exists, consumer skips processing to avoid duplicate side effects.

5. 5. If event ID does not exist, consumer processes event and applies side effects.

6. 6. After successful processing, consumer records event ID in Idempotency Store.

7. 7. Consumer acknowledges event to Event Broker to commit offset or remove message.

8. 8. Monitoring system collects metrics on processing success, latency, and errors.

Database Schema

Entities: - ProcessedEvent - event_id (PK, string): Unique identifier of the event - processed_at (timestamp): When the event was processed - status (string): Processing status (e.g., success, failed) Relationships: - No direct relationships needed; this table is used solely for idempotency checks.

Scaling Discussion

Bottlenecks

Idempotency Store becomes a hotspot due to frequent reads/writes for event ID checks

Event Consumer CPU or memory limits when processing high event throughput

Event Broker partitions or throughput limits

Network latency between consumer and idempotency store

Handling large event payloads causing slow processing

Solutions

Use a highly performant in-memory store like Redis with sharding for idempotency checks

Partition event consumption across multiple consumer instances for parallelism

Increase Event Broker partitions and tune throughput settings

Co-locate consumer and idempotency store or use caching layers to reduce latency

Optimize event payload size and processing logic; use asynchronous side effects if possible

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Explain the importance of idempotency in event-driven systems to avoid duplicate side effects

Describe how unique event IDs and an idempotency store prevent reprocessing

Discuss trade-offs between consistency, latency, and throughput

Mention failure handling with retries and dead-letter queues

Highlight monitoring to detect processing issues early