Producer-consumer pattern in HLD - System Design Exercise

Design: Producer-Consumer Messaging System

Design focuses on the messaging system core with producers, consumers, and message queue. Out of scope are producer and consumer internal logic and UI.

Functional Requirements

FR1: Allow multiple producers to send messages asynchronously

FR2: Allow multiple consumers to receive and process messages independently

FR3: Ensure messages are processed in the order they are produced

FR4: Support message durability to avoid data loss

FR5: Handle varying rates of production and consumption without losing messages

Non-Functional Requirements

NFR1: System should handle up to 10,000 messages per second

NFR2: Message processing latency p99 should be under 500ms

NFR3: System availability should be at least 99.9%

NFR4: Messages must not be lost even if consumers are temporarily down

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Message queue or broker

Producers (message senders)

Consumers (message processors)

Persistence storage for durability

Load balancer or dispatcher

Design Patterns

Message queue with FIFO ordering

Acknowledgement and retry mechanism

Backpressure handling

Load balancing among consumers

Dead letter queue for failed messages

Reference Architecture

  +------------+       +----------------+       +------------+
  |  Producer  | ----> | Message Queue  | ----> |  Consumer  |
  +------------+       +----------------+       +------------+
         |                     |                      |
         |                     |                      |
         +---- Multiple ----+  |  Multiple Consumers  |
                            |  |                      |
                      +----------------+              |
                      | Persistence DB | <------------+
                      +----------------+

Components

Producer

Any application or service

Creates and sends messages asynchronously to the message queue

Message Queue

RabbitMQ / Kafka / AWS SQS

Stores messages in order, supports asynchronous delivery, and ensures durability

Consumer

Any application or service

Receives messages from the queue and processes them independently

Persistence DB

Relational or NoSQL database

Stores messages durably to prevent data loss in case of queue failure

Request Flow

1. Producer creates a message and sends it to the Message Queue

2. Message Queue stores the message persistently and maintains order

3. Consumers subscribe to the Message Queue and receive messages asynchronously

4. Consumer processes the message and sends acknowledgement

5. Message Queue removes acknowledged messages from the queue

6. If consumer fails to acknowledge, message is retried or moved to dead letter queue

Database Schema

Entities: - Message: id (PK), content, status (pending, processing, done, failed), timestamp, producer_id - Producer: id (PK), name - Consumer: id (PK), name Relationships: - Message linked to Producer by producer_id - Message status tracks processing state - Messages stored in queue and persisted in DB for durability

Scaling Discussion

Bottlenecks

Message Queue throughput limits with very high message rates

Consumer processing speed slower than message production causing backlog

Persistence DB write latency affecting message durability

Network latency between producers, queue, and consumers

Solutions

Partition the message queue to distribute load (sharding or topics)

Add more consumer instances for parallel processing

Use faster storage or caching layers for persistence (e.g., SSD, Redis)

Implement backpressure to slow producers when consumers lag

Use regional deployment to reduce network latency

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain asynchronous decoupling of producers and consumers

Discuss message durability and ordering guarantees

Highlight handling of failure scenarios and retries

Describe scaling strategies for high throughput

Mention trade-offs between complexity and reliability