Producer-consumer pattern in HLD - System Design Exercise

Design: Producer-Consumer Messaging System

Design focuses on the messaging system core with producers, consumers, and message queue. Out of scope are producer and consumer internal logic and UI.

Functional Requirements

FR1: Allow multiple producers to send messages asynchronously

FR2: Allow multiple consumers to receive and process messages independently

FR3: Ensure messages are processed in the order they are produced

FR4: Support message durability to avoid data loss

FR5: Handle varying rates of production and consumption without losing messages

Non-Functional Requirements

NFR1: System should handle up to 10,000 messages per second

NFR2: Message processing latency p99 should be under 500ms

NFR3: System availability should be at least 99.9%

NFR4: Messages must not be lost even if consumers are temporarily down

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Message queue or broker

Producers (message senders)

Consumers (message processors)

Persistence storage for durability

Load balancer or dispatcher

Design Patterns

Message queue with FIFO ordering

Acknowledgement and retry mechanism

Backpressure handling

Load balancing among consumers

Dead letter queue for failed messages

Reference Architecture

  +------------+       +----------------+       +------------+
  |  Producer  | ----> | Message Queue  | ----> |  Consumer  |
  +------------+       +----------------+       +------------+
         |                     |                      |
         |                     |                      |
         +---- Multiple ----+  |  Multiple Consumers  |
                            |  |                      |
                      +----------------+              |
                      | Persistence DB | <------------+
                      +----------------+

Components

Producer

Any application or service

Creates and sends messages asynchronously to the message queue

Message Queue

RabbitMQ / Kafka / AWS SQS

Stores messages in order, supports asynchronous delivery, and ensures durability

Consumer

Any application or service

Receives messages from the queue and processes them independently

Persistence DB

Relational or NoSQL database

Stores messages durably to prevent data loss in case of queue failure

Request Flow

1. Producer creates a message and sends it to the Message Queue

2. Message Queue stores the message persistently and maintains order

3. Consumers subscribe to the Message Queue and receive messages asynchronously

4. Consumer processes the message and sends acknowledgement

5. Message Queue removes acknowledged messages from the queue

6. If consumer fails to acknowledge, message is retried or moved to dead letter queue

Database Schema

Entities: - Message: id (PK), content, status (pending, processing, done, failed), timestamp, producer_id - Producer: id (PK), name - Consumer: id (PK), name Relationships: - Message linked to Producer by producer_id - Message status tracks processing state - Messages stored in queue and persisted in DB for durability

Scaling Discussion

Bottlenecks

Message Queue throughput limits with very high message rates

Consumer processing speed slower than message production causing backlog

Persistence DB write latency affecting message durability

Network latency between producers, queue, and consumers

Solutions

Partition the message queue to distribute load (sharding or topics)

Add more consumer instances for parallel processing

Use faster storage or caching layers for persistence (e.g., SSD, Redis)

Implement backpressure to slow producers when consumers lag

Use regional deployment to reduce network latency

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain asynchronous decoupling of producers and consumers

Discuss message durability and ordering guarantees

Highlight handling of failure scenarios and retries

Describe scaling strategies for high throughput

Mention trade-offs between complexity and reliability

Practice

1. What is the main purpose of the producer-consumer pattern in system design?

easy

A. To eliminate the need for task queues entirely

B. To separate task creation from task processing using a queue

C. To combine multiple tasks into one for faster processing

D. To process tasks only when the system is idle

Solution

1. Step 1: Understand the roles of producer and consumer

   The producer creates tasks, and the consumer processes them separately.

2. Step 2: Recognize the use of a queue

   A queue buffers tasks to handle differences in speed between producing and consuming.

3. Final Answer:

   To separate task creation from task processing using a queue -> Option B

4. Quick Check:

   Producer-consumer = separate creation and processing [OK]

Hint: Producer creates, consumer processes, queue buffers tasks [OK]

Common Mistakes:

• Thinking producer and consumer do the same work
• Ignoring the role of the queue
• Assuming tasks are processed immediately without buffering

2. Which of the following correctly represents the basic structure of a producer-consumer system?

easy

A. Queue -> Producer -> Consumer

B. Consumer -> Producer -> Queue

C. Producer -> Queue -> Consumer

D. Producer -> Consumer -> Queue

Solution

1. Step 1: Identify the flow of tasks

   Tasks are created by the producer first.

2. Step 2: Understand the buffering role

   Tasks go into a queue before being processed by the consumer.

3. Final Answer:

   Producer -> Queue -> Consumer -> Option C

4. Quick Check:

   Task flow = Producer to Queue to Consumer [OK]

Hint: Tasks flow: Producer first, then queue, then consumer [OK]

Common Mistakes:

• Mixing the order of producer and consumer
• Placing queue after consumer
• Ignoring the queue as a middle step

3. Consider a system where the producer adds 5 tasks per second and the consumer processes 3 tasks per second. What will happen to the queue over time?

medium

A. The queue will grow indefinitely

B. The queue will remain stable

C. The queue will shrink to zero

D. The queue will alternate between empty and full

Solution

1. Step 1: Compare producer and consumer speeds

   Producer adds 5 tasks/sec, consumer processes 3 tasks/sec.

2. Step 2: Analyze queue behavior

   Since producer is faster, tasks accumulate, making the queue grow over time.

3. Final Answer:

   The queue will grow indefinitely -> Option A

4. Quick Check:

   Producer > Consumer speed = queue grows [OK]

Hint: If producer speed > consumer speed, queue grows [OK]

Common Mistakes:

• Assuming queue size stays constant
• Thinking consumer speed controls queue size alone
• Ignoring continuous task addition

4. A producer-consumer system uses a fixed-size queue of 10 tasks. The producer adds tasks without checking if the queue is full. What problem can occur?

medium

A. Consumer will stop processing tasks

B. Queue size will increase dynamically

C. Producer will slow down automatically

D. Queue overflow causing task loss or system crash

Solution

1. Step 1: Understand fixed-size queue limits

   The queue can hold only 10 tasks at a time.

2. Step 2: Analyze producer behavior without checks

   Producer keeps adding tasks even if queue is full, causing overflow.

3. Final Answer:

   Queue overflow causing task loss or system crash -> Option D

4. Quick Check:

   Fixed queue + unchecked producer = overflow [OK]

Hint: Fixed queue needs producer checks to avoid overflow [OK]

Common Mistakes:

• Assuming queue grows automatically
• Thinking consumer stops when queue is full
• Believing producer slows down without control

5. You are designing a high-traffic messaging system using the producer-consumer pattern. To handle sudden spikes in message volume without losing data, which approach is best?

hard

A. Use a scalable queue with dynamic size and multiple consumers

B. Limit the producer speed strictly to match consumer speed

C. Remove the queue and let consumers process directly

D. Use a single consumer to avoid complexity

Solution

1. Step 1: Identify system needs for spikes

   System must handle sudden high message volume without data loss.

2. Step 2: Choose scalable design

   A dynamic queue size and multiple consumers allow flexible processing and prevent backlog.

3. Final Answer:

   Use a scalable queue with dynamic size and multiple consumers -> Option A

4. Quick Check:

   Scalable queue + multiple consumers = handle spikes [OK]

Hint: Scale queue and consumers to handle spikes safely [OK]

Common Mistakes:

• Trying to limit producer speed strictly
• Removing queue causing lost buffering
• Using single consumer causing bottlenecks