Order processing pipeline in HLD - System Design Exercise

Design: Order Processing Pipeline

Includes order intake, validation, payment processing, inventory update, notification, and order tracking. Excludes product catalog management and delivery logistics.

Functional Requirements

FR1: Accept customer orders through a web or mobile interface

FR2: Validate order details including product availability and payment information

FR3: Process payments securely and reliably

FR4: Update inventory to reflect sold items

FR5: Generate order confirmation and notify customers

FR6: Support order status tracking by customers

FR7: Handle up to 10,000 concurrent orders per minute

FR8: Ensure order processing latency under 2 seconds for 99th percentile

FR9: Maintain 99.9% system availability

Non-Functional Requirements

NFR1: System must handle peak loads during sales events

NFR2: Data consistency is critical between inventory and orders

NFR3: Payment processing must comply with security standards (e.g., PCI DSS)

NFR4: Order status updates must be near real-time

NFR5: System should be designed for horizontal scalability

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

❓ Question 7

Key Components

API Gateway or Load Balancer

Order Service for intake and validation

Payment Service integrating with payment gateways

Inventory Service managing stock levels

Notification Service for customer communication

Order Tracking Service

Database(s) for orders, inventory, and transactions

Message Queue for asynchronous processing

Design Patterns

Event-driven architecture for decoupling services

Saga pattern for managing distributed transactions

Circuit breaker for payment gateway reliability

CQRS (Command Query Responsibility Segregation) for read/write separation

Caching for frequently accessed order status data

Reference Architecture

Client (Web/Mobile)
    |
    v
API Gateway / Load Balancer
    |
    v
+-------------------+       +------------------+       +------------------+
|   Order Service    |<----->| Payment Service  |<----->| Payment Gateway   |
+-------------------+       +------------------+       +------------------+
        |                          |
        v                          v
+-------------------+       +------------------+
| Inventory Service |       | Notification     |
+-------------------+       | Service          |
        |                   +------------------+
        v                          |
+-------------------+             v
| Order Database    |<----------> Message Queue
+-------------------+             |
        |                         v
        v                   +------------------+
+-------------------+       | Order Tracking   |
| Inventory Database |       | Service          |
+-------------------+       +------------------+

Components

API Gateway / Load Balancer

Nginx, AWS ALB, or similar

Route client requests to appropriate services and balance load

Order Service

RESTful API service (e.g., Node.js, Spring Boot)

Receive and validate orders, initiate processing

Payment Service

Microservice integrating with external payment gateways

Handle payment authorization and capture securely

Payment Gateway

Third-party payment processors (e.g., Stripe, PayPal)

Process actual payment transactions

Inventory Service

Microservice with database (e.g., PostgreSQL)

Manage stock levels and update inventory

Notification Service

Email/SMS/push notification system (e.g., AWS SNS, Twilio)

Send order confirmations and status updates to customers

Order Tracking Service

Service with read-optimized database or cache

Provide real-time order status to customers

Databases

Relational DB for orders and inventory (e.g., PostgreSQL), NoSQL or cache for tracking (e.g., Redis)

Store persistent order and inventory data

Message Queue

Kafka, RabbitMQ, or AWS SQS

Enable asynchronous communication between services for reliability and scalability

Request Flow

1. Client sends order request to API Gateway.

2. API Gateway forwards request to Order Service.

3. Order Service validates order details and checks product availability via Inventory Service.

4. If valid, Order Service sends payment request to Payment Service.

5. Payment Service processes payment through Payment Gateway.

6. On successful payment, Order Service updates order status and sends inventory update request to Inventory Service.

7. Inventory Service decrements stock and confirms update.

8. Order Service publishes order confirmation event to Message Queue.

9. Notification Service consumes event and sends confirmation to customer.

10. Order Tracking Service updates order status for customer queries.

11. Client can query Order Tracking Service for real-time status.

Database Schema

Entities: - Order: order_id (PK), customer_id, order_date, status, total_amount, payment_status - OrderItem: order_item_id (PK), order_id (FK), product_id, quantity, price - Inventory: product_id (PK), stock_quantity - PaymentTransaction: transaction_id (PK), order_id (FK), payment_method, status, amount, timestamp Relationships: - One Order has many OrderItems (1:N) - One Order has one PaymentTransaction (1:1) - Inventory tracks stock per product_id - Order references customer_id (not detailed here)

Scaling Discussion

Bottlenecks

Order Service CPU and memory under high concurrent order load

Payment Service latency due to external gateway calls

Inventory Service consistency under concurrent stock updates

Database write contention on orders and inventory tables

Notification Service throughput during peak order confirmations

Solutions

Scale Order Service horizontally behind load balancer; use stateless design

Implement circuit breaker and retries in Payment Service; use payment gateway with high SLA

Use optimistic locking or distributed locks in Inventory Service; consider eventual consistency with compensating transactions

Partition databases by customer or order ID; use write-ahead logs and batch writes

Use scalable message queues and multiple Notification Service instances; batch notifications when possible

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Emphasize importance of data consistency between orders and inventory

Discuss asynchronous processing to improve responsiveness and reliability

Highlight security considerations in payment processing

Explain how to handle failures and retries gracefully

Show understanding of scaling challenges and solutions

Mention monitoring and alerting for system health

Practice

1. What is the main purpose of an order processing pipeline in system design?

easy

A. To store all orders in a single database table

B. To break down order handling into clear, manageable steps

C. To process orders only during business hours

D. To send orders directly to customers without checks

Solution

1. Step 1: Understand the concept of order processing pipeline

   An order processing pipeline organizes the flow of orders into separate steps to improve clarity and management.

2. Step 2: Identify the main benefit

   This organization helps improve efficiency, scalability, and reliability by handling orders step-by-step.

3. Final Answer:

   To break down order handling into clear, manageable steps -> Option B

4. Quick Check:

   Order processing pipeline = clear, manageable steps [OK]

Hint: Order pipeline means splitting tasks into steps [OK]

Common Mistakes:

• Thinking it only stores orders
• Assuming orders are processed only at certain times
• Believing orders skip validation

2. Which component is typically used to decouple stages in an order processing pipeline?

easy

A. Message queues or event streams

B. Single-threaded processing loop

C. Synchronous HTTP requests only

D. Direct database calls between stages

Solution

1. Step 1: Identify decoupling methods in pipelines

   Decoupling means separating stages so they don't depend directly on each other.

2. Step 2: Recognize message queues as decouplers

   Message queues or event streams allow asynchronous communication, enabling stages to work independently.

3. Final Answer:

   Message queues or event streams -> Option A

4. Quick Check:

   Decoupling = message queues [OK]

Hint: Use queues to separate pipeline steps [OK]

Common Mistakes:

• Using direct DB calls causing tight coupling
• Assuming synchronous calls decouple well
• Thinking single-thread loops scale pipelines

3. Consider this simplified order pipeline code snippet:

orders = [1, 2, 3]
processed = []
for order in orders:
    if order % 2 == 1:
        processed.append(order * 10)
print(processed)

What is the output?

medium

A. [10, 20, 30]

B. [20]

C. [1, 3]

D. [10, 30]

Solution

1. Step 1: Analyze the loop and condition

   The loop goes through orders 1, 2, 3. It checks if order is odd (order % 2 == 1).

2. Step 2: Calculate processed list values

   Orders 1 and 3 are odd, so they are multiplied by 10 and added: 10 and 30.

3. Final Answer:

   [10, 30] -> Option D

4. Quick Check:

   Odd orders * 10 = [10, 30] [OK]

Hint: Check odd numbers and multiply by 10 [OK]

Common Mistakes:

• Including even numbers mistakenly
• Appending original orders instead of multiplied
• Confusing condition logic

4. In an order processing pipeline, a stage is failing to process orders because it reads from the queue but never acknowledges messages. What is the likely problem?

medium

A. Orders are lost because the queue deletes messages immediately

B. The pipeline processes orders twice due to duplicate acknowledgments

C. Orders pile up because messages are not acknowledged and re-delivered

D. The queue is empty because messages are acknowledged too early

Solution

1. Step 1: Understand message acknowledgment in queues

   Queues require consumers to acknowledge messages after processing to remove them.

2. Step 2: Identify effect of missing acknowledgments

   If messages are not acknowledged, the queue assumes failure and re-delivers, causing backlog.

3. Final Answer:

   Orders pile up because messages are not acknowledged and re-delivered -> Option C

4. Quick Check:

   No ack = message re-delivery and backlog [OK]

Hint: Always acknowledge queue messages after processing [OK]

Common Mistakes:

• Thinking messages are lost without ack
• Assuming duplicates come from acking
• Believing early ack empties queue

5. You need to design an order processing pipeline that can handle sudden spikes of 10,000 orders per minute without losing any orders. Which design choice best supports this requirement?

hard

A. Implement multiple pipeline stages connected by scalable message queues

B. Store all orders in a single database table and process them with one worker

C. Use a single monolithic service processing orders synchronously

D. Process orders directly on the client side to reduce server load

Solution

1. Step 1: Identify scalability needs for high order volume

   Handling 10,000 orders per minute requires the system to scale and avoid bottlenecks.

2. Step 2: Choose design supporting scalability and reliability

   Multiple pipeline stages with message queues allow asynchronous, parallel processing and buffering during spikes.

3. Final Answer:

   Implement multiple pipeline stages connected by scalable message queues -> Option A

4. Quick Check:

   Scalable queues + stages = handle spikes reliably [OK]

Hint: Use queues and stages to scale order processing [OK]

Common Mistakes:

• Using single service causing bottlenecks
• Relying on one worker limits throughput
• Processing on client risks data loss