Why delivery systems test service coordination in LLD - Design It to Understand It

Design: Delivery System Service Coordination Testing

Focus on testing coordination between services like order management, inventory, delivery tracking, and notifications. Out of scope: detailed unit tests of individual services.

Functional Requirements

FR1: Ensure multiple services in the delivery system work together correctly

FR2: Detect and handle failures in service communication

FR3: Validate timing and order of service interactions

FR4: Confirm data consistency across services

FR5: Support testing under realistic load and failure scenarios

Non-Functional Requirements

NFR1: Test environment should simulate real-world scale (e.g., 1000 concurrent deliveries)

NFR2: Tests must complete within reasonable time (e.g., under 30 minutes)

NFR3: High availability of test infrastructure (99.9% uptime)

NFR4: Low latency in test feedback to enable quick fixes

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Service orchestrator or coordinator

Message queues or event buses

Mock or stub services for testing

Monitoring and logging tools

Test automation framework

Design Patterns

Saga pattern for distributed transactions

Circuit breaker for failure handling

Event-driven architecture

Retry and timeout strategies

Canary testing and chaos engineering

Reference Architecture

  +----------------+       +----------------+       +----------------+
  | Order Service  | <---> | Inventory Svc  | <---> | Delivery Svc   |
  +----------------+       +----------------+       +----------------+
          |                        |                        |
          v                        v                        v
    +---------------------------------------------------------------+
    |                   Message Queue / Event Bus                   |
    +---------------------------------------------------------------+
                              |
                              v
                     +------------------+
                     | Notification Svc |
                     +------------------+

Test Coordinator triggers flows and monitors logs and metrics.

Components

Order Service

REST API

Handles order creation and updates

Inventory Service

gRPC

Manages stock availability and reservations

Delivery Service

REST API

Tracks delivery status and updates

Notification Service

Event-driven

Sends delivery status notifications to customers

Message Queue / Event Bus

Kafka or RabbitMQ

Enables asynchronous communication and event propagation

Test Coordinator

Custom test automation framework

Simulates workflows, injects failures, and verifies coordination

Request Flow

1. 1. Test Coordinator triggers an order creation in Order Service.

2. 2. Order Service sends reservation request to Inventory Service via synchronous call.

3. 3. Inventory Service confirms stock and publishes event to Message Queue.

4. 4. Delivery Service listens to stock reservation events and schedules delivery.

5. 5. Delivery Service updates status and publishes delivery events.

6. 6. Notification Service consumes delivery events and sends notifications.

7. 7. Test Coordinator monitors logs, metrics, and event sequences to verify correct coordination.

8. 8. Test Coordinator injects failure scenarios like service timeouts or message loss to validate system resilience.

Database Schema

Entities: - Order: id (PK), customer_id, status, created_at - InventoryItem: id (PK), product_id, quantity_available - Delivery: id (PK), order_id (FK), status, scheduled_time - Notification: id (PK), delivery_id (FK), type, status Relationships: - Order 1:N Delivery - Delivery 1:N Notification - InventoryItem linked to products (not shown)

Scaling Discussion

Bottlenecks

Message queue throughput limits under high event volume

Test Coordinator processing delays with many concurrent workflows

Service communication latency causing test timeouts

Difficulty simulating complex failure scenarios at scale

Solutions

Partition message queues and scale brokers horizontally

Distribute test coordination across multiple nodes

Use asynchronous calls and increase timeout thresholds carefully

Automate chaos testing with controlled failure injection tools

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and failure handling, 5 minutes summarizing key points.

Explain why testing service coordination is critical for delivery correctness

Describe how asynchronous messaging helps decouple services

Discuss failure scenarios and how tests validate resilience

Highlight importance of monitoring and observability in tests

Show awareness of scaling challenges and mitigation strategies