Bird
Raised Fist0

Design a Notification System - Push, SMS, Email Fan-Out at 10M+ Scale

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Notification System - Push, SMS, Email Fan-Out at 10M+ Scale
High-scale multi-channel notification delivery with fan-out and reliability
hardCommunication and Messaging SystemsFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed MessagingData Modeling for Scale
⚙️
Functional Requirements
  • Support multi-channel notification delivery: push notifications, SMS, and email.
  • Fan-out notifications to millions of users with personalized content.
  • Support scheduling and retry mechanisms for failed deliveries.
  • Provide APIs for clients to submit notification requests with metadata.
  • Support user preferences for notification channels and opt-outs.
  • Track delivery status and provide real-time metrics and logs.
📊
Non-Functional Requirements
  • Handle 10 million+ active users with peak write QPS of ~1 million notifications/sec.
  • Ensure 99.99% availability with multi-region failover.
  • End-to-end latency under 100ms P99 for push notifications; SMS and email within 5 seconds.
  • Data durability with replication factor of 3 and retention of 30 days for audit logs.
  • System must scale horizontally without downtime.
❓
Clarifying Questions to Ask
  • What is the expected peak notification volume per second?
  • Are notifications personalized per user or broadcast identical messages?
  • What are the SLAs for each channel (push, SMS, email)?
  • Is ordering of notifications per user required?
  • What user preference management capabilities are expected?
  • Should the system support transactional and bulk notifications?
Out of Scope
✗ Content generation or templating logic beyond basic personalization.
✗ User authentication and authorization mechanisms.
✗ Third-party SMS or email provider integration details.
✗ Analytics beyond delivery status (e.g., click-through rates).
📌
Assumptions
  • 10 million active users (DAU).
  • Average 40 notifications per user per day.
  • Average notification payload size: 100 bytes.
  • Retention period for logs and metadata: 30 days.
  • Replication factor for storage: 3.
⚡
QPS Calculation
write qps10M users × 40 notifications/day ÷ 86400 sec ≈ 4629 notifications/sec average
peak write qpsAssuming 5× peak factor: 4629 × 5 ≈ 23,145 notifications/sec peak
read qpsAssuming 2× read amplification for status queries: 23,145 × 2 ≈ 46,290 reads/sec
💾
Storage Estimate
daily storage10M × 40 × 100 bytes = 40 GB/day raw
retention storage40 GB/day × 30 days = 1.2 TB
with replication1.2 TB × 3 = 3.6 TB total storage
🖥️
Server Estimate
api serversAssuming 1000 RPS per server, need ~25 servers for peak write QPS
delivery workersAssuming each worker handles 1000 notifications/sec, need ~25 workers per channel (push, SMS, email) totaling ~75 workers
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          | Notification|
          |    API      |
          +------+------+
                 |
                 v
          +------+-------+
          | Kafka Topic  |
          | (Notification|
          |   Events)    |
          +------+-------+
                 |
    +------------+-------------+
    |            |             |
    v            v             v
 +------+    +--------+    +--------+
 |Push  |    | SMS    |    | Email  |
 |Worker|    |Worker  |    |Worker  |
 +------+    +--------+    +--------+
    |            |             |
    v            v             v
 +------+    +--------+    +--------+
 |Push  |    | SMS    |    | Email  |
 |Gateway|   |Gateway |    |Gateway |
 +------+    +--------+    +--------+
                 |
                 v
          +------+-------+
          | Delivery     |
          | Status Store |
          +--------------+
🧩
Components
Notification API
Receives notification requests from clients, validates, and publishes to Kafka.
Why thisStateless REST API servers behind load balancer scale horizontally; Kafka producer client supports high throughput and durability.
Why notDirect DB writes rejected due to write amplification and lack of decoupling; RabbitMQ rejected for lower throughput and weaker partitioning.
Kafka Topic (Notification Events)
Durable, partitioned log to decouple producers and consumers, enabling fan-out to multiple delivery channels.
Why thisKafka supports high throughput (millions of messages/sec), partitioning for parallelism, and retention for replay.
Why notAmazon SQS rejected due to lack of native partitioning and lower throughput; Redis Streams rejected due to limited persistence and scaling challenges.
Delivery Workers (Push, SMS, Email)
Consume notification events from Kafka, enrich with user preferences, and send to respective gateways.
Why thisDedicated worker pools per channel allow independent scaling and channel-specific logic; Kafka consumer groups ensure load balancing and fault tolerance.
Why notMonolithic delivery service rejected due to complexity and scaling bottlenecks; polling DB rejected due to latency and inefficiency.
Channel Gateways (Push Gateway, SMS Gateway, Email SMTP Relay)
Interface with external providers or platform services to deliver notifications.
Why thisAbstracts third-party integration, supports retries and backoff, and isolates external failures from core system.
Why notDirect worker-to-provider calls rejected due to lack of retry and failure isolation; in-house SMTP rejected due to complexity and deliverability issues.
Delivery Status Store (Cassandra)
Stores delivery status and logs for each notification with high write throughput and availability.
Why thisCassandra supports linear write scalability, tunable consistency, and time-series data modeling for status tracking.
Why notPostgreSQL rejected due to write bottlenecks and complex sharding; MongoDB rejected due to weaker consistency guarantees at scale.
🔄
Data Flow
1
Client → Notification API
Submit notification request with user IDs, message, and channel preferences.
API validates and enriches request metadata.
2
Notification API → Kafka Topic
Publish notification event asynchronously.
Kafka partitions by user ID for load balancing and ordering.
3
Kafka Topic → Delivery Workers
Workers consume events, enrich with preferences, and prepare for delivery.
Kafka consumer groups enable parallel processing and fault tolerance.
4
Delivery Workers → Channel Gateways
Send notifications to respective gateways for delivery.
Gateways handle retries and external provider interactions.
5
Channel Gateways → Delivery Status Store
Update delivery status and logs.
Status store supports real-time metrics and audit.

Practice

(1/5)
1. In the request flow of a global load balancing system, at Step 3 the GSLB receives a DNS query for a user in Europe. What does the GSLB do at this step?
easy
A. Returns the IP of the closest data center without checking health status
B. Checks data center health and latency metrics before returning the IP address
C. Forwards the DNS query to the local data center's DNS server for resolution
D. Caches the previous DNS response indefinitely to reduce load

Solution

  1. Step 1: Understand GSLB behavior at DNS query

    GSLB does not blindly return IPs (Returns the IP of the closest data center without checking health status) or forward queries (Forwards the DNS query to the local data center's DNS server for resolution). It actively checks health and latency to pick the best endpoint.

  2. Step 2: Evaluate caching behavior

    Indefinite caching (Caches the previous DNS response indefinitely to reduce load) would prevent dynamic failover and load balancing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    GSLB checks health and latency before DNS response [check]
Hint: GSLB checks health before DNS response [check]
Common Mistakes:
2. To handle personalized ranking at scale, should the system use a centralized ranking service with a global user profile store or a distributed ranking service co-located with Trie shards holding local user profiles? Consider latency and data freshness constraints.
medium
A. Centralized ranking with global profiles for consistent personalization
B. Distributed ranking co-located with Trie shards to reduce latency despite eventual consistency
C. Centralized ranking but cache user profiles locally to reduce latency
D. Distributed ranking with no user profiles, relying only on global popularity

Solution

  1. Step 1: Analyze latency and freshness trade-offs

    Centralized ranking with global profiles (A) adds network latency; caching (C) adds complexity and stale data risk.

  2. Step 2: Evaluate distributed ranking with local profiles

    Distributed ranking co-located with Trie shards (B) reduces latency by local data access, tolerating eventual consistency for freshness.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Distributed ranking + local profiles -> low latency, eventual consistency [check]
Hint: Local profiles with distributed ranking reduce latency despite stale data [check]
Common Mistakes:
3. If the Booking Service loses connectivity to the Calendar Availability Service during booking confirmation, what is the most likely failure mode experienced by the user?
medium
A. The booking is queued locally and confirmed later without user notification
B. The booking request times out and the user sees an error without confirmation
C. The booking is confirmed but the calendar is not updated, causing double bookings
D. The system automatically retries and confirms booking without user delay

Solution

  1. Step 1: Trace failure impact on booking flow

    Loss of connectivity prevents immediate calendar update, risking inconsistency.

  2. Step 2: Analyze failure handling

    Immediate confirmation without calendar update (A) risks double bookings; timeout with error (B) hurts UX; automatic retry without delay (D) is unrealistic under network failure. Local queuing (C) allows eventual consistency with delayed confirmation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Local queuing delays confirmation but prevents data loss [check]
Hint: Booking queues locally on calendar service failure, delaying confirmation [check]
Common Mistakes:
4. At 10x scale, which component in the file sync system is most likely to become the bottleneck first, and why?
hard
A. The client chunking module, because CPU usage grows linearly with file size and user count
B. The network layer, because bandwidth limits are hit before storage or compute
C. The storage backend, because disk I/O cannot keep up with chunk writes at scale
D. The chunk metadata index service, because it maintains global chunk existence state and faces high read/write contention

Solution

  1. Step 1: Identify global state bottleneck

    The chunk metadata index service maintains global chunk existence info, causing high contention as scale grows.

  2. Step 2: Compare other components

    Client chunking scales horizontally; network bandwidth can be increased; storage backends can be sharded; metadata index is centralized and harder to shard efficiently.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Metadata index contention is first bottleneck at scale [check]
Hint: Global metadata index faces contention first at scale [check]
Common Mistakes:
5. At 10x traffic scale, which component is most likely to fail first in a latency monitoring system that tracks P50, P95, and P99 latencies, and why?
hard
A. The data storage, because it cannot handle the increased volume of raw latency data
B. The aggregation service, because computing percentiles in real-time requires heavy CPU and memory resources
C. The alerting system, because it cannot process the increased number of alerts generated
D. The client instrumentation, because it cannot send telemetry fast enough

Solution

  1. Step 1: Identify bottleneck in percentile computation

    Computing P50, P95, and P99 latencies in real-time requires sorting or approximation algorithms that are CPU and memory intensive.

  2. Step 2: Why aggregation service fails first

    At 10x scale, the aggregation service faces exponential resource demands, making it the first to fail before storage or alerting.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Real-time percentile computation scales poorly -> aggregation service fails first [check]
Hint: Percentile computation is CPU-heavy; aggregation fails first [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard