Bird
Raised Fist0

Design a Notification System - Push, SMS, Email Fan-Out at 10M+ Scale

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Notification System - Push, SMS, Email Fan-Out at 10M+ Scale
45 min · senior
senior

You are building a notification system for a startup with 50M daily active users (DAU). The system must fan out notifications via push, SMS, and email to millions of users concurrently. Notifications include transactional alerts, marketing campaigns, and system messages. The system must guarantee end-to-end encryption, support offline delivery with retries, and maintain a P99 latency under 200ms for push notifications. SMS and email have relaxed latency but must handle high throughput. The system should be highly available and horizontally scalable.

Constraints
  • End-to-end encryption of notification content at rest and in transit
  • Offline delivery support with retry and backoff for push notifications
  • P99 latency under 200ms for push notifications
  • Support 10M+ concurrent fan-out with 50M DAU
  • Separate delivery pipelines for push, SMS, and email with different SLAs
📋
Guided Steps
1/5
Step 1
Choosing the Message Ingestion and Fan-Out Backbone
What technology and architecture will you use to ingest notifications and fan them out to multiple delivery channels at 10M+ scale?
💡 Hint
Consider a distributed log system that supports partitioning and ordering guarantees for high throughput and scalability.
Key Decision
Use Apache Kafka with topic partitioning keyed by notification_id to ensure ordered processing and scalable fan-out to delivery workers.
Interviewer Expectation
Mid-level: Names Kafka for ingestion. Senior: Explains partition key choice (notification_id) to maintain ordering and avoid cross-partition coordination.
⚠️Common Wrong Answer
Using a relational database for ingestion causes write bottlenecks and poor scalability at 10M+ QPS.
Step 2
Designing Delivery Pipelines for Push, SMS, and Email
How will you architect separate delivery pipelines for push, SMS, and email to meet different latency and reliability SLAs?
💡 Hint
Separate microservices consuming from Kafka topics with specialized retry and backoff logic per channel.
Key Decision
Implement dedicated delivery microservices per channel consuming from Kafka consumer groups, using Redis for push offline state and Twilio/SendGrid SDKs for SMS/email.
Interviewer Expectation
Mid-level: Describes separate consumers per channel. Senior: Details offline delivery using Redis TTL for push and explains why direct DB polling fails at scale.
⚠️Common Wrong Answer
Single monolithic delivery service causes resource contention and inability to tune retries per channel.
Step 3
Ensuring End-to-End Encryption and Secure Storage
How will you guarantee end-to-end encryption of notification content both at rest and in transit across the system?
💡 Hint
Use envelope encryption with client-side keys and TLS for transport security.
Key Decision
Encrypt notification payloads client-side using AWS KMS envelope encryption before sending to Kafka; use TLS for all network communication.
Interviewer Expectation
Mid-level: Mentions TLS and encryption. Senior: Explains client-side encryption to prevent server-side plaintext exposure and key rotation strategy.
⚠️Common Wrong Answer
Encrypting only at transport layer exposes plaintext in Kafka and downstream services.
Step 4
Designing Offline Delivery and Retry Mechanisms
How will you handle offline devices and retries for push notifications to meet P99 latency and delivery guarantees?
💡 Hint
Use Redis to track device online status with TTL and exponential backoff retry queues.
Key Decision
Use Redis with 30s TTL heartbeats to track device online status; delivery workers check Redis before sending push; failed deliveries enter Kafka retry topics with exponential backoff.
Interviewer Expectation
Mid-level: Describes retry queues. Senior: Explains Redis TTL heartbeat mechanism and how it prevents unnecessary retries to offline devices.
⚠️Common Wrong Answer
Blind retries without offline detection cause wasted resources and increased latency.
Step 5
API Design for Notification Submission, Status Query, and Acknowledgement
Design 3 core API endpoints for clients to submit notifications, query delivery status, and acknowledge receipt.
💡 Hint
Consider RESTful endpoints with idempotency keys and efficient status querying using Cassandra.
Key Decision
POST /notifications with idempotency key for submission; GET /notifications/{id}/status querying Cassandra by notification_id; POST /notifications/{id}/ack for client acknowledgements.
Interviewer Expectation
Mid-level: Names endpoints and basic methods. Senior: Explains idempotency, Cassandra schema for fast status queries, and acknowledgement impact on retry logic.
⚠️Common Wrong Answer
Polling delivery status from Kafka offsets causes high load and inconsistent views.
🔌
API Design Challenge
Sketch 3 core API endpoints for the notification system: sending a notification, querying delivery status, and acknowledging receipt.
Hints
  • How does the client submit a notification request with metadata?
  • How can clients query notification delivery or failure status efficiently?
  • How does the recipient device or service acknowledge notification receipt or failure?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with partition key notification_id for scalable ingestion and ordered fan-out
  • Separate delivery pipelines per channel with dedicated microservices
  • Client-side encryption with AWS KMS envelope encryption
  • Redis TTL heartbeat for offline device detection and retry optimization
  • RESTful API with idempotency and efficient status querying
Good to Have - 30%
  • Use Cassandra for delivery status storage with notification_id partition key
  • Exponential backoff retry mechanism using Kafka retry topics
  • TLS for all network communication
Bonus - 10%
  • Explain key rotation strategy for encryption keys
  • Design for multi-region deployment for high availability
Anti-patterns to Avoid
Using relational databases for ingestion causes write bottlenecks and poor scalability at 10M+ QPS, leading to system slowdowns and failures under high load
Monolithic delivery service leads to resource contention and inability to tune retries per channel, resulting in poor reliability and scalability
Encrypting only at transport layer exposes plaintext in Kafka and downstream services, risking data breaches and compliance violations
Blind retries without offline detection cause wasted resources and increased latency, degrading user experience and system efficiency
Polling delivery status from Kafka offsets causes high load and inconsistent views, overwhelming the system and providing unreliable status information
✅
Model Answer
▼ Reveal

The system uses Apache Kafka with notification_id partitioning for scalable ingestion and ordered fan-out to dedicated delivery microservices per channel. Client-side encryption with AWS KMS ensures end-to-end security. Redis tracks device online status with TTL heartbeats to optimize push notification retries. Cassandra stores notification metadata and delivery status for efficient querying. Exponential backoff retry topics in Kafka handle failed deliveries. RESTful APIs with idempotency keys enable reliable notification submission, status querying, and acknowledgements. TLS secures all network communication.

Key Decisions
  • Use Apache Kafka with notification_id as partition key to maintain ordering and scale ingestion to 10M+ QPS
  • Separate delivery pipelines per channel with dedicated microservices to meet different SLAs and enable independent scaling
  • Encrypt notification payloads client-side using AWS KMS envelope encryption to prevent plaintext exposure in transit and at rest
  • Track device online status using Redis TTL heartbeats to avoid unnecessary retries and reduce latency for push notifications
  • Store notification metadata and delivery status in Cassandra partitioned by notification_id for efficient and scalable queries
  • Implement exponential backoff retry mechanism using Kafka retry topics to ensure eventual delivery without overwhelming downstream services
  • Design RESTful APIs with idempotency keys to prevent duplicate notifications and support reliable client interactions
  • Use TLS for all network communication to secure data in transit and comply with security requirements
  • Apache Kafka for high-throughput ingestion and fan-out with notification_id partition key for ordering
  • Dedicated microservices per delivery channel (push, SMS, email) consuming from Kafka consumer groups
  • Redis with TTL-based heartbeat for device online/offline tracking to optimize push retries
  • AWS KMS envelope encryption for client-side payload encryption ensuring end-to-end security
  • Cassandra for storing notification metadata and delivery status with notification_id as partition key for efficient queries
  • Exponential backoff retry topics in Kafka for failed deliveries to ensure eventual consistency
  • RESTful API design with idempotency keys to prevent duplicate notifications
  • TLS for all network communication to secure data in transit

Practice

(1/5)
1. In an API gateway request flow, at step 3 the gateway inspects the Accept header to determine API version. What happens if the header is missing or malformed at this step?
easy
A. The gateway rejects the request with a 400 Bad Request error
B. The gateway falls back to a default version configured per service
C. The gateway defaults to the latest API version and forwards the request
D. The gateway forwards the request without versioning information, causing backend ambiguity

Solution

  1. Step 1: Trace gateway header validation at step 3

    The gateway requires a valid Accept header for content negotiation versioning; missing or malformed headers violate protocol.

  2. Step 2: Understand failure handling

    To avoid ambiguous routing or unintended version fallback, the gateway rejects invalid requests with 400 errors rather than guessing versions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing Accept header triggers 400 error at gateway [check]
Hint: Missing Accept header -> gateway rejects request [check]
Common Mistakes:
2. A system expects 1 million daily active users (DAU) with an average of 50 requests per user per day. Which component is best suited to handle the peak QPS estimation for capacity planning?
easy
A. Use a load balancer's metrics to estimate peak QPS from incoming traffic patterns
B. Use a relational database to directly measure QPS by counting requests in real-time
C. Use client-side logs aggregated asynchronously to estimate QPS after the fact
D. Use a CDN to estimate QPS by counting cache hits and misses

Solution

  1. Step 1: Identify component that reflects real-time incoming traffic

    Load balancer metrics provide immediate, accurate QPS data as they handle all incoming requests before backend processing.

  2. Step 2: Eliminate other options

    Relational DB counting is too slow and adds overhead; client logs are delayed and incomplete; CDN metrics reflect cached content, not all requests.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Load balancer metrics reflect real-time QPS accurately [check]
Hint: Load balancer metrics reflect real-time QPS [check]
Common Mistakes:
3. A messaging system must guarantee ordered processing of events per user session while supporting high throughput. Which Kafka feature ensures this ordering?
easy
A. Partitioning messages by user session key
B. Setting a short retention policy to reduce storage
C. Using multiple consumer groups to parallelize consumption
D. Using Kafka Streams for message transformation

Solution

  1. Step 1: Understand parallelism vs ordering

    Using multiple consumer groups increases parallelism but does not guarantee ordering per session.

  2. Step 2: Identify ordering guarantee

    Partitioning messages by user session key ensures ordering within partitions.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Multiple consumer groups increase throughput but ordering per session requires partitioning [check]
Hint: Multiple consumer groups increase throughput [OK]
Common Mistakes:
4. If the Geohash prefix index is unavailable, what exactly breaks in the nearby search flow?
medium
A. The system falls back to a full scan of all businesses, causing high latency but correct results
B. The system crashes because no fallback spatial index exists
C. Nearby search queries return incomplete results because spatial bucketing is missing, causing false negatives
D. The system returns all businesses regardless of location, causing false positives but no misses

Solution

  1. Step 1: Understand Geohash prefix role

    Geohash prefix index buckets locations; without it, filtering by location is impossible.

  2. Step 2: Trace failure impact

    Without the prefix index, the system cannot filter by location, so it returns all businesses (false positives), not incomplete results (false negatives).

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Missing Geohash prefix index causes unfiltered results (false positives) [check]
Hint: No Geohash prefix -> no filtering -> all results returned [check]
Common Mistakes:
5. For encrypting sensitive data at rest in a distributed database, which approach is best when the system requires frequent data reads with minimal latency and strong security guarantees?
medium
A. Use client-side encryption where clients encrypt data before sending to the database
B. Use Transparent Data Encryption (TDE) at the database layer with hardware security modules (HSMs)
C. Use application-layer encryption with keys stored in environment variables on app servers
D. Use file system-level encryption on database storage volumes

Solution

  1. Step 1: Evaluate encryption approaches for read latency and security

    TDE with HSMs provides strong security with minimal impact on read latency as encryption/decryption is transparent and hardware-accelerated.

  2. Step 2: Eliminate less suitable options

    Client-side encryption increases latency and complicates querying; app-layer encryption with env vars risks key exposure and adds latency; file system encryption protects at disk level but not granular data access.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    TDE + HSM balances security and low-latency reads [check]
Hint: TDE with HSMs -> strong security + low latency reads [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard