Bird
Raised Fist0

Alerting System Design - Threshold, Anomaly, Composite Alert Patterns

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Alerting System Design - Threshold, Anomaly, Composite Alert Patterns
Design a scalable alerting system supporting threshold, anomaly, and composite alerts for real-time monitoring at massive scale.
mediumMonitoring and ObservabilityFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed Systems BasicsStream Processing Concepts
⚙️
Functional Requirements
  • Support threshold-based alerts on metrics (e.g., CPU > 80%) with configurable thresholds per user or team.
  • Support anomaly detection alerts using statistical or ML models on time-series data.
  • Support composite alerts combining multiple alert conditions with logical operators (AND, OR).
  • Real-time alert evaluation and notification delivery with sub-second latency.
  • Allow users to define alert rules, manage alert lifecycle (enable/disable), and view alert history.
  • Integrate with multiple notification channels (email, SMS, Slack, PagerDuty).
📊
Non-Functional Requirements
  • Handle 2 billion daily active users generating monitoring data.
  • Support peak write throughput of ~925K alert evaluation events per second (derived from estimation).
  • Ensure alert evaluation and notification delivery with 99.99% availability and <100ms P99 latency.
  • Data retention of raw metrics and alert history for 30 days with efficient storage.
  • System must scale horizontally with minimal manual intervention.
❓
Clarifying Questions to Ask
  • What types of metrics and data sources are supported (metrics, logs, traces)?
  • What is the expected alert evaluation frequency (e.g., per second, per minute)?
  • Are alerts evaluated per user, per team, or globally aggregated?
  • What is the expected notification latency SLA?
  • Is anomaly detection done online or offline, and what ML models are used?
  • What are the expected notification channels and their throughput?
Out of Scope
✗ Design of metric ingestion pipelines and storage (assumed existing).
✗ Detailed ML model design for anomaly detection (assumed provided).
✗ User authentication and authorization mechanisms.
✗ Dashboard UI design for alert management.
📌
Assumptions
  • 2 billion daily active users (DAU).
  • Each user generates 40 metric data points per day relevant for alerting.
  • Average alert evaluation event size is 100 bytes.
  • Retention period for alert data is 30 days.
⚡
QPS Calculation
write qps2B users × 40 events/user/day ÷ 86400 sec/day ≈ 925,926 writes/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M writes/sec
read qpsAssuming 10% of writes are read for alert history and dashboard: 925,926 × 0.1 ≈ 92,593 reads/sec
💾
Storage Estimate
daily storage925,926 writes/sec × 100 bytes × 86400 sec ≈ 8 TB/day raw data
retention storage8 TB/day × 30 days = 240 TB
with replicationAssuming 3× replication for durability: 240 TB × 3 = 720 TB total
🖥️
Server Estimate
write pathKafka cluster with 50 brokers, each handling ~56K writes/sec (2.78M/50)
processingFlink cluster with 100 task managers to handle real-time alert evaluation at peak QPS
storageCassandra cluster with 200 nodes to store alert data with replication and low latency reads
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          | Metric Ingest  |
          +-------+--------+
                  |
          +-------v--------+
          | Kafka Cluster  |
          +-------+--------+
                  |
          +-------v--------+
          | Flink Stream   |
          | Processing     |
          +-------+--------+
                  |
          +-------v--------+
          | Alert Evaluator|
          +-------+--------+
                  |
          +-------v--------+
          | Cassandra DB   |
          +-------+--------+
                  |
          +-------v--------+
          | Notification   |
          | Delivery Svc   |
          +-------+--------+
                  |
          +-------v--------+
          | User Dashboard |
          +----------------+
🧩
Components
Kafka Cluster
Ingest and buffer metric and event streams for alert evaluation
Why thisKafka supports high-throughput, partitioned, durable log storage with exactly-once semantics for stream processing
Why notRabbitMQ lacks partitioning and throughput at 2.78M writes/sec; Kinesis has higher cost and vendor lock-in
Flink Stream Processing
Real-time alert evaluation engine for threshold, anomaly, and composite alerts
Why thisFlink provides low-latency, stateful stream processing with event time semantics and exactly-once guarantees
Why notSpark Streaming has higher latency and micro-batch model unsuitable for sub-second alerting
Alert Evaluator Service
Applies alert rules logic including threshold checks, anomaly detection triggers, and composite logic
Why thisMicroservice architecture allows independent scaling and deployment of alert logic with REST/gRPC APIs
Why notEmbedding logic directly in Flink jobs reduces flexibility and complicates rule updates
Cassandra DB
Stores alert definitions, evaluation results, and alert history with high write throughput and low latency reads
Why thisCassandra's wide-column model supports time-series data with tunable consistency and linear horizontal scalability
Why notPostgreSQL requires complex manual sharding at 2.78M writes/sec; B-tree writes don't scale linearly
Notification Delivery Service
Sends alerts to users via email, SMS, Slack, PagerDuty with retry and backoff
Why thisDedicated service handles integration with multiple channels and manages delivery guarantees and rate limiting
Why notDirect integration in alert evaluator mixes concerns and complicates scaling notification retries
User Dashboard
Allows users to create, manage alert rules and view alert history and status
Why thisWeb UI backed by API gateway provides user-friendly management and observability
Why notCLI-only or config-file based management is not user-friendly for large enterprise customers
🔄
Data Flow
1
Metric Ingest → Kafka Cluster
Metrics and events are published to Kafka topics partitioned by user or metric
Kafka buffers high-throughput data for downstream processing
2
Kafka Cluster → Flink Stream Processing
Flink consumes Kafka topics for real-time alert evaluation
Flink processes streams with event time semantics and stateful operators
3
Flink Stream Processing → Alert Evaluator Service
Alert rules are applied including threshold checks, anomaly detection, and composite logic
Microservice allows flexible rule updates and independent scaling
4
Alert Evaluator Service → Cassandra DB
Stores alert evaluation results and alert history for querying and retention
Cassandra provides scalable writes and low latency reads for time-series data
5
Alert Evaluator Service → Notification Delivery Service
Triggers notifications via multiple channels with retry and backoff
Dedicated service manages delivery guarantees and rate limiting
6
Notification Delivery Service → User Dashboard
Notification status and alert history are displayed to users
Dashboard provides observability and alert management UI

Practice

(1/5)
1. A system must ingest 1 million events per second with guaranteed ordering per user session and durable storage. Which component best handles this ingestion requirement?
easy
A. Use Redis Pub/Sub for low latency event streaming
B. Use a relational database with batch inserts for durability
C. Use Kafka with partitioning keyed by user session ID
D. Use a simple HTTP endpoint with in-memory queueing

Solution

  1. Step 1: Identify Kafka's role in ingestion

    Kafka supports high-throughput ingestion with partitioning by user session ID, ensuring ordering per session and durability via log storage.

  2. Step 2: Eliminate alternatives

    Relational DB batch inserts can't handle 1M events/sec with ordering guarantees; Redis Pub/Sub lacks durability and ordering guarantees; HTTP with in-memory queueing risks data loss and no ordering guarantees.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Kafka partitions keyed by session ensure ordering and durability [check]
Hint: Kafka partitions preserve ordering per key [check]
Common Mistakes:
2. In the request flow of the leaderboard system, at step 4 the Ranking Worker reads the updated score from Redis Sorted Sets. What state does the Ranking Worker observe if the score update was just written but the TTL expired simultaneously?
easy
A. The Ranking Worker sees no score entry because TTL expired and removed it
B. The Ranking Worker sees a partial score update causing inconsistent rank
C. The Ranking Worker sees the old score because Redis delays TTL eviction
D. The Ranking Worker blocks until Redis confirms the score is present

Solution

  1. Step 1: Understand Redis TTL eviction behavior

    Redis removes keys immediately after TTL expiration, so if TTL expired, the score entry is deleted.

  2. Step 2: Trace Ranking Worker read at step 4

    Ranking Worker reads Redis Sorted Set; if key expired, it finds no entry, not stale or partial data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Redis TTL expiry deletes keys before read, so no score entry found [check]
Hint: TTL expiry deletes keys before reads, no stale data returned [check]
Common Mistakes:
3. Without the distributed consensus mechanism in a NoSQL cluster, what exactly breaks during a network partition?
medium
A. Clients receive stale data but writes continue without errors
B. Writes may be lost or overwritten due to lack of coordination, causing data inconsistency
C. All read requests block until the partition heals
D. The cluster automatically switches to a relational database fallback

Solution

  1. Step 1: Understand distributed consensus role

    Consensus protocols ensure coordinated writes and consistency across nodes.

  2. Step 2: Trace failure without consensus during partition

    Without consensus, the cluster cannot safely coordinate writes and may fallback to a safe mode or external system.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    No consensus -> fallback to relational DB (plausible in some designs) [check]
Hint: No consensus -> fallback to safe storage [check]
Common Mistakes:
4. At 10x scale, which component in a WebRTC video calling system is most likely to become the bottleneck first, and why?
hard
A. Media Servers, because transcoding CPU load grows linearly with concurrent calls
B. Signaling Servers, because they maintain stateful WebSocket connections and cannot easily shard sessions
C. STUN/TURN Servers, because NAT traversal requests increase exponentially with users
D. Client devices, because battery and CPU limits prevent scaling user concurrency

Solution

  1. Step 1: Analyze component scalability at 10x load

    Media Servers scale with CPU but can be horizontally scaled; STUN/TURN scale well with stateless UDP; clients are outside system control.

  2. Step 2: Identify Signaling Server bottleneck

    Signaling Servers hold stateful WebSocket connections per user, making sharding and scaling challenging.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Stateful WebSocket connections limit Signaling Server horizontal scaling [check]
Hint: Stateful WebSocket limits Signaling Server scaling first [check]
Common Mistakes:
5. At 10x scale, which component is most likely to become the bottleneck first in a read-heavy CQRS system with read replicas, and why?
hard
A. Event queue, because increased write events overwhelm its processing capacity
B. Primary database, because write throughput scales linearly with read volume
C. Read replicas, because replication lag increases and they cannot keep up with read demand
D. Client-side caches, because cache invalidation traffic grows exponentially

Solution

  1. Step 1: Analyze scaling impact on components

    Write volume and event propagation increase 10x, stressing the event queue.

  2. Step 2: Compare bottlenecks

    Primary DB writes scale with writes, not reads; read replicas scale horizontally; client caches don't cause exponential invalidation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Event queue overwhelmed first due to write event surge [check]
Hint: Event queue chokes first as write events surge 10x [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard