Bird
Raised Fist0

Alerting System Design - Threshold, Anomaly, Composite Alert Patterns

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Alerting System Design - Threshold, Anomaly, Composite Alert Patterns
45 min · senior
senior

You are designing an alerting system for a 50 million daily active user (DAU) startup. The system must support threshold-based alerts (e.g., CPU > 80%), anomaly detection alerts (e.g., sudden spike in traffic), and composite alerts combining multiple conditions. Alerts must be delivered with P99 latency under 200ms. All alert data must be encrypted at rest and in transit. The system should support offline delivery (e.g., email or SMS) if real-time delivery fails. The system must scale to handle millions of metrics ingested per second and generate alerts in near real-time.

Constraints
  • P99 alert delivery latency < 200ms
  • Encryption at rest and in transit for all alert data
  • Support offline delivery fallback (email, SMS) if real-time delivery fails
  • Scale to ingest and process millions of metrics per second
  • Support complex composite alert patterns combining multiple conditions
📋
Guided Steps
1/5
Step 1
Metric Ingestion and Storage
How will you ingest and store millions of metrics per second with encryption and low latency?
💡 Hint
Consider a scalable, partitioned streaming platform with encryption support and a time-series database optimized for writes.
Key Decision
Use Apache Kafka with TLS encryption for metric ingestion to handle high throughput and partition by metric_id; store metrics in Apache Cassandra with encryption at rest, using time-series data modeling for efficient range queries.
Interviewer Expectation
Mid: Names Kafka and Cassandra for ingestion and storage. Senior: Explains Kafka partitioning by metric_id to ensure ordering and load balancing, and Cassandra's LSM-tree design for high write throughput and TTL-based data retention.
⚠️Common Wrong Answer
Using a relational database like MySQL for metric storage - it cannot handle millions of writes per second and causes high write amplification and latency.
Step 2
Alert Rule Evaluation Engine
How will you design the alert evaluation engine to support threshold, anomaly, and composite alerts at scale with low latency?
💡 Hint
Think about stream processing frameworks that can consume Kafka topics and support complex event processing with stateful operators.
Key Decision
Use Apache Flink with stateful stream processing to consume metrics from Kafka, evaluate threshold and anomaly detection rules in real-time, and support composite alert logic with event time windows.
Interviewer Expectation
Mid: Mentions stream processing with Flink or Spark Streaming. Senior: Details Flink’s exactly-once semantics, event-time processing for anomaly detection, and how composite alerts are implemented as DAGs of operators.
⚠️Common Wrong Answer
Batch processing with Hadoop - introduces high latency, violating the 200ms P99 delivery constraint.
Step 3
Alert Delivery and Offline Fallback
How will you deliver alerts to users in real-time and support offline fallback mechanisms securely?
💡 Hint
Consider a push notification system with guaranteed delivery and fallback queues for offline channels like email or SMS.
Key Decision
Use AWS SNS with encrypted topics for real-time push notifications; if delivery fails, enqueue alerts in Amazon SQS with encryption for offline delivery workers to send email/SMS via Amazon SES and Twilio respectively.
Interviewer Expectation
Mid: Names SNS and SQS for delivery and fallback. Senior: Explains encryption at rest and in transit for SNS/SQS, delivery retries, dead-letter queues, and how offline workers ensure eventual delivery.
⚠️Common Wrong Answer
Directly sending emails synchronously on alert generation - causes high latency and risk of alert loss if email service is down.
Step 4
API Design for Alert Configuration and Acknowledgment
Design 3 core API endpoints for clients to submit metrics, users to configure alert rules, and users to acknowledge alerts.
💡 Hint
Think REST or gRPC endpoints with authentication, validation, and idempotency for configuration and acknowledgment.
Key Decision
Expose REST APIs: POST /metrics for metric ingestion with batching and authentication; POST/PUT /alerts/rules for alert rule CRUD with validation; POST /alerts/{alert_id}/ack to acknowledge alerts, all secured with OAuth2 and TLS.
Interviewer Expectation
Mid: Defines basic endpoints and methods. Senior: Details request/response schemas, idempotency keys for retries, validation of composite rules, and security considerations.
⚠️Common Wrong Answer
Using unauthenticated HTTP endpoints or polling-based acknowledgment causing security risks and inefficient client-server communication.
Step 5
Scaling and Fault Tolerance
How will you ensure the system scales horizontally and remains fault tolerant under peak load?
💡 Hint
Consider partitioning, replication, and stateless components with checkpointing for stateful processing.
Key Decision
Kafka partitions scale ingestion; Cassandra clusters with replication factor 3 ensure durability; Flink jobs use checkpointing and savepoints for fault tolerance; alert delivery components are stateless and horizontally scalable behind load balancers.
Interviewer Expectation
Mid: Mentions replication and partitioning. Senior: Explains Kafka partition key choice for load balancing, Cassandra consistency levels, Flink exactly-once checkpointing, and how stateless delivery workers enable elastic scaling.
⚠️Common Wrong Answer
Single-node processing or no replication - leads to data loss and inability to handle traffic spikes.
🔌
API Design Challenge
Sketch 3 core API endpoints for the alerting system: how clients submit metrics, how users configure alert rules, and how alerts are delivered or acknowledged.
Hints
  • How does the client efficiently send high-volume metric data?
  • How can users define and update threshold, anomaly, and composite alert rules?
  • How does the system notify users and receive alert acknowledgments?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with TLS for metric ingestion to handle millions of metrics/sec
  • Use Cassandra with encryption at rest for scalable time-series storage
  • Use Flink for real-time alert evaluation supporting complex patterns
  • Design APIs for metric ingestion, alert rule config, and alert acknowledgment with security
  • Implement real-time alert delivery with fallback offline delivery queues
Good to Have - 30%
  • Explain Kafka partition key choice for load balancing and ordering
  • Describe Flink’s event-time processing and exactly-once semantics
  • Detail encryption in transit and at rest for all components
Bonus - 10%
    Anti-patterns to Avoid
    ✅
    Model Answer
    ▼ Reveal

    The alerting system ingests millions of encrypted metrics per second via Apache Kafka partitioned by metric_id, stores them in Apache Cassandra with encryption at rest, and processes alerts in real-time using Apache Flink's stateful stream processing. Alerts are delivered via AWS SNS with encrypted topics and fallback to Amazon SQS for offline delivery through email/SMS. REST APIs secured with OAuth2 and TLS enable metric submission, alert rule management, and alert acknowledgment. The system scales horizontally with Kafka partitions, Cassandra replication factor 3, and Flink checkpointing for fault tolerance.

    Key Decisions
    • Partition Kafka topics by metric_id to ensure ordering and load balancing
    • Use Cassandra's LSM-tree storage engine with TTL for efficient writes and data retention
    • Implement Flink jobs with exactly-once semantics and event-time windows for anomaly and composite alerts
    • Encrypt all data in transit (TLS) and at rest (Kafka, Cassandra, SNS, SQS)
    • Use SNS delivery retries and dead-letter queues in SQS to guarantee alert delivery
    • Design REST APIs with idempotency keys and validation for alert rule configuration and acknowledgment
    • Apache Kafka with TLS for high-throughput encrypted metric ingestion and partitioning by metric_id
    • Apache Cassandra with encryption at rest for scalable time-series metric storage and efficient range queries
    • Apache Flink for stateful, low-latency stream processing supporting threshold, anomaly, and composite alert evaluation
    • AWS SNS with encrypted topics for real-time alert push notifications with guaranteed delivery
    • Amazon SQS with encryption and dead-letter queues for offline fallback delivery via email/SMS
    • REST APIs secured with OAuth2 and TLS for metric submission, alert rule management, and alert acknowledgment
    • Kafka partitioning and Cassandra replication factor 3 for horizontal scalability and fault tolerance
    • Flink checkpointing and savepoints for exactly-once processing semantics and recovery

    Practice

    (1/5)
    1. A system must guarantee that user-uploaded files are stored durably and retrievable even after server crashes. Which component best addresses this non-functional requirement?
    easy
    A. In-memory cache like Redis for fast access
    B. WebSocket server for real-time upload progress
    C. Load balancer to distribute upload requests
    D. Distributed persistent storage like Amazon S3

    Solution

    1. Step 1: Identify durability requirement

      Durability requires persistent storage that survives crashes, ruling out volatile caches.

    2. Step 2: Match component to durability

      Amazon S3 provides distributed, durable storage with replication and fault tolerance, fitting the requirement.

    3. Final Answer:

      Option D -> Option D

    4. Quick Check:

      Persistent storage ensures data survives crashes [check]
    Hint: Durability needs persistent storage, not cache or load balancer [check]
    Common Mistakes:
    2. In the Snowflake ID generation flow, at Step 3 the sequence number is incremented after the timestamp and machine ID are set. What happens if the sequence number exceeds its maximum value within the same millisecond?
    easy
    A. The generator blocks and waits until the next millisecond to reset the sequence number.
    B. The system resets the sequence to zero and continues generating IDs immediately.
    C. The sequence number wraps around causing duplicate IDs within the same millisecond.
    D. The machine ID is incremented to avoid sequence overflow.

    Solution

    1. Step 1: Understand sequence number role in Snowflake

      The sequence number differentiates IDs generated within the same millisecond on the same machine.

    2. Step 2: Analyze overflow handling

      If the sequence exceeds its max (e.g., 4095), the generator must wait for the next millisecond to avoid duplicates. Resetting immediately (A) risks duplicates; wrapping (C) causes collisions; incrementing machine ID (D) is not part of the design.

    3. Final Answer:

      Option A -> Option A

    4. Quick Check:

      Sequence overflow -> wait next millisecond to avoid duplicates [check]
    Hint: Sequence overflow blocks until next millisecond [OK]
    Common Mistakes:
    3. Without a clear action item tracking system after a post-mortem, what specific failure occurs in the incident resolution process?
    medium
    A. The root cause analysis becomes inaccurate due to missing data
    B. Action items are forgotten or delayed, causing repeated incidents
    C. Incident detection slows down because alerts are not configured
    D. The blame culture intensifies as no one is held accountable

    Solution

    1. Step 1: Identify role of action item tracking

      Tracking ensures follow-through on improvements, preventing recurrence and reducing blame.

    2. Step 2: Trace failure without tracking

      Without tracking, accountability gaps appear, blame culture can worsen as no clear ownership exists.

    3. Final Answer:

      Option D -> Option D

    4. Quick Check:

      No tracking -> blame culture intensifies due to lack of ownership [check]
    Hint: No tracking -> blame culture grows, fixes stall [check]
    Common Mistakes:
    4. Without the message broker (e.g., Kafka) in a pub-sub system, what exactly breaks in the message delivery flow?
    medium
    A. Producers must synchronously send messages to each subscriber; if any subscriber is slow or down, the producer blocks or loses messages
    B. Subscribers receive messages out of order due to lack of broker ordering guarantees
    C. Messages are duplicated because the broker cannot deduplicate
    D. Messages are encrypted incorrectly causing delivery failures

    Solution

    1. Step 1: Role of the broker

      The broker decouples producers and subscribers, buffering messages and ensuring reliable delivery. Without it, producers must send directly to subscribers.

    2. Step 2: Consequences of no broker

      Direct synchronous sends cause producers to block if subscribers are slow or unavailable, risking message loss or backpressure. Duplication (Messages are duplicated because the broker cannot deduplicate) and ordering (Subscribers receive messages out of order due to lack of broker ordering guarantees) issues are secondary effects managed by the broker. Encryption (Messages are encrypted incorrectly causing delivery failures) is unrelated.

    3. Final Answer:

      Option A -> Option A

    4. Quick Check:

      No broker -> producer blocks or loses messages on slow subscribers [check]
    Hint: No broker -> producer blocks on slow subscribers, losing messages [check]
    Common Mistakes:
    5. At 10x scale, which component is most likely to fail first and why when estimating system bandwidth for a social media platform with rapidly growing DAU?
    hard
    A. Storage Estimator, because disk I/O cannot keep up with 10x data volume growth
    B. Bandwidth Estimator, because network metrics aggregation scales linearly and hits memory limits
    C. DAU Estimator, because tracking and aggregating active users in real-time becomes a bottleneck
    D. QPS Estimator, because load balancer metrics become too noisy and inaccurate at high scale

    Solution

    1. Step 1: Identify bottleneck at 10x DAU scale

      DAU Estimator must track millions of active users in real-time, requiring heavy aggregation and state management.

    2. Step 2: Eliminate other options

      Storage Estimator scales with disk but can be sharded; Bandwidth Estimator aggregates metrics but less stateful; QPS Estimator uses load balancer metrics which scale better.

    3. Final Answer:

      Option C -> Option C

    4. Quick Check:

      Real-time DAU tracking state grows fastest, causing bottleneck [check]
    Hint: Real-time DAU tracking state grows fastest at scale [check]
    Common Mistakes:
    Recommended

    Ready to prove it?

    Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

    10 questions~5 mineasy hard