Bird
Raised Fist0

SLO Error Budget Tracking - How Production Systems Measure Reliability

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
🏗️
SLO Error Budget Tracking - How Production Systems Measure Reliability
Design a scalable system to track SLO error budgets for production reliability measurement
mediumMonitoring and ObservabilityFreq: common
⚙️
Functional Requirements
  • Ingest real-time SLO measurement data (e.g., latency, availability) from multiple production services
  • Calculate error budget consumption continuously per SLO with configurable time windows (e.g., rolling 7-day)
  • Provide APIs and dashboards to query current error budget status and historical trends
  • Trigger alerts when error budget consumption crosses predefined thresholds
  • Support multi-tenant isolation for different teams and services
  • Allow configuration of SLO definitions, error budget policies, and alert thresholds
📊
Non-Functional Requirements
  • Support at least 2 billion daily SLO measurement events (2B DAU scale)
  • 99.99% availability for error budget query APIs and alerting system
  • P99 query latency under 100ms for dashboard and API responses
  • Data retention of at least 30 days for historical error budget analysis
  • System must handle peak write throughput spikes up to 2x average
❓
Clarifying Questions to Ask
  • What types of SLO metrics are ingested (latency, availability, throughput)?
  • Are SLOs defined per service, per endpoint, or per customer?
  • What is the expected write-to-read ratio for the system?
  • What alerting channels and integrations are required?
  • Is real-time streaming processing required or batch processing acceptable?
  • What is the acceptable staleness for error budget calculations?
Out of Scope
✗ Defining or instrumenting SLO metrics at the application code level
✗ Root cause analysis or automated remediation of SLO violations
✗ Long-term archival beyond 30 days or compliance reporting
📌
Assumptions
  • 2 billion SLO measurement events ingested daily
  • Each event is approximately 100 bytes (timestamp, SLO ID, metric value, tags)
  • Average 40 SLO measurement events per user per day
  • Retention period of 30 days for historical data
⚡
QPS Calculation
write qps2B events / 86400 seconds ≈ 23148 writes/sec
peak write qps23148 × 2 (peak factor) = 46296 writes/sec
read qpsAssuming 10 reads per write, 231480 reads/sec peak
💾
Storage Estimate
daily storage2B × 100 bytes = 200 GB/day
retention storage200 GB × 30 days = 6 TB
with replication6 TB × 3 replicas = 18 TB total storage
🖥️
Server Estimate
write ingestionKafka cluster with 10 brokers to handle 46K writes/sec peak (each broker 5K writes/sec capacity)
storageCassandra cluster with 30 nodes (each node 600 GB storage, 6000 writes/sec capacity) for 18 TB replicated data
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Production    |
          |  Services      |
          +-------+--------+
                  |
                  | SLO Metrics (latency, availability)
                  v
          +-------+--------+
          | Kafka Cluster  |
          | (10 brokers)   |
          +-------+--------+
                  |
                  | Stream of SLO events
                  v
          +-------+--------+
          | Stream Processing|
          | (Apache Flink)   |
          +-------+--------+
                  |
                  | Aggregated error budget data
                  v
          +-------+--------+
          | Cassandra DB   |
          | (30 nodes)     |
          +-------+--------+
                  |
                  | Query API
                  v
          +-------+--------+
          | API & Dashboard|
          +-------+--------+
                  |
                  | Alerts
                  v
          +-------+--------+
          | Alerting Svc   |
          +----------------+
🧩
Components
Kafka Cluster
Ingest and buffer high-throughput SLO measurement events from production services
Why thisKafka supports high-throughput, partitioned, durable log storage with strong ordering guarantees and consumer scalability
Why notRabbitMQ lacks partitioning and throughput at 46K writes/sec with 2B daily events; Kinesis is AWS-specific and less flexible
Apache Flink Stream Processing
Real-time aggregation and calculation of error budget consumption per SLO with windowing
Why thisFlink provides exactly-once semantics, low-latency event processing, and native support for time windows needed for rolling SLO calculations
Why notSpark Streaming has higher latency and micro-batch model unsuitable for near real-time error budget updates
Cassandra Database
Store aggregated error budget data with high write throughput and horizontal scalability
Why thisCassandra's wide-column model supports time-series data with efficient writes and tunable consistency for 30-node cluster at 18 TB replicated data
Why notPostgreSQL requires manual sharding and does not scale linearly at 46K writes/sec; HBase has higher operational complexity
API & Dashboard Service
Serve low-latency queries for current and historical error budget status and provide visualization
Why thisStateless REST API servers behind load balancer enable horizontal scaling and caching for sub-100ms P99 latency
Why notGraphQL adds unnecessary complexity and latency for simple SLO error budget queries
Alerting Service
Monitor error budget consumption thresholds and trigger alerts via email, Slack, PagerDuty
Why thisDedicated alerting service decouples alert logic from data ingestion and supports flexible notification channels
Why notEmbedding alerting in stream processing risks delayed or missed alerts due to backpressure or failures; dedicated services like PagerDuty or Opsgenie provide reliable alert delivery
🔄
Data Flow
1
Production Services → Kafka Cluster
Publish raw SLO measurement events (latency, availability) continuously
Kafka partitions events by SLO ID for parallelism and ordering
2
Kafka Cluster → Apache Flink
Consume event stream for real-time aggregation
Flink applies sliding time windows to compute error budget consumption
3
Apache Flink → Cassandra DB
Write aggregated error budget metrics per SLO and time window
Writes are idempotent and use Cassandra's time-series schema
4
API & Dashboard → Cassandra DB
Query current and historical error budget data
Queries optimized by partition key and clustering columns for fast retrieval
5
API & Dashboard → Alerting Service
Trigger alerts when error budget thresholds are crossed
Alerting service sends notifications via configured channels

Practice

(1/5)
1. A video upload service must handle millions of concurrent uploads and ensure durability before transcoding. Which component best handles temporary storage and durability of raw uploads before processing?
easy
A. Use an in-memory cache like Redis to store raw uploads temporarily
B. Use a CDN edge cache to hold raw uploads before transcoding
C. Use a distributed object storage system like Amazon S3 for raw video storage
D. Store raw uploads directly in a relational database for strong consistency

Solution

  1. Step 1: Identify durability and scale needs for raw uploads

    Raw uploads require durable, scalable storage that can handle large binary files and high concurrency. Redis (Use an in-memory cache like Redis to store raw uploads temporarily) is in-memory and not durable for large files. Relational DBs (Store raw uploads directly in a relational database for strong consistency) are inefficient for large blobs and scale poorly. CDN edge caches (Use a CDN edge cache to hold raw uploads before transcoding) are for delivery, not upload storage.
  2. Step 2: Recognize object storage suitability

    Distributed object storage like S3 provides durability, scalability, and cost-effective storage for large files, making it ideal for raw uploads before transcoding.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Object storage ensures durable, scalable raw upload storage [check]
Hint: Object storage for durable large file uploads [check]
Common Mistakes:
2. To support personalized course recommendations at scale, should the system use a batch offline ML pipeline or a real-time streaming recommendation engine? Consider latency and update frequency constraints.
medium
A. Use batch offline ML pipeline for real-time personalized recommendations
B. Use real-time streaming engine with complex models requiring heavy computation
C. Use real-time streaming engine for low-latency but less accurate recommendations
D. Use batch offline ML pipeline for periodic recommendations with high accuracy

Solution

  1. Step 1: Analyze latency and update frequency constraints

    Real-time streaming (Options C and D) offers low latency but may sacrifice model complexity or accuracy.
  2. Step 2: Evaluate batch offline pipeline suitability

    Batch offline ML (Use batch offline ML pipeline for periodic recommendations with high accuracy) provides higher accuracy with periodic updates, suitable when real-time freshness is not critical.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Batch offline pipeline balances accuracy and periodic updates [check]
Hint: Batch ML for accuracy, streaming for latency trade-off [OK]
Common Mistakes:
3. For job recommendation ranking, which architecture better supports real-time updates to user preferences and job postings with low latency and high throughput?
medium
A. Batch processing of job and user data nightly with offline model training
B. Online feature store with streaming data pipelines and real-time model scoring
C. Periodic REST API calls to external recommendation services with caching
D. Manual curation of job recommendations updated weekly by content teams

Solution

  1. Step 1: Analyze latency and throughput requirements

    Real-time updates require low latency and high throughput, which batch processing (Batch processing of job and user data nightly with offline model training) and manual curation (Manual curation of job recommendations updated weekly by content teams) cannot provide.
  2. Step 2: Compare online feature store vs periodic REST calls

    Online feature store with streaming pipelines (Online feature store with streaming data pipelines and real-time model scoring) supports continuous updates and real-time scoring, unlike periodic REST calls (Periodic REST API calls to external recommendation services with caching) which add latency and stale data.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Streaming pipelines + online store -> real-time low latency [check]
Hint: Streaming pipelines enable real-time model updates [OK]
Common Mistakes:
4. Without explicitly stating your assumptions during a design interview, what failure mode is most likely to occur in your explanation?
medium
A. The interviewer assumes your design is perfect and asks no questions
B. Your design appears inconsistent or incomplete, causing confusion and follow-up questions
C. Your design is accepted without challenge, even if it has flaws
D. The interviewer provides the solution for you to avoid confusion

Solution

  1. Step 1: Understand the role of stating assumptions

    Explicit assumptions clarify scope and constraints, preventing misunderstandings.
  2. Step 2: Trace failure when assumptions are omitted

    Without assumptions, interviewers may accept incomplete designs without challenge, missing flaws.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Omitting assumptions leads to unchallenged flawed designs [check]
Hint: State assumptions to avoid unchallenged flawed designs [check]
Common Mistakes:
5. At 10x scale, which component in the AI Code Assistant system is most likely to fail first due to stateful connection limits, and why?
hard
A. The API gateway, because it cannot handle the increased request routing throughput
B. The code generator service, because GPU resources saturate under concurrent requests
C. The WebSocket servers maintaining user sessions, because stateful connections cannot be sharded easily
D. The vector search engine, because its index size grows linearly and slows queries

Solution

  1. Step 1: Analyze vector search scaling

    Vector search scales with index size but can shard horizontally; query latency grows but not immediate failure (A incorrect).
  2. Step 2: Assess code generator scaling

    GPU saturation is a bottleneck but can be mitigated by batching and autoscaling (B less immediate).
  3. Step 3: Evaluate API gateway

    API gateways are stateless and horizontally scalable (C incorrect).
  4. Step 4: Identify WebSocket server bottleneck

    WebSocket servers hold stateful connections bound to server memory; sharding existing connections is complex, causing first failure at 10x scale (D correct).
  5. Final Answer:

    Option C -> Option C
  6. Quick Check:

    Stateful WebSocket connections limit server scaling [check]
Hint: Stateful WebSocket servers bottleneck scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard