Bird
Raised Fist0

SLO Error Budget Tracking - How Production Systems Measure Reliability

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
SLO Error Budget Tracking - How Production Systems Measure Reliability
45 min · senior
senior

You are tasked with designing an SLO (Service Level Objective) Error Budget Tracking system for a startup with 50 million daily active users. The system must ingest real-time production metrics from multiple microservices, calculate error budgets per SLO, and provide alerting and dashboards for reliability engineers. The system must handle high throughput, ensure data encryption at rest and in transit, and maintain P99 query latency under 200ms for dashboard queries.

Constraints
  • Must support ingestion of 10 million metric events per minute with horizontal scalability
  • All data must be encrypted at rest and in transit to comply with security policies
  • Dashboard queries must have P99 latency under 200ms for up to 1,000 concurrent users
  • Offline delivery of alerts to on-call engineers via email and SMS with guaranteed at-least-once delivery
  • System must tolerate partial failures without losing error budget data or alerting
📋
Guided Steps
1/6
Step 1
Ingestion Pipeline Design
How would you design the ingestion pipeline to handle 10 million metric events per minute with encryption and scalability?
💡 Hint
Consider a distributed messaging system that supports partitioning and encryption at rest and in transit.
Key Decision
Use Apache Kafka with TLS encryption and Kafka's built-in encryption at rest via integration with cloud KMS; partition by service_id to ensure ordered processing per service.
Interviewer Expectation
Mid-level: Names Kafka and TLS encryption. Senior: Explains partitioning by service_id to maintain ordering and why RabbitMQ or SQS would not scale linearly at this throughput.
⚠️Common Wrong Answer
Using HTTP REST endpoints directly for ingestion causing bottlenecks and lack of backpressure, leading to data loss under load.
Step 2
Storage and Aggregation of Metrics
Which storage system would you choose to store raw and aggregated metrics for fast error budget calculations and why?
💡 Hint
Think about time-series databases optimized for high write throughput and fast range queries with encryption support.
Key Decision
Use TimescaleDB (PostgreSQL extension) with data encrypted at rest via cloud provider and TLS in transit; it supports time-series compression and fast aggregation queries.
Interviewer Expectation
Mid-level: Suggests a time-series DB like InfluxDB or TimescaleDB. Senior: Explains why TimescaleDB's hypertables and compression outperform InfluxDB at scale and how encryption is integrated.
⚠️Common Wrong Answer
Using a NoSQL document store like MongoDB which lacks efficient time-series aggregation and leads to slow queries and high storage costs.
Step 3
Real-time Error Budget Calculation and Alerting
How would you implement real-time error budget calculation and alerting with offline delivery guarantees?
💡 Hint
Consider stream processing frameworks that integrate with Kafka and support exactly-once semantics and external alert delivery systems.
Key Decision
Use Apache Flink for stream processing with Kafka source and sink connectors, enabling exactly-once processing; integrate with Twilio and SendGrid for SMS and email alert delivery with retry and dead-letter queues.
Interviewer Expectation
Mid-level: Names Flink or Spark Streaming. Senior: Explains Flink's exactly-once semantics and why Spark Streaming's micro-batching adds latency; details offline delivery retry mechanisms.
⚠️Common Wrong Answer
Batch processing metrics hourly causing delayed alerting and missing critical error budget breaches.
Step 4
Dashboard Query API Design
How would you design the API and backend to serve dashboard queries with P99 latency under 200ms for 1,000 concurrent users?
💡 Hint
Consider caching layers and query optimization on aggregated data with horizontal scaling and encryption compliance.
Key Decision
Expose REST API backed by a horizontally scaled TimescaleDB read replica cluster with Redis caching for hot queries; all traffic encrypted via TLS and API gateway enforcing authentication.
Interviewer Expectation
Mid-level: Mentions caching and read replicas. Senior: Explains cache invalidation strategies, query pre-aggregation, and why a single DB instance or no caching would fail latency SLAs.
⚠️Common Wrong Answer
Directly querying the primary DB for every dashboard request causing high latency and DB overload.
Step 5
Security and Data Encryption
How do you ensure encryption at rest and in transit across all components while maintaining performance?
💡 Hint
Look for cloud-native encryption features and TLS everywhere with minimal overhead.
Key Decision
Enable TLS 1.3 for all network communication; use cloud provider-managed encryption keys (KMS) for Kafka, TimescaleDB, and object storage; leverage hardware acceleration for encryption to minimize latency.
Interviewer Expectation
Mid-level: Names TLS and encryption at rest. Senior: Details KMS integration, key rotation policies, and performance trade-offs of encryption overhead.
⚠️Common Wrong Answer
Encrypting only in transit but storing plaintext data at rest, risking compliance violations.
Step 6
API Design for Client Interaction
Sketch 3 core API endpoints for ingestion, querying error budgets, and alert acknowledgement. What are their inputs, outputs, and security considerations?
💡 Hint
Think about REST or gRPC endpoints with authentication, payload formats, and idempotency.
Key Decision
POST /metrics/ingest with JSON payload secured by OAuth2 bearer tokens; GET /errorbudgets?slo_id=... returns aggregated error budget status with pagination; POST /alerts/{alert_id}/ack to acknowledge alerts with idempotency tokens.
Interviewer Expectation
Mid-level: Defines basic endpoints and authentication. Senior: Explains idempotency, pagination, filtering, and how to handle partial failures and retries securely.
⚠️Common Wrong Answer
Using unauthenticated endpoints or lacking idempotency causing duplicate data and security risks.
🔌
API Design Challenge
Sketch 3 core API endpoints for the SLO Error Budget Tracking system: ingestion, querying error budgets, and alert acknowledgement.
Hints
  • How does the client send metric events securely and reliably?
  • How can users query aggregated error budget status efficiently?
  • How does the system allow on-call engineers to acknowledge alerts and update alert state?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of Kafka with partitioning and encryption for ingestion pipeline
  • Time-series optimized storage with encryption at rest and fast aggregation
  • Stream processing with exactly-once semantics for real-time error budget calculation
  • Caching and read replicas to meet dashboard query latency SLAs
  • Encryption at rest and in transit with KMS integration
  • Secure REST API design with authentication and idempotency
Good to Have - 30%
    Bonus - 10%
      Anti-patterns to Avoid
      ✅
      Model Answer
      ▼ Reveal

      This design uses Apache Kafka for scalable, encrypted ingestion of 10M metric events per minute, partitioned by service_id for ordering. TimescaleDB stores raw and aggregated metrics with encryption and fast queries. Apache Flink processes streams in real-time with exactly-once semantics to calculate error budgets and trigger alerts. Redis caching and horizontally scaled TimescaleDB read replicas ensure dashboard queries meet P99 latency under 200ms for 1,000 concurrent users. Alerts are delivered offline via Twilio and SendGrid with retry and dead-letter queues. All components use TLS 1.3 and cloud KMS for encryption key management. The REST API supports ingestion, querying, and alert acknowledgement with OAuth2 authentication, idempotency tokens, and pagination.

      Key Decisions
      • Partition Kafka topics by service_id to maintain ordering and scalability
      • Use TimescaleDB for time-series data with encryption at rest and TLS in transit
      • Implement real-time error budget calculation using Apache Flink with exactly-once semantics
      • Cache hot dashboard queries in Redis to meet latency SLAs
      • Use TLS 1.3 and cloud KMS for encryption key management and rotation
      • Design REST API endpoints with OAuth2 authentication, idempotency tokens, and pagination
      • Integrate Twilio and SendGrid with retry and dead-letter queues for reliable offline alert delivery
      • Scale TimescaleDB read replicas horizontally to support 1,000 concurrent dashboard users
      • Apache Kafka with TLS and KMS encryption for scalable, ordered, and secure ingestion
      • TimescaleDB for time-series storage with hypertables and compression for fast aggregation
      • Apache Flink for real-time stream processing with exactly-once semantics for error budget calculation
      • Redis caching layer to reduce dashboard query latency under 200ms at scale
      • Cloud provider KMS integration for encryption key management and rotation
      • REST API with OAuth2 authentication, idempotency tokens, and pagination for client interaction
      • Twilio and SendGrid integration with retry and dead-letter queues for offline alert delivery
      • Horizontally scaled read replicas for TimescaleDB to handle 1,000 concurrent dashboard users

      Practice

      (1/5)
      1. In the web crawler's request flow, at Step 5 the Duplicate Detection component receives a URL. What happens if this URL has been seen before?
      easy
      A. The URL is discarded and not sent to the Scheduler to avoid redundant crawling
      B. The URL is immediately re-queued to the Scheduler for crawling again
      C. The URL is marked for delayed re-crawl but still sent to the Scheduler
      D. The URL is sent to the Politeness Policy Manager to check domain delays

      Solution

      1. Step 1: Understand Duplicate Detection behavior at Step 5

        If the URL has been seen before, it is filtered out to prevent redundant crawling.

      2. Step 2: Eliminate other options

        Re-queuing duplicates wastes resources; marking for delayed re-crawl is a separate process; Politeness Policy Manager handles timing, not duplication.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Duplicate Detection discards seen URLs to avoid redundancy [check]
      Hint: Duplicate Detection discards duplicates before scheduling [OK]
      Common Mistakes:
      2. In the failover incident trace, at Step 4 the Database Proxy receives a client query after the primary node crashed. What state does the proxy enter before forwarding the query?
      easy
      A. Marks the primary as unavailable and routes query to the new leader after election
      B. Returns an error to the client without retrying
      C. Queues the query indefinitely until the old primary recovers
      D. Immediately forwards query to the old primary node, causing timeout

      Solution

      1. Step 1: Trace Database Proxy behavior at Step 4

        Upon detecting primary failure, the proxy marks it unavailable and waits for leader election to complete before routing queries to the new leader.

      2. Step 2: Eliminate other options

        Forwarding immediately to old primary (A) causes timeouts; queuing indefinitely (C) stalls clients; returning errors without retry (B) reduces availability.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Proxy reroutes queries only after leader election [check]
      Hint: Proxy reroutes only after leader election completes
      Common Mistakes:
      3. A company must expose internal web services securely to external clients while hiding internal server details and enabling SSL termination. Which proxy architecture is best suited for this?
      medium
      A. Use a reverse proxy to terminate SSL and hide internal server details
      B. Use a transparent proxy to intercept and forward requests without modification
      C. Use a forward proxy to route external client requests to internal servers
      D. Use a DNS proxy to resolve internal server names for external clients

      Solution

      1. Step 1: Identify proxy role for exposing internal services

        A reverse proxy sits in front of internal servers, hides their details, and can terminate SSL for external clients.

      2. Step 2: Eliminate other options

        Forward proxies act on clients, not servers; transparent proxies don't modify requests; DNS proxies only resolve names, not handle SSL or hide servers.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Reverse proxy hides servers and terminates SSL [check]
      Hint: Reverse proxy hides servers and handles SSL termination [check]
      Common Mistakes:
      4. If the recursive DNS server's cache is completely disabled, what is the most immediate failure mode experienced by users during domain resolution?
      medium
      A. Users' browsers fallback to using IP addresses directly, bypassing DNS
      B. Users receive stale IP addresses causing connection failures
      C. The recursive server returns IP addresses from its local hosts file instead of querying DNS servers
      D. The recursive server blocks user queries until authoritative servers respond, increasing latency and potential timeouts

      Solution

      1. Step 1: Understand recursive server caching role

        The recursive DNS server cache reduces query latency by storing recent results. Without caching, every query requires a full resolution chain traversal.

      2. Step 2: Trace failure impact

        Without cache, recursive server must synchronously query authoritative servers for every request, causing increased latency and possible timeouts, blocking user queries.

      3. Final Answer:

        Option D -> Option D

      4. Quick Check:

        No cache -> synchronous queries -> high latency and blocking [check]
      Hint: No cache means blocking queries until authoritative response [check]
      Common Mistakes:
      5. At 10x scale, which component in a system using TLS for encryption in transit is most likely to become the bottleneck first, and why?
      hard
      A. The TLS termination proxy, because it must handle CPU-intensive handshake operations for all new connections
      B. The Application Servers, because they decrypt and encrypt all traffic themselves causing CPU exhaustion
      C. The Network Load Balancer, because it cannot scale beyond a fixed number of connections
      D. The Database, because encrypted queries increase query processing time

      Solution

      1. Step 1: Identify TLS termination proxy role

        It handles all TLS handshakes which are CPU-intensive, especially at high connection rates.

      2. Step 2: Compare other components

        Application servers offload encryption to TLS proxy; load balancer scales horizontally; database impact is unrelated to TLS termination.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        TLS proxy CPU-bound on handshakes -> first bottleneck at scale [check]
      Hint: TLS handshake CPU load hits termination proxy first [check]
      Common Mistakes:
      Recommended

      Ready to prove it?

      Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

      10 questions~5 mineasy hard