Bird
Raised Fist0

SLO Error Budget Tracking - How Production Systems Measure Reliability

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Kafka Cluster

Ingest metrics

Receives raw production metrics from monitored services

Protocol:Kafka Producer
BeforeMetrics generated by production systems but not yet ingested
AfterMetrics durably stored in Kafka topic partitions
Apache Flink Stream Processing

Process stream

Consumes metrics from Kafka, computes error budget consumption in real-time

Protocol:Kafka Consumer
BeforeMetrics available in Kafka but not yet processed
AfterError budget computations updated and ready to be persisted
Cassandra Database

Persist results

Stores computed SLO error budget states for querying

Protocol:CQL Write
BeforeError budget data computed but not yet stored
AfterError budget states durably stored in Cassandra tables
API & Dashboard Service

Serve queries

Fetches SLO and error budget data from Cassandra to serve UI and API clients

Protocol:CQL Read / REST API
BeforeSLO data stored but not yet served to clients
AfterClients receive up-to-date SLO error budget metrics
Alerting Service

Evaluate thresholds

Polls Cassandra or subscribes to updates to trigger alerts if budgets breached

Protocol:CQL Read / Internal Event
BeforeError budget data available but no alert triggered
AfterAlerts sent to operators if error budget consumption exceeds threshold
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│       Kafka Cluster           │
│  (Ingests production metrics) │
└───────────────┬───────────────┘
                │ Kafka Stream
                ▼
┌───────────────────────────────┐
│ Apache Flink Stream Processing │
│ (Computes SLO error budgets)  │
└───────────────┬───────────────┘
                │ Writes results
                ▼
┌───────────────────────────────┐
│    Cassandra Database          │
│ (Stores SLO and error budget) │
└───────────────┬───────────────┘
                │ Queries via CQL
                ▼
┌───────────────────────────────┐
│  API & Dashboard Service       │
│ (Serves SLO data & UI)         │
└───────────────┬───────────────┘
                │ REST API calls
                ▼
┌───────────────────────────────┐
│     Alerting Service           │
│ (Triggers alerts on thresholds)│
└───────────────────────────────┘
Design Rationale

This architecture tracks SLO error budgets by ingesting production metrics into a Kafka Cluster, which acts as a durable, scalable message queue. Apache Flink processes these streams in real-time, computing error budget consumption continuously. The results are persisted in Cassandra, a wide-column database optimized for fast writes and queries at scale. The API & Dashboard Service queries Cassandra to provide up-to-date SLO metrics to users via REST APIs and UI dashboards. Meanwhile, the Alerting Service monitors stored error budget states and triggers alerts when thresholds are breached, enabling proactive reliability management. This vertical pipeline ensures reliable, scalable, and real-time SLO tracking.

Primary Focus

✓

Practice

(1/5)
1. A high-throughput messaging system must decouple 1 million writes per second from fan-out delivery to thousands of clients. Which component best handles this decoupling to ensure durability and scalability?
easy
A. Use a distributed log system like Kafka partitioned by conversation ID
B. Use a relational database with row-level locking to queue messages
C. Use Redis as an in-memory queue with TTL for message expiration
D. Use direct synchronous HTTP calls from sender to all recipients

Solution

  1. Step 1: Identify the decoupling requirement

    Relational DB with row-level locking (Use a relational database with row-level locking to queue messages) cannot handle 1M writes/sec due to locking and durability overhead. Redis (Use Redis as an in-memory queue with TTL for message expiration) lacks durability guarantees and TTL causes message loss risk. Direct synchronous calls (Use direct synchronous HTTP calls from sender to all recipients) block sender and don't scale.

  2. Step 2: Evaluate Kafka's suitability

    Kafka partitions by conversation ID ensure ordering and durability, enabling scalable fan-out with consumer groups.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka partitions -> durable, scalable decoupling [check]
Hint: Distributed logs decouple writes and fan-out reliably [check]
Common Mistakes:
2. In the typeahead system request flow, at step 4 the Ranking Service receives candidate suggestions from the Trie Cache. What does the Ranking Service do if the personalization data for the user is missing at this step?
easy
A. It returns an empty suggestion list immediately to avoid stale results
B. It blocks the request until personalization data is fetched synchronously
C. It falls back to global popularity-based ranking without personalization
D. It queries the Trie Cache again with a different prefix to compensate

Solution

  1. Step 1: Trace Ranking Service behavior at step 4

    Ranking Service combines candidate suggestions with personalization data; if missing, it must still return relevant results.

  2. Step 2: Evaluate fallback strategies

    Blocking for personalization data (C) causes latency; empty results (A) degrade UX; querying Trie again (D) is unrelated. Falling back to global popularity ranking ensures responsiveness.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Ranking Service fallback -> global popularity ranking [check]
Hint: Missing personalization -> fallback to global popularity ranking [check]
Common Mistakes:
3. If the Duplicate Detection component fails silently and stops filtering URLs, what is the most immediate failure observed in the crawler system?
medium
A. The URL Frontier crashes because it cannot handle duplicate URLs
B. The Politeness Policy Manager blocks all requests due to unexpected URL patterns
C. The Scheduler rejects all URLs due to overload and stops scheduling new crawls
D. The crawler repeatedly fetches the same URLs, wasting bandwidth and storage

Solution

  1. Step 1: Trace failure impact of Duplicate Detection silent failure

    Without filtering, duplicate URLs flood the system causing repeated fetches.

  2. Step 2: Eliminate other options

    Scheduler overload is secondary; Politeness Policy unaffected by duplicates; URL Frontier typically handles duplicates gracefully.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Duplicate Detection failure causes repeated URL fetches [check]
Hint: No duplicate filtering -> repeated URL fetches waste resources [OK]
Common Mistakes:
4. Without the graph database's relationship index, what exactly breaks during a shortest path query between two nodes?
medium
A. The query silently returns empty results without any error indication
B. The query falls back to scanning all nodes and edges, causing severe latency spikes
C. The query returns incorrect paths due to stale data in the index
D. The query crashes the database server due to unhandled exceptions

Solution

  1. Step 1: Role of relationship index

    The index enables efficient lookup of edges; without it, traversal cannot proceed correctly.

  2. Step 2: Behavior without index

    Queries relying on the index return empty results silently because no edges are found, not errors or scans.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing index causes silent empty results, not crashes or scans [check]
Hint: Missing index -> silent empty results, no errors or scans [check]
Common Mistakes:
5. At 10x scale, which component of a distributed Count-Min Sketch system is most likely to fail first and why?
hard
A. The network bandwidth saturates due to large sketch state synchronization
B. The sketch arrays in memory overflow due to fixed size limits
C. The hash functions become slower, increasing latency linearly
D. The centralized coordinator bottlenecks due to serializing all update requests

Solution

  1. Step 1: Identify bottleneck in distributed updates

    Centralized coordinator serializes updates to maintain consistency, limiting throughput.

  2. Step 2: Analyze other components

    Network bandwidth and hash function speed scale better; sketch arrays have fixed size but don't overflow.

  3. Step 3: Understand scale impact

    At 10x scale, coordinator becomes overwhelmed, causing update latency and failures.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Central coordinator serialization bottlenecks updates at scale [check]
Hint: Central coordinator serializes updates, bottlenecking at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard