Bird
Raised Fist0

Observability - Metrics, Logging, Tracing - The Three Pillars

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Observability - Metrics, Logging, Tracing - The Three Pillars
Design a scalable observability platform covering metrics, logging, and tracing for 2B DAU
mediumNon-Functional Requirements FrameworkFreq: high
⚙️
Functional Requirements
  • Collect and ingest metrics data (counters, gauges, histograms) from 2B DAU services
  • Collect and ingest structured logs with timestamps and metadata from all services
  • Collect distributed traces with spans and context propagation for request flows
  • Support real-time querying and alerting on metrics with sub-second latency
  • Support log search and filtering with time-range and metadata queries
  • Support trace lookup and visualization with span-level detail and dependency graphs
📊
Non-Functional Requirements
  • Handle ingestion write throughput of ~925K messages/sec (metrics+logs+traces combined)
  • Store at least 30 days of raw observability data with replication for durability
  • Provide 99.99% availability for ingestion and query APIs
  • Ensure query P99 latency under 100ms for metrics and under 500ms for logs and traces
  • Scale horizontally to support 2 billion daily active users generating 40 messages/user/day
❓
Clarifying Questions to Ask
  • What is the expected ratio of metrics, logs, and traces in the data volume?
  • Are metrics expected to be high cardinality or mostly low cardinality?
  • What is the SLA for data freshness in queries (near real-time vs batch)?
  • Is the system multi-tenant or single-tenant? Any isolation requirements?
  • What query types and aggregation functions must be supported for metrics?
  • Are logs structured (JSON) or unstructured text? Any indexing requirements?
Out of Scope
✗ Instrumentation libraries and SDK design for client-side data collection
✗ Alerting rule authoring UI and notification delivery mechanisms
✗ Long-term archival beyond 30 days (cold storage)
✗ Security and access control policies for multi-tenant isolation
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates on average 40 observability messages per day (metrics+logs+traces combined)
  • Average message size is 100 bytes
  • Retention period is 30 days
  • Replication factor for durability is 3
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86400 ≈ 925,926 messages/sec
peak write qpsAssuming 3x peak factor: 925,926 × 3 ≈ 2,777,778 messages/sec
read qpsAssuming 10% of write QPS for queries: 925,926 × 0.1 ≈ 92,593 queries/sec
💾
Storage Estimate
daily storage gb(2,000,000,000 × 40 × 100) / (1024^3) ≈ 7,450 GB/day
total storage 30d gb7,450 × 30 ≈ 223,500 GB (223.5 TB)
with replication gb223,500 × 3 ≈ 670,500 GB (670.5 TB)
🖥️
Server Estimate
ingestion serversAssuming each Kafka broker handles 50K msg/sec, need 2,778,000 / 50,000 ≈ 56 brokers
storage nodesAssuming each Cassandra node stores 5 TB with replication, need 670.5 TB / 5 TB ≈ 135 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Instrumented  |
          |   Services     |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |   Kafka Cluster |
          | (Ingestion Bus) |
          +-------+--------+
          /        |         \
         /         |          \
        v          v           v
 +-------+--+ +----+-------+ +----+-------+
 | Metrics  | | Logs Store | | Tracing    |
 | Storage  | | (Elasticsearch) | Storage  |
 | (Cassandra) | +-------------+ | (Cassandra) |
 +-------+--+                 +----+-------+
        |                          |
        v                          v
 +----------------+        +----------------+
 | Metrics Query  |        | Logs & Trace   |
 | API (Prometheus|        | Query API      |
 | compatible)    |        | (Elasticsearch +|
 +----------------+        | Jaeger UI)     |
                           +----------------+
🧩
Components
Kafka Cluster
Durable, scalable ingestion bus for all observability data streams
Why thisKafka supports high throughput (millions msg/sec), partitioning, and retention for buffering ingestion spikes
Why notRabbitMQ lacks horizontal scalability and throughput at 2.7M msg/sec peak; Kinesis is AWS-specific and less flexible
Metrics Storage (Apache Cassandra)
Store time-series metrics data with high write throughput and horizontal scalability
Why thisCassandra provides linear write scalability, tunable consistency, and efficient time-series data modeling with TTL
Why notPostgreSQL cannot handle 925K+ writes/sec without complex sharding; InfluxDB struggles with multi-tenant scale at 2B DAU
Logs Storage (Elasticsearch)
Index and store structured logs for fast full-text search and filtering
Why thisElasticsearch provides distributed inverted indexes, rich query DSL, and near real-time indexing for log search
Why notSplunk is proprietary and costly at this scale; Cassandra lacks efficient full-text search capabilities
Tracing Storage (Apache Cassandra)
Store distributed trace spans with high write throughput and fast retrieval by trace ID
Why thisCassandra's wide-row model suits trace spans grouped by trace ID; supports high write volume and TTL for retention
Why notRelational DBs have poor write scalability; Elasticsearch is inefficient for large span payloads and joins
Metrics Query API (Prometheus compatible)
Serve real-time metrics queries with sub-second latency and aggregation
Why thisPrometheus query language (PromQL) is industry standard; integration with Cassandra via custom adapters enables scalability
Why notGraphite lacks multi-tenancy and scalability; direct Cassandra queries are complex and slow for ad-hoc metrics
Logs & Trace Query API (Elasticsearch + Jaeger UI)
Provide search and visualization for logs and traces with rich filtering and span-level detail
Why thisElasticsearch supports fast log queries; Jaeger UI integrates with Cassandra for trace visualization and dependency graphs
Why notZipkin lacks scalability at this scale; custom UI without Elasticsearch would have poor query performance
🔄
Data Flow
1
Instrumented Services → Kafka Cluster
Services push metrics, logs, and traces asynchronously to Kafka topics
Kafka partitions data by service and data type for parallelism
2
Kafka Cluster → Metrics Storage (Cassandra)
Metrics consumer reads from Kafka and writes time-series data to Cassandra
Use wide-row schema with TTL for retention and efficient queries
3
Kafka Cluster → Logs Storage (Elasticsearch)
Logs consumer indexes structured logs into Elasticsearch for search
Use index rotation and shard allocation for scaling
4
Kafka Cluster → Tracing Storage (Cassandra)
Tracing consumer stores spans grouped by trace ID in Cassandra
Wide-row model enables fast trace retrieval and TTL enforces retention
5
Metrics Storage (Cassandra) → Metrics Query API
Query API serves PromQL queries by reading aggregated metrics from Cassandra
Custom adapters optimize query latency and aggregation
6
Logs Storage (Elasticsearch) and Tracing Storage (Cassandra) → Logs & Trace Query API
Query API provides search and visualization via Elasticsearch and Jaeger UI
Supports filtering, span-level detail, and dependency graphs

Practice

(1/5)
1. During a flash sale, the system must handle sudden bursts of 10x traffic within 60 seconds, ensuring order preservation of purchase requests per user session. Which component best handles this requirement?
easy
A. Use a relational database with row-level locking to serialize requests
B. Use Redis as a cache to store purchase requests temporarily
C. Use Kafka with partitioning keyed by user session ID to preserve order
D. Use a simple HTTP load balancer to distribute requests evenly

Solution

  1. Step 1: Identify ordering requirement per user session

    Kafka partitions keyed by user session ID guarantee ordering within that partition, ensuring requests are processed sequentially per user.

  2. Step 2: Evaluate alternatives

    Relational DB row-level locking (A) causes high contention and poor scalability under 10x burst. Redis (C) lacks strong ordering guarantees and durability. Load balancer (D) only distributes load but does not preserve order.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Kafka partition key -> per-session ordering guaranteed [check]
Hint: Kafka partitions preserve order per key [check]
Common Mistakes:
2. In the faceted search request flow, at Step 4 the Query Coordinator merges partial results from multiple shard nodes. What happens if one shard node responds with a partial result missing some facet counts due to a timeout?
easy
A. The Query Coordinator excludes the shard's data entirely from the final result
B. The Query Coordinator merges available partial results and marks missing facets as zero counts
C. The Query Coordinator returns an error to the user immediately
D. The Query Coordinator retries the shard query synchronously before merging

Solution

  1. Step 1: Understand Query Coordinator's merge logic

    The coordinator merges partial results from all shards; if a shard times out, its data is excluded to avoid blocking the entire query.

  2. Step 2: Eliminate other options

    Returning an error breaks availability; marking missing facets as zero is inaccurate; synchronous retries would increase latency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Timeout shard data excluded -> partial but available results [check]
Hint: Timeout shards excluded to keep query latency low [check]
Common Mistakes:
3. A web service receives highly variable traffic with some clients opening many simultaneous connections, while others open few. Which load balancing algorithm best distributes load evenly across servers?
easy
A. Round Robin, because it cycles through servers regardless of connection count
B. Weighted Round Robin, because it assigns requests based on server weights only
C. IP Hash, because it ensures requests from the same client always go to the same server
D. Least Connections, because it directs new requests to the server with the fewest active connections

Solution

  1. Step 1: Understand traffic variability

    Clients open varying numbers of connections, so servers can become unevenly loaded if requests are assigned blindly.

  2. Step 2: Evaluate load balancing algorithms

    Round Robin ignores current load, IP Hash pins clients to servers (may cause imbalance), Weighted Round Robin uses static weights ignoring dynamic load. Least Connections dynamically balances based on active connections.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Least Connections adapts to real-time load, balancing uneven client connections [check]
Hint: Least Connections adapts to active load, not just request count [check]
Common Mistakes:
4. You must design a distributed cache for a social media feed that tolerates network partitions and prioritizes availability over strict ordering. Which consistency model and architecture combination best fits this scenario?
medium
A. Eventual consistency with asynchronous replication and conflict resolution
B. Strong consistency with synchronous replication to all nodes
C. Causal consistency with vector clocks and dependency tracking
D. Read-your-writes consistency with client session affinity

Solution

  1. Step 1: Analyze availability and partition tolerance needs

    Eventual consistency with asynchronous replication maximizes availability during partitions, tolerating temporary inconsistencies.

  2. Step 2: Compare with other models

    Strong consistency blocks during partitions; causal consistency adds overhead and complexity not needed here; read-your-writes focuses on client guarantees, not global availability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Eventual consistency favors availability and partition tolerance [check]
Hint: Eventual consistency maximizes availability during partitions [check]
Common Mistakes:
5. At 10x scale, which component in a Redis system using Bloom filters to speed up key existence checks is most likely to fail first, and why?
hard
A. The disk subsystem fails due to increased Bloom filter false positives causing more lookups
B. The network bandwidth saturates due to Bloom filter synchronization traffic
C. The Bloom filter memory usage grows beyond available RAM, causing eviction and false positive spikes
D. The client-side cache becomes inconsistent due to Bloom filter update delays

Solution

  1. Step 1: Identify Bloom filter memory scaling

    Bloom filters require memory proportional to keyspace size; at 10x scale, memory usage grows and may exceed RAM limits.

  2. Step 2: Trace failure impact

    When memory is insufficient, Bloom filters evict bits or reduce size, increasing false positive rates, which causes more disk lookups and degrades performance.

  3. Step 3: Evaluate other components

    Network bandwidth and client cache inconsistencies are less likely first failures; disk subsystem is affected secondarily.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Bloom filter memory limits cause false positive spikes at scale [check]
Hint: Bloom filter memory hits RAM limit first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard