Bird
Raised Fist0

Observability - Metrics, Logging, Tracing - The Three Pillars

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
▶
Request Trace
Step 1/6
Kafka Cluster

Ingest observability data

Receives metrics, logs, and tracing spans from instrumented services

Protocol:Kafka protocol
BeforeObservability data generated but not yet centralized
AfterData durably stored in Kafka partitions for processing
Metrics Storage (Apache Cassandra)

Consume metrics from Kafka

Metrics consumer reads Kafka topics and writes to Cassandra

Protocol:Kafka consumer + Cassandra write
BeforeMetrics data only in Kafka partitions
AfterMetrics data persisted in Cassandra for query
Logs Storage (Elasticsearch)

Consume logs from Kafka

Logs consumer reads Kafka topics and indexes logs in Elasticsearch

Protocol:Kafka consumer + Elasticsearch indexing
BeforeLogs data only in Kafka partitions
AfterLogs indexed and searchable in Elasticsearch
Tracing Storage (Apache Cassandra)

Consume tracing spans from Kafka

Tracing consumer reads Kafka topics and writes spans to Cassandra

Protocol:Kafka consumer + Cassandra write
BeforeTracing spans only in Kafka partitions
AfterTracing spans persisted in Cassandra for query
Metrics Query API (Prometheus compatible)

Serve metrics query

Receives Prometheus query and fetches data from Cassandra

Protocol:HTTP/REST
BeforeMetrics data stored in Cassandra, no query issued
AfterMetrics query results returned to client
Logs & Trace Query API (Elasticsearch + Jaeger UI)

Serve logs and tracing query

Receives query, fetches logs from Elasticsearch and traces from Cassandra

Protocol:HTTP/REST
BeforeLogs and traces stored in Elasticsearch and Cassandra respectively
AfterLogs and trace results returned with UI rendering
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│     Metrics Query API          │
│  (Prometheus compatible)       │
└───────────────┬───────────────┘
                │ HTTP/REST Query
                ▼
┌───────────────────────────────┐
│ Metrics Storage (Apache        │
│       Cassandra)               │
└───────────────┬───────────────┘
                │ Kafka Ingest
                ▲
┌───────────────┴───────────────┐
│        Kafka Cluster           │
│ (Ingests Metrics, Logs, Trace)│
└───────────────┬───────────────┘
                │ Kafka Ingest
                ▼
┌───────────────────────────────┐
│ Logs Storage (Elasticsearch)  │
└───────────────┬───────────────┘
                │ Kafka Ingest
                ▼
┌───────────────────────────────┐
│ Tracing Storage (Apache       │
│       Cassandra)              │
└───────────────┬───────────────┘
                │ HTTP/REST Query
                ▼
┌───────────────────────────────┐
│ Logs & Trace Query API         │
│ (Elasticsearch + Jaeger UI)   │
└───────────────────────────────┘

This observability architecture centralizes metrics, logs, and tracing data ingestion via a Kafka cluster. Kafka acts as a durable, scalable queue that decouples data producers from storage consumers. Metrics and tracing spans are stored in Apache Cassandra for high write throughput and time-series queries, while logs are indexed in Elasticsearch for full-text search capabilities. Separate query APIs serve metrics (Prometheus compatible) and logs/traces (Elasticsearch + Jaeger UI), enabling specialized access patterns. This separation ensures scalability and optimized storage/query for each data type, while Kafka guarantees reliable ingestion and ordering.

Primary Focus

✓

Practice

(1/5)
1. A system must track millions of concurrent video playbacks and update user progress in near real-time without blocking the video stream. Which component is best suited to handle progress updates asynchronously?
easy
A. Use a message queue to buffer progress update events for asynchronous processing
B. Use a relational database with synchronous writes on every video event
C. Use a CDN edge cache to store progress state temporarily
D. Use client-side local storage exclusively to track progress

Solution

  1. Step 1: Identify asynchronous progress update requirement

    Relational DB synchronous writes (Use a relational database with synchronous writes on every video event) block video playback and don't scale well for millions of concurrent updates.
  2. Step 2: Evaluate buffering and decoupling

    Message queue (Use a message queue to buffer progress update events for asynchronous processing) decouples video playback from progress persistence, enabling scalable asynchronous processing.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Message queue buffers progress updates for async processing [check]
Hint: Buffer updates asynchronously with message queue [OK]
Common Mistakes:
2. For a social feed system with millions of users, which fan-out strategy is better when most users have a small number of followers but a few have millions, and why?
medium
A. Hybrid approach: fan-out on write for low-follower users, fan-out on read for high-follower users
B. Fan-out on read to reduce write amplification for all users
C. Fan-out on write for all users to guarantee low read latency
D. Fan-out on read for all users to minimize storage costs

Solution

  1. Step 1: Analyze follower distribution impact on fan-out

    Fan-out on write for users with millions of followers causes huge write amplification and storage overhead. Fan-out on read for low-follower users increases read latency unnecessarily.
  2. Step 2: Evaluate hybrid approach benefits

    Hybrid approach balances write amplification and read latency by applying fan-out on write only where follower count is manageable, and fan-out on read for heavy hitters.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Hybrid fan-out balances write cost and read latency [check]
Hint: Hybrid fan-out optimizes write/read trade-offs by follower count [check]
Common Mistakes:
3. For encrypting sensitive data at rest in a distributed database, which approach is best when the system requires frequent data reads with minimal latency and strong security guarantees?
medium
A. Use client-side encryption where clients encrypt data before sending to the database
B. Use Transparent Data Encryption (TDE) at the database layer with hardware security modules (HSMs)
C. Use application-layer encryption with keys stored in environment variables on app servers
D. Use file system-level encryption on database storage volumes

Solution

  1. Step 1: Evaluate encryption approaches for read latency and security

    TDE with HSMs provides strong security with minimal impact on read latency as encryption/decryption is transparent and hardware-accelerated.
  2. Step 2: Eliminate less suitable options

    Client-side encryption increases latency and complicates querying; app-layer encryption with env vars risks key exposure and adds latency; file system encryption protects at disk level but not granular data access.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    TDE + HSM balances security and low-latency reads [check]
Hint: TDE with HSMs -> strong security + low latency reads [check]
Common Mistakes:
4. At 10x scale, which component in a chat system fails first and why?
hard
A. The database fails first due to write amplification and disk I/O bottlenecks
B. The load balancer fails first because it cannot handle the increased connection count
C. Chat servers fail first because WebSocket connections are stateful and cannot be resharded easily
D. The message queue fails first due to partition leader overload

Solution

  1. Step 1: Analyze stateful components at scale

    Chat servers maintain WebSocket connections statefully; scaling requires connection migration or sticky sessions.
  2. Step 2: Identify bottleneck

    At 10x scale, chat servers hit connection limits and cannot shard existing connections, causing failure before DB or load balancer.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Stateful WebSocket servers -> scaling bottleneck due to connection affinity [check]
Hint: Stateful WebSocket servers block scaling before DB or load balancer [check]
Common Mistakes:
5. At 10x scale, which component is most likely to become the bottleneck in a WebSocket-based chat system, and why?
hard
A. The chat servers, because WebSocket connections are stateful and bound to server memory, making horizontal scaling and connection migration difficult.
B. The database, because it cannot handle the increased write volume from message persistence.
C. The load balancer, because it cannot efficiently distribute WebSocket connections due to their stateful nature.
D. The client browsers, because they cannot maintain more than a few WebSocket connections simultaneously.

Solution

  1. Step 1: Identify stateful components in WebSocket architecture

    WebSocket connections are stateful and maintained in server memory, binding clients to specific chat servers.
  2. Step 2: Analyze scaling challenges

    At 10x scale, chat servers face difficulty resharding connections or migrating them, limiting horizontal scaling. Databases and load balancers can be scaled more easily. Client browser limits are less impactful at system scale.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    WebSocket stateful connections -> chat servers bottleneck at scale [check]
Hint: Stateful WebSocket connections bind clients to servers -> chat servers bottleneck [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard