Bird
Raised Fist0

Observability - Metrics, Logging, Tracing - The Three Pillars

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Observability - Metrics, Logging, Tracing - The Three Pillars
45 min · senior
senior

You are designing an observability platform for a 50 million daily active user (DAU) startup. The system must collect, process, and query metrics, logs, and distributed traces from microservices deployed globally. The platform must support P99 query latency under 200ms for common queries, ensure end-to-end encryption of data in transit and at rest, and provide offline delivery guarantees for logs and traces from edge services with intermittent connectivity.

Constraints
  • End-to-end encryption for all observability data (metrics, logs, traces) in transit and at rest
  • P99 query latency under 200ms for common queries (e.g., last 1 hour metrics, trace lookup by trace ID)
  • Offline delivery support: edge services may be disconnected and must buffer data locally and sync later
  • High ingestion throughput: expect 10 million metrics points/sec, 1 million log lines/sec, 100K traces/sec
  • Retention: metrics for 30 days, logs for 7 days, traces for 15 days
📋
Guided Steps
1/5
Step 1
Data Ingestion Architecture
How would you design the ingestion pipeline to handle high throughput of metrics, logs, and traces with encryption and offline delivery?
💡 Hint
Consider scalable, partitioned messaging systems that support encryption and replay for offline sync.
Key Decision
Use Apache Kafka with TLS encryption for ingestion, partitioned by service and data type to scale writes and enable offline replay from local buffers.
Interviewer Expectation
Mid: Names Kafka with TLS. Senior: Explains partitioning by service and data type to isolate workloads and enable offline replay from edge buffers.
⚠️Common Wrong Answer
Using HTTP REST endpoints without buffering leads to data loss during edge disconnections and poor ingestion scalability.
Step 2
Storage Backend Selection
Which storage technologies would you choose for metrics, logs, and traces to meet retention and query latency requirements?
💡 Hint
Think about time-series databases for metrics, log stores optimized for append and search, and trace stores supporting span queries.
Key Decision
Use Prometheus remote write + Thanos for metrics (time-series, downsampling), Elasticsearch for logs (full-text search), and Jaeger with Cassandra backend for traces (LSM-tree writes, range queries).
Interviewer Expectation
Mid: Names Prometheus, Elasticsearch, Jaeger. Senior: Explains why Thanos enables long retention and global queries, why Elasticsearch suits logs with inverted indexes, and why Cassandra's partition key design supports trace span queries.
⚠️Common Wrong Answer
Using a single relational database for all data types causes query latency spikes and poor write scalability.
Step 3
Query API Design
How would you design the query APIs to support low-latency retrieval of metrics, logs, and traces with encryption and multi-tenant isolation?
💡 Hint
Consider gRPC or REST with authentication, caching layers, and query federation across storage backends.
Key Decision
Expose gRPC APIs with mTLS for authentication, implement query federation service that routes to Thanos, Elasticsearch, and Jaeger backends, and use Redis caching for hot queries to meet P99 < 200ms.
Interviewer Expectation
Mid: Describes gRPC with TLS. Senior: Details query federation logic, cache invalidation strategies, and tenant-aware access control.
⚠️Common Wrong Answer
Direct client queries to storage backends without caching or federation cause high latency and security risks.
Step 4
Offline Delivery and Data Sync
How would you ensure observability data from edge services with intermittent connectivity is reliably delivered and encrypted?
💡 Hint
Think about local buffering, encrypted storage, and replay mechanisms integrated with the ingestion pipeline.
Key Decision
Implement local encrypted RocksDB buffers on edge agents with Kafka Connect replay to central Kafka cluster over TLS, ensuring exactly-once delivery semantics.
Interviewer Expectation
Mid: Mentions local buffering. Senior: Explains encrypted local storage, replay offsets, and integration with Kafka for exactly-once ingestion.
⚠️Common Wrong Answer
No local buffering or unencrypted storage leads to data loss and security vulnerabilities during disconnections.
Step 5
Data Encryption and Security
How do you enforce end-to-end encryption and secure access across ingestion, storage, and query layers?
💡 Hint
Consider TLS, encryption at rest, key management, and role-based access control.
Key Decision
Use TLS 1.3 for all network communication, AES-256 encryption at rest via cloud KMS integration for storage backends, and OAuth 2.0 with RBAC for API access control.
Interviewer Expectation
Mid: Names TLS and encryption at rest. Senior: Details key rotation, zero-trust principles, and fine-grained RBAC policies per tenant and data type.
⚠️Common Wrong Answer
Relying only on network encryption without at-rest encryption or weak access controls risks data breaches.
🔌
API Design Challenge
Sketch 3 core API endpoints for the observability platform: one for ingesting data, one for querying data, and one for acknowledging data receipt or processing.
Hints
  • How does the client securely send metrics, logs, or traces with minimal latency?
  • How can clients query recent and historical observability data efficiently?
  • How does the system confirm successful ingestion or processing to clients or agents?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with TLS for ingestion to handle 10M metrics/sec and offline replay
  • Choose specialized storage: Prometheus+Thanos for metrics, Elasticsearch for logs, Jaeger+Cassandra for traces
  • Design query APIs with gRPC, mTLS, and caching to meet P99 < 200ms latency
  • Implement encrypted local buffering on edge agents for offline delivery
  • Enforce end-to-end encryption and RBAC for data security
Good to Have - 30%
  • Explain partitioning strategy in Kafka by service and data type
  • Describe query federation and cache invalidation mechanisms
  • Detail key rotation and zero-trust security principles
  • Discuss retention policies and data lifecycle management
Bonus - 10%
  • Propose autoscaling strategies for ingestion and query components
  • Suggest anomaly detection integration in metrics pipeline
  • Outline monitoring and alerting for observability platform health
Anti-patterns to Avoid
Using HTTP REST ingestion without buffering causes data loss during edge disconnections because the system cannot retry or replay data, leading to gaps in observability.
Storing all observability data in a single relational database leads to poor write scalability and high query latency due to mixed workload contention and lack of specialized indexing.
Allowing clients to query storage backends directly without caching or federation causes security risks and latency spikes as backend systems are exposed and overwhelmed by unoptimized queries.
Not encrypting data at rest or in transit exposes sensitive observability data to interception or unauthorized access, risking data breaches.
✅
Model Answer
▼ Reveal

The observability platform ingests high-volume metrics, logs, and traces securely and reliably from globally distributed microservices, supports low-latency queries under 200ms P99, and ensures offline delivery from edge services with intermittent connectivity. It enforces end-to-end encryption and fine-grained access control.

Key Decisions
  • Use Apache Kafka with TLS encryption, partitioned by service and data type, to handle 10M metrics/sec ingestion and enable offline replay from edge buffers.
  • Store metrics in Prometheus remote write with Thanos for long retention and downsampling; logs in Elasticsearch for full-text search; traces in Jaeger with Cassandra backend optimized for span queries.
  • Expose gRPC query APIs secured with mTLS, implement query federation routing to specialized backends, and use Redis caching to achieve P99 query latency under 200ms.
  • Implement encrypted RocksDB local buffers on edge agents to support offline delivery and replay data reliably over Kafka Connect with exactly-once semantics.
  • Enforce end-to-end encryption using TLS 1.3 for all network communication, AES-256 encryption at rest integrated with cloud KMS, and OAuth 2.0 with RBAC for secure multi-tenant access control.
  • Apache Kafka with TLS for scalable, encrypted ingestion and offline replay
  • Prometheus remote write + Thanos for scalable, long-term metrics storage with downsampling
  • Elasticsearch for logs enabling full-text search and fast retrieval
  • Jaeger with Cassandra backend for distributed tracing with efficient span queries
  • gRPC APIs with mTLS and query federation for low-latency, secure multi-tenant queries
  • Redis caching layer to meet P99 query latency under 200ms for hot queries
  • Encrypted RocksDB local buffers on edge agents for offline delivery and replay
  • AES-256 encryption at rest integrated with cloud KMS and OAuth 2.0 RBAC for security

Practice

(1/5)
1. In the request flow, at step 3 the API Gateway forwards the request to the BFF layer. What happens at step 4 inside the BFF before calling downstream microservices?
easy
A. BFF aggregates and transforms data from multiple microservices into a client-specific response
B. BFF caches the entire response from microservices before returning to client
C. BFF directly proxies the request without modification to microservices
D. BFF performs authentication and then sends request to a single microservice

Solution

  1. Step 1: Understand BFF responsibilities at step 4

    At step 4, the BFF aggregates data from multiple microservices and transforms it into a response tailored to the client's needs.

  2. Step 2: Eliminate incorrect options

    BFF caches the entire response from microservices before returning to client is wrong because caching happens after response, not before calling microservices. BFF directly proxies the request without modification to microservices is incorrect as BFF modifies requests/responses rather than proxying blindly. BFF performs authentication and then sends request to a single microservice is incomplete; authentication usually happens earlier at API Gateway, and BFF calls multiple microservices, not just one.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    BFF aggregates and transforms data before responding [check]
Hint: BFF aggregates and transforms data before responding [check]
Common Mistakes:
2. In a system where a Layer 7 load balancer receives an HTTP request at step 3, what happens at step 4 when it applies URL-based routing rules?
easy
A. The load balancer inspects HTTP headers and forwards to the matched backend pool
B. The load balancer forwards the raw TCP packet without modification
C. The load balancer performs DNS resolution to select backend IP
D. The load balancer encrypts the request before forwarding

Solution

  1. Step 1: Understand Layer 7 load balancer behavior at step 4

    At step 4, the Layer 7 load balancer inspects HTTP headers (like URL path) to decide routing.

  2. Step 2: Eliminate options inconsistent with Layer 7 behavior

    Forwarding raw TCP packets (A) is Layer 4 behavior; DNS resolution (C) is client-side; encryption (D) is unrelated here.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Layer 7 LB inspects HTTP headers and routes accordingly [check]
Hint: Layer 7 LB inspects HTTP headers before forwarding [OK]
Common Mistakes:
3. If the MongoDB cluster loses the shard key index on a sharded collection, what exactly breaks in query routing and data consistency?
medium
A. Queries continue normally but writes fail silently causing data loss.
B. The mongos router cannot route queries to correct shards, causing scatter-gather queries and potential stale reads.
C. Shard servers reject all queries until the index is rebuilt, causing total downtime.
D. Data replication stops between shards leading to inconsistent replicas.

Solution

  1. Step 1: Role of shard key index

    Shard key index is critical for routing queries to correct shards and maintaining data partitioning.

  2. Step 2: Impact of losing shard key index

    Without the index, shards cannot serve queries properly and reject operations until index rebuild, causing downtime.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Shard key index loss causes shard query rejection and downtime [check]
Hint: Shard key index loss causes shard query rejection, not silent failure [check]
Common Mistakes:
4. Without the DAU Estimator component, what exactly breaks in the system's capacity planning process?
medium
A. Bandwidth estimation fails because DAU directly affects total data transfer calculations
B. The system cannot estimate peak QPS, causing under-provisioning of servers
C. Storage estimation becomes inaccurate because DAU is a key multiplier for daily data volume
D. User authentication fails because DAU tracks active sessions

Solution

  1. Step 1: Identify DAU Estimator role

    DAU Estimator provides daily active user count critical for bandwidth calculations as bandwidth depends on total active users.

  2. Step 2: Eliminate other options

    The system cannot estimate peak QPS, causing under-provisioning of servers is incorrect because peak QPS depends on instantaneous requests, not just DAU; Storage estimation becomes inaccurate because DAU is a key multiplier for daily data volume is partially true but storage depends more on total objects, not DAU alone; User authentication fails because DAU tracks active sessions is unrelated to authentication.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bandwidth depends directly on DAU for total data transfer [check]
Hint: Bandwidth depends directly on DAU count [check]
Common Mistakes:
5. At 10x traffic scale, which component is most likely to fail first in a layered rate limiting system and why?
hard
A. API Gateway, because in-memory counters consume excessive memory and CPU per connection
B. Database, because synchronous counter updates cause write bottlenecks
C. Service Layer, because distributed caches cannot synchronize counters fast enough
D. Network Load Balancer, because it cannot handle the increased connection churn

Solution

  1. Step 1: Analyze API Gateway resource usage

    API Gateway maintains in-memory counters per user/request, which grow linearly with traffic, exhausting memory and CPU first.

  2. Step 2: Compare other components

    Databases can be sharded and scaled; service caches sync asynchronously; load balancers handle connection churn better than API Gateway resource limits.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    API Gateway memory/CPU bottleneck at 10x scale [check]
Hint: API Gateway memory/CPU hits first at 10x scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard