Bird
Raised Fist0

Performance - P50 P95 P99 Latency and Why Averages Lie

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Performance - P50 P95 P99 Latency and Why Averages Lie
45 min · senior
senior

You are building a performance monitoring system for a startup with 50 million daily active users (DAU). The system must ingest latency data from millions of requests per second, compute accurate P50, P95, and P99 latency percentiles in near real-time, and provide APIs for querying these metrics. The business requires P99 latency reporting within 200ms of data ingestion to enable fast alerting and troubleshooting. The system must also handle encrypted telemetry data and support offline data delivery from edge clients with intermittent connectivity.

Constraints
  • Must process and aggregate latency data at ~1M events per second with minimal delay
  • P99 latency metric must be available within 200ms of event ingestion
  • All telemetry data must be encrypted in transit and at rest
  • Support offline data delivery from edge devices with eventual consistency
  • APIs must support querying latency percentiles by service, endpoint, and time window
📋
Guided Steps
1/5
Step 1
Data Ingestion and Encryption Handling
How will you ingest encrypted latency telemetry data at 1M events/sec while ensuring minimal latency and data integrity?
💡 Hint
Consider a high-throughput, partitioned messaging system that supports encryption and ordering guarantees.
Key Decision
Use Apache Kafka with TLS encryption enabled for transport and Kafka's built-in support for partitioning by service-endpoint key to ensure ordered, scalable ingestion.
Interviewer Expectation
Mid: Names Kafka with TLS. Senior: Explains partition key choice to collocate related events for aggregation and why TLS is mandatory for compliance.
⚠️Common Wrong Answer
Using HTTP REST endpoints without batching or encryption leads to high overhead and insecure data transfer, collapsing under 1M events/sec.
Step 2
Real-time Percentile Computation
How will you compute P50, P95, and P99 latency percentiles in near real-time with 200ms freshness at this scale?
💡 Hint
Think about approximate algorithms and streaming aggregation frameworks that trade off accuracy for speed and memory efficiency.
Key Decision
Use Apache Flink with t-digest sketches for approximate percentile computation, enabling sub-second windowed aggregations with bounded memory and fast updates.
Interviewer Expectation
Mid: Mentions Flink or Spark Streaming. Senior: Details t-digest benefits over histogram buckets for accurate P99 with low memory and explains windowing strategy for 200ms freshness.
⚠️Common Wrong Answer
Batch processing with Hadoop or Spark jobs causes high latency and stale metrics, violating the 200ms freshness SLA.
Step 3
Storage and Querying of Aggregated Metrics
Where and how will you store aggregated percentile metrics to support fast queries by service, endpoint, and time window?
💡 Hint
Consider a time-series database optimized for high write throughput and low-latency range queries.
Key Decision
Store aggregated percentile sketches in TimescaleDB, leveraging its hypertables partitioned by service and time for efficient range scans and SQL querying.
Interviewer Expectation
Mid: Suggests a time-series DB like InfluxDB or TimescaleDB. Senior: Explains hypertable partitioning by service and time, and why TimescaleDB's compression and indexing optimize P99 query latency.
⚠️Common Wrong Answer
Using a generic relational DB without time-series optimizations leads to slow queries and poor scalability at 50M DAU scale.
Step 4
Offline Data Delivery and Eventual Consistency
How will you handle offline telemetry data delivery from edge clients that may be disconnected for extended periods?
💡 Hint
Think about local buffering, secure storage, and eventual synchronization with the central system.
Key Decision
Implement local encrypted storage on edge devices with a background sync service that batches and sends data to Kafka once connectivity is restored, ensuring idempotent writes and deduplication.
Interviewer Expectation
Mid: Mentions local buffering and retries. Senior: Details encryption at rest on device, idempotent Kafka producers, and deduplication strategies to avoid skewed latency metrics.
⚠️Common Wrong Answer
Dropping offline data or naive retries cause data loss or duplicate events, corrupting percentile calculations.
Step 5
API Design for Latency Metrics and Alerting
Design three core API endpoints to ingest latency data, query percentile metrics, and acknowledge alerts.
💡 Hint
Consider REST or gRPC endpoints with authentication and encryption, and how to paginate or filter queries efficiently.
Key Decision
Expose: (1) POST /latency/events for encrypted batch ingestion, (2) GET /latency/metrics with filters for service, endpoint, and time window returning P50/P95/P99, (3) POST /alerts/acknowledge to silence alerts, all secured with OAuth2 and TLS.
Interviewer Expectation
Mid: Defines basic endpoints with authentication. Senior: Explains batch ingestion payload design, query parameter indexing for low latency, and alert acknowledgement idempotency.
⚠️Common Wrong Answer
Using unencrypted HTTP endpoints or lacking filtering leads to security risks and poor query performance.
🔌
API Design Challenge
Sketch 3 core API endpoints for this latency monitoring system that support ingestion, querying, and alert acknowledgement.
Hints
  • How does the client send encrypted latency events to the system?
  • How can users query P50, P95, and P99 latency metrics efficiently?
  • How does the system allow users to acknowledge or silence alerts based on latency thresholds?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of Kafka with TLS for secure, scalable ingestion at 1M events/sec
  • Streaming approximate percentile computation with Apache Flink and t-digest
  • Time-series optimized storage (TimescaleDB) for fast percentile queries
  • Handling offline data with encrypted local storage and idempotent sync
  • API design covering ingestion, querying, and alert acknowledgement with security
Good to Have - 30%
  • Partitioning Kafka topics by service-endpoint key for ordered processing
  • Windowing strategy to achieve 200ms P99 latency freshness
  • Deduplication strategies to prevent skewed metrics from retries or duplicates
Bonus - 10%
  • Explanation of why average latency is misleading compared to percentiles
  • Discussion of trade-offs between accuracy and latency in percentile computation
  • Consideration of alerting thresholds and integration with monitoring tools
Anti-patterns to Avoid
Using batch processing frameworks like Hadoop for percentile computation causes stale metrics violating the 200ms freshness SLA, leading to delayed alerts and poor troubleshooting.
Using unencrypted HTTP endpoints exposes sensitive telemetry data, risking compliance violations and data breaches.
Dropping offline data or naive retries cause data loss or duplicate events, corrupting percentile calculations and skewing metrics.
Using a generic relational database without time-series optimizations leads to slow queries and poor scalability at 50M DAU scale.
HTTP polling at 50M DAU: 50M reconnects/min just to check for messages - collapses under load.
✅
Model Answer
▼ Reveal

This system ingests encrypted latency telemetry at 1M events/sec using Apache Kafka with TLS and partitioning by service-endpoint keys to ensure ordered, scalable ingestion. Apache Flink processes streaming data with t-digest sketches to compute approximate P50, P95, and P99 latency percentiles within 200ms freshness. Aggregated metrics are stored in TimescaleDB hypertables partitioned by service and time for efficient querying. Edge clients buffer encrypted data locally and sync idempotently to Kafka when online, ensuring eventual consistency. APIs expose secure endpoints for batch ingestion, filtered querying, and alert acknowledgement, all protected by OAuth2 and TLS. The design addresses accuracy, latency, security, and offline scenarios comprehensively.

Key Decisions
  • Use Kafka with TLS for encrypted, partitioned ingestion to handle 1M events/sec with ordering guarantees.
  • Employ Apache Flink with t-digest sketches for approximate percentile computation to meet 200ms freshness SLA with bounded memory.
  • Store aggregated metrics in TimescaleDB hypertables partitioned by service and time for efficient range queries and compression.
  • Implement encrypted local storage on edge devices with idempotent sync to Kafka to handle offline data and avoid duplicates.
  • Design APIs with batch ingestion, filtered querying by service/endpoint/time, and alert acknowledgement endpoints secured by OAuth2 and TLS.
  • Apache Kafka with TLS for encrypted, partitioned high-throughput ingestion
  • Apache Flink with t-digest sketches for low-latency approximate percentile computation
  • TimescaleDB hypertables partitioned by service and time for efficient storage and querying
  • Edge device local encrypted storage with idempotent Kafka producers for offline delivery
  • REST/gRPC APIs secured with OAuth2 and TLS for ingestion, querying, and alert acknowledgement
  • Use of partition keys (service-endpoint) in Kafka to ensure ordered event processing
  • Windowed streaming aggregation with 200ms sliding windows to meet freshness SLA
  • Deduplication and idempotency mechanisms to maintain data integrity in offline sync
  • Explanation of why average latency is misleading compared to percentiles: averages can be skewed by outliers and do not reflect tail latency critical for user experience.
  • Discussion of trade-offs between accuracy and latency in percentile computation: approximate algorithms like t-digest balance memory and speed with acceptable error margins.
  • Consideration of alerting thresholds and integration with monitoring tools to enable proactive incident response.

Practice

(1/5)
1. A company wants to conduct a post-mortem after a major outage without blaming individuals, focusing on systemic issues and actionable improvements. Which component best supports this blameless culture approach?
easy
A. A detailed blame assignment report highlighting individual errors
B. An automated alert system that escalates to management immediately
C. A structured timeline tool capturing events and decisions neutrally
D. A punitive review board to enforce accountability

Solution

  1. Step 1: Identify blameless culture requirements

    The approach requires neutral, factual event capture without assigning blame to individuals.

  2. Step 2: Evaluate components for neutrality and actionability

    A structured timeline tool capturing events and decisions neutrally provides a structured timeline tool that records events and decisions neutrally, enabling root cause analysis without blame.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Neutral event timelines enable blameless post-mortems [check]
Hint: Blameless means neutral facts, not blame assignment [check]
Common Mistakes:
2. Without the virtual nodes abstraction in consistent hashing, what exact failure occurs when a shard is removed?
medium
A. The hash ring becomes unbalanced, causing some shards to receive no traffic
B. Massive remapping of keys to remaining shards causes cache invalidation and latency spikes
C. All keys mapped to the removed shard are lost permanently with no fallback
D. Requests to the removed shard are automatically redirected to a backup shard

Solution

  1. Step 1: Understand role of virtual nodes

    Virtual nodes distribute keys evenly across physical shards; without them, shard removal causes uneven key distribution.

  2. Step 2: Trace failure impact

    Without virtual nodes, removing a shard leaves gaps in the hash ring, causing some shards to get no traffic (The hash ring becomes unbalanced, causing some shards to receive no traffic). Key loss (All keys mapped to the removed shard are lost permanently with no fallback) doesn't happen as keys remap; massive remapping (Massive remapping of keys to remaining shards causes cache invalidation and latency spikes) is reduced by virtual nodes but not eliminated; automatic redirection (Requests to the removed shard are automatically redirected to a backup shard) is not inherent.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Without virtual nodes, shard removal causes unbalanced hash ring [check]
Hint: No virtual nodes -> unbalanced ring, some shards get no traffic [OK]
Common Mistakes:
3. Without the asynchronous replication component in a PACELC system prioritizing latency, what exactly breaks during a network partition?
medium
A. Reads return stale data indefinitely without updates
B. Writes block until all replicas acknowledge, causing high latency
C. Local writes succeed but are lost if the node crashes before replication
D. The system automatically switches to a single primary mode

Solution

  1. Step 1: Identify role of asynchronous replication

    It propagates updates to other replicas to keep data fresh despite partitions.

  2. Step 2: Trace failure without async replication

    Without it, replicas do not get updates, so reads return stale data indefinitely.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    No async replication -> stale reads persist during partition [check]
Hint: No async replication -> stale reads never update during partition [check]
Common Mistakes:
4. At 10x scale, which component in a Redis system using Bloom filters to speed up key existence checks is most likely to fail first, and why?
hard
A. The disk subsystem fails due to increased Bloom filter false positives causing more lookups
B. The network bandwidth saturates due to Bloom filter synchronization traffic
C. The Bloom filter memory usage grows beyond available RAM, causing eviction and false positive spikes
D. The client-side cache becomes inconsistent due to Bloom filter update delays

Solution

  1. Step 1: Identify Bloom filter memory scaling

    Bloom filters require memory proportional to keyspace size; at 10x scale, memory usage grows and may exceed RAM limits.

  2. Step 2: Trace failure impact

    When memory is insufficient, Bloom filters evict bits or reduce size, increasing false positive rates, which causes more disk lookups and degrades performance.

  3. Step 3: Evaluate other components

    Network bandwidth and client cache inconsistencies are less likely first failures; disk subsystem is affected secondarily.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Bloom filter memory limits cause false positive spikes at scale [check]
Hint: Bloom filter memory hits RAM limit first at scale [check]
Common Mistakes:
5. At 10x scale, which component of the Snowflake ID generation system is most likely to become the bottleneck first, and why?
hard
A. The timestamp source, because clock synchronization across machines becomes impossible at scale.
B. The network bandwidth, because ID generation requires frequent coordination between machines.
C. The sequence number, because generating more than its max IDs per millisecond causes blocking delays.
D. The machine ID allocation, because limited machine ID bits cause ID collisions as machines increase.

Solution

  1. Step 1: Analyze timestamp and network constraints

    Timestamp synchronization (A) is challenging but tolerable with NTP; network coordination (D) is minimal as IDs are generated locally.

  2. Step 2: Evaluate sequence number limits

    Sequence number (B) limits per millisecond can cause blocking but can be mitigated by load balancing or sharding.

  3. Step 3: Identify machine ID allocation bottleneck

    Machine ID bits are fixed (e.g., 10 bits), limiting the number of unique machines. At 10x scale, running out of unique machine IDs causes collisions or requires complex reallocation.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Fixed machine ID bits limit scale -> collisions or reconfiguration needed [check]
Hint: Fixed machine ID bits limit scale before sequence or timestamp issues [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard