Bird
Raised Fist0

Availability Numbers - What 99.9 vs 99.99 vs 99.999 Percent Means in Practice

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
🎯
Availability Numbers - What 99.9 vs 99.99 vs 99.999 Percent Means in Practice
45 min · senior
senior

You are designing the availability guarantees for a messaging platform with 50 million daily active users (DAU). The system must support encrypted message delivery with offline support and maintain a P99 latency under 200ms for message delivery. The business requires clear understanding and trade-offs between 99.9%, 99.99%, and 99.999% availability SLAs, including how these affect architecture, cost, and user experience.

  • End-to-end encryption required for all messages
  • Offline message delivery with guaranteed eventual consistency
  • P99 message delivery latency under 200ms
  • Support 50M DAU with peak concurrency of 5M users
  • Clear SLA differentiation between 99.9%, 99.99%, and 99.999% availability
📋
Guided Steps
1/5
Step 1
Define availability SLA impact on architecture
How do different availability SLAs (99.9%, 99.99%, 99.999%) translate into architectural requirements and failure tolerance?
💡 Hint
Consider how downtime minutes per year differ and what redundancy or failover mechanisms are needed to meet each SLA.
Use multi-region active-active deployment with automated failover using Kubernetes and Consul for service discovery to meet 99.99% and above availability; 99.9% can tolerate single-region active-passive with manual failover.
Mid: Names multi-region deployment. Senior: Explains exact downtime minutes difference (e.g., 99.9% = ~8.76 hours/year vs 99.999% = ~5 minutes/year) and how that drives complexity in failover automation and data replication.
⚠️Common Wrong Answer
Assuming 99.9% and 99.999% availability require the same architecture, leading to under-provisioned failover and unexpected downtime.
Step 2
Design data replication and consistency for offline delivery
How do you design data replication and consistency to support offline message delivery while maintaining availability SLAs?
💡 Hint
Think about eventual consistency models, conflict resolution, and replication technologies that support multi-region writes.
Use Apache Cassandra with multi-region replication and lightweight transactions for conflict resolution to enable offline writes and eventual consistency with high availability.
Mid: Names Cassandra for replication. Senior: Explains why Cassandra’s tunable consistency and LWTs fit offline delivery and how it compares to alternatives like MongoDB or traditional RDBMS which have higher latency or weaker multi-region support.
⚠️Common Wrong Answer
Using a single-region relational database which blocks writes during failover, violating availability and offline delivery requirements.
Step 3
Choose messaging and delivery guarantees to meet latency and availability
What messaging infrastructure and delivery guarantees ensure P99 < 200ms latency and meet the availability SLAs?
💡 Hint
Consider partitioning, ordering guarantees, and exactly-once or at-least-once delivery semantics.
Use Apache Kafka with conversation_id as partition key to ensure ordered, low-latency message delivery and consumer groups for scalable processing; Kafka’s replication factor and ISR tuning enable 99.99%+ availability.
Mid: Names Kafka for messaging. Senior: Explains partition key choice for ordering, ISR quorum for availability, and why RabbitMQ or AWS SQS are less suitable at this scale and latency.
⚠️Common Wrong Answer
Using HTTP polling for message delivery which causes high latency and load spikes, breaking P99 latency and availability.
Step 4
Design API endpoints reflecting availability and offline constraints
Sketch 3 core API endpoints that support encrypted message send, offline sync, and delivery acknowledgement under the availability SLAs.
💡 Hint
Focus on request/response patterns that minimize latency and support offline retries and idempotency.
POST /messages/send with client-side encryption and immediate ACK; GET /messages/sync with cursor-based pagination for offline recovery; POST /messages/ack for recipient delivery confirmation enabling exactly-once semantics.
Mid: Names endpoints and basic functionality. Senior: Explains how idempotency keys, cursor-based sync, and encryption interplay to meet availability and offline delivery guarantees.
⚠️Common Wrong Answer
Designing synchronous blocking endpoints without offline sync support, causing poor UX and SLA violations during network partitions.
Step 5
Plan monitoring and alerting to maintain SLAs
What monitoring and alerting strategies ensure you detect and respond to availability degradations quickly?
💡 Hint
Consider real-time metrics, SLO error budgets, and automated remediation triggers.
Use Prometheus and Grafana for real-time latency and error rate monitoring, with PagerDuty integration for alerting on SLO breaches; implement automated failover triggers in Kubernetes based on health checks.
Mid: Mentions monitoring tools. Senior: Details SLO-based alerting, error budget policies, and automated remediation to prevent SLA violations proactively.
⚠️Common Wrong Answer
Relying solely on manual monitoring without automated alerts or remediation, leading to prolonged outages.
🔌
API Design Challenge
Sketch 3 core API endpoints that reflect the availability guarantees and offline delivery requirements.
  • How does the client send an encrypted message and receive acknowledgement?
  • How does the client sync missed messages after offline periods?
  • How does the recipient acknowledge message receipt to ensure end-to-end delivery?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Clear explanation of downtime minutes difference between 99.9%, 99.99%, and 99.999% availability and corresponding architectural impact
  • Use of multi-region deployment with automated failover for high availability
  • Use of Apache Cassandra for multi-region replication supporting offline delivery
  • Use of Apache Kafka with partition key for ordered, scalable message delivery
  • API design supporting encrypted send, offline sync, and delivery acknowledgement
Good to Have - 30%
  • Detailed monitoring and alerting strategy based on SLOs and error budgets
  • Discussion of idempotency and exactly-once delivery semantics
  • Consideration of cost and complexity trade-offs between availability tiers
Bonus - 10%
  • Deep failure mode analysis tracing exact failure sequences
✅
Model Answer
▼ Reveal

To meet different availability SLAs (99.9%, 99.99%, 99.999%), the system architecture must evolve from single-region active-passive with manual failover to multi-region active-active with automated failover. Apache Cassandra enables multi-region replication with lightweight transactions to support offline writes and eventual consistency. Apache Kafka with conversation_id partition key ensures ordered, low-latency message delivery meeting P99 < 200ms latency and high availability. API endpoints are designed for encrypted message sending, offline sync with cursor-based pagination, and delivery acknowledgements with idempotency to guarantee exactly-once semantics. Monitoring with Prometheus, Grafana, and PagerDuty ensures SLA adherence with automated remediation.

  • Use Kubernetes multi-region active-active deployment with Consul for automated failover to meet 99.99%+ availability
  • Use Apache Cassandra with multi-region replication and lightweight transactions for offline message writes and eventual consistency
  • Use Apache Kafka with conversation_id as partition key to ensure ordered, low-latency message delivery and high availability
  • Design API endpoints with client-side encryption, immediate ACKs, cursor-based sync for offline recovery, and idempotent delivery acknowledgements
  • Implement monitoring with Prometheus and Grafana for real-time metrics, PagerDuty for alerting, and Kubernetes health checks for automated remediation
  • Multi-region active-active Kubernetes clusters with Consul for service discovery to meet 99.99%+ availability
  • Apache Cassandra with multi-region replication and lightweight transactions for offline delivery and eventual consistency
  • Apache Kafka with conversation_id partition key for ordered, low-latency message delivery and high availability
  • Client-side end-to-end encryption integrated into API endpoints to meet security constraints
  • Cursor-based sync API endpoint enabling efficient offline message recovery
  • Prometheus and Grafana for real-time monitoring with PagerDuty for alerting on SLA breaches
  • Automated failover and remediation using Kubernetes health checks and custom controllers
  • Idempotent API design with acknowledgement endpoints to ensure exactly-once delivery semantics

Practice

(1/5)
1. For a high-throughput system where write latency must be minimized and occasional stale reads are acceptable, which caching pattern is the best choice?
medium
A. Write-Behind, to buffer writes and asynchronously update the database
B. Cache Aside, to update cache only on misses and write directly to the database
C. Write-Through, to ensure cache and database are always consistent synchronously
D. Read-Through, to load cache on reads and write directly to the database

Solution

  1. Step 1: Analyze write latency and consistency needs

    Write-Behind buffers writes, reducing write latency but allowing stale reads.
  2. Step 2: Eliminate other options

    Write-Through adds synchronous latency; Cache Aside and Read-Through write directly to DB, no buffering.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Write-Behind buffers writes for low latency, tolerates stale reads [check]
Hint: Write-Behind buffers writes, reduces latency, allows stale reads [check]
Common Mistakes:
2. If the distributed priority queue component fails and becomes unavailable, what is the immediate failure mode observed in the task scheduler?
medium
A. Worker nodes block waiting for tasks, causing execution delays and potential timeouts
B. Tasks are silently dropped without retries, causing data loss
C. Tasks continue executing but with incorrect priority ordering
D. The cron scheduler automatically reschedules tasks on all nodes

Solution

  1. Step 1: Understand role of distributed priority queue

    The queue is the source of truth for pending tasks; if unavailable, new tasks cannot be fetched.
  2. Step 2: Analyze failure impact

    Without the queue, tasks submitted may be lost if not persisted elsewhere; workers cannot fetch tasks, so blocking (B) is unlikely if queue is down; cron scheduler (D) does not reschedule tasks automatically.
  3. Step 3: Evaluate silent drops (C)

    Without queue durability, tasks may be lost silently if producers cannot enqueue or retries are not implemented.
  4. Final Answer:

    Option B -> Option B
  5. Quick Check:

    Queue failure causes silent task loss without retries [check]
Hint: Queue failure causes silent task loss without retries [check]
Common Mistakes:
3. For ranking answers in the feed, which approach is better given the need for real-time updates and complex ranking signals including user behavior and spam scores?
medium
A. Use batch processing with daily recomputation of rankings in a data warehouse
B. Use client-side ranking to reduce server load and latency
C. Use a simple heuristic ranking computed at write time with no model updates
D. Use an online feature store with real-time scoring and incremental model updates

Solution

  1. Step 1: Evaluate batch vs real-time ranking

    Batch (A) is too stale for real-time user behavior; heuristic (C) lacks adaptability; client-side (D) cannot access server-side signals securely.
  2. Step 2: Online feature store benefits

    Online feature stores enable real-time scoring with fresh data and incremental model updates, supporting complex signals and spam detection.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Online feature store supports real-time, complex ranking [check]
Hint: Real-time ranking needs online features, not stale batch [check]
Common Mistakes:
4. For real-time message delivery to mobile clients with intermittent connectivity, which architecture best balances latency and resource usage?
medium
A. Use gRPC streaming connections from server to mobile clients
B. Use WebSocket connections with heartbeat and fallback to push notifications
C. Use long polling HTTP requests from clients to server
D. Use MQTT protocol with persistent sessions for mobile clients

Solution

  1. Step 1: Evaluate WebSocket vs gRPC streaming for mobile clients

    gRPC streaming is efficient but poorly supported on many mobile platforms; WebSocket is widely supported with heartbeat to detect disconnects.
  2. Step 2: Consider fallback mechanisms

    WebSocket with push notification fallback handles intermittent connectivity better than long polling or MQTT, which have higher overhead or complexity.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    WebSocket + push fallback balances latency and mobile constraints [check]
Hint: WebSocket + push fallback handles mobile connectivity best [check]
Common Mistakes:
5. If the Kafka topic used for driver location updates is deleted accidentally, what is the immediate failure observed in the system?
medium
A. Driver location updates are lost permanently with no retry, causing stale driver positions
B. The system automatically switches to a backup topic with no data loss
C. Driver apps crash due to inability to send location updates
D. Matching service blocks waiting for location updates, causing request timeouts

Solution

  1. Step 1: Understand Kafka topic role

    Kafka topic buffers and streams driver location updates to downstream services.
  2. Step 2: Trace failure impact

    Deleting the topic causes producer writes to fail silently or drop messages; no automatic backup switch; driver apps continue sending but updates are lost; matching service receives stale data but does not block.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Kafka topic deletion causes permanent loss of location updates [check]
Hint: No Kafka topic means lost updates, no automatic failover [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard