Bird
Raised Fist0

Availability Numbers - What 99.9 vs 99.99 vs 99.999 Percent Means in Practice

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Availability Numbers - What 99.9 vs 99.99 vs 99.999 Percent Means in Practice
Understanding the practical implications of different availability SLAs and their impact on system design
mediumSystem Design Interview ApproachFreq: common
⚙️
Functional Requirements
  • Explain what 99.9%, 99.99%, and 99.999% availability mean in terms of downtime per year, month, and day
  • Quantify downtime in seconds/minutes/hours for each SLA level
  • Describe how availability impacts system design decisions such as redundancy and failover
  • Provide examples of failure scenarios that affect availability at each SLA level
  • Discuss monitoring and alerting requirements to detect SLA breaches
  • Explain trade-offs between cost, complexity, and availability targets
📊
Non-Functional Requirements
  • Support systems with up to 2 billion daily active users (DAU)
  • Target availability levels of 99.9%, 99.99%, and 99.999% (three, four, and five nines)
  • Ensure system response time P99 latency under 100ms to avoid user impact during failover
  • Design for continuous operation with minimal planned downtime (<1 hour/year)
  • Support automated failover and recovery within seconds to meet higher SLAs
❓
Clarifying Questions to Ask
  • What is the target time window for measuring availability (year, month, day)?
  • Are we considering planned maintenance downtime or only unplanned outages?
  • What is the acceptable recovery time objective (RTO) for each SLA?
  • Is the system globally distributed or single-region?
  • What are the critical components whose failure impacts availability?
  • Are we including network and infrastructure availability or only application-level?
Out of Scope
✗ Detailed implementation of specific failover mechanisms (e.g., DNS failover)
✗ Security and compliance aspects related to availability
✗ Cost optimization beyond availability trade-offs
✗ Design of specific distributed consensus algorithms
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 messages per day on average
  • Average message size is 100 bytes
  • System operates 24 hours per day continuously
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor → 1,851,852 msg/sec
read qpsAssuming 5 reads per write → 925,926 × 5 = 4,629,630 read/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024³) ≈ 7,450 GB/day
retention days30
total storage gb7,450 × 30 = 223,500 GB
with replication223,500 × 3 (3 replicas) = 670,500 GB ≈ 670.5 TB
🖥️
Server Estimate
write db nodesAssuming Cassandra nodes handle 10K writes/sec → 1,851,852 / 10,000 ≈ 186 nodes
read cache nodesRedis cluster nodes handle 50K reads/sec → 4,629,630 / 50,000 ≈ 93 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Client Apps   |
          +-------+--------+
                  |
          +-------v--------+
          |  Load Balancer |
          +-------+--------+
                  |
          +-------v--------+
          |  API Gateway   |
          +-------+--------+
                  |
          +-------v--------+
          |  Write Service |
          +-------+--------+
                  |
          +-------v--------+
          |  Kafka Cluster |
          +-------+--------+
                  |
          +-------v--------+
          | Cassandra DB  |
          +-------+--------+
                  |
          +-------v--------+
          | Redis Cache   |
          +----------------+
🧩
Components
API Gateway
Handles client requests, performs authentication, rate limiting, and routes to backend services
Why thisEnvoy proxy supports high throughput, low latency, and dynamic configuration for global scale
Why notNginx lacks native service discovery and dynamic config at 925K+ QPS scale
Kafka Cluster
Durable, distributed message queue for decoupling writes and enabling asynchronous processing
Why thisKafka supports high throughput (millions of msgs/sec), partitioning, and replication for durability
Why notRabbitMQ does not scale linearly beyond 100K msgs/sec and lacks partitioning at this scale
Cassandra DB
Primary storage for messages with high write throughput and multi-region replication
Why thisCassandra's peer-to-peer architecture and tunable consistency enable 1.8M writes/sec with 3 replicas
Why notPostgreSQL requires complex manual sharding and cannot sustain 925K+ writes/sec
Redis Cache
In-memory cache for low-latency reads and presence data with TTL-based expiry
Why thisRedis TTL natively expires stale data without cron jobs, supports sub-millisecond lookups
Why notMemcached lacks native TTL persistence and cluster management at this scale
Write Service
Validates and enriches incoming messages before publishing to Kafka
Why thisStateless microservice horizontally scalable to handle peak write QPS with autoscaling
Why notMonolithic service would create a single point of failure and bottleneck at 1.8M writes/sec
🔄
Data Flow
1
Client Apps → API Gateway
Send message write/read requests
API Gateway authenticates and routes requests
2
API Gateway → Write Service
Forward validated write requests
Write Service enriches and validates messages
3
Write Service → Kafka Cluster
Publish messages asynchronously
Kafka decouples ingestion from storage for durability
4
Kafka Cluster → Cassandra DB
Consume and persist messages
Cassandra stores data with replication for availability
5
Client Apps → API Gateway
Send read requests
Reads first hit Redis cache for low latency
6
API Gateway → Redis Cache
Query cached messages/presence
Cache TTL ensures freshness and automatic expiry
7
API Gateway → Cassandra DB
Fallback read if cache miss
Cassandra serves consistent data with tunable consistency

Practice

(1/5)
1. In a file storage system, at step 3 the client requests a file write. What happens inside the metadata server at this step?
easy
A. The metadata server sends the file data to object storage for persistence.
B. The metadata server writes the file data blocks directly to disk.
C. The metadata server updates the file's inode and directory entries to reflect the new write.
D. The metadata server caches the entire file in memory without updating disk metadata.

Solution

  1. Step 1: Understand metadata server role in file write

    The metadata server manages file metadata like inodes and directory entries, not the actual data blocks.

  2. Step 2: Trace what happens at step 3

    On a write request, the metadata server updates the inode and directory metadata to reflect changes; data blocks are handled separately by storage nodes.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Metadata server updates inode, not data blocks [check]
Hint: Metadata server updates metadata, not file data blocks [check]
Common Mistakes:
2. In the URL shortener system, at step 5 the Redirect Service receives a short code and queries the Cache before the Database. What happens if the Cache lookup misses at this step?
easy
A. The Redirect Service returns a 404 error immediately without querying the Database
B. The Redirect Service queries the Database for the original URL, then updates the Cache with the result
C. The Redirect Service generates a new short code and stores it in the Cache
D. The Redirect Service queues the request for later processing by an asynchronous worker

Solution

  1. Step 1: Trace Redirect Service behavior at Cache miss

    Cache miss means the short code is not in memory; the service must fetch the original URL from persistent storage.

  2. Step 2: Confirm Cache update after DB fetch

    After retrieving from DB, the service updates the Cache to speed up future requests.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Cache miss triggers DB lookup and Cache update [check]
Hint: Cache miss -> DB lookup -> Cache update [check]
Common Mistakes:
3. A location-based service must efficiently index millions of businesses worldwide to support fast nearby searches within a 5km radius. Which component is best suited to handle spatial indexing for this use case?
easy
A. Use a relational database with latitude and longitude columns and perform range queries with indexes
B. Use a Geohash-based spatial index to bucket locations into hierarchical grids
C. Use a simple key-value store with business IDs as keys and coordinates as values
D. Use a full-text search engine like Elasticsearch without spatial extensions

Solution

  1. Step 1: Identify spatial indexing needs

    Relational DBs with lat/lon indexes (Use a relational database with latitude and longitude columns and perform range queries with indexes) do not scale well for large radius queries and millions of points due to expensive range scans.

  2. Step 2: Evaluate Geohash suitability

    Geohash (Use a Geohash-based spatial index to bucket locations into hierarchical grids) encodes locations into hierarchical grids enabling efficient prefix queries for nearby searches, making it ideal for large-scale spatial indexing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Geohash buckets locations into grids for fast proximity queries [check]
Hint: Geohash buckets locations hierarchically for fast spatial queries [OK]
Common Mistakes:
4. If the Key Management Service (KMS) becomes unavailable, what exactly breaks in the system's encryption-at-rest workflow?
medium
A. New encryption keys cannot be generated, but existing data remains accessible and decryptable
B. Data encryption stops, causing all writes to fail immediately
C. Decryption of existing data fails, causing read operations to block or error out
D. The system falls back to unencrypted storage, risking data exposure

Solution

  1. Step 1: Understand KMS role in key lifecycle

    KMS generates and manages encryption keys; if unavailable, new keys can't be created but existing keys cached locally allow decryption.

  2. Step 2: Eliminate incorrect failure modes

    Encryption doesn't stop immediately; decryption continues with cached keys; system does not fallback to unencrypted storage automatically.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    KMS outage blocks new key generation, existing keys still usable [check]
Hint: KMS down -> no new keys, existing keys still decrypt [check]
Common Mistakes:
5. At 10x scale, which component in a distributed file system is most likely to become the bottleneck first, and why?
hard
A. Metadata Servers, because maintaining global namespace and handling metadata requests is stateful and hard to shard
B. Data Nodes, because disk I/O throughput saturates with increased client requests
C. Network switches, because increased data traffic causes packet loss and latency spikes
D. Client caches, because cache invalidation overhead grows exponentially with scale

Solution

  1. Step 1: Identify stateful bottlenecks

    Metadata Servers maintain global namespace state, which is difficult to shard and scale linearly.

  2. Step 2: Compare other components

    Data Nodes scale horizontally better; network and client caches are less immediate bottlenecks at metadata scale.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Metadata Servers' statefulness limits scaling first [check]
Hint: Metadata Servers' global state limits scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard