Bird
Raised Fist0

GitHub Database Failover Incident - What Went Wrong and Prevention

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
▶
Request Trace
Step 1/6
Load Balancer (HAProxy)

Receives client DB request

Load balancer routes request to current primary cluster

Protocol:MySQL Protocol (TCP)
BeforePrimary cluster is healthy and accepting connections
AfterRequest forwarded to primary cluster for processing
MySQL Group Replication Primary Cluster

Processes write request

Primary commits transaction and replicates to replicas

Protocol:MySQL Protocol (TCP)
BeforePrimary cluster is active and consistent
AfterTransaction committed on primary and replication started
MySQL Group Replication Replica Cluster

Receives replication data

Replica applies transaction to stay in sync

Protocol:Replication Protocol
BeforeReplica cluster is behind primary by one transaction
AfterReplica cluster updated with latest transaction
Failover Orchestrator (Consul + Custom Logic)

Detects primary cluster failure

Health checks fail; orchestrator initiates failover

Protocol:Health Checks (HTTP)
BeforePrimary cluster is unresponsive
AfterFailover process started; replica promoted to primary
Load Balancer (HAProxy)

Updates routing to new primary

Load balancer switches traffic to promoted replica cluster

Protocol:MySQL Protocol (TCP)
BeforeLoad balancer routes to failed primary cluster
AfterLoad balancer routes requests to new primary cluster
Monitoring & Alerting (Prometheus + Grafana)

Sends alert on failover event

Alerts engineers about primary cluster failure and failover

Protocol:Alerting Protocol
BeforeNo failover alerts active
AfterFailover alert triggered and visible on dashboards
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────────────────────┐
│       Load Balancer (HAProxy)                 │
└─────────────────────────┬─────────────────────┘
                          │ MySQL Protocol (TCP)
                          ▼
          ┌────────────────────────────────────┐
          │ MySQL Group Replication Primary    │
          │ Cluster                           │
          └───────────────┬────────────────────┘
                          │ Replication Protocol
                          ▼
          ┌────────────────────────────────────┐
          │ MySQL Group Replication Replica    │
          │ Cluster                           │
          └────────────────────────────────────┘
                          ▲
                          │
┌─────────────────────────┴─────────────────────┐
│ Failover Orchestrator (Consul + Custom Logic) │
└─────────────────────────┬─────────────────────┘
                          │ Health Checks (HTTP)
                          ▼
┌───────────────────────────────────────────────┐
│ Monitoring & Alerting (Prometheus + Grafana)  │
└───────────────────────────────────────────────┘

This architecture diagram illustrates GitHub's database failover process during a primary cluster outage. The Load Balancer (HAProxy) initially routes client requests to the MySQL Group Replication Primary Cluster, which handles writes and replicates data to the Replica Cluster. The Failover Orchestrator continuously monitors the primary's health using Consul and custom logic. Upon detecting failure, it triggers promotion of the replica to primary and updates the Load Balancer routing accordingly. Monitoring & Alerting tools like Prometheus and Grafana provide visibility and alert engineers to the failover event. This design ensures minimal downtime by automating failover and maintaining data consistency through group replication.

Primary Focus

✓

Practice

(1/5)
1. A social media platform needs to store user posts with high write throughput and ensure efficient retrieval by user and timestamp. Which component best handles this workload?
easy
A. A wide-column store like Cassandra with partition keys on user ID
B. A relational database with normalized tables and joins
C. A document store like MongoDB with embedded user posts
D. An in-memory cache like Redis without persistence

Solution

  1. Step 1: Identify write throughput and query pattern

    The system requires high write throughput and efficient retrieval by user and timestamp, which suits wide-column stores that partition data by user ID.
  2. Step 2: Evaluate options against constraints

    Relational DBs struggle with high write throughput and joins; document stores lack efficient range queries by timestamp; Redis lacks durability and is not ideal for large persistent datasets.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Wide-column store partitions by user ID -> high write throughput and efficient user queries [check]
Hint: Partition by user ID for high write and read throughput [check]
Common Mistakes:
2. In a system using Kafka for write-ahead logging to ensure strong consistency, what exactly breaks if Kafka becomes unavailable during a write operation?
medium
A. The system falls back to eventual consistency, causing stale reads temporarily
B. The client's write blocks indefinitely until Kafka recovers, preventing data loss
C. The write is accepted locally but lost permanently if the server crashes before Kafka recovers
D. The system switches to causal consistency mode automatically to maintain availability

Solution

  1. Step 1: Trace failure impact on write path

    Without Kafka, writes accepted locally but not durably logged can be lost if the server crashes before Kafka recovers.
  2. Step 2: Eliminate other options

    The system falls back to eventual consistency, causing stale reads temporarily is incorrect because fallback to eventual consistency is not automatic; The client's write blocks indefinitely until Kafka recovers, preventing data loss is wrong because writes may not block indefinitely; The system switches to causal consistency mode automatically to maintain availability is false as consistency mode does not switch automatically.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Writes accepted locally but lost if server crashes before Kafka recovers [check]
Hint: Without Kafka, local writes risk loss if server crashes [check]
Common Mistakes:
3. If the system loses the HyperLogLog register state due to a crash before merging sketches, what exactly breaks in the cardinality estimation process?
medium
A. The system will continue to provide exact counts from raw user IDs stored elsewhere
B. The system will undercount unique users because lost registers cause missing data in the union
C. The system will overcount unique users because duplicates are counted multiple times
D. The system will fail to detect new users but still count old users correctly

Solution

  1. Step 1: Understand HyperLogLog state loss impact

    Losing the register state means the approximate count for that shard is lost; no exact user IDs are stored elsewhere.
  2. Step 2: Check consequences on cardinality

    Without the register, the system cannot include that shard's data, causing undercounting, not overcounting or exact counts.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Lost registers -> missing data -> undercounting unique users [check]
Hint: Lost registers cause undercount, no exact backup [check]
Common Mistakes:
4. For a social media feed requiring low-latency reads and flexible schema, which database choice is better and why?
medium
A. Use a relational database with normalized tables for consistency
B. Use a wide-column NoSQL store to denormalize and optimize read latency
C. Use a graph database to model user relationships for feed generation
D. Use an in-memory cache only to serve all feed requests

Solution

  1. Step 1: Analyze read latency and schema flexibility needs

    Social feeds require fast reads and flexible schema to handle diverse content types.
  2. Step 2: Compare database options

    Relational DBs have join overhead; graph DBs excel at relationships but not high-volume feed reads; in-memory cache alone lacks persistence.
  3. Step 3: Justify wide-column NoSQL choice

    Wide-column stores denormalize data for fast reads and support flexible schemas, fitting feed requirements.
  4. Final Answer:

    Option B -> Option B
  5. Quick Check:

    Wide-column NoSQL -> denormalized, low-latency reads [check]
Hint: Low-latency feed -> denormalized wide-column store [check]
Common Mistakes:
5. At 10x scale of global traffic, which component in the BGP-based routing infrastructure is most likely to fail first, and why?
hard
A. Edge routers, because their CPU cannot handle the increased packet forwarding rate
B. Route reflectors, because they become overwhelmed maintaining full routing tables and propagating updates
C. Network monitoring systems, because data volume exceeds processing capacity causing delayed alerts
D. Client devices, because they cannot process rapid route changes at scale

Solution

  1. Step 1: Identify bottleneck at scale

    Edge routers handle packet forwarding at line rate; at 10x traffic, their CPU and forwarding ASICs are stressed first.
  2. Step 2: Evaluate other components

    Route reflectors (A) scale better with hierarchical design; monitoring systems (C) can be scaled horizontally; client devices (D) are downstream and less critical for routing stability.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Edge routers' CPU overload causes first failure at 10x traffic [check]
Hint: Edge routers hit CPU limits first at massive scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard