Bird
Raised Fist0

Facebook October 2021 Outage - BGP Misconfiguration and Global Recovery

HLDhardInterview freq: rare
Choose your preparation mode4 modes available
🏗️
Facebook October 2021 Outage - BGP Misconfiguration and Global Recovery
Designing resilient global network infrastructure and recovery mechanisms to prevent and mitigate BGP misconfiguration outages at Facebook scale
hardSystem Failures and Post-MortemsFreq: rare
Prerequisites:Distributed Systems BasicsObservabilityNetwork Protocols (BGP)Incident Response and Post-Mortems
⚙️
Functional Requirements
  • Detect BGP route misconfigurations in real-time across Facebook's global network
  • Automatically isolate and mitigate faulty BGP announcements to prevent global traffic blackholing
  • Provide rapid failover and rerouting to maintain service availability during network disruptions
  • Enable coordinated global recovery and rollback of BGP configurations after incident containment
  • Offer detailed observability and alerting on BGP state changes and network health
  • Support manual override and incident response workflows for network engineers
📊
Non-Functional Requirements
  • Support 2 billion daily active users with 99.99% global network availability
  • Detect and respond to BGP anomalies within 30 seconds to minimize outage impact
  • Handle peak BGP update rates of 100K updates/sec without data loss
  • Provide end-to-end network telemetry with 100ms P99 latency for alerting pipelines
  • Ensure recovery procedures complete within 15 minutes to restore global connectivity
❓
Clarifying Questions to Ask
  • What is the scale and topology of Facebook's global network and BGP deployment?
  • What existing monitoring and alerting tools are in place for BGP and network health?
  • Are there automated rollback mechanisms for BGP configurations or is it manual?
  • What is the expected maximum BGP update rate during normal and peak conditions?
  • How is traffic engineering handled across multiple data centers and regions?
  • What SLAs exist for network availability and incident recovery times?
Out of Scope
✗ Design of Facebook's internal application-layer failover mechanisms
✗ Detailed BGP protocol specification or router hardware design
✗ User-facing application error handling during outages
✗ Third-party ISP network management outside Facebook's control
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 network-related events per day (BGP updates, telemetry)
  • Average BGP update message size is 100 bytes
  • Retention period for network telemetry data is 7 days
⚡
QPS Calculation
write qps2B users × 40 events/user/day ÷ 86400 sec/day ≈ 925,926 writes/sec
peak write qpsAssuming 3× peak factor during incidents: 925,926 × 3 ≈ 2.78M writes/sec
read qpsMonitoring dashboards and alerting consume ~10% of write QPS: 925,926 × 0.1 ≈ 92,593 reads/sec
💾
Storage Estimate
daily storage925,926 writes/sec × 100 bytes × 86400 sec ≈ 8 TB/day
retention storage8 TB/day × 7 days = 56 TB
with replication56 TB × 3 replicas = 168 TB total storage
🖥️
Server Estimate
BGP Monitoring ClusterHandles 3M writes/sec peak; assuming 50K writes/sec per server → 60 servers
Telemetry Storage ClusterStores 168 TB replicated data; assuming 4 TB per server → 42 servers
🏗️
Architecture Diagram
Architecture Diagram
       +----------------+
       | Global BGP Routers|
       +--------+-------+
                | BGP Updates
                v
       +----------------+
       | BGP Monitoring |
       |   Cluster      |
       +--------+-------+
                | Filtered Alerts & Telemetry
                v
       +----------------+
       | Telemetry Store|
       | (Cassandra +   |
       |  Kafka for ETL)|
       +--------+-------+
                | Data Feeds
                v
       +----------------+
       | Alerting &     |
       | Incident Mgmt  |
       +--------+-------+
                | Control Cmds
                v
       +----------------+
       | Network Control|
       | Plane (BGP     |
       | Config Rollback|
       +----------------+
🧩
Components
Global BGP Routers
Exchange BGP route announcements globally across Facebook's network
Why thisPhysical routers running BGP provide real-time route propagation and peering
Why notSoftware-only BGP simulators lack real-time global scale and hardware optimizations
BGP Monitoring Cluster
Ingest and analyze BGP updates in real-time to detect anomalies and misconfigurations
Why thisApache Kafka for high-throughput ingestion + Apache Flink for real-time stream processing enables sub-30s detection
Why notTraditional RDBMS cannot handle 3M writes/sec with low latency; batch processing too slow
Telemetry Store
Persist BGP update history and network telemetry for 7-day retention and analytics
Why thisApache Cassandra provides linear scalability, high write throughput, and tunable consistency for time-series data
Why notPostgreSQL requires complex manual sharding at 3M writes/sec; B-tree writes don't scale linearly
Alerting & Incident Management
Generate alerts, notify engineers, and coordinate incident response workflows
Why thisPagerDuty integrated with custom dashboards provides reliable alerting and escalation
Why notEmail-only alerts cause delays and lack structured incident tracking
Network Control Plane
Apply automated and manual BGP configuration changes and rollbacks to contain incidents
Why thisCustom control plane with automated rollback scripts reduces human error and recovery time
Why notManual CLI-only changes are error-prone and slow at Facebook's scale
🔄
Data Flow
1
Global BGP Routers → BGP Monitoring Cluster
Stream BGP update messages in real-time
Continuous feed of route announcements and withdrawals
2
BGP Monitoring Cluster → Telemetry Store
Persist filtered BGP updates and metadata
Store for historical analysis and audit
3
BGP Monitoring Cluster → Alerting & Incident Management
Trigger anomaly alerts on suspicious BGP patterns
Detect route leaks, hijacks, or misconfigurations
4
Alerting & Incident Management → Network Control Plane
Issue automated or manual rollback commands
Contain faulty BGP announcements quickly
5
Network Control Plane → Global BGP Routers
Apply configuration changes and rollbacks
Restore correct routing state globally

Practice

(1/5)
1. A distributed database requires that every write is confirmed by a majority of replicas before acknowledging success to the client. Which replication strategy best fits this requirement?
easy
A. Asynchronous replication with eventual consistency
B. Single-leader replication without acknowledgments
C. Synchronous replication with quorum writes
D. Master-slave replication with delayed slave updates

Solution

  1. Step 1: Identify replication strategy matching majority confirmation

    Synchronous replication with quorum writes requires majority replicas to acknowledge before success.
  2. Step 2: Eliminate asynchronous and single-leader without ack

    Asynchronous replication and single-leader without acknowledgments do not guarantee majority confirmation before success.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Quorum writes ensure majority ack before success [check]
Hint: Quorum writes wait for majority acks before success [OK]
Common Mistakes:
2. In a fraud detection system ingesting thousands of transactions per second, which component is best suited to decouple the high-throughput transaction ingestion from downstream real-time scoring to avoid backpressure?
easy
A. Use a relational database with batch inserts to buffer transactions
B. Use direct synchronous HTTP calls from ingestion to scoring service
C. Use an in-memory cache like Redis to store incoming transactions temporarily
D. Use a distributed message queue like Kafka with partitioning by user ID

Solution

  1. Step 1: Identify decoupling mechanism

    Relational DB batch inserts (A) add latency and don't handle high write throughput well. Redis (C) lacks durability guarantees for buffering. Direct synchronous calls (D) cause backpressure and tight coupling.
  2. Step 2: Evaluate Kafka suitability

    Kafka supports high-throughput, partitioned by user ID to maintain ordering per user, and durable storage, enabling asynchronous processing and backpressure handling.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Kafka partitions decouple ingestion and scoring with durability and ordering [check]
Hint: Kafka partitions buffer high-throughput streams reliably [check]
Common Mistakes:
3. A web API needs to enforce a rate limit of 100 requests per minute per user, allowing bursts but avoiding sudden resets at minute boundaries. Which rate limiting approach best fits this requirement?
easy
A. Fixed window counter, resetting counts exactly every minute
B. Sliding window counter, tracking requests over the last 60 seconds continuously
C. Token bucket with fixed refill intervals every minute
D. Leaky bucket with fixed drain rate per minute

Solution

  1. Step 1: Identify burst tolerance and smooth rate enforcement

    Fixed window counters reset abruptly at minute boundaries, causing spikes. Token and leaky buckets enforce steady rates but may not track exact recent usage.
  2. Step 2: Sliding window counters track requests continuously over the last 60 seconds, allowing bursts without sudden resets

    Sliding window counters provide smoother rate limiting by considering recent activity, avoiding boundary effects.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Sliding window smooths rate limits over time, avoiding fixed window spikes [check]
Hint: Sliding window smooths counts over time, fixed resets abruptly [check]
Common Mistakes:
4. If the idempotency key store becomes unavailable, what is the most likely failure mode for the payment API when a client retries a payment request?
medium
A. The payment API processes each retry as a new payment, causing duplicate charges
B. The payment API queues requests until the store recovers, delaying all payments
C. The payment API rejects all requests with 503 Service Unavailable errors
D. The payment API returns cached responses from a fallback in-memory store

Solution

  1. Step 1: Understand idempotency store role in retries

    Without the store, the API cannot verify if a request is a retry, so it must delay processing to avoid duplicates.
  2. Step 2: Eliminate other options

    Rejecting all requests is possible but not typical; processing retries as new causes duplicates, which is avoided; fallback caches are uncommon and inconsistent.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Unavailable idempotency store causes request queuing to prevent duplicates [check]
Hint: No idempotency store -> queue retries to avoid duplicates [check]
Common Mistakes:
5. At 10x traffic scale, which CDN component is most likely to become the bottleneck first and why?
hard
A. Origin server, because it must handle all cache misses and cannot be scaled horizontally easily
B. Edge nodes, because their cache storage and network bandwidth scale linearly with traffic
C. DNS servers, because increased queries cause lookup latency and resolution failures
D. Load balancers, because they maintain stateful connections that cannot be sharded

Solution

  1. Step 1: Analyze scaling characteristics of CDN components

    Origin servers can be scaled horizontally; DNS is stateless and highly distributed; load balancers can be scaled or replaced.
  2. Step 2: Identify edge node bottleneck

    Edge nodes have fixed cache storage and network bandwidth per PoP; scaling traffic 10x stresses these resources first.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Edge nodes limited by cache and bandwidth -> bottleneck at 10x scale [check]
Hint: Edge nodes limited by cache and bandwidth first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard