Bird
Raised Fist0

Facebook October 2021 Outage - BGP Misconfiguration and Global Recovery

HLDhardInterview freq: rare
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Global BGP Routers

Propagate BGP route updates

Routers exchange BGP routes globally; misconfiguration causes route withdrawal

Protocol:BGP
BeforeRouters have stable BGP routes with normal prefixes
AfterRouters withdraw routes due to misconfiguration, causing reachability loss
Network Control Plane

Receive and process BGP updates

Processes route withdrawals and updates internal state

Protocol:BGP Updates
BeforeControl Plane has consistent routing info matching routers
AfterControl Plane detects missing routes and updates telemetry data
BGP Monitoring Cluster

Analyze telemetry for anomalies

Consumes telemetry data, detects abnormal route withdrawals

Protocol:gRPC Telemetry
BeforeMonitoring cluster receives normal telemetry streams
AfterMonitoring cluster flags abnormal route loss patterns
Alerting & Incident Mgmt

Generate alerts and incidents

Receives anomaly events, triggers alert workflows

Protocol:Kafka
BeforeNo active alerts for BGP route issues
AfterAlerts created and incident management workflows initiated
Telemetry Store

Store telemetry and alert data

Persist telemetry metrics and alert logs for postmortem

Protocol:REST API
BeforeTelemetry store contains historical data only
AfterTelemetry store updated with current outage metrics and alerts
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────┐
│   Global BGP Routers  │
└────────────┬──────────┘
             │ BGP Updates
             ▼
┌───────────────────────────┐
│   Network Control Plane   │
└────────────┬──────────────┘
             │ Telemetry Data (gRPC)
             ▼
┌───────────────────────────┐
│    BGP Monitoring Cluster │
└────────────┬──────────────┘
             │ Alert Events (Kafka)
             ▼
┌───────────────────────────────┐
│ Alerting & Incident Mgmt      │
└────────────┬──────────────────┘
             │ Store Metrics (REST)
             ▼
┌───────────────────┐
│   Telemetry Store │
└───────────────────┘
Design Rationale

During the Facebook October 2021 outage, a BGP misconfiguration caused global route withdrawals at the Global BGP Routers. These route changes propagated to the Network Control Plane, which updated its routing state and generated telemetry data reflecting the loss of reachability. The BGP Monitoring Cluster consumed this telemetry via gRPC, detecting abnormal route withdrawal patterns indicative of a network outage. It then sent alert events through Kafka to the Alerting & Incident Mgmt system, which generated alerts and initiated incident workflows. Finally, all telemetry and alert data were stored in the Telemetry Store via REST API for analysis and postmortem. This architecture highlights the critical path of BGP route propagation, telemetry generation, anomaly detection, and alerting during a global network outage.

Primary Focus

✓

Practice

(1/5)
1. A global e-commerce platform requires zero downtime during data center failures and immediate failover without manual intervention. Which availability pattern best supports this requirement?
easy
A. Active-Active failover with synchronous replication
B. Single active node with periodic backups
C. Active-Passive failover with asynchronous replication
D. Active-Passive failover with manual DNS switch

Solution

  1. Step 1: Identify failover requirements

    Zero downtime and immediate failover require both sites to be active and data synchronized synchronously to avoid data loss.

  2. Step 2: Evaluate options

    Active-Active with synchronous replication ensures both data centers serve traffic simultaneously and failover is automatic without data loss.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Active-Active with sync replication -> zero downtime and instant failover [check]
Hint: Active-Active sync = zero downtime, instant failover [check]
Common Mistakes:
2. A high-traffic e-commerce platform expects sudden spikes of 1 million orders per minute. Which component best handles decoupling order intake from downstream inventory updates to maintain responsiveness?
easy
A. An in-memory cache with TTL eviction to buffer incoming orders
B. A relational database with strong ACID transactions for immediate consistency
C. A distributed message queue with partitioning and durable storage
D. A REST API gateway that throttles requests to downstream services

Solution

  1. Step 1: Identify decoupling need

    Immediate consistency via relational DB (A relational database with strong ACID transactions for immediate consistency) blocks on writes, causing latency spikes. In-memory cache (An in-memory cache with TTL eviction to buffer incoming orders) lacks durability and ordering guarantees. API gateway throttling (A REST API gateway that throttles requests to downstream services) limits throughput but doesn't decouple processing.

  2. Step 2: Durable queue with partitioning

    A distributed message queue (A distributed message queue with partitioning and durable storage) buffers spikes, partitions workload, and guarantees durability, enabling asynchronous downstream processing.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Distributed queue partitions spikes, ensures durability, decouples processing [check]
Hint: Use durable queues to buffer spikes and decouple processing [OK]
Common Mistakes:
3. If the Kafka cluster responsible for photo upload event streaming goes down, what exactly breaks in the system?
medium
A. Photo uploads fail immediately with HTTP 500 errors to users
B. Feed generation continues normally but stories fail to update
C. Photo metadata extraction and thumbnail generation stop; uploads succeed but processing is delayed indefinitely
D. User notifications about new photos are lost but photos appear in feed

Solution

  1. Step 1: Understand Kafka's role in event streaming

    Kafka streams photo upload events to downstream processors like metadata extraction and feed generation.

  2. Step 2: Trace failure impact

    If Kafka is down, photo uploads still succeed because upload service writes to storage directly. Feed generation depends on Kafka events for stories update; without Kafka, stories updates stop, but feed generation for photos continues from DB. Metadata extraction is delayed but not necessarily indefinite. Notifications are handled separately.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Kafka outage stops stories update but feed photo generation continues [check]
Hint: Kafka outage blocks stories update, feed photos still served [check]
Common Mistakes:
4. For a system requiring strong consistency on writes but extremely high read throughput, which architecture is best suited?
medium
A. Use CQRS with separate write and read models, updating read model asynchronously
B. Use synchronous replication with read replicas and route reads only to replicas after commit
C. Use a single primary database with aggressive caching on the client side
D. Use asynchronous replication with eventual consistency on read replicas

Solution

  1. Step 1: Analyze consistency and throughput needs

    Strong write consistency with high read throughput favors CQRS separating write/read models.

  2. Step 2: Evaluate options

    Use asynchronous replication with eventual consistency on read replicas sacrifices strong consistency; Use synchronous replication with read replicas and route reads only to replicas after commit synchronous replication slows writes; Use a single primary database with aggressive caching on the client side risks stale cache and write bottleneck.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    CQRS separates concerns, enabling strong writes and scalable reads [check]
Hint: CQRS separates write/read for strong consistency and scale [check]
Common Mistakes:
5. At 10x scale, which component in a Blue-Green deployment system is most likely to fail first and why?
hard
A. Deployment pipeline because build servers cannot handle 10x concurrent builds, delaying releases
B. Green environment servers because maintaining two full production environments doubles resource usage and costs
C. Load balancer because it must instantly switch traffic for millions of users, causing connection storms
D. Feature flag service because it cannot handle 10x toggle evaluations per request

Solution

  1. Step 1: Identify scaling bottlenecks

    Blue-Green requires two full environments; load balancer scales horizontally; feature flags scale independently.

  2. Step 2: Analyze deployment pipeline

    Build servers face 10x concurrent builds, causing queueing and delays, blocking releases.

  3. Step 3: Compare other components

    Load balancer and feature flags can be sharded; doubling environment cost is expensive but not a failure.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Build servers bottleneck at 10x concurrent builds, delaying deployments [check]
Hint: Build servers choke first at 10x concurrent builds, blocking deploys [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard