Bird
Raised Fist0

Facebook October 2021 Outage - BGP Misconfiguration and Global Recovery

HLDhardInterview freq: rare
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Facebook October 2021 Outage - BGP Misconfiguration and Global Recovery
45 min · senior
senior

You are tasked with designing a global network routing and recovery system for a large-scale social media platform with 2.9B monthly active users. The system must handle BGP route misconfigurations and enable rapid global recovery from routing failures that can cause multi-hour outages affecting billions of users worldwide.

Constraints
  • Must detect and isolate BGP misconfigurations within 5 minutes to minimize outage impact
  • Global scale: 100+ data centers interconnected via BGP with millions of routes
  • Recovery must avoid cascading failures and ensure eventual consistency of routing tables
  • System must provide real-time visibility and alerting with P99 detection latency < 1 minute
  • All control plane communications must be encrypted to prevent route hijacking
  • Recovery actions must be auditable and reversible to prevent human error escalation
📋
Guided Steps
1/4
Step 1
BGP Anomaly Detection Architecture
How would you design a system to detect BGP route anomalies and misconfigurations at global scale?
💡 Hint
Consider streaming telemetry from routers and real-time analytics to detect route leaks or withdrawals.
Key Decision
Use Apache Kafka to ingest streaming BGP route updates from all data centers, combined with Apache Flink for real-time anomaly detection based on route origin validation and prefix consistency.
Interviewer Expectation
Mid: Names Kafka and Flink for streaming and analytics. Senior: Explains why Kafka partitions by data center and Flink jobs correlate multi-source route updates to detect global inconsistencies.
⚠️Common Wrong Answer
Polling router configs periodically - leads to detection delays of tens of minutes, too slow for outage mitigation.
Step 2
Encrypted Control Plane Communication
How do you secure control plane communications between network devices and the detection system to prevent hijacking?
💡 Hint
Think about encryption protocols and authentication mechanisms for BGP sessions and telemetry streams.
Key Decision
Use TLS 1.3 with mutual authentication for all BGP session communications and gRPC streams carrying telemetry data, ensuring confidentiality and integrity.
Interviewer Expectation
Mid: Mentions TLS encryption. Senior: Details mutual TLS with certificate rotation and automated key management to prevent stale credentials.
⚠️Common Wrong Answer
Relying on IPsec tunnels only - complex to manage at scale and lacks fine-grained authentication controls.
Step 3
Automated Recovery and Rollback Workflow
Design a system to automate recovery from detected BGP misconfigurations while allowing safe rollback.
💡 Hint
Consider workflow orchestration tools and audit logging for operator actions.
Key Decision
Use Apache Airflow to orchestrate recovery workflows that trigger prefix withdrawals or peer resets via Netconf/YANG APIs, with all actions logged in Elasticsearch for audit and rollback.
Interviewer Expectation
Mid: Names Airflow for orchestration. Senior: Explains integration with network device APIs and how audit logs enable safe rollback and forensic analysis.
⚠️Common Wrong Answer
Manual CLI commands without orchestration - error-prone and slow, increasing outage duration.
Step 4
Real-time Visibility and Alerting
How would you provide network operators with real-time visibility and alerting on BGP anomalies and recovery status?
💡 Hint
Think about dashboards, alerting thresholds, and integration with incident management.
Key Decision
Use Grafana connected to Prometheus metrics exported from Flink jobs and Airflow workflows, with PagerDuty integration for P99 alerting under 1 minute latency.
Interviewer Expectation
Mid: Mentions Grafana dashboards. Senior: Details metric instrumentation, alert thresholds, and automated incident escalation workflows.
⚠️Common Wrong Answer
Email alerts only - slow and unreliable for critical outage detection.
🔌
API Design Challenge
Sketch 3 core API endpoints for the network operations team to detect, isolate, and recover from BGP misconfigurations.
Hints
  • How does the system report detected anomalies and route inconsistencies?
  • How can operators isolate affected prefixes or peers programmatically?
  • How do operators trigger and monitor recovery workflows and rollback actions?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Streaming ingestion of BGP updates with Kafka to handle global scale
  • Real-time anomaly detection with Flink or equivalent streaming analytics
  • Encrypted and authenticated control plane communication using TLS 1.3 mutual TLS
  • Automated recovery workflows with orchestration and audit logging
  • Real-time visibility dashboards and alerting with sub-minute latency
Good to Have - 30%
  • Prefix filtering and route validation policies to isolate failures
  • Rollback mechanisms integrated with recovery workflows
  • Centralized certificate and key management for control plane security
Bonus - 10%
  • Integration with incident management tools like PagerDuty
  • Use of Netconf/YANG APIs for network device automation
Anti-patterns to Avoid
Polling router configs instead of streaming telemetry - causes slow detection and prolonged outages due to delayed anomaly identification
Manual CLI recovery without orchestration - error-prone and delays recovery, increasing outage duration and risk of human error
Unencrypted BGP sessions - exposes system to route hijacking and man-in-the-middle attacks compromising network integrity
Email-only alerting - slow and unreliable for critical outage response, leading to delayed incident handling
Using IPsec tunnels alone for control plane security - complex to manage at scale and lacks fine-grained authentication controls, increasing risk of unauthorized access
No audit logging of recovery actions - prevents rollback and forensic analysis, making error correction and accountability difficult
✅
Model Answer
▼ Reveal

The system ingests global BGP route updates via Apache Kafka, analyzes them in real-time with Apache Flink to detect anomalies, and secures all control plane communications with mutual TLS. Automated recovery workflows orchestrated by Apache Airflow trigger network device actions via Netconf/YANG APIs, with all steps logged in Elasticsearch for audit and rollback. Operators monitor the system through Grafana dashboards with Prometheus metrics and receive sub-minute alerts via PagerDuty.

Key Decisions
  • Partition Kafka topics by data center to scale ingestion and maintain ordering guarantees
  • Use Flink jobs to correlate multi-source route updates for global anomaly detection within 1 minute
  • Implement mutual TLS with automated certificate rotation for all control plane communications
  • Orchestrate recovery workflows with Airflow triggering Netconf/YANG API calls to network devices
  • Log all recovery and rollback actions in Elasticsearch for audit and forensic analysis
  • Provide real-time metrics via Prometheus and dashboards via Grafana with PagerDuty alerts for sub-minute detection latency
  • Apache Kafka for ingesting global BGP route updates with partitioning by data center to scale to millions of routes
  • Apache Flink for real-time streaming analytics to detect route leaks, withdrawals, and origin validation failures within 1 minute
  • TLS 1.3 with mutual authentication securing BGP sessions and telemetry streams to prevent route hijacking
  • Apache Airflow orchestrating automated recovery workflows via Netconf/YANG APIs with audit logging in Elasticsearch
  • Grafana dashboards and Prometheus metrics providing real-time visibility and PagerDuty integration for sub-minute alerting
  • Use of prefix filtering and route validation policies to isolate affected prefixes and prevent cascading failures
  • Automated rollback mechanisms triggered by operator approval to safely revert recovery actions
  • Centralized certificate management system for secure key rotation and credential lifecycle management

Practice

(1/5)
1. In the request flow, at step 3 the API Gateway forwards the request to the BFF layer. What happens at step 4 inside the BFF before calling downstream microservices?
easy
A. BFF aggregates and transforms data from multiple microservices into a client-specific response
B. BFF caches the entire response from microservices before returning to client
C. BFF directly proxies the request without modification to microservices
D. BFF performs authentication and then sends request to a single microservice

Solution

  1. Step 1: Understand BFF responsibilities at step 4

    At step 4, the BFF aggregates data from multiple microservices and transforms it into a response tailored to the client's needs.

  2. Step 2: Eliminate incorrect options

    BFF caches the entire response from microservices before returning to client is wrong because caching happens after response, not before calling microservices. BFF directly proxies the request without modification to microservices is incorrect as BFF modifies requests/responses rather than proxying blindly. BFF performs authentication and then sends request to a single microservice is incomplete; authentication usually happens earlier at API Gateway, and BFF calls multiple microservices, not just one.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    BFF aggregates and transforms data before responding [check]
Hint: BFF aggregates and transforms data before responding [check]
Common Mistakes:
2. A ticket booking system must prevent double booking by temporarily reserving seats during payment processing. Which component is best suited to handle this seat locking with a timeout?
easy
A. Use a message queue to serialize seat booking requests and process them sequentially
B. Use a relational database transaction with row-level locks to hold seats until payment completes
C. Use a distributed in-memory cache with TTL (e.g., Redis) to lock seats temporarily
D. Use client-side locking where the user's browser holds the seat reservation state

Solution

  1. Step 1: Identify seat locking requirements

    Seat locking must be temporary with automatic expiration to avoid indefinite holds if payment fails.

  2. Step 2: Evaluate options for temporary locks

    Relational DB locks block resources and can cause deadlocks; message queues serialize but add latency and don't expire locks; client-side locking is unreliable and insecure. Distributed cache with TTL provides automatic expiration and fast access.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Redis TTL auto-expires locks, preventing stale seat holds [check]
Hint: Temporary locks need auto-expiry; Redis TTL fits best [OK]
Common Mistakes:
3. In a distributed chat system, at Step 4 the Delivery Worker reads from Kafka and attempts to deliver a message to Bob's device. What happens if Bob's device is offline at this step?
easy
A. The Delivery Worker retries delivery asynchronously and keeps the message in Kafka until acknowledged
B. The Delivery Worker immediately marks the message as delivered and removes it from Kafka
C. The Delivery Worker blocks all other deliveries until Bob's device comes online
D. The Delivery Worker deletes the message permanently to avoid blocking the queue

Solution

  1. Step 1: Trace Delivery Worker decision at Step 4 -- Bob's device offline

    The Delivery Worker reads the message from Kafka but cannot deliver it immediately because Bob is offline.

  2. Step 2: Understand retry and message retention behavior

    The worker retries delivery asynchronously and keeps the message in Kafka until Bob acknowledges receipt. It does not block other deliveries or delete messages prematurely.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Delivery Worker retries and retains message until ack [check]
Hint: Offline devices cause retries, not message loss or blocking [check]
Common Mistakes:
4. At 10x scale, which component in a real-time chat system implementing CAP theorem fails first and why?
hard
A. The database fails first due to increased write amplification
B. The message queue fails first due to partition key hotspots causing uneven load
C. Chat servers fail first because WebSocket connections are stateful and cannot be resharded easily
D. The load balancer fails first due to inability to handle increased connection churn

Solution

  1. Step 1: Identify stateful bottleneck

    Chat servers maintain WebSocket connections statefully, limiting horizontal scaling.

  2. Step 2: Understand scaling impact

    At 10x scale, existing WebSocket connections cannot be moved, causing chat servers to saturate first.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful WebSocket servers bottleneck scaling before DB or load balancer [check]
Hint: Stateful WebSocket servers bottleneck scaling first [check]
Common Mistakes:
5. At 10x scale, which component in a Blue-Green deployment system is most likely to fail first and why?
hard
A. Deployment pipeline because build servers cannot handle 10x concurrent builds, delaying releases
B. Green environment servers because maintaining two full production environments doubles resource usage and costs
C. Load balancer because it must instantly switch traffic for millions of users, causing connection storms
D. Feature flag service because it cannot handle 10x toggle evaluations per request

Solution

  1. Step 1: Identify scaling bottlenecks

    Blue-Green requires two full environments; load balancer scales horizontally; feature flags scale independently.

  2. Step 2: Analyze deployment pipeline

    Build servers face 10x concurrent builds, causing queueing and delays, blocking releases.

  3. Step 3: Compare other components

    Load balancer and feature flags can be sharded; doubling environment cost is expensive but not a failure.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Build servers bottleneck at 10x concurrent builds, delaying deployments [check]
Hint: Build servers choke first at 10x concurrent builds, blocking deploys [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard