Leader election in HLD - System Design Exercise

Design: Distributed Leader Election System

Design focuses on leader election mechanism and fault tolerance. Does not cover full distributed consensus or data replication.

Functional Requirements

FR1: Elect a single leader node among multiple distributed nodes

FR2: Handle node failures and re-elect leader if current leader fails

FR3: Ensure only one leader exists at any time (no split-brain)

FR4: Support dynamic addition and removal of nodes

FR5: Provide fast leader election to minimize downtime

FR6: Allow nodes to detect leader status and act accordingly

Non-Functional Requirements

NFR1: System must handle up to 1000 nodes

NFR2: Leader election latency should be under 5 seconds

NFR3: System availability target is 99.9% uptime

NFR4: Network partitions may occur and must be handled gracefully

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Node communication protocol (heartbeat, messaging)

Failure detection mechanism

Leader election algorithm implementation

State storage for leader info (in-memory or distributed store)

Timeout and retry logic

Design Patterns

Bully algorithm

Ring algorithm

Raft leader election phase

Paxos leader election

Heartbeat and timeout pattern

Reference Architecture

  +-------------------+       +-------------------+       +-------------------+
  |      Node 1       |<----->|      Node 2       |<----->|      Node 3       |
  +-------------------+       +-------------------+       +-------------------+
           |                          |                          |
           | Heartbeats & Election Messages                      |
           +----------------------------------------------------+
                                   |
                           Leader Election Logic
                                   |
                          +-------------------+
                          |   Leader Node      |
                          +-------------------+

Components

Nodes

Any distributed system nodes (servers, containers)

Participate in leader election and perform leader or follower roles

Communication Layer

TCP/UDP or RPC messaging

Exchange heartbeat and election messages between nodes

Failure Detector

Timeout-based heartbeat monitoring

Detect node failures by missing heartbeats

Leader Election Module

Algorithm implementation (e.g., Bully algorithm)

Run election process to select a single leader

State Store

In-memory or distributed key-value store

Store current leader identity and election state

Request Flow

1. 1. Each node periodically sends heartbeat messages to other nodes.

2. 2. Nodes monitor heartbeats to detect failures.

3. 3. When a node detects leader failure or startup, it initiates leader election.

4. 4. Nodes exchange election messages according to the chosen algorithm.

5. 5. Nodes agree on a single leader based on priority or ID.

6. 6. The elected leader broadcasts its status to all nodes.

7. 7. Nodes update their state to recognize the leader.

8. 8. If leader fails, process repeats.

Database Schema

Entities: - Node: node_id (PK), status (active, failed), priority - Leader: leader_node_id (FK to Node), election_term, timestamp Relationships: - One leader per election_term - Nodes participate in election terms This schema supports tracking current leader and node statuses.

Scaling Discussion

Bottlenecks

Network congestion due to many heartbeat and election messages

Slow failure detection with large number of nodes

Split-brain scenarios during network partitions

Leader election latency increases with node count

Solutions

Use hierarchical or partitioned election groups to reduce message overhead

Implement adaptive heartbeat intervals and failure detection thresholds

Use quorum-based election algorithms to avoid split-brain

Optimize election algorithm to reduce message rounds (e.g., use Bully algorithm with priority)

Leverage distributed consensus protocols like Raft for stronger guarantees if needed

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Clarify assumptions about network and node behavior

Explain choice of leader election algorithm and why

Describe failure detection and handling of node crashes

Discuss how to avoid split-brain and ensure single leader

Address scaling challenges and solutions

Mention trade-offs between simplicity and consistency