HDFS high availability in Hadoop - Step-by-Step Execution

Concept Flow - HDFS high availability

Start HDFS Cluster

↓

Configure Two NameNodes

↓

One Active, One Standby

↓

Shared Storage or Quorum Journal

↓

Failover Triggered?

No→Continue Normal Operation

Yes↓

Standby Becomes Active

↓

Clients Redirected to New Active

↓

Cluster Continues Without Downtime

HDFS high availability uses two NameNodes where one is active and the other standby. If the active fails, the standby takes over seamlessly.

Execution Sample

Hadoop

1. Start two NameNodes
2. Active NameNode serves clients
3. Standby monitors active
4. On failure, standby becomes active
5. Clients connect to new active

This sequence shows how HDFS switches between active and standby NameNodes to keep the system running.

Execution Table

Step	Action	Active NameNode State	Standby NameNode State	Client Connection	Result
1	Start both NameNodes	Active and serving	Standby and monitoring	Connected to active	Normal operation
2	Active NameNode running	Serving client requests	Ready to take over	Connected to active	Normal operation
3	Active NameNode fails	Down	Detects failure	Connection lost	Failover triggered
4	Standby becomes active	Down or standby	Becomes active and serves	Reconnects to new active	Service restored
5	Old active recovers	Down or standby	Active serving clients	Connected to active	High availability maintained
6	Normal operation continues	Active serving	Standby monitoring	Connected to active	Cluster stable

💡 Execution stops as cluster stabilizes with one active and one standby NameNode.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
Active NameNode	Running	Serving clients	Failed	Down or standby	Down or standby	Active serving
Standby NameNode	Monitoring	Ready to take over	Detects failure	Becomes active	Active serving	Monitoring standby
Client Connection	Connected to active	Connected to active	Lost connection	Reconnected to new active	Connected to active	Connected to active

Key Moments - 3 Insights

Why does the standby NameNode not serve clients until failover?

The standby NameNode stays in monitoring mode to avoid split-brain issues. It only becomes active after detecting failure, as shown in execution_table step 3 and 4.

How do clients know to connect to the new active NameNode after failover?

Clients use a virtual IP or a failover controller that redirects them to the active NameNode, demonstrated by client reconnection in execution_table step 4.

What happens if the old active NameNode recovers after failover?

It either stays down or becomes the standby to avoid conflicts, ensuring only one active NameNode, as seen in execution_table step 5.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the standby NameNode become active?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Standby NameNode State' column in execution_table rows.

According to variable_tracker, what is the client connection state after the active NameNode fails?

A Lost connection

B Connected to active

C Reconnected to new active

D Disconnected permanently

💡 Hint

Look at 'Client Connection' variable after Step 3 in variable_tracker.

If the failover did not happen, what would be the client connection state at Step 4?

A Connected to active

B Lost connection

C Reconnected to new active

D Connected to standby

💡 Hint

Refer to execution_table Step 3 and 4 client connection states.

Concept Snapshot

HDFS High Availability:
- Two NameNodes: one active, one standby
- Standby monitors active, no client serving
- On active failure, standby takes over
- Clients reconnect via virtual IP
- Ensures no downtime in NameNode failure

Full Transcript

HDFS high availability uses two NameNodes to avoid downtime. One NameNode is active and serves clients. The other is standby and monitors the active. If the active fails, the standby detects this and becomes active. Clients then reconnect to the new active NameNode. This failover process keeps the HDFS cluster running without interruption. The old active either stays down or becomes standby after recovery. This setup prevents split-brain and ensures continuous service.