HDFS high availability in Hadoop - Step-by-Step Execution

Concept Flow - HDFS high availability

Start HDFS Cluster

↓

Configure Two NameNodes

↓

One Active, One Standby

↓

Shared Storage or Quorum Journal

↓

Failover Triggered?

No→Continue Normal Operation

Yes↓

Standby Becomes Active

↓

Clients Redirected to New Active

↓

Cluster Continues Without Downtime

HDFS high availability uses two NameNodes where one is active and the other standby. If the active fails, the standby takes over seamlessly.

Execution Sample

Hadoop

1. Start two NameNodes
2. Active NameNode serves clients
3. Standby monitors active
4. On failure, standby becomes active
5. Clients connect to new active

This sequence shows how HDFS switches between active and standby NameNodes to keep the system running.

Execution Table

Step	Action	Active NameNode State	Standby NameNode State	Client Connection	Result
1	Start both NameNodes	Active and serving	Standby and monitoring	Connected to active	Normal operation
2	Active NameNode running	Serving client requests	Ready to take over	Connected to active	Normal operation
3	Active NameNode fails	Down	Detects failure	Connection lost	Failover triggered
4	Standby becomes active	Down or standby	Becomes active and serves	Reconnects to new active	Service restored
5	Old active recovers	Down or standby	Active serving clients	Connected to active	High availability maintained
6	Normal operation continues	Active serving	Standby monitoring	Connected to active	Cluster stable

💡 Execution stops as cluster stabilizes with one active and one standby NameNode.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
Active NameNode	Running	Serving clients	Failed	Down or standby	Down or standby	Active serving
Standby NameNode	Monitoring	Ready to take over	Detects failure	Becomes active	Active serving	Monitoring standby
Client Connection	Connected to active	Connected to active	Lost connection	Reconnected to new active	Connected to active	Connected to active

Key Moments - 3 Insights

Why does the standby NameNode not serve clients until failover?

The standby NameNode stays in monitoring mode to avoid split-brain issues. It only becomes active after detecting failure, as shown in execution_table step 3 and 4.

How do clients know to connect to the new active NameNode after failover?

Clients use a virtual IP or a failover controller that redirects them to the active NameNode, demonstrated by client reconnection in execution_table step 4.

What happens if the old active NameNode recovers after failover?

It either stays down or becomes the standby to avoid conflicts, ensuring only one active NameNode, as seen in execution_table step 5.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the standby NameNode become active?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Standby NameNode State' column in execution_table rows.

According to variable_tracker, what is the client connection state after the active NameNode fails?

A Lost connection

B Connected to active

C Reconnected to new active

D Disconnected permanently

💡 Hint

Look at 'Client Connection' variable after Step 3 in variable_tracker.

If the failover did not happen, what would be the client connection state at Step 4?

A Connected to active

B Lost connection

C Reconnected to new active

D Connected to standby

💡 Hint

Refer to execution_table Step 3 and 4 client connection states.

Concept Snapshot

HDFS High Availability:
- Two NameNodes: one active, one standby
- Standby monitors active, no client serving
- On active failure, standby takes over
- Clients reconnect via virtual IP
- Ensures no downtime in NameNode failure

Full Transcript

HDFS high availability uses two NameNodes to avoid downtime. One NameNode is active and serves clients. The other is standby and monitors the active. If the active fails, the standby detects this and becomes active. Clients then reconnect to the new active NameNode. This failover process keeps the HDFS cluster running without interruption. The old active either stays down or becomes standby after recovery. This setup prevents split-brain and ensures continuous service.

Practice

1. What is the main purpose of HDFS high availability (HA)?

easy

A. To speed up data processing by parallel tasks

B. To avoid downtime by using two active NameNodes

C. To increase storage capacity by adding more DataNodes

D. To encrypt data stored in HDFS automatically

Solution

1. Step 1: Understand HDFS HA concept

   HDFS HA uses two NameNodes to ensure one is always available if the other fails.

2. Step 2: Identify the main goal

   The goal is to avoid downtime, not to increase storage or speed processing.

3. Final Answer:

   To avoid downtime by using two active NameNodes -> Option B

4. Quick Check:

   HDFS HA = Avoid downtime [OK]

Hint: Remember: HA means no downtime with two NameNodes [OK]

Common Mistakes:

• Confusing HA with storage scaling
• Thinking HA speeds up processing
• Assuming HA encrypts data

2. Which component is essential for automatic failover in HDFS high availability?

easy

A. ZooKeeper

B. HDFS DataNode

C. YARN ResourceManager

D. MapReduce JobTracker

Solution

1. Step 1: Identify failover mechanism in HDFS HA

   ZooKeeper coordinates and manages automatic failover between NameNodes.

2. Step 2: Exclude unrelated components

   DataNode stores data, ResourceManager manages resources, JobTracker is for MapReduce jobs, none handle failover.

3. Final Answer:

   ZooKeeper -> Option A

4. Quick Check:

   Failover = ZooKeeper [OK]

Hint: ZooKeeper manages failover in HDFS HA [OK]

Common Mistakes:

• Choosing DataNode as failover manager
• Confusing YARN with failover
• Selecting MapReduce components

3. Given the following simplified HDFS HA configuration snippet, what is the role of dfs.nameservices?

dfs.nameservices=cluster1

medium

A. Lists all DataNodes in the cluster

B. Specifies the IP address of the active NameNode

C. Defines the logical name for the HDFS cluster used by clients

D. Sets the replication factor for data blocks

Solution

1. Step 1: Understand dfs.nameservices meaning

   It defines a logical name for the HDFS cluster, grouping NameNodes under one namespace.

2. Step 2: Eliminate other options

   It does not specify IPs, DataNodes, or replication factors.

3. Final Answer:

   Defines the logical name for the HDFS cluster used by clients -> Option C

4. Quick Check:

   dfs.nameservices = cluster name [OK]

Hint: dfs.nameservices names the cluster logically [OK]

Common Mistakes:

• Thinking it sets IP addresses
• Confusing with DataNode list
• Mixing with replication settings

4. You configured HDFS HA but the failover does not happen automatically. Which configuration is most likely missing?

medium

A. Setting dfs.ha.automatic-failover.enabled to true

B. Increasing the replication factor to 5

C. Adding more DataNodes to the cluster

D. Disabling the NameNode checkpoint

Solution

1. Step 1: Identify failover configuration

   Automatic failover requires dfs.ha.automatic-failover.enabled set to true.

2. Step 2: Exclude unrelated settings

   Replication factor, DataNodes count, and checkpoint settings do not control failover.

3. Final Answer:

   Setting dfs.ha.automatic-failover.enabled to true -> Option A

4. Quick Check:

   Auto failover enabled = true [OK]

Hint: Enable dfs.ha.automatic-failover.enabled for auto failover [OK]

Common Mistakes:

• Changing replication factor to fix failover
• Adding DataNodes expecting failover fix
• Disabling checkpoint thinking it helps failover

5. You want to configure HDFS HA with two NameNodes: nn1 and nn2. Which configuration snippet correctly sets the failover proxy provider for cluster1?

hard

A. dfs.replication=3

B. dfs.nameservices=nn1,nn2

C. dfs.ha.automatic-failover.enabled=false

D. dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

Solution

1. Step 1: Understand failover proxy provider setting

   This property tells the client how to switch between NameNodes during failover.

2. Step 2: Check correctness of options

   dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

   correctly sets the proxy provider for cluster1.

   dfs.nameservices=nn1,nn2

   wrongly lists NameNodes as nameservices.

   dfs.ha.automatic-failover.enabled=false

   disables failover.

   dfs.replication=3

   sets replication, unrelated to failover.

3. Final Answer:

   dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider -> Option D

4. Quick Check:

   Failover proxy provider =

   dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

   [OK]

Hint: Set failover proxy provider for cluster1 correctly [OK]

Common Mistakes:

• Confusing nameservices with NameNodes list
• Disabling automatic failover by mistake
• Mixing replication with failover settings