HDFS high availability in Hadoop

Introduction

HDFS high availability helps keep your data safe and accessible even if one part of the system fails. It stops your system from going offline.

When you want your data to be available all the time without interruptions.

When you have many users or applications accessing data and cannot afford downtime.

When you want to avoid losing data if a NameNode server crashes.

When running important business applications that need constant data access.

When you want to improve fault tolerance in your Hadoop cluster.

Syntax

Hadoop

1. Set up two NameNodes: one active and one standby.
2. Configure a shared storage called JournalNode for edit logs.
3. Use ZooKeeper to manage automatic failover between NameNodes.
4. Update hdfs-site.xml with HA settings including dfs.nameservices, dfs.ha.namenodes, dfs.namenode.rpc-address, dfs.namenode.http-address, dfs.namenode.shared.edits.dir, and dfs.client.failover.proxy.provider.
5. Start JournalNodes, ZooKeeper, and both NameNodes.
6. Use hdfs commands to check the status and switch active NameNode if needed.

This setup requires careful configuration of multiple components.

ZooKeeper helps decide which NameNode is active to avoid conflicts.

Examples

This XML config shows how to define two NameNodes and shared edits for HA.

Hadoop

# Example hdfs-site.xml snippet for HA
<configuration>
  <property>
    <name>dfs.nameservices</name>
    <value>mycluster</value>
  </property>
  <property>
    <name>dfs.ha.namenodes.mycluster</name>
    <value>nn1,nn2</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn1</name>
    <value>host1:8020</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn2</name>
    <value>host2:8020</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.mycluster.nn1</name>
    <value>host1:50070</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.mycluster.nn2</name>
    <value>host2:50070</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://host1:8485;host2:8485;host3:8485/mycluster</value>
  </property>
  <property>
    <name>dfs.client.failover.proxy.provider.mycluster</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
</configuration>

This setup has no standby NameNode, so no high availability.

Hadoop

# Edge case: Only one NameNode configured (no HA)
<configuration>
  <property>
    <name>dfs.nameservices</name>
    <value>mycluster</value>
  </property>
  <property>
    <name>dfs.ha.namenodes.mycluster</name>
    <value>nn1</value>
  </property>
</configuration>

If standby fails, active keeps working. This shows fault tolerance.

Hadoop

# Edge case: Standby NameNode is down
# The active NameNode continues serving data.
# Failover can be manual or automatic if ZooKeeper is configured.

Sample Program

This script shows how to check which NameNode is active or standby and how to switch active NameNode manually.

Hadoop

# This is a shell script example to check HDFS HA status

# Check the status of NameNodes
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2

# Manually failover from nn1 to nn2
hdfs haadmin -failover nn1 nn2

# Check status again
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2

Important Notes

Time complexity: HA setup adds some overhead but keeps system responsive.

Space complexity: Requires extra storage for shared edits and standby NameNode.

Common mistake: Not configuring ZooKeeper correctly can cause failover to fail.

Use HA when you need zero downtime. For small clusters, simple single NameNode may be enough.

Summary

HDFS high availability uses two NameNodes to avoid downtime.

Shared storage and ZooKeeper help manage failover automatically.

Proper configuration is key to keep data safe and accessible.

Practice

1. What is the main purpose of HDFS high availability (HA)?

easy

A. To speed up data processing by parallel tasks

B. To avoid downtime by using two active NameNodes

C. To increase storage capacity by adding more DataNodes

D. To encrypt data stored in HDFS automatically

Solution

1. Step 1: Understand HDFS HA concept

   HDFS HA uses two NameNodes to ensure one is always available if the other fails.

2. Step 2: Identify the main goal

   The goal is to avoid downtime, not to increase storage or speed processing.

3. Final Answer:

   To avoid downtime by using two active NameNodes -> Option B

4. Quick Check:

   HDFS HA = Avoid downtime [OK]

Hint: Remember: HA means no downtime with two NameNodes [OK]

Common Mistakes:

• Confusing HA with storage scaling
• Thinking HA speeds up processing
• Assuming HA encrypts data

2. Which component is essential for automatic failover in HDFS high availability?

easy

A. ZooKeeper

B. HDFS DataNode

C. YARN ResourceManager

D. MapReduce JobTracker

Solution

1. Step 1: Identify failover mechanism in HDFS HA

   ZooKeeper coordinates and manages automatic failover between NameNodes.

2. Step 2: Exclude unrelated components

   DataNode stores data, ResourceManager manages resources, JobTracker is for MapReduce jobs, none handle failover.

3. Final Answer:

   ZooKeeper -> Option A

4. Quick Check:

   Failover = ZooKeeper [OK]

Hint: ZooKeeper manages failover in HDFS HA [OK]

Common Mistakes:

• Choosing DataNode as failover manager
• Confusing YARN with failover
• Selecting MapReduce components

3. Given the following simplified HDFS HA configuration snippet, what is the role of dfs.nameservices?

dfs.nameservices=cluster1

medium

A. Lists all DataNodes in the cluster

B. Specifies the IP address of the active NameNode

C. Defines the logical name for the HDFS cluster used by clients

D. Sets the replication factor for data blocks

Solution

1. Step 1: Understand dfs.nameservices meaning

   It defines a logical name for the HDFS cluster, grouping NameNodes under one namespace.

2. Step 2: Eliminate other options

   It does not specify IPs, DataNodes, or replication factors.

3. Final Answer:

   Defines the logical name for the HDFS cluster used by clients -> Option C

4. Quick Check:

   dfs.nameservices = cluster name [OK]

Hint: dfs.nameservices names the cluster logically [OK]

Common Mistakes:

• Thinking it sets IP addresses
• Confusing with DataNode list
• Mixing with replication settings

4. You configured HDFS HA but the failover does not happen automatically. Which configuration is most likely missing?

medium

A. Setting dfs.ha.automatic-failover.enabled to true

B. Increasing the replication factor to 5

C. Adding more DataNodes to the cluster

D. Disabling the NameNode checkpoint

Solution

1. Step 1: Identify failover configuration

   Automatic failover requires dfs.ha.automatic-failover.enabled set to true.

2. Step 2: Exclude unrelated settings

   Replication factor, DataNodes count, and checkpoint settings do not control failover.

3. Final Answer:

   Setting dfs.ha.automatic-failover.enabled to true -> Option A

4. Quick Check:

   Auto failover enabled = true [OK]

Hint: Enable dfs.ha.automatic-failover.enabled for auto failover [OK]

Common Mistakes:

• Changing replication factor to fix failover
• Adding DataNodes expecting failover fix
• Disabling checkpoint thinking it helps failover

5. You want to configure HDFS HA with two NameNodes: nn1 and nn2. Which configuration snippet correctly sets the failover proxy provider for cluster1?

hard

A. dfs.replication=3

B. dfs.nameservices=nn1,nn2

C. dfs.ha.automatic-failover.enabled=false

D. dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

Solution

1. Step 1: Understand failover proxy provider setting

   This property tells the client how to switch between NameNodes during failover.

2. Step 2: Check correctness of options

   dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

   correctly sets the proxy provider for cluster1.

   dfs.nameservices=nn1,nn2

   wrongly lists NameNodes as nameservices.

   dfs.ha.automatic-failover.enabled=false

   disables failover.

   dfs.replication=3

   sets replication, unrelated to failover.

3. Final Answer:

   dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider -> Option D

4. Quick Check:

   Failover proxy provider =

   dfs.client.failover.proxy.provider.cluster1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

   [OK]

Hint: Set failover proxy provider for cluster1 correctly [OK]

Common Mistakes:

• Confusing nameservices with NameNodes list
• Disabling automatic failover by mistake
• Mixing replication with failover settings