Node decommissioning and scaling in Hadoop

Introduction

Node decommissioning helps safely remove a server from a Hadoop cluster without losing data. Scaling adds or removes nodes to match the workload.

When a server needs maintenance or replacement without stopping the cluster

When the data or workload grows and you need more storage or processing power

When you want to reduce costs by removing unused servers

When balancing the cluster to improve performance

When upgrading hardware or software on cluster nodes

Syntax

Hadoop

1. Add nodes to the cluster by updating configuration files and starting services.
2. To decommission a node:
   - Add the node's hostname to the exclude file (e.g., dfs.exclude).
   - Refresh the NameNode with: hdfs dfsadmin -refreshNodes
   - Wait for data to replicate off the node.
   - Stop the node safely after decommission completes.

The exclude file lists nodes to remove safely.

Refreshing nodes tells Hadoop to start decommissioning listed nodes.

Examples

This file lists nodes to be decommissioned.

Hadoop

# Example exclude file content
node1.example.com
node2.example.com

This command tells Hadoop to start decommissioning nodes in the exclude file.

Hadoop

hdfs dfsadmin -refreshNodes

Steps to scale up by adding a new node.

Hadoop

# Adding a new node
# Update configuration with new node hostname
# Start DataNode service on new node

Sample Program

This script shows how to safely remove a node from a Hadoop cluster.

Hadoop

# This is a shell script example to decommission a node
# Step 1: Add node to exclude file
 echo 'node3.example.com' >> /etc/hadoop/conf/dfs.exclude

# Step 2: Refresh nodes
 hdfs dfsadmin -refreshNodes

# Step 3: Check decommission status
 hdfs dfsadmin -report

# Step 4: After decommission completes, stop DataNode on node3
 ssh node3.example.com 'sudo systemctl stop hadoop-hdfs-datanode'

# Step 5: Remove node3 from cluster configuration if desired

Important Notes

Decommissioning can take time depending on data size to replicate.

Always monitor cluster health during scaling or decommissioning.

Scaling down requires careful planning to avoid data loss.

Summary

Node decommissioning safely removes nodes without data loss.

Scaling adjusts cluster size to workload needs.

Use exclude files and refresh commands to manage nodes.