Node decommissioning and scaling in Hadoop - Step-by-Step Execution

Concept Flow - Node decommissioning and scaling

Start Decommission Request

↓

Mark Node as Decommissioning

↓

Rebalance Data Blocks

↓

Wait for Data Replication

↓

Confirm No Active Tasks

↓

Remove Node from Cluster

↓

Update Cluster State

↓

Scaling: Add or Remove Nodes

↓

Rebalance Data and Tasks

↓

End

The flow shows how a node is safely removed by replicating data and stopping tasks, then how scaling adjusts cluster size and rebalances.

Execution Sample

Hadoop

1. Mark node as decommissioning
2. Replicate data blocks to other nodes
3. Wait until replication completes
4. Confirm no active tasks on node
5. Remove node from cluster
6. Update cluster metadata
7. Add new nodes if scaling up

This sequence safely removes a node and optionally adds nodes to scale the cluster.

Execution Table

Step	Action	Node State	Data Blocks Status	Cluster Tasks	Cluster Size
1	Mark node as decommissioning	Decommissioning	Original on node	Running	5
2	Start replicating data blocks	Decommissioning	Replicating	Running	5
3	Wait for replication to complete	Decommissioning	Replicated elsewhere	Running	5
4	Confirm no active tasks on node	Decommissioning	Replicated elsewhere	No tasks on node	5
5	Remove node from cluster	Removed	Data safe	Running	4
6	Update cluster metadata	Removed	Data safe	Running	4
7	Add new node for scaling	Active	New node empty	Running	5
8	Rebalance data and tasks	Active	Balanced	Running	5
9	End	Active	Balanced	Running	5

💡 Node removed after data replication and task migration; cluster scaled by adding new node and rebalancing.

Variable Tracker

Variable	Start	After Step 3	After Step 5	After Step 7	Final
Node State	Active	Decommissioning	Removed	Active (new node)	Active
Data Blocks Status	Original on node	Replicated elsewhere	Data safe	New node empty	Balanced
Cluster Size	5	5	4	5	5
Cluster Tasks	Running	Running	Running	Running	Running

Key Moments - 3 Insights

Why do we wait for data replication before removing the node?

Because the data blocks must be safely copied to other nodes to avoid data loss, as shown in steps 2 and 3 of the execution_table.

What happens to cluster size during decommissioning and scaling?

Cluster size decreases by one when the node is removed (step 5) and increases when a new node is added (step 7), as tracked in the variable_tracker.

Why do we check for active tasks before removing the node?

To ensure no running jobs are interrupted, which is confirmed in step 4 where cluster tasks on the node become zero.

Visual Quiz - 3 Questions

Look at the execution_table, what is the node state after step 5?

A Removed

B Decommissioning

C Active

D Unknown

💡 Hint

Check the 'Node State' column at step 5 in the execution_table.

At which step does the cluster size decrease?

A Step 3

B Step 5

C Step 7

D Step 9

💡 Hint

Look at the 'Cluster Size' column in the execution_table and find where it changes from 5 to 4.

If data replication is not complete, what should happen next?

A Remove node immediately

B Add new nodes

C Wait until replication completes

D Stop cluster tasks

💡 Hint

Refer to steps 2 and 3 in the execution_table where replication status is 'Replicating' and 'Replicated elsewhere'.

Concept Snapshot

Node decommissioning safely removes a node by replicating data and migrating tasks.
Steps: mark node, replicate data, wait, remove node, update cluster.
Scaling adjusts cluster size by adding/removing nodes and rebalancing.
Always ensure data safety and no active tasks before removal.
Cluster metadata updates keep cluster state consistent.

Full Transcript

Node decommissioning and scaling in Hadoop involves marking a node for removal, replicating its data blocks to other nodes to prevent data loss, waiting for replication to finish, ensuring no active tasks run on the node, then removing it from the cluster and updating cluster metadata. Scaling means adding or removing nodes and rebalancing data and tasks to maintain cluster health. The process ensures data safety and continuous operation.

Practice

1. What is the main purpose of node decommissioning in a Hadoop cluster?

easy

A. To safely remove a node without losing data

B. To add more nodes to the cluster

C. To restart all nodes simultaneously

D. To backup data from the cluster

Solution

1. Step 1: Understand node decommissioning

   Node decommissioning is the process of removing a node from the cluster safely.

2. Step 2: Identify the purpose

   The goal is to ensure no data loss while removing the node.

3. Final Answer:

   To safely remove a node without losing data -> Option A

4. Quick Check:

   Node decommissioning = safe node removal [OK]

Hint: Decommission means safe removal without data loss [OK]

Common Mistakes:

• Confusing decommissioning with adding nodes
• Thinking it restarts nodes
• Assuming it backs up data

2. Which command is used to refresh the list of nodes after updating the exclude file for decommissioning?

easy

A. hdfs namenode -format

B. hdfs dfsadmin -refreshNodes

C. hdfs datanode -report

D. hdfs fsck /

Solution

1. Step 1: Identify the command for refreshing nodes

   After updating the exclude file, the cluster needs to refresh node status.

2. Step 2: Match the correct command

   The command hdfs dfsadmin -refreshNodes tells the NameNode to reload the exclude file.

3. Final Answer:

   hdfs dfsadmin -refreshNodes -> Option B

4. Quick Check:

   Refresh nodes command = dfsadmin -refreshNodes [OK]

Hint: Refresh nodes after exclude file update with dfsadmin -refreshNodes [OK]

Common Mistakes:

• Using namenode -format which erases metadata
• Confusing datanode -report with refresh
• Running fsck which checks file system health

3. Given the following exclude file content:
node1.example.com node3.example.com
and the cluster nodes are node1, node2, node3, node4.
After running hdfs dfsadmin -refreshNodes, which nodes will be decommissioned?

medium

A. All nodes

B. node2 and node4

C. node1, node2, node3

D. node1 and node3

Solution

1. Step 1: Understand exclude file role

   The exclude file lists nodes to be decommissioned.

2. Step 2: Identify nodes in exclude file

   Nodes node1.example.com and node3.example.com are listed, so these two will be decommissioned.

3. Final Answer:

   node1 and node3 -> Option D

4. Quick Check:

   Exclude file nodes = decommissioned nodes [OK]

Hint: Exclude file lists nodes to decommission [OK]

Common Mistakes:

• Assuming nodes not in exclude file are decommissioned
• Thinking all nodes are decommissioned
• Confusing node hostnames

4. You updated the exclude file to remove a node but forgot to run hdfs dfsadmin -refreshNodes. What is the likely result?

medium

A. The node remains active and is not decommissioned

B. The node is immediately decommissioned

C. The cluster shuts down

D. The exclude file is ignored and deleted

Solution

1. Step 1: Understand refreshNodes role

   The refreshNodes command tells the NameNode to reload the exclude file.

2. Step 2: Consequence of not running refresh

   If refreshNodes is not run, the NameNode does not know about the updated exclude file, so the node stays active.

3. Final Answer:

   The node remains active and is not decommissioned -> Option A

4. Quick Check:

   No refreshNodes = no decommission [OK]

Hint: Always run refreshNodes after exclude file update [OK]

Common Mistakes:

• Assuming node decommissions automatically
• Thinking cluster shuts down
• Believing exclude file deletes itself

5. You want to scale your Hadoop cluster by adding 3 new nodes and safely remove 2 old nodes. Which sequence of actions is correct?

hard

A. Add new nodes, restart cluster, update exclude file

B. Add new nodes, update exclude file with old nodes, run hdfs dfsadmin -refreshNodes

C. Update exclude file with old nodes, run hdfs dfsadmin -refreshNodes, add new nodes

D. Remove old nodes physically, then add new nodes

Solution

1. Step 1: Decommission old nodes first

   Update the exclude file with old nodes to remove and run refreshNodes to start decommissioning safely.

2. Step 2: Add new nodes after old nodes are decommissioned

   Once old nodes are safely removed, add new nodes to scale the cluster.

3. Final Answer:

   Update exclude file with old nodes, run hdfs dfsadmin -refreshNodes, add new nodes -> Option C

4. Quick Check:

   Decommission old nodes before adding new ones [OK]

Hint: Decommission old nodes before adding new ones [OK]

Common Mistakes:

• Adding new nodes before decommissioning old ones
• Physically removing nodes without decommissioning
• Restarting cluster unnecessarily