Node decommissioning and scaling in Hadoop - Step-by-Step Execution

Concept Flow - Node decommissioning and scaling

Start Decommission Request

↓

Mark Node as Decommissioning

↓

Rebalance Data Blocks

↓

Wait for Data Replication

↓

Confirm No Active Tasks

↓

Remove Node from Cluster

↓

Update Cluster State

↓

Scaling: Add or Remove Nodes

↓

Rebalance Data and Tasks

↓

End

The flow shows how a node is safely removed by replicating data and stopping tasks, then how scaling adjusts cluster size and rebalances.

Execution Sample

Hadoop

1. Mark node as decommissioning
2. Replicate data blocks to other nodes
3. Wait until replication completes
4. Confirm no active tasks on node
5. Remove node from cluster
6. Update cluster metadata
7. Add new nodes if scaling up

This sequence safely removes a node and optionally adds nodes to scale the cluster.

Execution Table

Step	Action	Node State	Data Blocks Status	Cluster Tasks	Cluster Size
1	Mark node as decommissioning	Decommissioning	Original on node	Running	5
2	Start replicating data blocks	Decommissioning	Replicating	Running	5
3	Wait for replication to complete	Decommissioning	Replicated elsewhere	Running	5
4	Confirm no active tasks on node	Decommissioning	Replicated elsewhere	No tasks on node	5
5	Remove node from cluster	Removed	Data safe	Running	4
6	Update cluster metadata	Removed	Data safe	Running	4
7	Add new node for scaling	Active	New node empty	Running	5
8	Rebalance data and tasks	Active	Balanced	Running	5
9	End	Active	Balanced	Running	5

💡 Node removed after data replication and task migration; cluster scaled by adding new node and rebalancing.

Variable Tracker

Variable	Start	After Step 3	After Step 5	After Step 7	Final
Node State	Active	Decommissioning	Removed	Active (new node)	Active
Data Blocks Status	Original on node	Replicated elsewhere	Data safe	New node empty	Balanced
Cluster Size	5	5	4	5	5
Cluster Tasks	Running	Running	Running	Running	Running

Key Moments - 3 Insights

Why do we wait for data replication before removing the node?

Because the data blocks must be safely copied to other nodes to avoid data loss, as shown in steps 2 and 3 of the execution_table.

What happens to cluster size during decommissioning and scaling?

Cluster size decreases by one when the node is removed (step 5) and increases when a new node is added (step 7), as tracked in the variable_tracker.

Why do we check for active tasks before removing the node?

To ensure no running jobs are interrupted, which is confirmed in step 4 where cluster tasks on the node become zero.

Visual Quiz - 3 Questions

Look at the execution_table, what is the node state after step 5?

A Removed

B Decommissioning

C Active

D Unknown

💡 Hint

Check the 'Node State' column at step 5 in the execution_table.

At which step does the cluster size decrease?

A Step 3

B Step 5

C Step 7

D Step 9

💡 Hint

Look at the 'Cluster Size' column in the execution_table and find where it changes from 5 to 4.

If data replication is not complete, what should happen next?

A Remove node immediately

B Add new nodes

C Wait until replication completes

D Stop cluster tasks

💡 Hint

Refer to steps 2 and 3 in the execution_table where replication status is 'Replicating' and 'Replicated elsewhere'.

Concept Snapshot

Node decommissioning safely removes a node by replicating data and migrating tasks.
Steps: mark node, replicate data, wait, remove node, update cluster.
Scaling adjusts cluster size by adding/removing nodes and rebalancing.
Always ensure data safety and no active tasks before removal.
Cluster metadata updates keep cluster state consistent.

Full Transcript

Node decommissioning and scaling in Hadoop involves marking a node for removal, replicating its data blocks to other nodes to prevent data loss, waiting for replication to finish, ensuring no active tasks run on the node, then removing it from the cluster and updating cluster metadata. Scaling means adding or removing nodes and rebalancing data and tasks to maintain cluster health. The process ensures data safety and continuous operation.