Node decommissioning and scaling in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to start the decommissioning process for a node in Hadoop.

Hadoop

hdfs dfsadmin -[1] <node-hostname>

Drag options to blanks, or click blank then click option'

Adecommission

BrefreshNodes

CstartDecommission

DstopNode

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'decommission' instead of 'refreshNodes' flag.

Trying to decommission without updating the exclude file.

The command 'hdfs dfsadmin -refreshNodes' tells Hadoop to reload the list of nodes to decommission.

2fill in blank

medium

Complete the code to add a new node to the Hadoop cluster by updating the include file.

Hadoop

echo '[1]' >> /etc/hadoop/conf/dfs.include

Drag options to blanks, or click blank then click option'

Anew-node-hostname

Bnamenode-host

Cold-node-hostname

Ddatanode-host

Attempts:

3 left

💡 Hint

Common Mistakes

Adding the namenode hostname instead of the new node.

Editing the exclude file instead of the include file.

You add the new node's hostname to the dfs.include file to allow it to join the cluster.

3fill in blank

hard

Fix the error in the command to check the decommission status of nodes.

Hadoop

hdfs dfsadmin -[1]

Drag options to blanks, or click blank then click option'

Acheck

Bstatus

Creport

Dlist

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'status' or 'check' which are not valid dfsadmin flags.

Trying to list nodes with an incorrect flag.

The 'hdfs dfsadmin -report' command shows the status of all nodes including decommissioning progress.

4fill in blank

hard

Fill both blanks to update the exclude file and refresh nodes to decommission a node.

Hadoop

echo '[1]' >> /etc/hadoop/conf/dfs.exclude
hdfs dfsadmin -[2]

Drag options to blanks, or click blank then click option'

Anode-to-remove-hostname

BrefreshNodes

CdecommissionNode

DstopNode

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong flags like 'decommissionNode' instead of 'refreshNodes'.

Not updating the exclude file before refreshing nodes.

You add the node hostname to dfs.exclude and then run 'hdfs dfsadmin -refreshNodes' to start decommissioning.

5fill in blank

hard

Fill all three blanks to create a new node, add it to the include file, and refresh nodes to scale the cluster.

Hadoop

ssh [1] 'sudo systemctl start hadoop-datanode'
echo '[2]' >> /etc/hadoop/conf/dfs.include
hdfs dfsadmin -[3]

Drag options to blanks, or click blank then click option'

Anew-node-hostname

CrefreshNodes

DstartNode

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'startNode' instead of 'refreshNodes' to apply changes.

Not starting the datanode service before adding to include file.

You start the datanode service on the new node, add it to dfs.include, then refresh nodes to scale.

Practice

1. What is the main purpose of node decommissioning in a Hadoop cluster?

easy

A. To safely remove a node without losing data

B. To add more nodes to the cluster

C. To restart all nodes simultaneously

D. To backup data from the cluster

Solution

1. Step 1: Understand node decommissioning

   Node decommissioning is the process of removing a node from the cluster safely.

2. Step 2: Identify the purpose

   The goal is to ensure no data loss while removing the node.

3. Final Answer:

   To safely remove a node without losing data -> Option A

4. Quick Check:

   Node decommissioning = safe node removal [OK]

Hint: Decommission means safe removal without data loss [OK]

Common Mistakes:

• Confusing decommissioning with adding nodes
• Thinking it restarts nodes
• Assuming it backs up data

2. Which command is used to refresh the list of nodes after updating the exclude file for decommissioning?

easy

A. hdfs namenode -format

B. hdfs dfsadmin -refreshNodes

C. hdfs datanode -report

D. hdfs fsck /

Solution

1. Step 1: Identify the command for refreshing nodes

   After updating the exclude file, the cluster needs to refresh node status.

2. Step 2: Match the correct command

   The command hdfs dfsadmin -refreshNodes tells the NameNode to reload the exclude file.

3. Final Answer:

   hdfs dfsadmin -refreshNodes -> Option B

4. Quick Check:

   Refresh nodes command = dfsadmin -refreshNodes [OK]

Hint: Refresh nodes after exclude file update with dfsadmin -refreshNodes [OK]

Common Mistakes:

• Using namenode -format which erases metadata
• Confusing datanode -report with refresh
• Running fsck which checks file system health

3. Given the following exclude file content:
node1.example.com node3.example.com
and the cluster nodes are node1, node2, node3, node4.
After running hdfs dfsadmin -refreshNodes, which nodes will be decommissioned?

medium

A. All nodes

B. node2 and node4

C. node1, node2, node3

D. node1 and node3

Solution

1. Step 1: Understand exclude file role

   The exclude file lists nodes to be decommissioned.

2. Step 2: Identify nodes in exclude file

   Nodes node1.example.com and node3.example.com are listed, so these two will be decommissioned.

3. Final Answer:

   node1 and node3 -> Option D

4. Quick Check:

   Exclude file nodes = decommissioned nodes [OK]

Hint: Exclude file lists nodes to decommission [OK]

Common Mistakes:

• Assuming nodes not in exclude file are decommissioned
• Thinking all nodes are decommissioned
• Confusing node hostnames

4. You updated the exclude file to remove a node but forgot to run hdfs dfsadmin -refreshNodes. What is the likely result?

medium

A. The node remains active and is not decommissioned

B. The node is immediately decommissioned

C. The cluster shuts down

D. The exclude file is ignored and deleted

Solution

1. Step 1: Understand refreshNodes role

   The refreshNodes command tells the NameNode to reload the exclude file.

2. Step 2: Consequence of not running refresh

   If refreshNodes is not run, the NameNode does not know about the updated exclude file, so the node stays active.

3. Final Answer:

   The node remains active and is not decommissioned -> Option A

4. Quick Check:

   No refreshNodes = no decommission [OK]

Hint: Always run refreshNodes after exclude file update [OK]

Common Mistakes:

• Assuming node decommissions automatically
• Thinking cluster shuts down
• Believing exclude file deletes itself

5. You want to scale your Hadoop cluster by adding 3 new nodes and safely remove 2 old nodes. Which sequence of actions is correct?

hard

A. Add new nodes, restart cluster, update exclude file

B. Add new nodes, update exclude file with old nodes, run hdfs dfsadmin -refreshNodes

C. Update exclude file with old nodes, run hdfs dfsadmin -refreshNodes, add new nodes

D. Remove old nodes physically, then add new nodes

Solution

1. Step 1: Decommission old nodes first

   Update the exclude file with old nodes to remove and run refreshNodes to start decommissioning safely.

2. Step 2: Add new nodes after old nodes are decommissioned

   Once old nodes are safely removed, add new nodes to scale the cluster.

3. Final Answer:

   Update exclude file with old nodes, run hdfs dfsadmin -refreshNodes, add new nodes -> Option C

4. Quick Check:

   Decommission old nodes before adding new ones [OK]

Hint: Decommission old nodes before adding new ones [OK]

Common Mistakes:

• Adding new nodes before decommissioning old ones
• Physically removing nodes without decommissioning
• Restarting cluster unnecessarily