Node troubleshooting in Kubernetes - Mini Project: Build & Apply

Node troubleshooting in Kubernetes

📖 Scenario: You are a Kubernetes administrator. One of the worker nodes in your cluster is not ready. You need to check the node status, describe the node to find issues, and then get logs from the kubelet service to understand the problem.

🎯 Goal: Learn how to check node status, describe a node, and fetch kubelet logs to troubleshoot node issues in Kubernetes.

📋 What You'll Learn

Use kubectl get nodes to check node status

Use kubectl describe node to get detailed node info

Use journalctl -u kubelet to view kubelet logs on the node

💡 Why This Matters

🌍 Real World

Kubernetes clusters often have nodes that become NotReady due to network issues, resource exhaustion, or kubelet failures. Troubleshooting nodes quickly helps keep applications running smoothly.

💼 Career

Kubernetes administrators and DevOps engineers must know how to check node health and diagnose node problems to maintain cluster stability and performance.

1

Check the status of all nodes

Run the command kubectl get nodes to list all nodes and their status in the cluster.

Kubernetes

# Run kubectl get nodes to see node status
# Your code here

Hint

This command shows all nodes and their current status like Ready or NotReady.

2

Describe the problematic node

Use kubectl describe node replacing <node_name> with the exact name of the node that is NotReady to get detailed information about it.

Kubernetes

kubectl get nodes
# Use kubectl describe node  to get detailed info about the NotReady node
# Your code here

Hint

Look for the node name exactly as shown in the first command output.

3

Check kubelet logs on the node

On the problematic node, run journalctl -u kubelet to view the kubelet service logs and find any errors or warnings.

Kubernetes

kubectl get nodes
kubectl describe node worker-node-1
# Run journalctl -u kubelet on the node to check kubelet logs
# Your code here

Hint

This command shows logs from the kubelet service which manages the node.

4

Display the first 20 lines of kubelet logs

Run journalctl -u kubelet -n 20 to display only the last 20 lines of the kubelet logs for a quick overview.

Kubernetes

kubectl get nodes
kubectl describe node worker-node-1
journalctl -u kubelet
# Run journalctl -u kubelet -n 20 to show last 20 lines of logs
# Your code here

Hint

This command limits the output to the most recent 20 log lines for easier reading.

Practice

1. What command shows the current status of all nodes in a Kubernetes cluster?

easy

A. kubectl get nodes

B. kubectl describe pods

C. kubectl get pods

D. kubectl top pods

Solution

1. Step 1: Understand the command purpose

   kubectl get nodes lists all nodes and their status in the cluster.

2. Step 2: Compare with other commands

   Other commands focus on pods, not nodes, so they don't show node status.

3. Final Answer:

   kubectl get nodes -> Option A

4. Quick Check:

   Node status = kubectl get nodes [OK]

Hint: Use 'kubectl get nodes' to see node status quickly [OK]

Common Mistakes:

• Confusing pods with nodes
• Using describe instead of get for quick status
• Trying 'kubectl top pods' for node info

2. Which command syntax correctly shows detailed information about a specific node named node-1?

easy

A. kubectl describe node node-1

B. kubectl get node node-1

C. kubectl get nodes node-1

D. kubectl describe nodes node-1

Solution

1. Step 1: Identify correct command for detailed info

   kubectl describe node node-1 shows detailed info about the node named node-1.

2. Step 2: Check syntax correctness

   Singular 'node' is correct here; plural 'nodes' is invalid for describing a single node. 'get' shows summary, not details.

3. Final Answer:

   kubectl describe node node-1 -> Option A

4. Quick Check:

   Detailed node info = kubectl describe node [OK]

Hint: Use singular 'node' with describe for a specific node [OK]

Common Mistakes:

• Using plural 'nodes' with describe for a single node
• Using 'get' instead of 'describe' for details
• Omitting the node name

3. What is the expected output of the command kubectl top node?

medium

A. A list of pods with their resource requests

B. A list of nodes with CPU and memory usage metrics

C. Detailed node configuration and labels

D. A list of nodes with their IP addresses only

Solution

1. Step 1: Understand the purpose of 'kubectl top node'

   This command shows resource usage like CPU and memory for each node.

2. Step 2: Differentiate from other outputs

   It does not show pod info, detailed config, or just IP addresses.

3. Final Answer:

   A list of nodes with CPU and memory usage metrics -> Option B

4. Quick Check:

   Resource usage per node = kubectl top node [OK]

Hint: Top command shows resource usage, not config or IPs [OK]

Common Mistakes:

• Confusing node metrics with pod metrics
• Expecting detailed config from 'top' command
• Thinking it shows only IP addresses

4. You run kubectl describe node node-2 and see the node is in NotReady state. What is the best first step to troubleshoot?

medium

A. Run kubectl get pods to check pod status

B. Delete the node from the cluster immediately

C. Restart all pods on the node manually

D. Check the node's events section for errors or warnings

Solution

1. Step 1: Review node events for clues

   The events section in the describe output shows recent errors or warnings causing NotReady state.

2. Step 2: Avoid premature actions

   Deleting node or restarting pods without info can cause disruption; checking events is safer first step.

3. Final Answer:

   Check the node's events section for errors or warnings -> Option D

4. Quick Check:

   Check events first when node NotReady [OK]

Hint: Look at node events to find issues first [OK]

Common Mistakes:

• Deleting node without diagnosis
• Restarting pods blindly
• Checking pods instead of node events first

5. A node shows high CPU usage and pods are evicted frequently. Which combined steps help troubleshoot and fix this?

hard

A. Scale down all deployments to zero immediately

B. Delete the node and recreate it to reset CPU usage

C. Use kubectl top node to confirm CPU load, then check pod resource requests and limits

D. Run kubectl describe pod on all pods to find errors

Solution

1. Step 1: Confirm node CPU usage

   Run kubectl top node to verify high CPU load on the node.

2. Step 2: Check pod resource settings

   Review pods' resource requests and limits to see if they are causing CPU overload and evictions.

3. Step 3: Adjust resources or scale pods

   Based on findings, adjust pod resource limits or scale workloads to reduce CPU pressure.

4. Final Answer:

   Use kubectl top node to confirm CPU load, then check pod resource requests and limits -> Option C

5. Quick Check:

   Check CPU usage and pod limits to fix evictions [OK]

Hint: Check node CPU then pod limits to fix evictions [OK]

Common Mistakes:

• Deleting node without analysis
• Scaling down all deployments blindly
• Checking pods errors without resource context