Node troubleshooting in Kubernetes - Step-by-Step Execution

Process Flow - Node troubleshooting

Detect Node Issue

↓

Check Node Status

↓

Review Node Events

↓

Inspect Node Logs

↓

Check Node Resources

↓

Restart Node Components

↓

Verify Node Recovery

↓

END

This flow shows the steps to find and fix problems on a Kubernetes node, starting from detecting the issue to verifying recovery.

Execution Sample

Kubernetes

kubectl get nodes
kubectl describe node <node-name>
journalctl -u kubelet
kubectl cordon <node-name>
kubectl drain <node-name>
kubectl uncordon <node-name>

Commands to check node status, see details, view logs, and manage node availability during troubleshooting.

Process Table

Step	Command	Action	Output/Result
1	kubectl get nodes	Check overall node status	List of nodes with Ready/NotReady status
2	kubectl describe node node1	View detailed node info and events	Node conditions, recent events showing errors or warnings
3	journalctl -u kubelet	Inspect kubelet logs on node	Logs showing errors or warnings related to node services
4	kubectl cordon node1	Mark node unschedulable	Node marked as unschedulable to prevent new pods
5	kubectl drain node1	Evict pods safely	Pods evicted, node prepared for maintenance
6	systemctl restart kubelet	Restart kubelet service	Kubelet restarted, errors cleared in logs
7	kubectl uncordon node1	Allow scheduling on node	Node marked schedulable again
8	kubectl get nodes	Verify node status	Node status shows Ready
9	-	End troubleshooting	Node is healthy and ready

💡 Node status is Ready, indicating successful troubleshooting and recovery

Status Tracker

Variable	Start	After Step 1	After Step 4	After Step 5	After Step 7	Final
Node Status	Unknown	NotReady or Ready	NotReady (cordoned)	NotReady (drained)	Ready (uncordoned)	Ready
Pods on Node	Running	Running	Running	Evicted	None	Running

Key Moments - 3 Insights

Why do we cordon the node before draining it?

Cordon marks the node unschedulable to stop new pods from being assigned while draining safely evicts existing pods. See execution_table rows 4 and 5.

What does 'kubectl describe node' help us find?

It shows detailed node conditions and recent events that can reveal errors causing node problems. Refer to execution_table row 2.

Why check kubelet logs during troubleshooting?

Kubelet logs contain error messages about node services that help identify the root cause. See execution_table row 3.

Visual Quiz - 3 Questions

Look at the execution_table, what is the node status immediately after step 4 (cordon)?

A Ready

B Drained

C NotReady (cordoned)

D Scheduling Allowed

💡 Hint

Check variable_tracker row 'Node Status' after Step 4

At which step are pods evicted from the node?

A Step 5

B Step 3

C Step 7

D Step 2

💡 Hint

Look at execution_table row describing 'kubectl drain' command

If the node remains NotReady after restarting kubelet, what should you check next?

A Drain the node again

B Node events and logs for errors

C Uncordon the node immediately

D Delete the node from cluster

💡 Hint

Refer to execution_table steps 2 and 3 for checking node events and logs

Concept Snapshot

Node Troubleshooting in Kubernetes:
- Use 'kubectl get nodes' to check node status
- 'kubectl describe node <name>' shows detailed info and events
- Check kubelet logs with 'journalctl -u kubelet'
- Cordon node to stop scheduling: 'kubectl cordon <name>'
- Drain node to evict pods safely: 'kubectl drain <name>'
- After fixes, uncordon node: 'kubectl uncordon <name>'
- Verify node is Ready before resuming workloads

Full Transcript

Node troubleshooting in Kubernetes starts by detecting the issue, usually by checking node status with 'kubectl get nodes'. If a node is NotReady, use 'kubectl describe node' to see detailed conditions and events that might explain the problem. Next, inspect kubelet logs on the node using 'journalctl -u kubelet' to find service errors. To safely fix the node, first cordon it to prevent new pods from scheduling, then drain it to evict existing pods. After restarting or fixing node services like kubelet, uncordon the node to allow scheduling again. Finally, verify the node status is Ready to confirm recovery. This step-by-step approach helps isolate and resolve node issues effectively.

Practice

1. What command shows the current status of all nodes in a Kubernetes cluster?

easy

A. kubectl get nodes

B. kubectl describe pods

C. kubectl get pods

D. kubectl top pods

Solution

1. Step 1: Understand the command purpose

   kubectl get nodes lists all nodes and their status in the cluster.

2. Step 2: Compare with other commands

   Other commands focus on pods, not nodes, so they don't show node status.

3. Final Answer:

   kubectl get nodes -> Option A

4. Quick Check:

   Node status = kubectl get nodes [OK]

Hint: Use 'kubectl get nodes' to see node status quickly [OK]

Common Mistakes:

• Confusing pods with nodes
• Using describe instead of get for quick status
• Trying 'kubectl top pods' for node info

2. Which command syntax correctly shows detailed information about a specific node named node-1?

easy

A. kubectl describe node node-1

B. kubectl get node node-1

C. kubectl get nodes node-1

D. kubectl describe nodes node-1

Solution

1. Step 1: Identify correct command for detailed info

   kubectl describe node node-1 shows detailed info about the node named node-1.

2. Step 2: Check syntax correctness

   Singular 'node' is correct here; plural 'nodes' is invalid for describing a single node. 'get' shows summary, not details.

3. Final Answer:

   kubectl describe node node-1 -> Option A

4. Quick Check:

   Detailed node info = kubectl describe node [OK]

Hint: Use singular 'node' with describe for a specific node [OK]

Common Mistakes:

• Using plural 'nodes' with describe for a single node
• Using 'get' instead of 'describe' for details
• Omitting the node name

3. What is the expected output of the command kubectl top node?

medium

A. A list of pods with their resource requests

B. A list of nodes with CPU and memory usage metrics

C. Detailed node configuration and labels

D. A list of nodes with their IP addresses only

Solution

1. Step 1: Understand the purpose of 'kubectl top node'

   This command shows resource usage like CPU and memory for each node.

2. Step 2: Differentiate from other outputs

   It does not show pod info, detailed config, or just IP addresses.

3. Final Answer:

   A list of nodes with CPU and memory usage metrics -> Option B

4. Quick Check:

   Resource usage per node = kubectl top node [OK]

Hint: Top command shows resource usage, not config or IPs [OK]

Common Mistakes:

• Confusing node metrics with pod metrics
• Expecting detailed config from 'top' command
• Thinking it shows only IP addresses

4. You run kubectl describe node node-2 and see the node is in NotReady state. What is the best first step to troubleshoot?

medium

A. Run kubectl get pods to check pod status

B. Delete the node from the cluster immediately

C. Restart all pods on the node manually

D. Check the node's events section for errors or warnings

Solution

1. Step 1: Review node events for clues

   The events section in the describe output shows recent errors or warnings causing NotReady state.

2. Step 2: Avoid premature actions

   Deleting node or restarting pods without info can cause disruption; checking events is safer first step.

3. Final Answer:

   Check the node's events section for errors or warnings -> Option D

4. Quick Check:

   Check events first when node NotReady [OK]

Hint: Look at node events to find issues first [OK]

Common Mistakes:

• Deleting node without diagnosis
• Restarting pods blindly
• Checking pods instead of node events first

5. A node shows high CPU usage and pods are evicted frequently. Which combined steps help troubleshoot and fix this?

hard

A. Scale down all deployments to zero immediately

B. Delete the node and recreate it to reset CPU usage

C. Use kubectl top node to confirm CPU load, then check pod resource requests and limits

D. Run kubectl describe pod on all pods to find errors

Solution

1. Step 1: Confirm node CPU usage

   Run kubectl top node to verify high CPU load on the node.

2. Step 2: Check pod resource settings

   Review pods' resource requests and limits to see if they are causing CPU overload and evictions.

3. Step 3: Adjust resources or scale pods

   Based on findings, adjust pod resource limits or scale workloads to reduce CPU pressure.

4. Final Answer:

   Use kubectl top node to confirm CPU load, then check pod resource requests and limits -> Option C

5. Quick Check:

   Check CPU usage and pod limits to fix evictions [OK]

Hint: Check node CPU then pod limits to fix evictions [OK]

Common Mistakes:

• Deleting node without analysis
• Scaling down all deployments blindly
• Checking pods errors without resource context