Debugging with kubectl debug in Kubernetes - Deep Dive

Overview - Debugging with kubectl debug

What is it?

kubectl debug is a command in Kubernetes that helps you find and fix problems in your running containers or pods. It lets you create a temporary copy of a pod or add a new container to an existing pod for troubleshooting. This way, you can inspect the environment, run commands, and understand what is going wrong without stopping your application.

Why it matters

Without kubectl debug, fixing issues in Kubernetes pods can be slow and risky because you might have to stop or change running containers directly. This could cause downtime or data loss. kubectl debug provides a safe way to investigate problems live, making it easier to keep applications running smoothly and reduce downtime.

Where it fits

Before learning kubectl debug, you should understand basic Kubernetes concepts like pods, containers, and kubectl commands. After mastering debugging, you can explore advanced troubleshooting tools like logging, monitoring, and tracing in Kubernetes clusters.

Mental Model

Core Idea

kubectl debug creates a safe, temporary environment inside or alongside your running pod to explore and fix issues without disrupting the original workload.

Think of it like...

It's like having a spare car identical to yours that you can take for a test drive to figure out why your main car is acting up, without risking damage to your daily ride.

┌─────────────────────────────┐
│       Original Pod           │
│  ┌───────────────┐          │
│  │ Main Container│          │
│  └───────────────┘          │
│                             │
│  kubectl debug creates:     │
│  ┌───────────────┐          │
│  │ Debug Container│ <───────┤
│  └───────────────┘          │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Pods

Concept: Learn what a pod is and how containers run inside it.

A pod is the smallest unit in Kubernetes that holds one or more containers. Containers inside a pod share the same network and storage. When you run an application, it usually runs inside a pod.

Result

You know that pods are the basic units where your apps run in Kubernetes.

Understanding pods is essential because debugging happens at the pod level, not just containers.

2

Foundation

Basic kubectl Commands

Concept: Learn how to use kubectl to interact with Kubernetes resources.

kubectl is the command-line tool to manage Kubernetes. Commands like 'kubectl get pods' show running pods, and 'kubectl exec' lets you run commands inside a container.

Result

You can list pods and run commands inside containers using kubectl.

Knowing kubectl basics is necessary before using advanced commands like kubectl debug.

3

Intermediate

What kubectl debug Does

🤔Before reading on: do you think kubectl debug modifies your original pod or creates a new environment? Commit to your answer.

Concept: kubectl debug creates a new pod or adds a debug container without changing the original pod's state.

When you run 'kubectl debug', it either clones the pod with debugging tools or adds a new container to the existing pod. This lets you inspect the environment safely.

Result

You get a temporary pod or container where you can run troubleshooting commands.

Knowing that kubectl debug does not alter the original pod prevents accidental disruptions.

4

Intermediate

Using kubectl debug to Clone Pods

🤔Before reading on: do you think cloning a pod copies its current state or just its configuration? Commit to your answer.

Concept: kubectl debug can create a copy of a pod's configuration but not its live state or data.

Example command: kubectl debug pod/myapp-pod --image=busybox --copy-to=debug-myapp This creates a new pod named debug-myapp with the same configuration but a different image for debugging.

Result

A new pod runs with debugging tools, separate from the original pod.

Understanding that cloning copies configuration but not live data helps set expectations for debugging.

5

Intermediate

Adding Debug Containers to Running Pods

🤔Before reading on: do you think adding a debug container requires restarting the pod? Commit to your answer.

Concept: kubectl debug can add a new container to a running pod without restarting it.

Example command: kubectl debug pod/myapp-pod -it --image=busybox --target=myapp-container This adds a debug container to the pod and attaches you to it interactively.

Result

You get a shell inside the debug container running alongside the original container.

Knowing you can add containers live avoids downtime during troubleshooting.

6

Advanced

Debugging Network and Storage Issues

🤔Before reading on: do you think kubectl debug can help inspect network and storage inside pods? Commit to your answer.

Concept: Using debug containers, you can run tools to check network connections and storage mounts inside the pod environment.

Inside the debug container, you can run commands like 'ping', 'nslookup', or 'df -h' to check network and disk status. This helps find connectivity or storage problems.

Result

You identify network or storage issues affecting your application.

Knowing how to use debug containers for environment inspection expands your troubleshooting toolkit.

7

Expert

Limitations and Security Considerations

🤔Before reading on: do you think kubectl debug bypasses all security restrictions on pods? Commit to your answer.

Concept: kubectl debug respects Kubernetes security policies and may be limited by PodSecurityPolicies or RBAC settings.

If your cluster restricts adding containers or running privileged images, kubectl debug might fail or be limited. You must have proper permissions and cluster setup to use it fully.

Result

You understand when kubectl debug can or cannot be used due to security controls.

Knowing security limits prevents confusion and helps plan debugging strategies.

Under the Hood

kubectl debug works by interacting with the Kubernetes API server to create new pods or patch existing pods with additional containers. It uses the pod's specification as a template and injects debugging images or containers. The debug container shares the pod's network and storage namespaces, allowing inspection of the running environment without stopping the original containers.

Why designed this way?

This design allows safe, non-disruptive debugging by isolating troubleshooting tools in separate containers or pods. It avoids modifying live containers directly, which could cause instability. Alternatives like exec into containers risk changing the running state or require pre-installed tools, which kubectl debug overcomes by injecting fresh debug containers.

┌───────────────────────────────┐
│ Kubernetes API Server          │
│                               │
│  ┌───────────────┐            │
│  │ Original Pod  │            │
│  │ ┌───────────┐ │            │
│  │ │ Container │ │            │
│  │ └───────────┘ │            │
│  └───────────────┘            │
│           ▲                   │
│           │ kubectl debug     │
│           │ request           │
│  ┌────────┴────────┐          │
│  │ Debug Container │          │
│  └─────────────────┘          │
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does kubectl debug change the original pod's containers? Commit yes or no.

Common Belief: kubectl debug modifies the original pod's containers directly to add debugging tools.

Reality: kubectl debug creates new pods or adds separate debug containers without changing the original containers.

Why it matters: Believing it modifies original containers can lead to accidental disruptions or downtime.

Quick: Does cloning a pod with kubectl debug copy its live data? Commit yes or no.

Common Belief: Cloning a pod copies its current running state and data exactly.

Reality: Cloning copies only the pod's configuration, not its live memory or data volumes.

Why it matters: Expecting live data in clones can cause confusion when debugging stateful issues.

Quick: Can kubectl debug bypass all Kubernetes security policies? Commit yes or no.

Common Belief: kubectl debug can override any security restrictions to allow debugging.

Reality: kubectl debug respects cluster security policies and requires proper permissions to work.

Why it matters: Assuming it bypasses security can cause failed debugging attempts and wasted time.

Quick: Does adding a debug container require restarting the pod? Commit yes or no.

Common Belief: You must restart the pod to add a debug container.

Reality: kubectl debug can add debug containers live without restarting the pod.

Why it matters: Thinking a restart is needed may delay urgent troubleshooting.

Expert Zone

1

Debug containers share the pod's namespaces, so changes inside debug containers can affect the pod environment, which requires careful handling.

2

kubectl debug can be combined with ephemeral containers, a Kubernetes feature that allows adding temporary containers for debugging without changing pod specs permanently.

3

Some clusters restrict debug container images or require specific security contexts, so understanding cluster policies is crucial for effective debugging.

When NOT to use

kubectl debug is not suitable when debugging requires access to persistent live data that cannot be cloned or when cluster security policies forbid adding debug containers. In such cases, use logging, monitoring tools, or recreate pods with enhanced debugging configurations.

Production Patterns

In production, kubectl debug is used to quickly add ephemeral containers with troubleshooting tools like busybox or curl to live pods. Teams often automate debug container creation in incident response playbooks and combine it with logs and metrics for root cause analysis.

Connections

Ephemeral Containers

kubectl debug uses ephemeral containers as a core feature to add temporary debugging environments.

Understanding ephemeral containers clarifies how kubectl debug can add containers without restarting pods.

Linux Namespaces

kubectl debug containers share Linux namespaces like network and mount with the original pod containers.

Knowing namespaces explains how debug containers can inspect and affect the pod environment safely.

Incident Response in IT Operations

kubectl debug is a tool used during incident response to quickly diagnose live system issues.

Recognizing kubectl debug as part of incident response helps integrate it into broader operational workflows.

Common Pitfalls

#1 Trying to debug a pod without proper permissions.

Wrong approach:

kubectl debug pod/myapp-pod --image=busybox

Correct approach:

Ensure you have RBAC permissions to create ephemeral containers or pods before running kubectl debug.

Root cause:

Lack of understanding of Kubernetes security and RBAC leads to permission errors.

#2 Expecting cloned pods to have the same live data as the original.

Wrong approach:

kubectl debug pod/myapp-pod --copy-to=debug-pod
# Then expecting debug-pod to have current in-memory state

Correct approach:

Use logging or persistent storage to capture live data; understand cloned pods only copy configuration.

Root cause:

Misunderstanding the difference between configuration and runtime state.

#3 Adding debug container but targeting wrong container name.

Wrong approach:

kubectl debug pod/myapp-pod -it --image=busybox --target=wrong-container

Correct approach:

kubectl debug pod/myapp-pod -it --image=busybox --target=myapp-container

Root cause:

Not verifying container names inside pods causes command failures.

Key Takeaways

kubectl debug lets you safely troubleshoot Kubernetes pods by creating temporary debug containers or cloned pods without disrupting the original workload.

It works by adding ephemeral containers that share the pod's environment, allowing inspection of network, storage, and processes live.

Understanding Kubernetes pods, containers, and security policies is essential to use kubectl debug effectively.

kubectl debug respects cluster security and requires proper permissions, so it may not work in all environments without configuration.

Using kubectl debug as part of incident response accelerates problem solving and reduces downtime in production systems.

Practice

1. What is the main purpose of the kubectl debug command in Kubernetes?

easy

A. To delete a pod and recreate it with debug options

B. To permanently change the pod's container image

C. To scale the number of pod replicas for debugging

D. To add a temporary container with debugging tools to a running pod

Solution

1. Step 1: Understand the purpose of kubectl debug

   This command adds a temporary container to a running pod for debugging without changing the original pod.

2. Step 2: Compare other options

   Options A, B, and D describe permanent or unrelated actions, not the temporary debugging feature.

3. Final Answer:

   To add a temporary container with debugging tools to a running pod -> Option D

4. Quick Check:

   Temporary debug container = C [OK]

Hint: Remember: debug adds temporary tools, no permanent pod change [OK]

Common Mistakes:

• Thinking debug changes pod image permanently
• Confusing debug with pod scaling
• Assuming debug deletes pods

2. Which of the following is the correct syntax to start a debug session on a pod named webapp-123?

easy

A. kubectl debug --pod webapp-123 --image=busybox

B. kubectl debug webapp-123 -image=busybox

C. kubectl debug pod/webapp-123 --image=busybox

D. kubectl debug pod webapp-123 --image=busybox

Solution

1. Step 1: Recall correct kubectl debug syntax

   The correct syntax uses kubectl debug pod/NAME --image=IMAGE to specify the pod and debug container image.

2. Step 2: Analyze options

   kubectl debug pod/webapp-123 --image=busybox matches the correct syntax exactly. Options A, B, and C have syntax errors or missing slashes.

3. Final Answer:

   kubectl debug pod/webapp-123 --image=busybox -> Option C

4. Quick Check:

   Correct syntax includes 'pod/' prefix = D [OK]

Hint: Use 'pod/NAME' format with --image option for debug [OK]

Common Mistakes:

• Omitting 'pod/' prefix before pod name
• Using spaces instead of slash between 'pod' and name
• Misplacing --image option

3. What will be the output of this command if the pod api-server-1 does not exist?

kubectl debug pod/api-server-1 --image=busybox

medium

A. Error: pods "api-server-1" not found

B. A new debug container starts successfully

C. Pod is deleted and recreated with debug image

D. Command runs but no debug container is added

Solution

1. Step 1: Understand pod existence requirement

   The kubectl debug command requires the target pod to exist to add a debug container.

2. Step 2: Predict command behavior if pod missing

   If the pod does not exist, kubectl returns an error indicating the pod was not found.

3. Final Answer:

   Error: pods "api-server-1" not found -> Option A

4. Quick Check:

   Missing pod causes 'not found' error = A [OK]

Hint: Check pod exists before debug; else 'not found' error [OK]

Common Mistakes:

• Assuming debug creates pod if missing
• Expecting debug to run silently without pod
• Confusing debug with pod creation commands

4. You run kubectl debug pod/myapp --image=busybox but get an error: error: container busybox not found. What is the likely cause?

medium

A. The debug container image is missing or misspelled

B. The pod does not exist

C. The pod already has a container named busybox

D. kubectl debug requires root privileges

Solution

1. Step 1: Analyze the error message

   The error says 'container busybox not found', which usually means the image name is incorrect or missing.

2. Step 2: Check common causes

   The debug container image is missing or misspelled fits because a misspelled or missing image causes this error. The pod existence or privileges are unrelated.

3. Final Answer:

   The debug container image is missing or misspelled -> Option A

4. Quick Check:

   Image name errors cause 'container not found' message [OK]

Hint: Verify debug image name spelling and availability [OK]

Common Mistakes:

• Assuming pod absence causes this error
• Thinking debug needs root privileges
• Confusing container name with image name

5. You want to debug a running pod backend-0 but need to run a shell with root privileges inside the debug container. Which command correctly achieves this?

hard

A. kubectl debug pod/backend-0 --image=busybox --target=backend-0 -- sh

B. kubectl debug pod/backend-0 --image=busybox --privileged -- sh

C. kubectl debug pod/backend-0 --image=busybox --container=debugger -- sh

D. kubectl debug pod/backend-0 --image=busybox --as-root -- sh

Solution

1. Step 1: Identify how to run debug container with root

   The --privileged flag allows the debug container to run with root privileges.

2. Step 2: Check command correctness

   kubectl debug pod/backend-0 --image=busybox --privileged -- sh uses --privileged and runs shell sh, which is correct. Other options misuse flags or do not enable root.

3. Final Answer:

   kubectl debug pod/backend-0 --image=busybox --privileged -- sh -> Option B

4. Quick Check:

   Use --privileged to get root shell in debug container [OK]

Hint: Add --privileged flag to run debug container as root [OK]

Common Mistakes:

• Using nonexistent flags like --as-root
• Confusing --target with privilege escalation
• Omitting --privileged when root shell needed