kubectl logs for debugging in Kubernetes - Deep Dive

Overview - kubectl logs for debugging

What is it?

kubectl logs is a command used to view the output logs of containers running inside Kubernetes pods. Logs show what the container has printed to its standard output and error streams, which helps understand what the application inside the container is doing. This command is essential for troubleshooting and debugging issues in Kubernetes environments. It allows you to see real-time or past logs to find errors or unexpected behavior.

Why it matters

Without kubectl logs, it would be very hard to know what is happening inside containers running in Kubernetes. You would have no direct way to see error messages or debug information from your applications. This would make fixing problems slow and frustrating, especially in complex systems with many containers. Logs provide the vital clues needed to quickly identify and solve issues, keeping applications reliable and users happy.

Where it fits

Before learning kubectl logs, you should understand basic Kubernetes concepts like pods, containers, and how to use kubectl to interact with the cluster. After mastering logs, you can move on to advanced debugging techniques like using kubectl exec to run commands inside containers or using monitoring tools that collect and analyze logs automatically.

Mental Model

Core Idea

kubectl logs lets you peek inside a running container to see what it is saying, like reading a diary of its recent actions and errors.

Think of it like...

Imagine a container as a worker in a factory who keeps a notebook of everything they do and any problems they face. kubectl logs is like reading that notebook to understand what happened during their shift.

┌───────────────┐
│ Kubernetes Pod│
│ ┌───────────┐ │
│ │ Container │ │
│ │  Logs     │ │
│ └───────────┘ │
└───────┬───────┘
        │
        ▼
┌───────────────────┐
│ kubectl logs       │
│ (shows container  │
│  output)          │
└───────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Pods and Containers

Concept: Learn what pods and containers are in Kubernetes and how they run applications.

A pod is the smallest unit in Kubernetes that can hold one or more containers. Containers are like small packages that run your application code. Each container has its own environment and runs independently inside the pod. Knowing this helps you understand where logs come from.

Result

You know that logs come from containers inside pods, so you can target the right container when debugging.

Understanding the pod-container relationship is key to knowing where logs originate and how to access them.

2

Foundation

Basic kubectl Command Usage

Concept: Learn how to use kubectl to interact with Kubernetes resources.

kubectl is the command-line tool to communicate with Kubernetes clusters. You can list pods with 'kubectl get pods' and describe them with 'kubectl describe pod '. These commands help you find the pod you want to check logs for.

Result

You can find the exact pod name and status before fetching logs.

Knowing how to find pods is essential before you can view their logs.

3

Intermediate

Fetching Logs from a Single Container Pod

🤔Before reading on: do you think 'kubectl logs ' shows logs from all containers in the pod or just one? Commit to your answer.

Concept: Learn how to get logs from a pod with a single container using kubectl logs.

Run 'kubectl logs ' to see the logs from the container inside that pod. This shows the standard output and error streams of the container since it started or restarted.

Result

You see the container's output logs in your terminal, helping you understand what the app is doing.

Knowing the default behavior of kubectl logs helps avoid confusion when pods have multiple containers.

4

Intermediate

Fetching Logs from Multi-Container Pods

🤔Before reading on: do you think kubectl logs can show logs from all containers in a pod at once? Commit to your answer.

Concept: Learn how to specify which container's logs to view in pods with multiple containers.

Use 'kubectl logs -c ' to get logs from a specific container inside a pod. This is necessary because pods can have more than one container, each with its own logs.

Result

You get logs only from the container you specify, avoiding mixed or confusing output.

Understanding container selection prevents mistakes when debugging multi-container pods.

5

Intermediate

Viewing Previous Container Logs After Crash

🤔Before reading on: do you think kubectl logs shows logs from previous container instances by default? Commit to your answer.

Concept: Learn how to see logs from a container that crashed or restarted recently.

Use 'kubectl logs -p' to view logs from the previous container instance before it crashed or restarted. This helps find errors that caused the crash.

Result

You can see error messages or output from the container before it stopped, aiding root cause analysis.

Knowing how to access previous logs is crucial for diagnosing crashes and restarts.

6

Advanced

Streaming Logs in Real-Time

🤔Before reading on: do you think kubectl logs can show live logs as they happen? Commit to your answer.

Concept: Learn how to watch logs live as the container runs using kubectl logs.

Add the '-f' or '--follow' flag: 'kubectl logs -f ' to stream logs in real-time. This is useful when you want to watch what the app is doing as it runs.

Result

You see new log lines appear live in your terminal, like tailing a file.

Streaming logs helps catch issues as they happen and understand app behavior dynamically.

7

Expert

Limitations and Alternatives to kubectl logs

🤔Before reading on: do you think kubectl logs can handle logs from very large or long-running applications efficiently? Commit to your answer.

Concept: Understand when kubectl logs is not enough and what tools to use instead.

kubectl logs only shows logs from the current or previous container instance and can be limited by buffer size or pod lifecycle. For large-scale or long-term logging, use centralized logging solutions like Fluentd, Elasticsearch, and Kibana (EFK stack) or cloud logging services. These collect, store, and analyze logs from many pods over time.

Result

You know when to switch from kubectl logs to professional logging tools for better debugging and monitoring.

Recognizing kubectl logs' limits prevents wasted time and encourages scalable logging practices.

Under the Hood

When you run kubectl logs, the command contacts the Kubernetes API server, which then communicates with the kubelet on the node where the pod runs. The kubelet reads the container's log files stored on the node's filesystem, usually under /var/log/containers or /var/lib/docker/containers, and streams this data back to kubectl. Logs are captured from the container's standard output and error streams, which the container runtime redirects to these files.

Why designed this way?

This design keeps logs decentralized on nodes to avoid a single point of failure and reduce network overhead. It leverages existing container runtime logging mechanisms without adding complexity inside containers. Centralized logging is optional and built on top of this foundation for scalability.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ kubectl CLI   │──────▶│ Kubernetes API│──────▶│ Kubelet on    │
│ (user client) │       │ Server        │       │ Node          │
└───────────────┘       └───────────────┘       └───────────────┘
                                                      │
                                                      ▼
                                         ┌─────────────────────┐
                                         │ Container Log Files  │
                                         │ (/var/log/containers)│
                                         └─────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does 'kubectl logs ' show logs from all containers in the pod by default? Commit yes or no.

Common Belief: kubectl logs shows logs from all containers in a pod automatically.

Reality: kubectl logs without specifying a container only shows logs from the first container in the pod or fails if multiple containers exist.

Why it matters: Assuming all logs are shown can cause you to miss errors in other containers, leading to incomplete debugging.

Quick: Does kubectl logs keep all logs forever? Commit yes or no.

Common Belief: kubectl logs stores and shows all historical logs from containers indefinitely.

Reality: kubectl logs only shows logs from the current and optionally the previous container instance; older logs are lost when containers restart or pods are deleted.

Why it matters: Relying solely on kubectl logs can cause loss of important historical data, making it hard to diagnose intermittent or past issues.

Quick: Can kubectl logs show logs from containers that have never started? Commit yes or no.

Common Belief: kubectl logs can show logs even if the container never started successfully.

Reality: kubectl logs shows logs only if the container has started and produced output; if the container never started, no logs are available.

Why it matters: Expecting logs from failed-to-start containers wastes time and can mislead troubleshooting efforts.

Quick: Does streaming logs with '-f' always show all logs without missing any? Commit yes or no.

Common Belief: Using 'kubectl logs -f' guarantees no log lines are missed during streaming.

Reality: Log streaming can miss lines if the container restarts or if network interruptions occur; it is not a guaranteed complete log stream.

Why it matters: Assuming perfect streaming can cause missed clues during live debugging, leading to incomplete understanding of issues.

Expert Zone

1

kubectl logs reads logs from the node's filesystem, so if the node crashes or logs rotate aggressively, logs may be lost even if the pod is running.

2

When multiple containers write logs simultaneously, their outputs are separate; mixing them requires external tools or manual aggregation.

3

Using the --since and --tail flags with kubectl logs can limit output size and improve performance, but improper use can hide important information.

When NOT to use

kubectl logs is not suitable for long-term log storage, complex querying, or aggregating logs from many pods. In such cases, use centralized logging systems like EFK (Elasticsearch, Fluentd, Kibana), Loki, or cloud logging services that collect and index logs for analysis.

Production Patterns

In production, kubectl logs is mainly used for quick, on-the-spot debugging during development or incident response. Teams often integrate log shipping agents in pods or nodes to forward logs to centralized systems for monitoring, alerting, and historical analysis.

Connections

Centralized Logging Systems

kubectl logs is a simple, direct way to access logs, while centralized logging systems collect and analyze logs at scale.

Understanding kubectl logs helps grasp the foundation of log collection before moving to complex centralized solutions.

Container Runtime Logging

kubectl logs relies on container runtime mechanisms that capture stdout and stderr streams into log files.

Knowing how container runtimes handle logs clarifies why kubectl logs behaves the way it does and its limitations.

Forensic Investigation

Both kubectl logs debugging and forensic investigation involve analyzing recorded evidence to understand past events.

Seeing logs as a form of digital evidence helps appreciate their role in diagnosing and preventing failures.

Common Pitfalls

#1 Trying to get logs from a pod with multiple containers without specifying the container.

Wrong approach:

kubectl logs my-pod

Correct approach:

kubectl logs my-pod -c my-container

Root cause:

Not knowing that kubectl logs defaults to the first container or errors out if multiple containers exist.

#2 Expecting kubectl logs to show logs from a container that crashed and restarted multiple times without using the -p flag.

Wrong approach:

kubectl logs my-pod

Correct approach:

kubectl logs my-pod -p

Root cause:

Not understanding that kubectl logs shows only current container logs unless -p is used for previous instance.

#3 Using kubectl logs to try to analyze logs from pods that have been deleted or evicted.

Wrong approach:

kubectl logs deleted-pod

Correct approach:

Use centralized logging or persistent storage solutions instead.

Root cause:

Assuming kubectl logs can retrieve logs from pods no longer running or present in the cluster.

Key Takeaways

kubectl logs is a simple but powerful tool to see what containers inside pods are outputting, essential for debugging Kubernetes applications.

You must specify the container name when pods have multiple containers to get the correct logs.

Use the -p flag to view logs from previous container instances, especially after crashes or restarts.

Streaming logs with -f lets you watch live output, but it can miss lines if containers restart or network issues occur.

For large-scale, long-term, or complex logging needs, centralized logging systems are necessary beyond kubectl logs.