Docker logs for troubleshooting - Deep Dive

Overview - Docker logs for troubleshooting

What is it?

Docker logs are the recorded output messages from running containers. They capture what a containerized application prints to its standard output and error streams. These logs help you see what is happening inside a container, especially when something goes wrong. By reading these logs, you can understand errors, warnings, or normal operation messages.

Why it matters

Without Docker logs, troubleshooting container problems would be like trying to fix a broken machine without seeing or hearing what is wrong. Logs provide the clues needed to identify issues quickly and fix them. Without logs, developers and operators would waste time guessing, leading to longer downtimes and frustrated users.

Where it fits

Before learning Docker logs, you should understand basic Docker concepts like containers and images. After mastering logs, you can explore advanced monitoring, centralized logging systems, and alerting tools that build on logs to keep applications healthy.

Mental Model

Core Idea

Docker logs are the window into a container’s real-time and past behavior, showing what the container is doing or what errors it encounters.

Think of it like...

Docker logs are like the dashboard lights and sounds in a car that tell you if the engine is running well or if something needs attention.

┌───────────────┐
│ Docker Engine │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Running       │
│ Container    │
│ (App inside)  │
└──────┬────────┘
       │ stdout/stderr
       ▼
┌───────────────┐
│ Docker Logs   │
│ (Captured    │
│ Output)      │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What Are Docker Logs

Concept: Introduce what Docker logs are and where they come from.

Docker containers run applications that print messages to the screen. These messages include normal information and error messages. Docker captures these messages as logs, which you can view anytime to understand what the container did or is doing.

Result

You understand that Docker logs are the text output from containers, saved by Docker for review.

Knowing that logs come from the container’s output streams helps you realize logs are the first place to check when something goes wrong.

2

Foundation

Basic Command to View Logs

Concept: Learn the main Docker command to see logs from a container.

Use the command: docker logs This shows all the logs the container has produced so far. For example: $ docker logs myapp [INFO] Server started [ERROR] Failed to connect to database This command reads the stored logs and prints them on your screen.

Result

You can run a simple command to see what a container has printed so far.

Knowing the exact command to fetch logs is the first practical step to troubleshooting.

3

Intermediate

Following Logs in Real-Time

🤔Before reading on: do you think 'docker logs -f' shows past logs or updates live? Commit to your answer.

Concept: Learn how to watch logs as they happen, not just past logs.

Add the -f option to the logs command to follow logs live: docker logs -f This keeps the terminal open and shows new log lines as the container produces them, like watching a live feed.

Result

You can monitor a container’s activity live, which helps catch errors as they occur.

Understanding live log streaming lets you react immediately to problems instead of waiting for batch log reviews.

4

Intermediate

Limiting Logs with Tail Option

🤔Before reading on: do you think 'docker logs --tail 10' shows the first 10 or last 10 lines? Commit to your answer.

Concept: Learn how to limit the amount of logs shown to focus on recent messages.

Use --tail to see only the last few lines: docker logs --tail 10 This shows just the last 10 lines, which is useful when logs are very long and you want recent info quickly.

Result

You can quickly access the most recent log entries without scrolling through everything.

Knowing how to limit logs saves time and helps focus on the most relevant information.

5

Intermediate

Combining Follow and Tail Options

🤔Before reading on: what happens if you combine '-f' and '--tail 5'? Predict the behavior.

Concept: Learn how to watch recent logs live by combining options.

Run: docker logs -f --tail 5 This shows the last 5 lines immediately and then continues to show new lines as they appear. It’s like jumping into a live broadcast just in time.

Result

You get a quick snapshot of recent activity plus live updates.

Combining options creates powerful, efficient troubleshooting commands.

6

Advanced

Understanding Log Drivers and Storage

🤔Before reading on: do you think Docker stores logs inside the container or outside? Commit to your answer.

Concept: Learn how Docker stores logs and how different log drivers affect this.

Docker uses log drivers to decide where and how to save logs. The default is 'json-file', which stores logs as JSON files on the host machine. Other drivers send logs to external systems like syslog or journald. This affects how you access and manage logs.

Result

You understand that logs are stored outside containers and can be configured to go to different places.

Knowing log drivers helps you plan for scalable logging and troubleshooting in complex environments.

7

Expert

Troubleshooting with Logs in Production

🤔Before reading on: do you think logs alone are enough for production troubleshooting? Commit to your answer.

Concept: Explore how logs fit into real-world production troubleshooting and their limitations.

In production, logs are often centralized using tools like ELK stack or Fluentd. Logs alone may not show the full picture; combining logs with metrics and tracing gives better insights. Also, log rotation and retention policies prevent disk space issues. Understanding these practices is key to effective troubleshooting.

Result

You see that logs are part of a bigger system for monitoring and troubleshooting in production.

Recognizing logs’ role in a larger observability strategy prevents overreliance on logs alone and improves incident response.

Under the Hood

Docker captures container output by connecting to the container’s standard output (stdout) and standard error (stderr) streams. These streams are redirected by Docker to log files or external systems depending on the configured log driver. The default 'json-file' driver writes logs as JSON objects to files on the host, allowing Docker commands to read and display them. This separation means logs persist even if the container stops or crashes.

Why designed this way?

Docker was designed to isolate containers from the host system but still provide access to their output. Using stdout/stderr streams is a universal way to capture logs without modifying applications. The pluggable log driver system allows flexibility to integrate with various logging solutions, balancing simplicity and scalability.

┌───────────────┐
│ Container App │
│ stdout/stderr │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Docker Engine │
│ Log Driver    │
│ (json-file,   │
│ syslog, etc.) │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Host Log File │
│ or External   │
│ Logging Sys.  │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does 'docker logs' show logs from all containers or just one? Commit to your answer.

Common Belief: Docker logs command shows logs from all running containers at once.

Reality: Docker logs shows logs only from the specified container, not all containers.

Why it matters: Trying to see all logs at once with 'docker logs' leads to confusion and missed information, slowing troubleshooting.

Quick: Do you think logs are stored inside the container filesystem? Commit to your answer.

Common Belief: Logs are stored inside the container’s filesystem and disappear when the container is removed.

Reality: Logs are stored on the host machine outside the container, so they persist even if the container stops or is deleted, depending on log driver and retention.

Why it matters: Misunderstanding log storage can cause loss of important logs if users delete containers expecting logs to be saved inside.

Quick: Does 'docker logs -f' show only new logs or all logs plus new ones? Commit to your answer.

Common Belief: 'docker logs -f' shows only new logs generated after the command runs.

Reality: 'docker logs -f' shows all existing logs first, then continues to show new logs as they appear.

Why it matters: Expecting only new logs can cause users to miss important past errors that appear immediately when following logs.

Quick: Can you rely solely on Docker logs for full production troubleshooting? Commit to your answer.

Common Belief: Docker logs alone are enough to diagnose and fix all production issues.

Reality: Logs are important but insufficient alone; combining logs with metrics, tracing, and alerts provides a complete view.

Why it matters: Relying only on logs can delay problem detection and resolution in complex systems.

Expert Zone

1

Log drivers can impact performance; some drivers buffer logs causing delays in visibility.

2

Log rotation settings prevent disk space exhaustion but misconfiguration can cause log loss or excessive disk use.

3

Multi-container applications require centralized logging to correlate events across containers effectively.

When NOT to use

Docker logs are not suitable for long-term log storage or analysis in large-scale systems. Instead, use centralized logging solutions like ELK stack, Fluentd, or cloud logging services that aggregate, index, and analyze logs from many containers and hosts.

Production Patterns

In production, teams use docker logs for quick debugging on a single host but rely on centralized logging pipelines for monitoring. Logs are combined with metrics and tracing in observability platforms to detect, diagnose, and resolve issues efficiently.

Connections

System Logging (syslog)

Docker log drivers can send container logs to syslog, integrating container logs with system logs.

Understanding syslog helps grasp how Docker logs can be centralized and managed alongside other system logs.

Observability

Docker logs are one pillar of observability, alongside metrics and tracing.

Knowing how logs fit into observability helps build better monitoring and troubleshooting strategies.

Customer Support Ticketing

Logs provide the technical evidence needed to resolve customer issues reported in support tickets.

Recognizing logs as a communication bridge between users and engineers improves problem resolution workflows.

Common Pitfalls

#1 Trying to view logs without specifying a container.

Wrong approach:

docker logs

Correct approach:

docker logs

Root cause:

Not understanding that 'docker logs' requires a container identifier to know which logs to show.

#2 Assuming logs update automatically without following.

Wrong approach:

docker logs

Correct approach:

docker logs -f

Root cause:

Not knowing that the default logs command shows only past logs and does not stream new output.

#3 Ignoring log rotation leading to disk full errors.

Wrong approach:

No log rotation configured, letting logs grow indefinitely.

Correct approach:

Configure log rotation in Docker daemon or container logging options to limit log file size and number.

Root cause:

Overlooking the need to manage log file size and retention in production environments.

Key Takeaways

Docker logs capture the output of containerized applications and are essential for troubleshooting.

The 'docker logs' command shows logs from a specific container, with options to follow live output and limit lines.

Logs are stored outside containers on the host, managed by configurable log drivers.

In production, logs are part of a broader observability strategy including metrics and tracing.

Proper log management, including rotation and centralization, is critical to avoid performance and storage issues.