Container logging architecture in Kubernetes - Step-by-Step Execution

Process Flow - Container logging architecture

Container writes logs

↓

Logs stored in container stdout/stderr

↓

Kubelet reads logs from container runtime

↓

Logs saved to node's log files

↓

Log collector agent (e.g., Fluentd) reads node logs

↓

Logs forwarded to central storage or analysis system

↓

User queries logs from central system

Logs flow from containers to node files, then collected by agents and sent to central storage for user access.

Execution Sample

Kubernetes

# Container writes logs
kubectl logs <pod-name>
# Fluentd collects logs
# Logs sent to Elasticsearch
# User queries logs via Kibana

Shows the path of logs from container output to user query in a Kubernetes cluster.

Process Table

Step	Action	Source	Destination	Result
1	Container writes logs	Application inside container	Container stdout/stderr	Logs generated and output
2	Kubelet reads logs	Container runtime	Node log files	Logs saved on node disk
3	Log collector reads logs	Node log files	Log collector agent	Logs collected for forwarding
4	Logs forwarded	Log collector agent	Central storage (e.g., Elasticsearch)	Logs stored centrally
5	User queries logs	Central storage	User interface (e.g., Kibana)	Logs displayed to user
6	Exit	N/A	N/A	Logging process complete

💡 Logging ends after user retrieves logs from central system

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Logs	None	Generated in container stdout/stderr	Saved in node log files	Collected by log agent	Stored in central system	Available for user query

Key Moments - 3 Insights

Why don't containers write logs directly to central storage?

Containers write logs to stdout/stderr which are stored locally on the node; log collectors then forward logs to central storage. This separation allows containers to remain lightweight and decouples logging from application code, as shown in execution_table steps 1 to 4.

What role does the kubelet play in logging?

The kubelet reads logs from the container runtime and saves them to node log files, acting as the bridge between container output and node storage (execution_table step 2).

How does the user access logs from multiple containers?

Users query logs from a central storage system where logs from all nodes and containers are aggregated by log collectors, as shown in execution_table steps 4 and 5.

Visual Quiz - 3 Questions

Look at the execution table, at which step are logs saved on the node disk?

A Step 4

B Step 2

C Step 1

D Step 5

💡 Hint

Check the 'Destination' column for node log files in execution_table row with Step 2.

According to variable_tracker, what is the state of logs after Step 3?

A Logs generated in container stdout/stderr

B Logs stored in central system

C Logs collected by log agent

D Logs saved in node log files

💡 Hint

Look at the 'Logs' row under 'After Step 3' in variable_tracker.

If the log collector agent fails, which step in the execution_table would be directly affected?

A Step 3

B Step 2

C Step 1

D Step 5

💡 Hint

Step 3 describes the log collector reading logs; failure here stops forwarding.

Concept Snapshot

Container logging architecture:
- Containers write logs to stdout/stderr
- Kubelet saves logs to node files
- Log collector agents gather logs
- Logs forwarded to central storage
- Users query logs centrally
This decouples logging from apps and centralizes log management.

Full Transcript

In Kubernetes container logging architecture, containers write logs to their standard output streams. The kubelet reads these logs from the container runtime and saves them as files on the node. A log collector agent, such as Fluentd, reads these node log files and forwards the logs to a central storage system like Elasticsearch. Finally, users can query and view these logs through interfaces such as Kibana. This flow ensures logs are collected reliably and centralized for easy access and analysis.

Practice

1. In Kubernetes, where do containers typically write their logs?

easy

A. Directly to files inside the container's filesystem

B. To a database inside the container

C. To a remote logging server

D. To stdout and stderr streams

Solution

1. Step 1: Understand container logging basics

   Containers are designed to write logs to standard output (stdout) and standard error (stderr) streams instead of files inside the container.

2. Step 2: Recall Kubernetes logging capture method

   Kubernetes captures these stdout and stderr streams from containers to manage logs effectively.

3. Final Answer:

   To stdout and stderr streams -> Option D

4. Quick Check:

   Container logs = stdout/stderr [OK]

Hint: Remember containers log to stdout/stderr, not files [OK]

Common Mistakes:

• Thinking logs are stored inside container files
• Assuming logs go directly to remote servers
• Confusing stdout/stderr with database logging

2. Which of the following is the correct way Kubernetes stores container logs on a node?

easy

A. As log files under /var/log/containers directory on the node

B. In a centralized database on the master node

C. Inside the container's writable layer

D. In memory only, not persisted on disk

Solution

1. Step 1: Identify Kubernetes node log storage

   Kubernetes stores container logs as files on the node, typically under the /var/log/containers directory.

2. Step 2: Eliminate incorrect options

   Logs are not stored in a centralized database on the master, nor inside the container writable layer, and they are persisted on disk, not just in memory.

3. Final Answer:

   As log files under /var/log/containers directory on the node -> Option A

4. Quick Check:

   Node logs path = /var/log/containers [OK]

Hint: Kubernetes logs are files under /var/log/containers on nodes [OK]

Common Mistakes:

• Assuming logs are stored only in memory
• Thinking logs are inside container writable layer
• Believing logs are centralized on master node

3. Given a Kubernetes cluster with a logging agent running on each node, what is the primary role of this agent?

medium

A. To collect container logs from node files and send them to a central system

B. To create log files inside each container

C. To delete old logs from the container filesystem

D. To restart containers when logs grow too large

Solution

1. Step 1: Understand logging agent function

   Logging agents run on nodes to gather logs from container log files stored on the node.

2. Step 2: Identify agent's purpose

   The agent sends collected logs to a central logging system for easy access and analysis.

3. Final Answer:

   To collect container logs from node files and send them to a central system -> Option A

4. Quick Check:

   Logging agent = collect and forward logs [OK]

Hint: Logging agents gather and forward logs to central systems [OK]

Common Mistakes:

• Thinking agents create logs inside containers
• Assuming agents delete logs automatically
• Believing agents restart containers based on log size

4. You notice that your Kubernetes logging agent is not forwarding logs to the central system. Which of the following is the most likely cause?

medium

A. Containers are writing logs to stdout/stderr

B. The logging agent cannot access the /var/log/containers directory on the node

C. The central logging system is storing logs on the node

D. Kubernetes does not support logging agents

Solution

1. Step 1: Analyze logging agent failure

   If the agent cannot access the node's log directory, it cannot read logs to forward them.

2. Step 2: Check other options for correctness

   Containers writing to stdout/stderr is normal; Kubernetes supports logging agents; central system storing logs on node is unrelated to forwarding failure.

3. Final Answer:

   The logging agent cannot access the /var/log/containers directory on the node -> Option B

4. Quick Check:

   Agent access to logs = critical [OK]

Hint: Check logging agent's access to node log files first [OK]

Common Mistakes:

• Blaming containers writing to stdout/stderr
• Assuming Kubernetes lacks logging agent support
• Confusing central system storage with forwarding issues

5. You want to implement a centralized logging solution in Kubernetes. Which combination correctly describes the container logging flow?

hard

A. Containers write logs to stdout/stderr -> Kubernetes stores logs in etcd -> Logging agent collects logs from etcd

B. Containers write logs to files inside container -> Kubernetes copies files to master -> Logging agent forwards logs

C. Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs

D. Containers send logs directly to central server -> Kubernetes stores logs on node -> Logging agent deletes logs

Solution

1. Step 1: Understand container log writing

   Containers write logs to stdout/stderr streams, not files inside the container.

2. Step 2: Trace Kubernetes log handling

   Kubernetes captures these logs and stores them as files on the node.

3. Step 3: Identify logging agent role

   Logging agents collect these node log files and forward them to a central logging system.

4. Final Answer:

   Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs -> Option C

5. Quick Check:

   Logging flow = stdout -> node files -> agent -> central [OK]

Hint: Follow logs: stdout -> node storage -> agent -> central system [OK]

Common Mistakes:

• Thinking logs are stored in etcd
• Assuming containers write logs to files inside container
• Believing logging agent deletes logs