Container logging architecture in Kubernetes - Commands & Configuration

Introduction

When you run applications inside containers, you need a way to collect and view their logs. Container logging architecture helps gather logs from many containers so you can see what is happening inside each one and troubleshoot problems easily.

When you want to see error messages from your app running inside a Kubernetes pod.

When you need to collect logs from multiple containers running on different nodes in a cluster.

When you want to store logs outside the container for long-term analysis.

When you want to monitor your app's behavior in real time.

When you want to debug issues without accessing the container directly.

Commands

This command fetches and shows the logs from the container inside the pod named 'my-app-pod'. It helps you see what the app is printing to its standard output.

Terminal

kubectl logs my-app-pod

Expected Output

2024-06-01T12:00:00Z Starting application

2024-06-01T12:00:05Z Application ready to accept requests

→

-f - Follow the log output in real time

→

-c - Specify which container's logs to show if the pod has multiple containers

This command streams the logs from 'my-app-pod' live, so you can watch new log entries as they happen.

Terminal

kubectl logs -f my-app-pod

Expected Output

2024-06-01T12:00:00Z Starting application

2024-06-01T12:00:05Z Application ready to accept requests

2024-06-01T12:01:00Z Received new user request

→

-f - Follow the log output in real time

This command lists all pod names across all namespaces, helping you find the pods you want to check logs for.

Terminal

kubectl get pods --all-namespaces -o jsonpath='{.items[*].metadata.name}'

Expected Output

my-app-pod other-app-pod database-pod

→

--all-namespaces - Show pods from all namespaces

→

-o jsonpath - Format output to show only pod names

Key Concept

If you remember nothing else from container logging, remember: logs are collected from container standard output and can be viewed using kubectl logs commands.

Common Mistakes

Trying to access logs of a pod that does not exist or is misspelled.

The command will fail with an error because Kubernetes cannot find the pod.

Use 'kubectl get pods' to check the exact pod name before running the logs command.

Not specifying the container name when the pod has multiple containers.

kubectl logs will fail or show logs from the wrong container.

Use the '-c container-name' flag to specify which container's logs to view.

Expecting logs to be stored permanently inside the container.

Container logs are ephemeral and lost if the container restarts or is deleted.

Set up a centralized logging system like Fluentd or Elasticsearch to store logs outside containers.

Summary

Use 'kubectl logs pod-name' to view logs from a container inside a pod.

Use the '-f' flag to follow logs live as they are generated.

Always verify pod names and specify container names if needed to get the correct logs.

Practice

1. In Kubernetes, where do containers typically write their logs?

easy

A. Directly to files inside the container's filesystem

B. To a database inside the container

C. To a remote logging server

D. To stdout and stderr streams

Solution

1. Step 1: Understand container logging basics

   Containers are designed to write logs to standard output (stdout) and standard error (stderr) streams instead of files inside the container.

2. Step 2: Recall Kubernetes logging capture method

   Kubernetes captures these stdout and stderr streams from containers to manage logs effectively.

3. Final Answer:

   To stdout and stderr streams -> Option D

4. Quick Check:

   Container logs = stdout/stderr [OK]

Hint: Remember containers log to stdout/stderr, not files [OK]

Common Mistakes:

• Thinking logs are stored inside container files
• Assuming logs go directly to remote servers
• Confusing stdout/stderr with database logging

2. Which of the following is the correct way Kubernetes stores container logs on a node?

easy

A. As log files under /var/log/containers directory on the node

B. In a centralized database on the master node

C. Inside the container's writable layer

D. In memory only, not persisted on disk

Solution

1. Step 1: Identify Kubernetes node log storage

   Kubernetes stores container logs as files on the node, typically under the /var/log/containers directory.

2. Step 2: Eliminate incorrect options

   Logs are not stored in a centralized database on the master, nor inside the container writable layer, and they are persisted on disk, not just in memory.

3. Final Answer:

   As log files under /var/log/containers directory on the node -> Option A

4. Quick Check:

   Node logs path = /var/log/containers [OK]

Hint: Kubernetes logs are files under /var/log/containers on nodes [OK]

Common Mistakes:

• Assuming logs are stored only in memory
• Thinking logs are inside container writable layer
• Believing logs are centralized on master node

3. Given a Kubernetes cluster with a logging agent running on each node, what is the primary role of this agent?

medium

A. To collect container logs from node files and send them to a central system

B. To create log files inside each container

C. To delete old logs from the container filesystem

D. To restart containers when logs grow too large

Solution

1. Step 1: Understand logging agent function

   Logging agents run on nodes to gather logs from container log files stored on the node.

2. Step 2: Identify agent's purpose

   The agent sends collected logs to a central logging system for easy access and analysis.

3. Final Answer:

   To collect container logs from node files and send them to a central system -> Option A

4. Quick Check:

   Logging agent = collect and forward logs [OK]

Hint: Logging agents gather and forward logs to central systems [OK]

Common Mistakes:

• Thinking agents create logs inside containers
• Assuming agents delete logs automatically
• Believing agents restart containers based on log size

4. You notice that your Kubernetes logging agent is not forwarding logs to the central system. Which of the following is the most likely cause?

medium

A. Containers are writing logs to stdout/stderr

B. The logging agent cannot access the /var/log/containers directory on the node

C. The central logging system is storing logs on the node

D. Kubernetes does not support logging agents

Solution

1. Step 1: Analyze logging agent failure

   If the agent cannot access the node's log directory, it cannot read logs to forward them.

2. Step 2: Check other options for correctness

   Containers writing to stdout/stderr is normal; Kubernetes supports logging agents; central system storing logs on node is unrelated to forwarding failure.

3. Final Answer:

   The logging agent cannot access the /var/log/containers directory on the node -> Option B

4. Quick Check:

   Agent access to logs = critical [OK]

Hint: Check logging agent's access to node log files first [OK]

Common Mistakes:

• Blaming containers writing to stdout/stderr
• Assuming Kubernetes lacks logging agent support
• Confusing central system storage with forwarding issues

5. You want to implement a centralized logging solution in Kubernetes. Which combination correctly describes the container logging flow?

hard

A. Containers write logs to stdout/stderr -> Kubernetes stores logs in etcd -> Logging agent collects logs from etcd

B. Containers write logs to files inside container -> Kubernetes copies files to master -> Logging agent forwards logs

C. Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs

D. Containers send logs directly to central server -> Kubernetes stores logs on node -> Logging agent deletes logs

Solution

1. Step 1: Understand container log writing

   Containers write logs to stdout/stderr streams, not files inside the container.

2. Step 2: Trace Kubernetes log handling

   Kubernetes captures these logs and stores them as files on the node.

3. Step 3: Identify logging agent role

   Logging agents collect these node log files and forward them to a central logging system.

4. Final Answer:

   Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs -> Option C

5. Quick Check:

   Logging flow = stdout -> node files -> agent -> central [OK]

Hint: Follow logs: stdout -> node storage -> agent -> central system [OK]

Common Mistakes:

• Thinking logs are stored in etcd
• Assuming containers write logs to files inside container
• Believing logging agent deletes logs