Container logging architecture in Kubernetes - Mini Project: Build & Apply

Container Logging Architecture in Kubernetes

📖 Scenario: You are working as a DevOps engineer managing a Kubernetes cluster. You want to set up a simple container logging architecture to collect logs from your application pods and store them centrally for easy access and troubleshooting.

🎯 Goal: Build a basic container logging setup in Kubernetes by creating a pod with a container that writes logs, configure a log path variable, collect logs using a sidecar container, and finally display the collected logs.

📋 What You'll Learn

Create a pod manifest with a main container that writes logs to a file

Add a configuration variable for the log file path

Add a sidecar container that reads the log file and outputs logs

Display the logs collected by the sidecar container

💡 Why This Matters

🌍 Real World

Centralized logging helps DevOps teams monitor and troubleshoot applications running in Kubernetes clusters efficiently.

💼 Career

Understanding container logging architecture is essential for roles like DevOps engineer, site reliability engineer, and cloud administrator.

1

Create a pod manifest with a main container that writes logs

Create a Kubernetes pod manifest named logging-pod.yaml with a pod called logger. It should have one container named app using the image busybox. The container should run the command sh -c "while true; do echo 'Log entry' >> /var/log/app.log; sleep 5; done" to write logs continuously to /var/log/app.log. Use an emptyDir volume named log-volume mounted at /var/log.

Kubernetes

# Create the pod manifest with the specified container and volume
# Your code here

Hint

Use emptyDir volume to share logs inside the pod. The app container writes logs to /var/log/app.log.

2

Add a configuration variable for the log file path

Add an environment variable named LOG_PATH with the value /var/log/app.log to the app container in the logging-pod.yaml manifest.

Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: logger
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "while true; do echo 'Log entry' >> /var/log/app.log; sleep 5; done"]
    volumeMounts:
    - name: log-volume
      mountPath: /var/log
    # Add environment variable LOG_PATH here
    env:
    # Your code here
  volumes:
  - name: log-volume
    emptyDir: {}

Hint

Use the env field inside the app container to add the LOG_PATH variable.

3

Add a sidecar container to read and output logs

Add a second container named log-reader to the pod. Use the image busybox. It should mount the same log-volume at /var/log and run the command sh -c "tail -f /var/log/app.log" to continuously output the logs.

Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: logger
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "while true; do echo 'Log entry' >> /var/log/app.log; sleep 5; done"]
    volumeMounts:
    - name: log-volume
      mountPath: /var/log
    env:
    - name: LOG_PATH
      value: "/var/log/app.log"
  # Add the log-reader container below
  - name: log-reader
    # Your code here
  volumes:
  - name: log-volume
    emptyDir: {}

Hint

The log-reader container shares the log-volume and tails the log file to output logs.

4

Display the logs collected by the sidecar container

Run the command kubectl logs logger -c log-reader to display the logs output by the log-reader container.

Kubernetes

# Run the kubectl command below to show logs from the log-reader container
# Your code here

Hint

This command fetches logs from the log-reader container inside the logger pod.

Practice

1. In Kubernetes, where do containers typically write their logs?

easy

A. Directly to files inside the container's filesystem

B. To a database inside the container

C. To a remote logging server

D. To stdout and stderr streams

Solution

1. Step 1: Understand container logging basics

   Containers are designed to write logs to standard output (stdout) and standard error (stderr) streams instead of files inside the container.

2. Step 2: Recall Kubernetes logging capture method

   Kubernetes captures these stdout and stderr streams from containers to manage logs effectively.

3. Final Answer:

   To stdout and stderr streams -> Option D

4. Quick Check:

   Container logs = stdout/stderr [OK]

Hint: Remember containers log to stdout/stderr, not files [OK]

Common Mistakes:

• Thinking logs are stored inside container files
• Assuming logs go directly to remote servers
• Confusing stdout/stderr with database logging

2. Which of the following is the correct way Kubernetes stores container logs on a node?

easy

A. As log files under /var/log/containers directory on the node

B. In a centralized database on the master node

C. Inside the container's writable layer

D. In memory only, not persisted on disk

Solution

1. Step 1: Identify Kubernetes node log storage

   Kubernetes stores container logs as files on the node, typically under the /var/log/containers directory.

2. Step 2: Eliminate incorrect options

   Logs are not stored in a centralized database on the master, nor inside the container writable layer, and they are persisted on disk, not just in memory.

3. Final Answer:

   As log files under /var/log/containers directory on the node -> Option A

4. Quick Check:

   Node logs path = /var/log/containers [OK]

Hint: Kubernetes logs are files under /var/log/containers on nodes [OK]

Common Mistakes:

• Assuming logs are stored only in memory
• Thinking logs are inside container writable layer
• Believing logs are centralized on master node

3. Given a Kubernetes cluster with a logging agent running on each node, what is the primary role of this agent?

medium

A. To collect container logs from node files and send them to a central system

B. To create log files inside each container

C. To delete old logs from the container filesystem

D. To restart containers when logs grow too large

Solution

1. Step 1: Understand logging agent function

   Logging agents run on nodes to gather logs from container log files stored on the node.

2. Step 2: Identify agent's purpose

   The agent sends collected logs to a central logging system for easy access and analysis.

3. Final Answer:

   To collect container logs from node files and send them to a central system -> Option A

4. Quick Check:

   Logging agent = collect and forward logs [OK]

Hint: Logging agents gather and forward logs to central systems [OK]

Common Mistakes:

• Thinking agents create logs inside containers
• Assuming agents delete logs automatically
• Believing agents restart containers based on log size

4. You notice that your Kubernetes logging agent is not forwarding logs to the central system. Which of the following is the most likely cause?

medium

A. Containers are writing logs to stdout/stderr

B. The logging agent cannot access the /var/log/containers directory on the node

C. The central logging system is storing logs on the node

D. Kubernetes does not support logging agents

Solution

1. Step 1: Analyze logging agent failure

   If the agent cannot access the node's log directory, it cannot read logs to forward them.

2. Step 2: Check other options for correctness

   Containers writing to stdout/stderr is normal; Kubernetes supports logging agents; central system storing logs on node is unrelated to forwarding failure.

3. Final Answer:

   The logging agent cannot access the /var/log/containers directory on the node -> Option B

4. Quick Check:

   Agent access to logs = critical [OK]

Hint: Check logging agent's access to node log files first [OK]

Common Mistakes:

• Blaming containers writing to stdout/stderr
• Assuming Kubernetes lacks logging agent support
• Confusing central system storage with forwarding issues

5. You want to implement a centralized logging solution in Kubernetes. Which combination correctly describes the container logging flow?

hard

A. Containers write logs to stdout/stderr -> Kubernetes stores logs in etcd -> Logging agent collects logs from etcd

B. Containers write logs to files inside container -> Kubernetes copies files to master -> Logging agent forwards logs

C. Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs

D. Containers send logs directly to central server -> Kubernetes stores logs on node -> Logging agent deletes logs

Solution

1. Step 1: Understand container log writing

   Containers write logs to stdout/stderr streams, not files inside the container.

2. Step 2: Trace Kubernetes log handling

   Kubernetes captures these logs and stores them as files on the node.

3. Step 3: Identify logging agent role

   Logging agents collect these node log files and forward them to a central logging system.

4. Final Answer:

   Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs -> Option C

5. Quick Check:

   Logging flow = stdout -> node files -> agent -> central [OK]

Hint: Follow logs: stdout -> node storage -> agent -> central system [OK]

Common Mistakes:

• Thinking logs are stored in etcd
• Assuming containers write logs to files inside container
• Believing logging agent deletes logs