Container logging architecture in Kubernetes - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding container log storage locations

Where does Kubernetes typically store container logs on the node by default?

A /home/user/.kube/logs directory

B /etc/kubernetes/logs directory

C /usr/local/bin/logs directory

D /var/log/containers directory

💡 Hint

Think about where Docker or container runtimes write logs on the host.

Kubernetes stores container logs in the /var/log/containers directory on the node by default. This directory contains symlinks to the actual log files managed by the container runtime.

💻 Command Output

intermediate

Output of kubectl logs command

What is the output of the command kubectl logs my-pod if the pod has a container that writes 'Hello World' to stdout?

A Error: pod not found

B No logs found for container

C Hello World

D kubectl: command not found

💡 Hint

This command shows the container's standard output logs.

The kubectl logs command fetches the logs from the container's stdout and stderr streams. If the container wrote 'Hello World', that text will appear as output.

❓ Configuration

advanced

Configuring a Fluentd DaemonSet for cluster-wide logging

Which configuration snippet correctly deploys Fluentd as a DaemonSet to collect logs from all nodes?

A

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd
spec:
  replicas: 3
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

B

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

C

apiVersion: v1
kind: Pod
metadata:
  name: fluentd
spec:
  containers:
  - name: fluentd
    image: fluent/fluentd
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    hostPath:
      path: /var/log

D

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      name: fluentd
  serviceName: fluentd
  replicas: 1
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

💡 Hint

DaemonSets run one pod per node to collect logs from all nodes.

Option B correctly defines a DaemonSet with a container mounting the host's /var/log directory to collect logs from all nodes. Other options use Pod, Deployment, or StatefulSet which do not guarantee one pod per node.

❓ Troubleshoot

advanced

Diagnosing missing logs in Kubernetes cluster

You notice that kubectl logs returns empty output for a pod, but the application writes logs to stdout. What is the most likely cause?

A The container runtime is not writing logs to the expected location

B The pod is running on a node without network connectivity

C The pod's container image is missing the logging library

D The Kubernetes API server is down

💡 Hint

Think about where logs come from when you run kubectl logs.

kubectl logs fetches logs from the container runtime's log files. If the runtime is not writing logs properly, the command returns empty output even if the app writes to stdout.

🔀 Workflow

expert

Order of steps to implement centralized logging in Kubernetes

Arrange the steps in the correct order to set up centralized logging for a Kubernetes cluster using Fluentd and Elasticsearch.

A 3,2,1,4

B 2,3,1,4

C 3,1,2,4

D 1,3,2,4

💡 Hint

Think about creating a clean space first, then collecting logs, storing them, and finally visualizing.

First create a namespace to organize logging components (3). Then deploy Fluentd as a DaemonSet to collect logs (2). Next deploy Elasticsearch to store logs (1). Finally set up Kibana to visualize logs (4).

Practice

1. In Kubernetes, where do containers typically write their logs?

easy

A. Directly to files inside the container's filesystem

B. To a database inside the container

C. To a remote logging server

D. To stdout and stderr streams

Solution

1. Step 1: Understand container logging basics

   Containers are designed to write logs to standard output (stdout) and standard error (stderr) streams instead of files inside the container.

2. Step 2: Recall Kubernetes logging capture method

   Kubernetes captures these stdout and stderr streams from containers to manage logs effectively.

3. Final Answer:

   To stdout and stderr streams -> Option D

4. Quick Check:

   Container logs = stdout/stderr [OK]

Hint: Remember containers log to stdout/stderr, not files [OK]

Common Mistakes:

• Thinking logs are stored inside container files
• Assuming logs go directly to remote servers
• Confusing stdout/stderr with database logging

2. Which of the following is the correct way Kubernetes stores container logs on a node?

easy

A. As log files under /var/log/containers directory on the node

B. In a centralized database on the master node

C. Inside the container's writable layer

D. In memory only, not persisted on disk

Solution

1. Step 1: Identify Kubernetes node log storage

   Kubernetes stores container logs as files on the node, typically under the /var/log/containers directory.

2. Step 2: Eliminate incorrect options

   Logs are not stored in a centralized database on the master, nor inside the container writable layer, and they are persisted on disk, not just in memory.

3. Final Answer:

   As log files under /var/log/containers directory on the node -> Option A

4. Quick Check:

   Node logs path = /var/log/containers [OK]

Hint: Kubernetes logs are files under /var/log/containers on nodes [OK]

Common Mistakes:

• Assuming logs are stored only in memory
• Thinking logs are inside container writable layer
• Believing logs are centralized on master node

3. Given a Kubernetes cluster with a logging agent running on each node, what is the primary role of this agent?

medium

A. To collect container logs from node files and send them to a central system

B. To create log files inside each container

C. To delete old logs from the container filesystem

D. To restart containers when logs grow too large

Solution

1. Step 1: Understand logging agent function

   Logging agents run on nodes to gather logs from container log files stored on the node.

2. Step 2: Identify agent's purpose

   The agent sends collected logs to a central logging system for easy access and analysis.

3. Final Answer:

   To collect container logs from node files and send them to a central system -> Option A

4. Quick Check:

   Logging agent = collect and forward logs [OK]

Hint: Logging agents gather and forward logs to central systems [OK]

Common Mistakes:

• Thinking agents create logs inside containers
• Assuming agents delete logs automatically
• Believing agents restart containers based on log size

4. You notice that your Kubernetes logging agent is not forwarding logs to the central system. Which of the following is the most likely cause?

medium

A. Containers are writing logs to stdout/stderr

B. The logging agent cannot access the /var/log/containers directory on the node

C. The central logging system is storing logs on the node

D. Kubernetes does not support logging agents

Solution

1. Step 1: Analyze logging agent failure

   If the agent cannot access the node's log directory, it cannot read logs to forward them.

2. Step 2: Check other options for correctness

   Containers writing to stdout/stderr is normal; Kubernetes supports logging agents; central system storing logs on node is unrelated to forwarding failure.

3. Final Answer:

   The logging agent cannot access the /var/log/containers directory on the node -> Option B

4. Quick Check:

   Agent access to logs = critical [OK]

Hint: Check logging agent's access to node log files first [OK]

Common Mistakes:

• Blaming containers writing to stdout/stderr
• Assuming Kubernetes lacks logging agent support
• Confusing central system storage with forwarding issues

5. You want to implement a centralized logging solution in Kubernetes. Which combination correctly describes the container logging flow?

hard

A. Containers write logs to stdout/stderr -> Kubernetes stores logs in etcd -> Logging agent collects logs from etcd

B. Containers write logs to files inside container -> Kubernetes copies files to master -> Logging agent forwards logs

C. Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs

D. Containers send logs directly to central server -> Kubernetes stores logs on node -> Logging agent deletes logs

Solution

1. Step 1: Understand container log writing

   Containers write logs to stdout/stderr streams, not files inside the container.

2. Step 2: Trace Kubernetes log handling

   Kubernetes captures these logs and stores them as files on the node.

3. Step 3: Identify logging agent role

   Logging agents collect these node log files and forward them to a central logging system.

4. Final Answer:

   Containers write logs to stdout/stderr -> Kubernetes stores logs on node -> Logging agent collects and forwards logs -> Option C

5. Quick Check:

   Logging flow = stdout -> node files -> agent -> central [OK]

Hint: Follow logs: stdout -> node storage -> agent -> central system [OK]

Common Mistakes:

• Thinking logs are stored in etcd
• Assuming containers write logs to files inside container
• Believing logging agent deletes logs