DaemonSets for per-node workloads in Kubernetes - Commands & Configuration

Introduction

Sometimes you need to run a small program on every computer in your cluster. DaemonSets help you do this by making sure a copy of your program runs on each node automatically.

When you want to collect logs from every node in your cluster.

When you need to monitor the health of each node with a small agent.

When you want to run a network tool on every node to manage traffic.

When you need to apply security scanning software on all nodes.

When you want to run a storage driver on every node to handle disks.

Config File - daemonset.yaml

daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
  labels:
    app: node-agent
spec:
  selector:
    matchLabels:
      app: node-agent
  template:
    metadata:
      labels:
        app: node-agent
    spec:
      containers:
      - name: node-agent
        image: busybox
        command: ["sh", "-c", "while true; do echo Hello from node $(hostname); sleep 30; done"]

This file creates a DaemonSet named node-agent. It runs a simple container on every node that prints a message every 30 seconds. The selector and template.metadata.labels ensure the DaemonSet manages the right pods. The command runs a loop printing the node's name.

Commands

This command creates the DaemonSet in your Kubernetes cluster, starting the pods on all nodes.

Terminal

kubectl apply -f daemonset.yaml

Expected Output

daemonset.apps/node-agent created

This command lists all DaemonSets in the current namespace to verify your DaemonSet is running.

Terminal

kubectl get daemonsets

Expected Output

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE

node-agent   3         3         3       3            3           <none>          10s

This command shows the pods created by the DaemonSet and on which nodes they are running.

Terminal

kubectl get pods -l app=node-agent -o wide

Expected Output

NAME              READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES

node-agent-abc12   1/1     Running   0          15s   10.244.1.5   worker-node-1  <none>           <none>

node-agent-def34   1/1     Running   0          15s   10.244.2.5   worker-node-2  <none>           <none>

node-agent-ghi56   1/1     Running   0          15s   10.244.3.5   worker-node-3  <none>           <none>

→

-l app=node-agent - Filter pods by label to show only DaemonSet pods

→

-o wide - Show extra details including node names

This command shows detailed information about the DaemonSet, including events and pod status.

Terminal

kubectl describe daemonset node-agent

Expected Output

Name:           node-agent

Namespace:      default

Selector:       app=node-agent

Labels:         app=node-agent

Desired Number of Nodes Scheduled: 3

Current Number of Nodes Scheduled: 3

Number of Nodes Misscheduled: 0

Pods Status:    3 Running / 0 Waiting / 0 Succeeded / 0 Failed

Events:

  Type    Reason            Age   From                   Message

  ----    ------            ----  ----                   -------

  Normal  SuccessfulCreate  1m    daemonset-controller   Created pod: node-agent-abc12

  Normal  SuccessfulCreate  1m    daemonset-controller   Created pod: node-agent-def34

  Normal  SuccessfulCreate  1m    daemonset-controller   Created pod: node-agent-ghi56

Key Concept

A DaemonSet ensures one copy of a pod runs on every node automatically and stays running.

Common Mistakes

Not matching the selector labels with the pod template labels in the DaemonSet spec.

Kubernetes won't know which pods belong to the DaemonSet, so it won't manage them properly.

Make sure the selector.matchLabels exactly matches the labels in template.metadata.labels.

Applying the DaemonSet YAML without checking node readiness or taints.

Pods may not schedule on nodes that are not ready or have taints blocking them.

Verify nodes are ready and adjust tolerations in the DaemonSet spec if needed.

Using a container image that does not run or exits immediately.

Pods will crash or restart constantly, causing instability.

Use a container with a long-running process or a command that keeps running.

Summary

Create a DaemonSet YAML file that defines a pod to run on every node.

Apply the DaemonSet with kubectl apply to start pods on all nodes.

Use kubectl get daemonsets and kubectl get pods to verify pods are running on each node.

Use kubectl describe daemonset to see detailed status and events.

Practice

1. What is the main purpose of a DaemonSet in Kubernetes?

easy

A. To expose services outside the cluster

B. To manage persistent storage volumes

C. To run one pod on every node in the cluster

D. To schedule batch jobs at specific times

Solution

1. Step 1: Understand DaemonSet functionality

   A DaemonSet ensures that a copy of a pod runs on all or some nodes in the cluster.

2. Step 2: Compare with other Kubernetes objects

   Other objects like Jobs or Services do not guarantee one pod per node; DaemonSet does.

3. Final Answer:

   To run one pod on every node in the cluster -> Option C

4. Quick Check:

   DaemonSet = one pod per node [OK]

Hint: DaemonSet always runs pods on all nodes [OK]

Common Mistakes:

• Confusing DaemonSet with Deployment
• Thinking DaemonSet manages storage
• Assuming DaemonSet schedules batch jobs

2. Which of the following is the correct way to specify a DaemonSet selector in the YAML manifest?

easy

A. selector:
labels:
app: my-daemon

B. selector:
matchLabels:
app: my-daemon

C. selector:
matchExpressions:
- key: app
operator: Equals
values: [my-daemon]

D. selector:
matchLabels:
app: my-daemon
matchExpressions:
- key: tier
operator: In
values: [backend]

Solution

1. Step 1: Recall DaemonSet selector syntax

   The selector must use matchLabels or matchExpressions under selector.

2. Step 2: Identify correct syntax

   selector: matchLabels: app: my-daemon uses matchLabels correctly with key-value pairs. selector: labels: app: my-daemon is invalid because it uses labels instead of matchLabels. selector: matchExpressions: - key: app operator: Equals values: [my-daemon] has wrong operator case and structure. selector: matchLabels: app: my-daemon matchExpressions: - key: tier operator: In values: [backend] mixes both but DaemonSet selector only supports one selector type.

3. Final Answer:

   selector: matchLabels: app: my-daemon -> Option B

4. Quick Check:

   DaemonSet selector uses matchLabels [OK]

Hint: Use matchLabels under selector for DaemonSet [OK]

Common Mistakes:

• Using 'labels' instead of 'matchLabels'
• Incorrect operator casing in matchExpressions
• Mixing matchLabels and matchExpressions in selector

3. Given this DaemonSet YAML snippet, what will happen when a new node joins the cluster?

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
spec:
  selector:
    matchLabels:
      app: log-agent
  template:
    metadata:
      labels:
        app: log-agent
    spec:
      containers:
      - name: log-agent
        image: log-agent:latest

medium

A. A pod named log-agent will be created on the new node automatically

B. No pod will be created until manually triggered

C. The DaemonSet will fail due to missing node selector

D. Pods will be created only on nodes labeled 'log-agent'

Solution

1. Step 1: Understand DaemonSet behavior on new nodes

   DaemonSets automatically create pods on every node that matches the selector, including new nodes joining the cluster.

2. Step 2: Check node selector usage

   This DaemonSet has no node selector restricting nodes, so it applies to all nodes.

3. Final Answer:

   A pod named log-agent will be created on the new node automatically -> Option A

4. Quick Check:

   DaemonSet pods auto-create on new nodes [OK]

Hint: DaemonSet pods auto-launch on all nodes by default [OK]

Common Mistakes:

• Assuming manual pod creation on new nodes
• Thinking node selector is mandatory
• Believing pods only run on labeled nodes

4. You created a DaemonSet but no pods are running on any node. Which of these is the most likely cause?

medium

A. The DaemonSet is created in the wrong namespace

B. The DaemonSet spec is missing the container image

C. The cluster has no nodes available

D. The pod template labels do not match the DaemonSet selector

Solution

1. Step 1: Check DaemonSet selector and pod template labels

   DaemonSet requires the selector to match the pod template labels exactly; otherwise, pods won't be created.

2. Step 2: Evaluate other options

   Missing container image causes pod creation failure but usually shows pod errors. No nodes means no pods but is rare. Wrong namespace means DaemonSet exists but unrelated to nodes.

3. Final Answer:

   The pod template labels do not match the DaemonSet selector -> Option D

4. Quick Check:

   Selector and pod labels must match [OK]

Hint: Match pod labels with selector to run DaemonSet pods [OK]

Common Mistakes:

• Ignoring label mismatch between selector and template
• Assuming missing image stops pod creation silently
• Not verifying cluster node availability

5. You want to run a logging agent on all Linux nodes but exclude Windows nodes in your Kubernetes cluster. How should you configure the DaemonSet?

hard

A. Add a nodeSelector with kubernetes.io/os: linux in the pod spec

B. Add a tolerations for Windows taints in the pod spec

C. Set the DaemonSet selector to os: linux label

D. Use a Deployment instead of DaemonSet with node affinity

Solution

1. Step 1: Identify how to target Linux nodes only

   Kubernetes labels nodes with kubernetes.io/os set to their OS. Using nodeSelector with kubernetes.io/os: linux restricts pods to Linux nodes.

2. Step 2: Evaluate other options

   Tolerations are for taints, not OS filtering. Selector matches pod labels, not nodes. Deployment does not guarantee one pod per node.

3. Final Answer:

   Add a nodeSelector with kubernetes.io/os: linux in the pod spec -> Option A

4. Quick Check:

   Use nodeSelector to filter nodes by OS [OK]

Hint: Use nodeSelector with OS label to filter nodes [OK]

Common Mistakes:

• Using selector to filter nodes instead of nodeSelector
• Confusing tolerations with node filtering
• Using Deployment instead of DaemonSet for per-node pods