DaemonSets for per-node workloads in Kubernetes - Step-by-Step Execution

Process Flow - DaemonSets for per-node workloads

Create DaemonSet YAML

↓

kubectl apply DaemonSet

↓

Kubernetes Scheduler

↓

DaemonSet Controller creates Pod on each Node

↓

Pods run on all current Nodes

↓

New Node joins Cluster?

Yes↓

DaemonSet Controller adds Pod to new Node

↓

Pods run on all Nodes including new one

↓

Node removed?

Yes↓

Pod on removed Node deleted

↓

DaemonSet maintains one Pod per Node

This flow shows how a DaemonSet ensures one Pod runs on each node, including new nodes joining the cluster.

Execution Sample

Kubernetes

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
spec:
  selector:
    matchLabels:
      name: node-monitor
  template:
    metadata:
      labels:
        name: node-monitor
    spec:
      containers:
      - name: monitor
        image: busybox
        command: ["sh", "-c", "while true; do echo monitoring; sleep 30; done"]

This YAML creates a DaemonSet named node-monitor that runs a simple monitoring container on every node.

Process Table

Step	Action	Result	Pods per Node
1	Apply DaemonSet YAML with kubectl	DaemonSet resource created	0
2	DaemonSet Controller detects DaemonSet	Starts creating Pods	0
3	Create Pod on Node1	Pod running on Node1	Node1:1
4	Create Pod on Node2	Pod running on Node2	Node1:1, Node2:1
5	New Node3 joins cluster	DaemonSet Controller creates Pod on Node3	Node1:1, Node2:1, Node3:1
6	Node2 removed from cluster	Pod on Node2 deleted	Node1:1, Node3:1
7	DaemonSet maintains Pods on all nodes	Pods running on all current nodes	Node1:1, Node3:1

💡 DaemonSet ensures exactly one Pod per node; stops when no more nodes exist or DaemonSet deleted.

Status Tracker

Variable	Start	After Step 3	After Step 4	After Step 5	After Step 6	Final
Pods per Node	{}	{"Node1":1}	{"Node1":1, "Node2":1}	{"Node1":1, "Node2":1, "Node3":1}	{"Node1":1, "Node3":1}	{"Node1":1, "Node3":1}

Key Moments - 3 Insights

Why does the DaemonSet create a new Pod when a new node joins?

Because the DaemonSet controller watches the cluster nodes and ensures one Pod per node. When a new node appears (see Step 5), it creates a Pod there automatically.

What happens to Pods on nodes that are removed from the cluster?

Pods on removed nodes are deleted (Step 6) because the node no longer exists. The DaemonSet controller updates the Pod list accordingly.

Can a DaemonSet create more than one Pod per node?

No. The DaemonSet ensures exactly one Pod per node, as shown in the Pods per Node column in the execution table.

Visual Quiz - 3 Questions

Look at the execution table, how many Pods are running after Step 4?

A 1 Pod on Node1 and 1 Pod on Node2

B 1 Pod on Node1

C 1 Pod on Node1, Node2, and Node3

D No Pods running yet

💡 Hint

Check the 'Pods per Node' column at Step 4 in the execution table.

At which step does the DaemonSet add a Pod to a new node?

A Step 4

B Step 5

C Step 3

D Step 6

💡 Hint

Look for when Node3 first appears with a Pod in the execution table.

If Node2 is removed, what happens to the Pod on Node2?

A It remains running

B It moves to another node

C It is deleted

D It duplicates on Node1

💡 Hint

See Step 6 in the execution table where Node2 is removed.

Concept Snapshot

DaemonSet runs one Pod on each node in a Kubernetes cluster.
Apply a DaemonSet YAML with kubectl to create it.
Pods are automatically added to new nodes.
Pods on removed nodes are deleted.
Ensures per-node workload consistency.

Full Transcript

A DaemonSet in Kubernetes ensures that a specific Pod runs on every node in the cluster. When you create a DaemonSet using kubectl apply, the DaemonSet controller watches the cluster nodes and creates one Pod per node. If a new node joins, the controller adds a Pod there automatically. If a node is removed, the Pod on that node is deleted. This way, the DaemonSet keeps exactly one Pod running on each node, useful for per-node tasks like monitoring or logging.

Practice

1. What is the main purpose of a DaemonSet in Kubernetes?

easy

A. To expose services outside the cluster

B. To manage persistent storage volumes

C. To run one pod on every node in the cluster

D. To schedule batch jobs at specific times

Solution

1. Step 1: Understand DaemonSet functionality

   A DaemonSet ensures that a copy of a pod runs on all or some nodes in the cluster.

2. Step 2: Compare with other Kubernetes objects

   Other objects like Jobs or Services do not guarantee one pod per node; DaemonSet does.

3. Final Answer:

   To run one pod on every node in the cluster -> Option C

4. Quick Check:

   DaemonSet = one pod per node [OK]

Hint: DaemonSet always runs pods on all nodes [OK]

Common Mistakes:

• Confusing DaemonSet with Deployment
• Thinking DaemonSet manages storage
• Assuming DaemonSet schedules batch jobs

2. Which of the following is the correct way to specify a DaemonSet selector in the YAML manifest?

easy

A. selector:
labels:
app: my-daemon

B. selector:
matchLabels:
app: my-daemon

C. selector:
matchExpressions:
- key: app
operator: Equals
values: [my-daemon]

D. selector:
matchLabels:
app: my-daemon
matchExpressions:
- key: tier
operator: In
values: [backend]

Solution

1. Step 1: Recall DaemonSet selector syntax

   The selector must use matchLabels or matchExpressions under selector.

2. Step 2: Identify correct syntax

   selector: matchLabels: app: my-daemon uses matchLabels correctly with key-value pairs. selector: labels: app: my-daemon is invalid because it uses labels instead of matchLabels. selector: matchExpressions: - key: app operator: Equals values: [my-daemon] has wrong operator case and structure. selector: matchLabels: app: my-daemon matchExpressions: - key: tier operator: In values: [backend] mixes both but DaemonSet selector only supports one selector type.

3. Final Answer:

   selector: matchLabels: app: my-daemon -> Option B

4. Quick Check:

   DaemonSet selector uses matchLabels [OK]

Hint: Use matchLabels under selector for DaemonSet [OK]

Common Mistakes:

• Using 'labels' instead of 'matchLabels'
• Incorrect operator casing in matchExpressions
• Mixing matchLabels and matchExpressions in selector

3. Given this DaemonSet YAML snippet, what will happen when a new node joins the cluster?

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
spec:
  selector:
    matchLabels:
      app: log-agent
  template:
    metadata:
      labels:
        app: log-agent
    spec:
      containers:
      - name: log-agent
        image: log-agent:latest

medium

A. A pod named log-agent will be created on the new node automatically

B. No pod will be created until manually triggered

C. The DaemonSet will fail due to missing node selector

D. Pods will be created only on nodes labeled 'log-agent'

Solution

1. Step 1: Understand DaemonSet behavior on new nodes

   DaemonSets automatically create pods on every node that matches the selector, including new nodes joining the cluster.

2. Step 2: Check node selector usage

   This DaemonSet has no node selector restricting nodes, so it applies to all nodes.

3. Final Answer:

   A pod named log-agent will be created on the new node automatically -> Option A

4. Quick Check:

   DaemonSet pods auto-create on new nodes [OK]

Hint: DaemonSet pods auto-launch on all nodes by default [OK]

Common Mistakes:

• Assuming manual pod creation on new nodes
• Thinking node selector is mandatory
• Believing pods only run on labeled nodes

4. You created a DaemonSet but no pods are running on any node. Which of these is the most likely cause?

medium

A. The DaemonSet is created in the wrong namespace

B. The DaemonSet spec is missing the container image

C. The cluster has no nodes available

D. The pod template labels do not match the DaemonSet selector

Solution

1. Step 1: Check DaemonSet selector and pod template labels

   DaemonSet requires the selector to match the pod template labels exactly; otherwise, pods won't be created.

2. Step 2: Evaluate other options

   Missing container image causes pod creation failure but usually shows pod errors. No nodes means no pods but is rare. Wrong namespace means DaemonSet exists but unrelated to nodes.

3. Final Answer:

   The pod template labels do not match the DaemonSet selector -> Option D

4. Quick Check:

   Selector and pod labels must match [OK]

Hint: Match pod labels with selector to run DaemonSet pods [OK]

Common Mistakes:

• Ignoring label mismatch between selector and template
• Assuming missing image stops pod creation silently
• Not verifying cluster node availability

5. You want to run a logging agent on all Linux nodes but exclude Windows nodes in your Kubernetes cluster. How should you configure the DaemonSet?

hard

A. Add a nodeSelector with kubernetes.io/os: linux in the pod spec

B. Add a tolerations for Windows taints in the pod spec

C. Set the DaemonSet selector to os: linux label

D. Use a Deployment instead of DaemonSet with node affinity

Solution

1. Step 1: Identify how to target Linux nodes only

   Kubernetes labels nodes with kubernetes.io/os set to their OS. Using nodeSelector with kubernetes.io/os: linux restricts pods to Linux nodes.

2. Step 2: Evaluate other options

   Tolerations are for taints, not OS filtering. Selector matches pod labels, not nodes. Deployment does not guarantee one pod per node.

3. Final Answer:

   Add a nodeSelector with kubernetes.io/os: linux in the pod spec -> Option A

4. Quick Check:

   Use nodeSelector to filter nodes by OS [OK]

Hint: Use nodeSelector with OS label to filter nodes [OK]

Common Mistakes:

• Using selector to filter nodes instead of nodeSelector
• Confusing tolerations with node filtering
• Using Deployment instead of DaemonSet for per-node pods