Taints and tolerations in Kubernetes - Deep Dive

Overview - Taints and tolerations

What is it?

Taints and tolerations are Kubernetes features that control which pods can be scheduled on which nodes. A taint on a node marks it as special or restricted, preventing pods from running there unless they have a matching toleration. Tolerations are set on pods to allow them to be scheduled on nodes with specific taints. This system helps manage workloads by controlling pod placement based on node conditions or roles.

Why it matters

Without taints and tolerations, Kubernetes would schedule pods on any node without restrictions, which can cause problems like running critical workloads on unsuitable nodes or mixing incompatible pods. This could lead to resource conflicts, degraded performance, or security risks. Taints and tolerations solve this by giving cluster administrators fine control over pod placement, improving reliability and efficiency.

Where it fits

Before learning taints and tolerations, you should understand basic Kubernetes concepts like nodes, pods, and scheduling. After mastering this topic, you can explore advanced scheduling features like node affinity, pod affinity/anti-affinity, and custom schedulers to further control workload placement.

Mental Model

Core Idea

Taints mark nodes as restricted, and tolerations on pods allow them to ignore those restrictions to be scheduled there.

Think of it like...

Imagine a parking lot where some spots have 'Reserved' signs (taints). Only cars with a special permit (tolerations) can park in those spots. Without the permit, cars must park elsewhere.

Nodes with taints ──┐
                     │
Pods with matching tolerations ──> Allowed to schedule
                     │
Pods without tolerations ──> Not scheduled on tainted nodes

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Nodes and Pods

Concept: Learn what nodes and pods are in Kubernetes and how pods get assigned to nodes.

Nodes are machines (virtual or physical) where Kubernetes runs your applications. Pods are the smallest units that hold your application containers. The Kubernetes scheduler decides which pod runs on which node based on available resources and constraints.

Result

You know that pods need nodes to run and that scheduling decides pod placement.

Understanding nodes and pods is essential because taints and tolerations control how pods get placed on nodes.

2

Foundation

What Are Node Taints?

Concept: Taints are special labels on nodes that repel pods unless the pods tolerate them.

A taint has three parts: a key, a value, and an effect (NoSchedule, PreferNoSchedule, or NoExecute). For example, a node can be tainted with key=key1, value=value1, effect=NoSchedule. This means no pod can schedule on this node unless it has a matching toleration.

Result

Nodes with taints reject pods without matching tolerations.

Knowing that taints repel pods helps you control which nodes can run certain pods.

3

Intermediate

How Pod Tolerations Work

🤔Before reading on: do you think tolerations remove taints from nodes or just allow pods to ignore them? Commit to your answer.

Concept: Tolerations on pods allow them to be scheduled on nodes with matching taints by 'tolerating' the taint without removing it.

A toleration specifies the key, value, and effect it tolerates. If a pod has a toleration matching a node's taint, the scheduler allows the pod to run there. For example, a pod with toleration key=key1, value=value1, effect=NoSchedule can be scheduled on a node tainted with the same.

Result

Pods with matching tolerations can be scheduled on tainted nodes; others cannot.

Understanding that tolerations let pods ignore taints without changing nodes is key to flexible scheduling.

4

Intermediate

Different Taint Effects and Their Impact

🤔Before reading on: which taint effect do you think immediately evicts pods—NoSchedule or NoExecute? Commit to your answer.

Concept: Taints have three effects: NoSchedule prevents new pods without tolerations from scheduling; PreferNoSchedule tries to avoid scheduling but doesn't guarantee it; NoExecute evicts existing pods without tolerations and prevents new ones.

NoSchedule means pods without tolerations won't be placed on the node. PreferNoSchedule is a soft preference to avoid scheduling pods there. NoExecute actively removes pods without tolerations from the node and blocks new ones.

Result

You can control pod scheduling and eviction behavior precisely using taint effects.

Knowing the difference between effects helps you manage node usage and pod lifecycle effectively.

5

Intermediate

Applying Taints and Tolerations in Practice

Concept: Learn how to add taints to nodes and tolerations to pods using kubectl and YAML manifests.

To taint a node: kubectl taint nodes node1 key=value:NoSchedule To add tolerations to a pod, include this in the pod spec: tolerations: - key: "key" operator: "Equal" value: "value" effect: "NoSchedule" This setup allows the pod to schedule on the tainted node.

Result

You can control pod placement by applying taints and tolerations in your cluster.

Knowing the commands and YAML structure empowers you to enforce scheduling policies.

6

Advanced

Combining Taints with Node Affinity

🤔Before reading on: do you think node affinity overrides taints or works alongside them? Commit to your answer.

Concept: Taints and tolerations work together with node affinity to provide layered scheduling controls, where all conditions must be met for pod placement.

Node affinity lets you specify preferred or required node labels for pods. Even if a pod matches node affinity, it still needs tolerations for any taints on that node. This combination allows precise control over pod placement based on node labels and taints.

Result

Pods are scheduled only on nodes that satisfy both affinity and toleration rules.

Understanding how multiple scheduling constraints combine helps design robust placement strategies.

7

Expert

Taints and Tolerations Internals and Edge Cases

🤔Before reading on: do you think a pod with no tolerations can ever run on a tainted node with PreferNoSchedule? Commit to your answer.

Concept: The Kubernetes scheduler checks taints and tolerations during scheduling and eviction phases, with subtle behaviors for PreferNoSchedule and NoExecute effects that can surprise even experienced users.

PreferNoSchedule is a soft preference; the scheduler tries to avoid placing pods without tolerations but may do so if no alternatives exist. NoExecute causes immediate eviction of pods without tolerations. Also, tolerations can use operators like Exists to tolerate any value for a key. These details affect pod stability and scheduling decisions.

Result

You understand nuanced scheduler behavior and can predict pod placement and eviction accurately.

Knowing scheduler internals and edge cases prevents unexpected pod evictions and scheduling failures in production.

Under the Hood

When the Kubernetes scheduler runs, it checks each node's taints against the pod's tolerations. If the pod tolerates all taints on a node, it can be scheduled there. For NoExecute taints, the kubelet on the node also monitors running pods and evicts those without matching tolerations. The scheduler uses taints as hard or soft constraints depending on the effect, influencing its decision-making process.

Why designed this way?

Taints and tolerations were designed to give cluster operators a flexible way to isolate workloads, handle special hardware or node conditions, and enforce policies without modifying pod specs globally. Alternatives like node labels alone were insufficient because they only attract pods but cannot repel them. Taints provide a repelling mechanism, balancing control and flexibility.

┌─────────────┐       ┌───────────────┐       ┌───────────────┐
│   Node 1    │──────▶│  Has Taints   │──────▶│ Scheduler     │
│ (with taint)│       │ key=value:NoSchedule │  │ checks pod's  │
└─────────────┘       └───────────────┘       │ tolerations   │
                                                └───────────────┘
                                                      │
                                                      ▼
                                             ┌─────────────────┐
                                             │ Pod tolerates?  │
                                             ├─────────┬───────┤
                                             │ Yes     │ No    │
                                             ▼         ▼       
                                      Schedule pod  Reject pod
                                      on node       from node

Myth Busters - 4 Common Misconceptions

Quick: Does a toleration remove a taint from a node? Commit yes or no.

Common Belief: Tolerations remove taints from nodes so pods can schedule there.

Reality: Tolerations do not remove taints; they only allow pods to ignore taints when scheduling.

Why it matters: Thinking tolerations remove taints leads to confusion about node states and unexpected pod placements.

Quick: Does PreferNoSchedule guarantee pods won't schedule on tainted nodes? Commit yes or no.

Common Belief: PreferNoSchedule taints strictly prevent pods without tolerations from scheduling on nodes.

Reality: PreferNoSchedule is a soft preference; pods without tolerations may still schedule if no better nodes exist.

Why it matters: Misunderstanding this can cause surprise when pods appear on nodes thought to be off-limits.

Quick: Can a pod without any tolerations run on a node with a NoExecute taint? Commit yes or no.

Common Belief: Pods without tolerations can stay running on nodes with NoExecute taints indefinitely.

Reality: NoExecute taints cause immediate eviction of pods without matching tolerations.

Why it matters: Ignoring this can cause critical pods to be unexpectedly evicted, disrupting services.

Quick: Does a pod need to tolerate all taints on a node to be scheduled there? Commit yes or no.

Common Belief: Pods only need to tolerate some taints on a node to schedule there.

Reality: Pods must tolerate every taint on a node to be scheduled there.

Why it matters: Partial toleration leads to scheduling failures and confusion about pod placement.

Expert Zone

1

Tolerations can use the 'Exists' operator to tolerate any value for a taint key, enabling flexible pod scheduling without specifying exact values.

2

NoExecute taints affect both scheduling and runtime pod eviction, requiring coordination between the scheduler and kubelet for consistent behavior.

3

PreferNoSchedule taints influence scheduler scoring but do not guarantee pod placement avoidance, which can cause subtle scheduling outcomes under resource pressure.

When NOT to use

Avoid using taints and tolerations when simple node labeling and affinity rules suffice, as they add complexity. For fine-grained resource isolation, consider using Kubernetes namespaces and resource quotas instead. Also, custom schedulers or admission controllers may be better for complex placement policies.

Production Patterns

In production, taints and tolerations are used to isolate critical workloads on dedicated nodes, mark nodes with special hardware (like GPUs), or quarantine nodes undergoing maintenance. They are combined with node affinity and pod priority to ensure high availability and efficient resource use.

Connections

Node Affinity

Works alongside taints and tolerations to control pod placement by specifying preferred or required node labels.

Understanding how node affinity and taints combine helps design precise scheduling policies that consider both attraction and repulsion.

Access Control Lists (ACLs)

Similar pattern of allowing or denying access based on matching rules.

Knowing ACLs in networking helps grasp how taints repel pods and tolerations grant exceptions, reflecting a common control pattern.

Traffic Lights in Road Systems

Taints act like red lights preventing traffic (pods) from entering nodes unless they have permission (tolerations).

Recognizing this control mechanism clarifies how Kubernetes manages pod flow to nodes, balancing safety and efficiency.

Common Pitfalls

#1 Applying a taint without adding matching tolerations to pods that must run there.

Wrong approach:

kubectl taint nodes node1 key=value:NoSchedule
# Deploy pod without tolerations

Correct approach:

kubectl taint nodes node1 key=value:NoSchedule
# Pod spec includes:
# tolerations:
# - key: "key"
# operator: "Equal"
# value: "value"
# effect: "NoSchedule"

Root cause:

Not understanding that pods need explicit tolerations to run on tainted nodes causes scheduling failures.

#2 Using PreferNoSchedule taint expecting strict pod exclusion.

Wrong approach:

kubectl taint nodes node1 key=value:PreferNoSchedule
# Expect pods without tolerations never to schedule here

Correct approach:

Use NoSchedule effect for strict exclusion:
kubectl taint nodes node1 key=value:NoSchedule

Root cause:

Confusing soft preference (PreferNoSchedule) with hard exclusion (NoSchedule) leads to unexpected pod placements.

#3 Assuming tolerations remove taints from nodes.

Wrong approach:

Pod spec with tolerations but no taint removal commands on nodes

Correct approach:

To remove taints:
kubectl taint nodes node1 key=value:NoSchedule-

Root cause:

Misunderstanding the difference between tolerating a taint and removing it causes misconfiguration.

Key Takeaways

Taints repel pods from nodes unless pods have matching tolerations, enabling controlled pod placement.

Tolerations allow pods to ignore node taints but do not remove the taints themselves.

Different taint effects (NoSchedule, PreferNoSchedule, NoExecute) control scheduling and eviction behaviors with varying strictness.

Combining taints and tolerations with node affinity provides powerful, layered scheduling controls.

Understanding scheduler behavior and taint effects prevents unexpected pod evictions and placement issues in production.