Node affinity and anti-affinity in Kubernetes - Deep Dive

Overview - Node affinity and anti-affinity

What is it?

Node affinity and anti-affinity are rules in Kubernetes that help decide which nodes a pod should or should not run on. Node affinity lets you specify preferred or required conditions for nodes where pods can be scheduled. Anti-affinity is the opposite; it tells Kubernetes to avoid placing pods on certain nodes based on labels or other criteria. These rules help control pod placement to improve performance, availability, and resource use.

Why it matters

Without node affinity and anti-affinity, Kubernetes might place pods randomly or inefficiently, causing resource waste or failures. For example, critical pods might end up on the same node, risking downtime if that node fails. These rules let you guide Kubernetes to spread pods out or group them, improving reliability and making sure your app runs smoothly.

Where it fits

Before learning node affinity, you should understand basic Kubernetes concepts like pods, nodes, and labels. After mastering node affinity and anti-affinity, you can explore more advanced scheduling features like pod affinity, taints and tolerations, and custom schedulers.

Mental Model

Core Idea

Node affinity and anti-affinity are like setting preferences and restrictions for where your app pieces (pods) can live inside a cluster of computers (nodes).

Think of it like...

Imagine you are organizing guests at a party. Node affinity is like saying 'I want my friends to sit at tables with windows' (preferred spots), while anti-affinity is like saying 'I don’t want my noisy friends sitting at the same table' (avoid certain neighbors).

┌───────────────┐       ┌───────────────┐
│   Kubernetes  │       │     Nodes     │
│   Scheduler   │──────▶│  Node A (has  │
│               │       │  labels: zone=1│
│               │       └───────────────┘
│               │       ┌───────────────┐
│               │       │  Node B (has  │
│               │       │  labels: zone=2│
└───────────────┘       └───────────────┘

Pod with node affinity: zone=1
Pod with node anti-affinity: avoid zone=2

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Nodes and Pods

Concept: Learn what nodes and pods are in Kubernetes and how pods run on nodes.

In Kubernetes, a node is a worker machine where your applications run. A pod is the smallest unit that holds one or more containers. Pods need to be scheduled onto nodes to run. By default, Kubernetes schedules pods on any available node without special rules.

Result

You understand that pods run on nodes and that scheduling decides which node a pod uses.

Knowing the basic relationship between pods and nodes is essential before controlling where pods run.

2

Foundation

Using Labels to Identify Nodes

Concept: Nodes can have labels, which are key-value pairs used to describe their attributes.

You can add labels to nodes, like 'zone=us-east' or 'ssd=true'. These labels help Kubernetes and users identify nodes with certain features. Labels are the foundation for affinity rules because they let you select nodes based on their characteristics.

Result

Nodes have labels that can be used to select or avoid them when scheduling pods.

Labels turn nodes into searchable items, enabling targeted scheduling decisions.

3

Intermediate

Defining Node Affinity Rules

🤔Before reading on: do you think node affinity can only require nodes, or can it also prefer nodes? Commit to your answer.

Concept: Node affinity lets you specify required or preferred conditions for nodes where pods should be scheduled.

Node affinity has two types: requiredDuringSchedulingIgnoredDuringExecution (hard requirement) and preferredDuringSchedulingIgnoredDuringExecution (soft preference). Required means the pod will only run on nodes matching the rule. Preferred means Kubernetes tries to schedule on matching nodes but can choose others if needed. Example YAML snippet: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: zone operator: In values: - us-east preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: ssd operator: In values: - "true"

Result

Pods are scheduled only on nodes in 'us-east' zone and preferably on nodes with SSDs.

Understanding the difference between required and preferred affinity helps you balance strictness and flexibility in scheduling.

4

Intermediate

Using Node Anti-Affinity to Avoid Nodes

🤔Before reading on: do you think anti-affinity only applies to pods or can it also apply to nodes? Commit to your answer.

Concept: Node anti-affinity tells Kubernetes to avoid scheduling pods on nodes with certain labels or conditions.

Node anti-affinity is similar to node affinity but works in reverse. It prevents pods from running on nodes that match specified labels. This helps spread pods out or avoid problematic nodes. Example YAML snippet: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disktype operator: NotIn values: - hdd

Result

Pods will not be scheduled on nodes labeled with 'disktype=hdd'.

Knowing how to exclude nodes helps prevent resource conflicts and improve reliability.

5

Intermediate

Combining Affinity and Anti-Affinity Rules

Concept: You can combine node affinity and anti-affinity rules to precisely control pod placement.

Kubernetes allows you to specify multiple affinity and anti-affinity rules together. For example, you can require pods to run in a certain zone but avoid nodes with low memory. This combination gives fine control over scheduling. Example YAML snippet: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: zone operator: In values: - us-west - matchExpressions: - key: memory operator: NotIn values: - low

Result

Pods run only on nodes in 'us-west' zone that do not have 'memory=low' label.

Combining rules lets you tailor scheduling to complex real-world needs.

6

Advanced

How Kubernetes Scheduler Uses Affinity Rules

🤔Before reading on: do you think the scheduler applies affinity rules before or after checking node resources? Commit to your answer.

Concept: The Kubernetes scheduler filters and scores nodes based on affinity rules along with other factors to decide pod placement.

When scheduling a pod, Kubernetes first filters nodes that do not meet required affinity rules. Then it scores the remaining nodes based on preferred affinity and other criteria like resource availability. The highest scoring node is chosen. This process balances affinity preferences with cluster health and resource use.

Result

Pods are placed on nodes that satisfy hard rules and best match soft preferences while considering resources.

Understanding the scheduler’s decision process helps you write effective affinity rules that work well in real clusters.

7

Expert

Surprising Effects of Affinity in Large Clusters

🤔Before reading on: do you think strict affinity rules can cause pods to remain unscheduled? Commit to your answer.

Concept: Strict node affinity or anti-affinity can cause pods to stay pending if no nodes match, affecting availability.

In large clusters, overly strict affinity rules can reduce scheduling flexibility. If no nodes meet the required conditions, pods remain unscheduled, causing downtime. Also, combining multiple strict rules can fragment the cluster, reducing resource utilization. Experts monitor scheduling events and adjust rules to balance control and availability.

Result

Pods may remain pending if affinity rules are too restrictive, leading to resource waste and outages.

Knowing the risks of strict affinity prevents common production issues and guides better rule design.

Under the Hood

The Kubernetes scheduler uses a multi-step process: it first filters nodes that do not meet required affinity or anti-affinity rules. Then it scores the remaining nodes based on preferred affinity weights and other factors like resource availability. The scheduler picks the node with the highest score. Affinity rules are evaluated by matching pod selectors against node labels using logical operators like In, NotIn, Exists, and DoesNotExist.

Why designed this way?

Affinity and anti-affinity were designed to give users flexible control over pod placement without hardcoding node names. The separation into required and preferred rules balances strictness and flexibility. This design allows Kubernetes to optimize cluster utilization while respecting user constraints. Alternatives like fixed node assignment were too rigid and did not scale well.

┌───────────────┐
│ Pod to schedule│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Filter nodes   │
│ (required     │
│ affinity/anti-│
│ affinity)     │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Score nodes   │
│ (preferred   │
│ affinity)    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Select node   │
│ with highest  │
│ score         │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does node affinity guarantee pods will always run on preferred nodes? Commit yes or no.

Common Belief: Node affinity always guarantees pods run on preferred nodes.

Reality: Preferred node affinity is a soft rule; pods may run on other nodes if preferred ones are unavailable.

Why it matters: Assuming preferred affinity is strict can lead to confusion when pods run on unexpected nodes.

Quick: Can node anti-affinity be used to avoid nodes based on pod labels? Commit yes or no.

Common Belief: Node anti-affinity can only avoid nodes based on node labels.

Reality: Node anti-affinity only works with node labels, not pod labels; pod anti-affinity handles pod labels.

Why it matters: Mixing node and pod anti-affinity causes scheduling failures or unexpected pod placement.

Quick: Does combining multiple required affinity rules always increase scheduling success? Commit yes or no.

Common Belief: More required affinity rules always help schedule pods better.

Reality: Adding multiple required affinity rules can reduce eligible nodes, causing pods to remain unscheduled.

Why it matters: Overly strict rules reduce cluster flexibility and can cause downtime.

Quick: Is node affinity evaluated continuously after pod scheduling? Commit yes or no.

Common Belief: Node affinity is enforced continuously, so pods move if nodes change labels.

Reality: Node affinity is checked only during scheduling; pods do not move if node labels change later.

Why it matters: Expecting pods to move automatically can cause surprises in cluster behavior.

Expert Zone

1

Preferred affinity weights can be tuned to influence scheduler decisions subtly without blocking scheduling.

2

Node affinity rules are ignored during pod execution, so node label changes after scheduling do not affect running pods.

3

Combining node affinity with taints and tolerations provides powerful control over pod placement and node usage.

When NOT to use

Avoid strict required node affinity in highly dynamic clusters where nodes frequently join or leave. Instead, use preferred affinity or pod affinity to maintain flexibility. For workload spreading, consider pod anti-affinity or topology spread constraints as alternatives.

Production Patterns

In production, node affinity is used to place workloads on nodes with special hardware like GPUs or SSDs. Anti-affinity helps spread replicas across failure zones to improve availability. Teams combine affinity with taints and tolerations to isolate workloads and optimize resource usage.

Connections

Pod affinity and anti-affinity

Builds-on

Understanding node affinity helps grasp pod affinity, which controls pod placement relative to other pods, adding another layer of scheduling control.

Taints and tolerations

Complementary

Node affinity controls where pods prefer or avoid nodes, while taints and tolerations enforce strict node acceptance policies; together they provide full scheduling control.

Constraint satisfaction problems (CSP) in computer science

Same pattern

Node affinity and anti-affinity are practical examples of CSP, where the scheduler solves constraints to find valid pod placements, similar to puzzles or resource allocation problems.

Common Pitfalls

#1 Using required node affinity with labels that no node has.

Wrong approach:

spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nonexistent-label
operator: In
values:
- value1

Correct approach:

spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: nonexistent-label
operator: In
values:
- value1

Root cause:

Confusing required and preferred affinity causes pods to remain unscheduled if no nodes match.

#2 Mixing node anti-affinity with pod labels instead of node labels.

Wrong approach:

spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: app
operator: In
values:
- frontend

Correct approach:

spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- frontend
topologyKey: kubernetes.io/hostname

Root cause:

Confusing node labels with pod labels leads to wrong affinity type and scheduling errors.

#3 Assuming pods will move if node labels change after scheduling.

Wrong approach:

Changing node labels expecting pods to reschedule automatically.

Correct approach:

Manually evict pods or restart them to reschedule after node label changes.

Root cause:

Misunderstanding that affinity is only checked at scheduling time, not continuously.

Key Takeaways

Node affinity and anti-affinity let you control where pods run by specifying node label rules.

Required affinity rules are strict and must be met; preferred rules guide scheduling but allow flexibility.

Anti-affinity helps avoid nodes with certain labels, improving workload distribution and reliability.

Overly strict affinity rules can cause pods to remain unscheduled, so balance control with flexibility.

Affinity works with other Kubernetes features like taints and pod affinity to optimize cluster scheduling.