Node affinity and anti-affinity in Kubernetes - Deep Dive

Overview - Node affinity and anti-affinity

What is it?

Node affinity and anti-affinity are rules in Kubernetes that help decide which nodes a pod should or should not run on. Node affinity lets you specify preferred or required conditions for nodes where pods can be scheduled. Anti-affinity is the opposite; it tells Kubernetes to avoid placing pods on certain nodes based on labels or other criteria. These rules help control pod placement to improve performance, availability, and resource use.

Why it matters

Without node affinity and anti-affinity, Kubernetes might place pods randomly or inefficiently, causing resource waste or failures. For example, critical pods might end up on the same node, risking downtime if that node fails. These rules let you guide Kubernetes to spread pods out or group them, improving reliability and making sure your app runs smoothly.

Where it fits

Before learning node affinity, you should understand basic Kubernetes concepts like pods, nodes, and labels. After mastering node affinity and anti-affinity, you can explore more advanced scheduling features like pod affinity, taints and tolerations, and custom schedulers.

Mental Model

Core Idea

Node affinity and anti-affinity are like setting preferences and restrictions for where your app pieces (pods) can live inside a cluster of computers (nodes).

Think of it like...

Imagine you are organizing guests at a party. Node affinity is like saying 'I want my friends to sit at tables with windows' (preferred spots), while anti-affinity is like saying 'I don’t want my noisy friends sitting at the same table' (avoid certain neighbors).

┌───────────────┐       ┌───────────────┐
│   Kubernetes  │       │     Nodes     │
│   Scheduler   │──────▶│  Node A (has  │
│               │       │  labels: zone=1│
│               │       └───────────────┘
│               │       ┌───────────────┐
│               │       │  Node B (has  │
│               │       │  labels: zone=2│
└───────────────┘       └───────────────┘

Pod with node affinity: zone=1
Pod with node anti-affinity: avoid zone=2

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Nodes and Pods

Concept: Learn what nodes and pods are in Kubernetes and how pods run on nodes.

In Kubernetes, a node is a worker machine where your applications run. A pod is the smallest unit that holds one or more containers. Pods need to be scheduled onto nodes to run. By default, Kubernetes schedules pods on any available node without special rules.

Result

You understand that pods run on nodes and that scheduling decides which node a pod uses.

Knowing the basic relationship between pods and nodes is essential before controlling where pods run.

2

Foundation

Using Labels to Identify Nodes

Concept: Nodes can have labels, which are key-value pairs used to describe their attributes.

You can add labels to nodes, like 'zone=us-east' or 'ssd=true'. These labels help Kubernetes and users identify nodes with certain features. Labels are the foundation for affinity rules because they let you select nodes based on their characteristics.

Result

Nodes have labels that can be used to select or avoid them when scheduling pods.

Labels turn nodes into searchable items, enabling targeted scheduling decisions.

3

Intermediate

Defining Node Affinity Rules

🤔Before reading on: do you think node affinity can only require nodes, or can it also prefer nodes? Commit to your answer.

Concept: Node affinity lets you specify required or preferred conditions for nodes where pods should be scheduled.

Node affinity has two types: requiredDuringSchedulingIgnoredDuringExecution (hard requirement) and preferredDuringSchedulingIgnoredDuringExecution (soft preference). Required means the pod will only run on nodes matching the rule. Preferred means Kubernetes tries to schedule on matching nodes but can choose others if needed. Example YAML snippet: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: zone operator: In values: - us-east preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: ssd operator: In values: - "true"

Result

Pods are scheduled only on nodes in 'us-east' zone and preferably on nodes with SSDs.

Understanding the difference between required and preferred affinity helps you balance strictness and flexibility in scheduling.

4

Intermediate

Using Node Anti-Affinity to Avoid Nodes

🤔Before reading on: do you think anti-affinity only applies to pods or can it also apply to nodes? Commit to your answer.

Concept: Node anti-affinity tells Kubernetes to avoid scheduling pods on nodes with certain labels or conditions.

Node anti-affinity is similar to node affinity but works in reverse. It prevents pods from running on nodes that match specified labels. This helps spread pods out or avoid problematic nodes. Example YAML snippet: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disktype operator: NotIn values: - hdd

Result

Pods will not be scheduled on nodes labeled with 'disktype=hdd'.

Knowing how to exclude nodes helps prevent resource conflicts and improve reliability.

5

Intermediate

Combining Affinity and Anti-Affinity Rules

Concept: You can combine node affinity and anti-affinity rules to precisely control pod placement.

Kubernetes allows you to specify multiple affinity and anti-affinity rules together. For example, you can require pods to run in a certain zone but avoid nodes with low memory. This combination gives fine control over scheduling. Example YAML snippet: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: zone operator: In values: - us-west - matchExpressions: - key: memory operator: NotIn values: - low

Result

Pods run only on nodes in 'us-west' zone that do not have 'memory=low' label.

Combining rules lets you tailor scheduling to complex real-world needs.

6

Advanced

How Kubernetes Scheduler Uses Affinity Rules

🤔Before reading on: do you think the scheduler applies affinity rules before or after checking node resources? Commit to your answer.

Concept: The Kubernetes scheduler filters and scores nodes based on affinity rules along with other factors to decide pod placement.

When scheduling a pod, Kubernetes first filters nodes that do not meet required affinity rules. Then it scores the remaining nodes based on preferred affinity and other criteria like resource availability. The highest scoring node is chosen. This process balances affinity preferences with cluster health and resource use.

Result

Pods are placed on nodes that satisfy hard rules and best match soft preferences while considering resources.

Understanding the scheduler’s decision process helps you write effective affinity rules that work well in real clusters.

7

Expert

Surprising Effects of Affinity in Large Clusters

🤔Before reading on: do you think strict affinity rules can cause pods to remain unscheduled? Commit to your answer.

Concept: Strict node affinity or anti-affinity can cause pods to stay pending if no nodes match, affecting availability.

In large clusters, overly strict affinity rules can reduce scheduling flexibility. If no nodes meet the required conditions, pods remain unscheduled, causing downtime. Also, combining multiple strict rules can fragment the cluster, reducing resource utilization. Experts monitor scheduling events and adjust rules to balance control and availability.

Result

Pods may remain pending if affinity rules are too restrictive, leading to resource waste and outages.

Knowing the risks of strict affinity prevents common production issues and guides better rule design.

Under the Hood

The Kubernetes scheduler uses a multi-step process: it first filters nodes that do not meet required affinity or anti-affinity rules. Then it scores the remaining nodes based on preferred affinity weights and other factors like resource availability. The scheduler picks the node with the highest score. Affinity rules are evaluated by matching pod selectors against node labels using logical operators like In, NotIn, Exists, and DoesNotExist.

Why designed this way?

Affinity and anti-affinity were designed to give users flexible control over pod placement without hardcoding node names. The separation into required and preferred rules balances strictness and flexibility. This design allows Kubernetes to optimize cluster utilization while respecting user constraints. Alternatives like fixed node assignment were too rigid and did not scale well.

┌───────────────┐
│ Pod to schedule│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Filter nodes   │
│ (required     │
│ affinity/anti-│
│ affinity)     │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Score nodes   │
│ (preferred   │
│ affinity)    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Select node   │
│ with highest  │
│ score         │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does node affinity guarantee pods will always run on preferred nodes? Commit yes or no.

Common Belief: Node affinity always guarantees pods run on preferred nodes.

Reality: Preferred node affinity is a soft rule; pods may run on other nodes if preferred ones are unavailable.

Why it matters: Assuming preferred affinity is strict can lead to confusion when pods run on unexpected nodes.

Quick: Can node anti-affinity be used to avoid nodes based on pod labels? Commit yes or no.

Common Belief: Node anti-affinity can only avoid nodes based on node labels.

Reality: Node anti-affinity only works with node labels, not pod labels; pod anti-affinity handles pod labels.

Why it matters: Mixing node and pod anti-affinity causes scheduling failures or unexpected pod placement.

Quick: Does combining multiple required affinity rules always increase scheduling success? Commit yes or no.

Common Belief: More required affinity rules always help schedule pods better.

Reality: Adding multiple required affinity rules can reduce eligible nodes, causing pods to remain unscheduled.

Why it matters: Overly strict rules reduce cluster flexibility and can cause downtime.

Quick: Is node affinity evaluated continuously after pod scheduling? Commit yes or no.

Common Belief: Node affinity is enforced continuously, so pods move if nodes change labels.

Reality: Node affinity is checked only during scheduling; pods do not move if node labels change later.

Why it matters: Expecting pods to move automatically can cause surprises in cluster behavior.

Expert Zone

1

Preferred affinity weights can be tuned to influence scheduler decisions subtly without blocking scheduling.

2

Node affinity rules are ignored during pod execution, so node label changes after scheduling do not affect running pods.

3

Combining node affinity with taints and tolerations provides powerful control over pod placement and node usage.

When NOT to use

Avoid strict required node affinity in highly dynamic clusters where nodes frequently join or leave. Instead, use preferred affinity or pod affinity to maintain flexibility. For workload spreading, consider pod anti-affinity or topology spread constraints as alternatives.

Production Patterns

In production, node affinity is used to place workloads on nodes with special hardware like GPUs or SSDs. Anti-affinity helps spread replicas across failure zones to improve availability. Teams combine affinity with taints and tolerations to isolate workloads and optimize resource usage.

Connections

Pod affinity and anti-affinity

Builds-on

Understanding node affinity helps grasp pod affinity, which controls pod placement relative to other pods, adding another layer of scheduling control.

Taints and tolerations

Complementary

Node affinity controls where pods prefer or avoid nodes, while taints and tolerations enforce strict node acceptance policies; together they provide full scheduling control.

Constraint satisfaction problems (CSP) in computer science

Same pattern

Node affinity and anti-affinity are practical examples of CSP, where the scheduler solves constraints to find valid pod placements, similar to puzzles or resource allocation problems.

Common Pitfalls

#1 Using required node affinity with labels that no node has.

Wrong approach:

spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nonexistent-label
operator: In
values:
- value1

Correct approach:

spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: nonexistent-label
operator: In
values:
- value1

Root cause:

Confusing required and preferred affinity causes pods to remain unscheduled if no nodes match.

#2 Mixing node anti-affinity with pod labels instead of node labels.

Wrong approach:

spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: app
operator: In
values:
- frontend

Correct approach:

spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- frontend
topologyKey: kubernetes.io/hostname

Root cause:

Confusing node labels with pod labels leads to wrong affinity type and scheduling errors.

#3 Assuming pods will move if node labels change after scheduling.

Wrong approach:

Changing node labels expecting pods to reschedule automatically.

Correct approach:

Manually evict pods or restart them to reschedule after node label changes.

Root cause:

Misunderstanding that affinity is only checked at scheduling time, not continuously.

Key Takeaways

Node affinity and anti-affinity let you control where pods run by specifying node label rules.

Required affinity rules are strict and must be met; preferred rules guide scheduling but allow flexibility.

Anti-affinity helps avoid nodes with certain labels, improving workload distribution and reliability.

Overly strict affinity rules can cause pods to remain unscheduled, so balance control with flexibility.

Affinity works with other Kubernetes features like taints and pod affinity to optimize cluster scheduling.

Practice

1. What does nodeAffinity in Kubernetes primarily control?

easy

A. Which nodes a pod can be scheduled on based on node labels

B. The order in which pods start in a deployment

C. The network policies applied to pods

D. The resource limits of a pod

Solution

1. Step 1: Understand nodeAffinity purpose

   Node affinity is used to specify rules about which nodes a pod can run on, based on labels assigned to nodes.

2. Step 2: Compare options to nodeAffinity function

   Options B, C, and D relate to pod startup order, network policies, and resource limits, which are unrelated to node affinity.

3. Final Answer:

   Which nodes a pod can be scheduled on based on node labels -> Option A

4. Quick Check:

   nodeAffinity controls pod placement on nodes [OK]

Hint: Node affinity = pod placement by node labels [OK]

Common Mistakes:

• Confusing node affinity with pod resource limits
• Thinking node affinity controls pod startup order
• Mixing node affinity with network policies

2. Which of the following is the correct syntax to specify a required node affinity in a pod spec?

easy

A. "nodeAffinity": {"preferredDuringSchedulingIgnoredDuringExecution": [{"weight": 1, "preference": {"matchExpressions": [{"key": "disktype", "operator": "In", "values": ["ssd"]}]}}]}

B. "podAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"labelSelector": {"matchLabels": {"disktype": "ssd"}}}}

C. "nodeSelector": {"disktype": "ssd"}

D. "nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "disktype", "operator": "In", "values": ["ssd"]}]}]}}

Solution

1. Step 1: Identify required node affinity syntax

   Required node affinity uses the key requiredDuringSchedulingIgnoredDuringExecution with nodeSelectorTerms and matchExpressions to specify node labels.

2. Step 2: Check options for correct structure

   "nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "disktype", "operator": "In", "values": ["ssd"]}]}]}} matches the required node affinity syntax. "nodeAffinity": {"preferredDuringSchedulingIgnoredDuringExecution": [{"weight": 1, "preference": {"matchExpressions": [{"key": "disktype", "operator": "In", "values": ["ssd"]}]}}]} is preferred affinity, C is nodeSelector (different syntax), and D is pod affinity, not node affinity.

3. Final Answer:

   "nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "disktype", "operator": "In", "values": ["ssd"]}]}]}} -> Option D

4. Quick Check:

   Required node affinity uses requiredDuringSchedulingIgnoredDuringExecution [OK]

Hint: Required node affinity uses requiredDuringSchedulingIgnoredDuringExecution [OK]

Common Mistakes:

• Confusing preferred and required affinity syntax
• Using podAffinity instead of nodeAffinity
• Mixing nodeSelector with nodeAffinity syntax

3. Given this pod spec snippet, what will happen if a node has label zone: us-east-1a but no label disk: ssd?

{
  "affinity": {
    "nodeAffinity": {
      "requiredDuringSchedulingIgnoredDuringExecution": {
        "nodeSelectorTerms": [
          {
            "matchExpressions": [
              {"key": "zone", "operator": "In", "values": ["us-east-1a"]},
              {"key": "disk", "operator": "In", "values": ["ssd"]}
            ]
          }
        ]
      }
    }
  }
}

medium

A. The pod will be scheduled only if preferredDuringScheduling is set

B. The pod will not be scheduled on the node because it lacks the required disk label

C. The pod will be scheduled but with a warning

D. The pod will be scheduled because the zone label matches

Solution

1. Step 1: Analyze required node affinity conditions

   Required node affinity means all match expressions must be true for scheduling. The node must have both zone: us-east-1a and disk: ssd labels.

2. Step 2: Check node labels against requirements

   The node has zone: us-east-1a but lacks disk: ssd. This fails the required condition, so the pod cannot be scheduled there.

3. Final Answer:

   The pod will not be scheduled on the node because it lacks the required disk label -> Option B

4. Quick Check:

   Required affinity needs all labels matched [OK]

Hint: Required affinity means all labels must match [OK]

Common Mistakes:

• Assuming partial label match is enough
• Confusing required with preferred affinity
• Thinking pod schedules with warnings on missing labels

4. You wrote this pod spec with pod anti-affinity but pods are still scheduled on the same node. What is the likely error?

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchLabels:
            app: myapp
        topologyKey: "kubernetes.io/hostname"

medium

A. Incorrect labelSelector syntax causes failure

B. Missing nodeSelector in the pod spec

C. Using preferred instead of required anti-affinity allows scheduling on same node

D. topologyKey should be set to "zone" instead of hostname

Solution

1. Step 1: Understand pod anti-affinity types

   Preferred anti-affinity suggests avoiding co-location but does not enforce it strictly. Required anti-affinity enforces pods not to be scheduled together.

2. Step 2: Analyze the pod spec anti-affinity type

   The spec uses preferredDuringSchedulingIgnoredDuringExecution, which is a soft rule. This allows pods to be scheduled on the same node if needed.

3. Final Answer:

   Using preferred instead of required anti-affinity allows scheduling on same node -> Option C

4. Quick Check:

   Preferred anti-affinity is a soft rule [OK]

Hint: Preferred anti-affinity is soft; use required to enforce [OK]

Common Mistakes:

• Confusing preferred and required anti-affinity
• Assuming labelSelector syntax is wrong without checking
• Thinking topologyKey must be zone for anti-affinity

5. You want to ensure pods with label app=frontend never run on the same node as pods with label app=backend. Which affinity configuration achieves this?

hard

A. Use podAntiAffinity with requiredDuringSchedulingIgnoredDuringExecution and labelSelector matching app=backend

B. Use nodeAffinity with requiredDuringSchedulingIgnoredDuringExecution and labelSelector matching app=backend

C. Use podAffinity with preferredDuringSchedulingIgnoredDuringExecution and labelSelector matching app=backend

D. Use nodeSelector with label app=backend

Solution

1. Step 1: Identify requirement to prevent co-location of pods

   Pod anti-affinity prevents pods from running on the same node as other pods matching certain labels.

2. Step 2: Choose required anti-affinity for strict enforcement

   Using requiredDuringSchedulingIgnoredDuringExecution enforces pods never co-locate on the same node.

3. Step 3: Match labelSelector to pods to avoid

   The labelSelector should match pods with app=backend to prevent frontend pods from scheduling on same node.

4. Final Answer:

   Use podAntiAffinity with requiredDuringSchedulingIgnoredDuringExecution and labelSelector matching app=backend -> Option A

5. Quick Check:

   Pod anti-affinity + required = strict pod separation [OK]

Hint: Use required podAntiAffinity to strictly separate pods [OK]

Common Mistakes:

• Using nodeAffinity instead of podAntiAffinity for pod separation
• Using preferred instead of required anti-affinity for strict rules
• Confusing podAffinity with podAntiAffinity