Pod priority and preemption in Kubernetes - Deep Dive

Overview - Pod priority and preemption

What is it?

Pod priority and preemption is a Kubernetes feature that helps decide which pods get to run when resources are limited. Each pod can be assigned a priority value, and when the cluster runs out of resources, lower priority pods can be stopped (preempted) to make room for higher priority pods. This ensures important workloads get the resources they need. It works automatically based on the priority values set by the user or system.

Why it matters

Without pod priority and preemption, all pods would compete equally for resources, causing critical applications to slow down or fail when the cluster is busy. This could lead to downtime or poor user experience. Priority and preemption solve this by guaranteeing that important pods run first, improving reliability and resource use in busy clusters.

Where it fits

Before learning pod priority and preemption, you should understand basic Kubernetes concepts like pods, nodes, and resource requests/limits. After this, you can learn about advanced scheduling, resource quotas, and cluster autoscaling to manage resources efficiently.

Mental Model

Core Idea

Pod priority and preemption lets Kubernetes decide which pods to run first by assigning importance levels and stopping less important pods when needed.

Think of it like...

Imagine a busy elevator with limited space. People with VIP passes (high priority pods) get to enter first, and if the elevator is full, regular passengers (low priority pods) might be asked to wait outside to make room.

┌───────────────┐
│   Kubernetes  │
│   Scheduler   │
└──────┬────────┘
       │
       ▼
┌───────────────┐       ┌───────────────┐
│ Pod Priority  │──────▶│ Preemption    │
│ Assigned per  │       │ Stops low     │
│ Pod           │       │ priority pods │
└───────────────┘       └───────────────┘
       │                      ▲
       └───────────────┬──────┘
                       ▼
               ┌───────────────┐
               │ Resource      │
               │ Availability  │
               └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Pods

Concept: Pods are the smallest units that run containers in Kubernetes.

A pod is like a small box that holds one or more containers. Kubernetes runs pods on nodes (computers). Each pod requests some resources like CPU and memory to run properly.

Result

You know what a pod is and that it needs resources to run on a node.

Understanding pods is essential because priority and preemption work by managing pods based on their resource needs and importance.

2

Foundation

Resource Requests and Limits Basics

Concept: Pods declare how much CPU and memory they need to run using requests and limits.

When you create a pod, you specify resource requests (minimum needed) and limits (maximum allowed). The scheduler uses requests to decide where to place pods. If the node doesn't have enough resources, the pod won't be scheduled.

Result

You can predict if a pod will fit on a node based on its resource requests.

Knowing resource requests is key because priority and preemption depend on resource availability to decide which pods run or get stopped.

3

Intermediate

Assigning Pod Priority Classes

🤔Before reading on: do you think pod priority is a number or a label? Commit to your answer.

Concept: Pod priority is set using PriorityClasses, which assign numeric values to pods.

Kubernetes uses PriorityClass objects to define priority levels. Each PriorityClass has a name and a numeric value. Higher numbers mean higher priority. Pods reference a PriorityClass by name to get their priority.

Result

You can create PriorityClasses and assign them to pods to control their importance.

Understanding PriorityClasses lets you control pod importance explicitly, which is the foundation for preemption decisions.

4

Intermediate

How Preemption Works in Scheduling

🤔Before reading on: do you think preemption stops pods before or after scheduling fails? Commit to your answer.

Concept: Preemption happens when the scheduler can't place a high priority pod due to lack of resources, so it stops lower priority pods to free space.

When a high priority pod can't be scheduled, Kubernetes looks for lower priority pods to evict (stop). It chooses pods that free enough resources to fit the high priority pod. Evicted pods are terminated and may be restarted later.

Result

High priority pods get scheduled by removing lower priority pods if needed.

Knowing preemption timing helps you understand how Kubernetes balances fairness and importance during resource shortages.

5

Intermediate

Configuring Priority and Preemption in Pods

Concept: You set pod priority by referencing a PriorityClass in the pod spec, enabling preemption automatically.

In the pod YAML, add the field 'priorityClassName' with the name of the PriorityClass. Kubernetes uses this to assign priority. Preemption is enabled by default but can be disabled per pod with 'preemptionPolicy'.

Result

Pods run with assigned priorities and can preempt others if needed.

Configuring priority in pod specs is how you tell Kubernetes which pods matter most in your workloads.

6

Advanced

Handling Pod Disruption and Grace Periods

🤔Before reading on: do you think preempted pods are stopped instantly or given time to shut down? Commit to your answer.

Concept: Preempted pods receive a termination notice and have a grace period to shut down cleanly before being killed.

When a pod is preempted, Kubernetes sends a SIGTERM signal and waits for the pod's terminationGracePeriodSeconds before forcefully stopping it. This allows pods to save state or clean up.

Result

Pods can handle preemption gracefully, reducing data loss or corruption.

Understanding pod termination during preemption helps design resilient applications that handle interruptions smoothly.

7

Expert

Priority and Preemption Impact on Cluster Stability

🤔Before reading on: do you think aggressive preemption always improves cluster performance? Commit to your answer.

Concept: Excessive preemption can cause instability by repeatedly stopping pods, so careful priority design and limits are needed.

If many pods have high priority or preemption is too aggressive, pods may be evicted frequently, causing thrashing and degraded performance. Kubernetes administrators balance priorities and use features like PodDisruptionBudgets to limit disruptions.

Result

Clusters remain stable and workloads run smoothly with balanced priority and preemption settings.

Knowing the tradeoffs of preemption prevents misconfiguration that harms cluster reliability and user experience.

Under the Hood

Kubernetes scheduler evaluates pod priorities as numeric values. When scheduling a pod, if resources are insufficient, it searches for lower priority pods to evict. It selects pods that free enough resources with minimal disruption. Evicted pods receive termination signals and are removed from nodes. The scheduler then places the high priority pod. This process repeats dynamically as cluster state changes.

Why designed this way?

This design balances fairness and importance by allowing critical workloads to run even in resource scarcity. Alternatives like static reservations waste resources or manual intervention. Preemption automates resource reallocation while respecting pod importance and graceful shutdown.

┌───────────────┐
│ Pod to Schedule│
│ (High Priority)│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Scheduler     │
│ Checks Node   │
│ Resources     │
└──────┬────────┘
       │
       ▼
┌───────────────┐       ┌───────────────┐
│ Enough Space? │──No──▶│ Find Lower    │
└──────┬────────┘       │ Priority Pods │
       │Yes             └──────┬────────┘
       ▼                      │
┌───────────────┐             ▼
│ Schedule Pod  │       ┌───────────────┐
└───────────────┘       │ Evict Pods    │
                        │ (Preemption) │
                        └──────┬────────┘
                               │
                               ▼
                      ┌───────────────┐
                      │ Pod Scheduled │
                      └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does a higher priority pod always preempt all lower priority pods immediately? Commit yes or no.

Common Belief: Higher priority pods instantly stop all lower priority pods when scheduled.

Reality: Preemption only happens if the scheduler cannot find enough free resources; it evicts only enough pods to fit the high priority pod, not all lower priority pods.

Why it matters: Thinking preemption is immediate can lead to overestimating its impact and misconfiguring priorities, causing unexpected pod evictions.

Quick: Can a pod without a priorityClass preempt others? Commit yes or no.

Common Belief: All pods can preempt others regardless of priority settings.

Reality: Only pods with assigned priorityClass and higher priority than running pods can preempt others; pods without priorityClass have default low priority and cannot preempt.

Why it matters: Assuming all pods can preempt leads to confusion about scheduling behavior and resource allocation.

Quick: Does disabling preemption on a pod prevent it from being evicted? Commit yes or no.

Common Belief: Disabling preemption on a pod means it cannot be evicted by others.

Reality: Disabling preemption only stops the pod from preempting others; it can still be evicted if lower priority than incoming pods.

Why it matters: Misunderstanding this causes wrong assumptions about pod protection and can lead to unexpected pod terminations.

Quick: Does preemption guarantee that a high priority pod will always be scheduled? Commit yes or no.

Common Belief: Preemption ensures high priority pods always get scheduled immediately.

Reality: Preemption helps but does not guarantee scheduling if cluster resources are insufficient even after evictions.

Why it matters: Overreliance on preemption can cause false confidence in workload availability and poor capacity planning.

Expert Zone

1

Priority values are integers but their relative difference matters more than absolute numbers; small gaps can cause unexpected preemptions.

2

PodDisruptionBudgets can limit preemption impact by preventing eviction of too many pods at once, balancing availability and priority.

3

Preemption decisions consider pod topology and affinity rules, so evicting a pod may not always free usable resources for the high priority pod.

When NOT to use

Avoid using pod priority and preemption in small clusters with stable workloads where manual scheduling suffices. Instead, use resource quotas and node taints for simpler control. Also, do not rely on preemption for bursty workloads without autoscaling, as it can cause instability.

Production Patterns

In production, teams define multiple PriorityClasses for system-critical, business-critical, and best-effort workloads. They combine priority with PodDisruptionBudgets and cluster autoscaling to maintain stability. Preemption is monitored to avoid thrashing, and critical pods often have high priority with reserved resources.

Connections

Operating System Process Scheduling

Pod priority and preemption in Kubernetes is similar to how OS schedulers prioritize processes and preempt lower priority ones.

Understanding OS scheduling helps grasp how Kubernetes balances workload importance and resource contention dynamically.

Traffic Management in Networks

Priority and preemption resemble Quality of Service (QoS) in networks, where important data packets get priority over less critical ones.

Knowing network QoS concepts clarifies how Kubernetes ensures critical workloads get resources first under congestion.

Emergency Room Triage

Pod priority and preemption works like triage in hospitals, where patients with severe conditions get treated first, even if it means delaying others.

This real-world triage analogy helps understand the fairness and urgency balance in resource allocation.

Common Pitfalls

#1 Assigning all pods the same high priority, causing no effective preemption.

Wrong approach:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000
globalDefault: true

# All pods use 'high-priority' class

Correct approach:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: system-critical
value: 1000

---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: batch-jobs
value: 100

Root cause:

Misunderstanding that priority needs meaningful differences to enable preemption.

#2 Disabling preemption on a pod expecting it to be protected from eviction.

Wrong approach:

spec:
priorityClassName: high-priority
preemptionPolicy: Never

Correct approach:

spec:
priorityClassName: high-priority
# preemptionPolicy omitted or set to 'PreemptLowerPriority' to allow preemption

Root cause:

Confusing preemptionPolicy's effect on preempting others versus being preempted.

#3 Not setting terminationGracePeriodSeconds, causing pods to be killed abruptly on preemption.

Wrong approach:

spec:
terminationGracePeriodSeconds: 0

Correct approach:

spec:
terminationGracePeriodSeconds: 30

Root cause:

Ignoring pod shutdown behavior leads to data loss or corruption during preemption.

Key Takeaways

Pod priority and preemption let Kubernetes decide which pods run first by assigning importance levels and evicting less important pods when resources are tight.

PriorityClasses assign numeric priorities to pods, and higher priority pods can preempt lower priority ones if needed to free resources.

Preemption is a careful process that only evicts enough pods to schedule higher priority pods and respects pod termination grace periods.

Misconfiguring priorities or preemption policies can cause instability or unexpected pod evictions, so careful planning and monitoring are essential.

Understanding pod priority and preemption helps ensure critical workloads run reliably even in busy clusters, improving overall system stability.