Priority classes for critical workloads in Kubernetes - Deep Dive

Overview - Priority classes for critical workloads

What is it?

Priority classes in Kubernetes are a way to tell the system which workloads are more important than others. They assign a priority value to pods, so the system knows which pods to keep running when resources are tight. This helps critical workloads get the resources they need before less important ones. Without priority classes, all workloads would be treated equally, which could cause important tasks to be delayed or stopped.

Why it matters

Without priority classes, Kubernetes might stop or delay important workloads during resource shortages, causing failures in critical services. Priority classes ensure that essential applications keep running smoothly even when the cluster is busy. This improves reliability and user experience for important services. It also helps teams manage resources better by clearly defining workload importance.

Where it fits

Before learning priority classes, you should understand basic Kubernetes concepts like pods, scheduling, and resource limits. After mastering priority classes, you can explore advanced topics like pod disruption budgets, taints and tolerations, and cluster autoscaling to manage workloads more effectively.

Mental Model

Core Idea

Priority classes assign importance levels to workloads so Kubernetes knows which pods to keep running first when resources are limited.

Think of it like...

Imagine a hospital emergency room where patients are treated based on how urgent their condition is. Critical patients get immediate attention, while less urgent cases wait. Priority classes work like this triage system for workloads in Kubernetes.

┌───────────────────────────────┐
│ Kubernetes Scheduler           │
├───────────────┬───────────────┤
│ Priority Class│ Pod Priority  │
├───────────────┼───────────────┤
│ high          │ 1000000       │
│ medium        │ 500000        │
│ low           │ 100000        │
└───────────────┴───────────────┘
        ↓
┌───────────────────────────────┐
│ Pod Eviction Decision          │
│ (Evict lower priority pods)   │
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Pods

Concept: Pods are the smallest deployable units in Kubernetes that run containers.

A pod is like a small box that holds one or more containers. Kubernetes schedules pods on nodes to run your applications. Each pod has resources like CPU and memory assigned to it.

Result

You know what a pod is and how Kubernetes runs your application inside it.

Understanding pods is essential because priority classes assign importance to these units.

2

Foundation

What is Scheduling and Resource Pressure

Concept: Kubernetes schedules pods on nodes based on available resources and priorities.

When many pods want to run but nodes have limited resources, Kubernetes must decide which pods to run or stop. This is called scheduling and eviction. Without priorities, Kubernetes treats all pods equally.

Result

You understand why Kubernetes needs a way to decide which pods to keep when resources are tight.

Knowing scheduling basics helps you see why priority classes are needed to manage resource conflicts.

3

Intermediate

Defining Priority Classes

🤔Before reading on: do you think priority classes are assigned per pod or per container? Commit to your answer.

Concept: Priority classes are Kubernetes objects that define priority values and names for pods.

You create a PriorityClass resource with a name and a numeric value. Higher values mean higher priority. Pods reference a priority class by name to get their priority. For example, a 'high' priority class might have value 1000000.

Result

You can create and assign priority classes to pods to control their importance.

Understanding that priority classes are separate objects lets you reuse and manage priorities consistently across pods.

4

Intermediate

How Priority Affects Pod Eviction

🤔Before reading on: do you think Kubernetes evicts higher priority pods first or lower priority pods first? Commit to your answer.

Concept: Kubernetes evicts pods with lower priority first when nodes run out of resources.

When a node is under pressure, Kubernetes looks at pod priorities. It evicts pods starting from the lowest priority to free resources. This protects critical workloads from being stopped.

Result

You understand eviction order depends on pod priority, protecting important pods.

Knowing eviction respects priority prevents surprises when critical pods stay running during resource shortages.

5

Intermediate

Using Preemption with Priority Classes

🤔Before reading on: do you think preemption allows a low priority pod to stop a high priority pod? Commit to your answer.

Concept: Preemption lets higher priority pods evict lower priority pods to get scheduled.

If a high priority pod cannot be scheduled due to lack of resources, Kubernetes can evict lower priority pods to make room. This process is called preemption. It ensures critical pods get resources first.

Result

You know how Kubernetes makes space for important pods by removing less important ones.

Understanding preemption helps you design clusters that prioritize critical workloads effectively.

6

Advanced

Customizing Priority Classes for Production

🤔Before reading on: do you think all critical workloads should share one priority class or have multiple? Commit to your answer.

Concept: You can create multiple priority classes to finely control workload importance in production.

In real clusters, you might have several priority classes like 'system-critical', 'business-critical', and 'batch'. Assigning different values helps Kubernetes make nuanced decisions during resource pressure. You can also mark some priority classes as global defaults.

Result

You can tailor priority classes to match your organization's workload importance levels.

Knowing how to customize priority classes lets you balance resource use and workload importance precisely.

7

Expert

Priority Classes and Cluster Autoscaling Interaction

🤔Before reading on: do you think cluster autoscaling respects pod priorities when adding nodes? Commit to your answer.

Concept: Priority classes influence pod scheduling but cluster autoscalers may not always consider them directly.

Cluster autoscalers add nodes when pods cannot be scheduled. However, autoscalers typically react to unschedulable pods regardless of priority. This can cause low priority pods to trigger scaling. Advanced setups combine priority classes with custom autoscaler filters or use pod disruption budgets to control scaling behavior.

Result

You understand the subtle interaction between priority classes and autoscaling in production.

Knowing this interaction helps prevent unexpected costs or resource waste in large clusters.

Under the Hood

Kubernetes stores priority classes as API objects with a numeric value. When scheduling, the scheduler assigns each pod a priority from its class. During resource pressure, the eviction manager compares pod priorities and selects lower priority pods for eviction. Preemption allows the scheduler to remove lower priority pods to schedule higher priority ones. This process uses internal queues sorted by priority values.

Why designed this way?

Priority classes were introduced to solve the problem of fair resource allocation in multi-tenant clusters. Before them, Kubernetes had no built-in way to express workload importance, leading to critical pods being evicted unexpectedly. The numeric priority system is simple, extensible, and integrates well with existing scheduling and eviction mechanisms.

┌───────────────────────────────┐
│ API Server                    │
│ ┌─────────────────────────┐ │
│ │ PriorityClass Objects    │ │
│ └─────────────────────────┘ │
└─────────────┬─────────────────┘
              │
              ▼
┌───────────────────────────────┐
│ Scheduler                     │
│ ┌─────────────────────────┐ │
│ │ Assigns pod.priority     │ │
│ └─────────────────────────┘ │
└─────────────┬─────────────────┘
              │
              ▼
┌───────────────────────────────┐
│ Eviction Manager              │
│ ┌─────────────────────────┐ │
│ │ Evicts pods by priority  │ │
│ └─────────────────────────┘ │
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does a higher numeric priority value mean a pod is less important? Commit yes or no.

Common Belief: Higher numeric priority values mean lower importance.

Reality: Higher numeric priority values mean higher importance and pods with higher values are preferred.

Why it matters: Misunderstanding this can cause you to assign wrong priorities, leading to critical pods being evicted first.

Quick: Can a pod without a priority class still run in Kubernetes? Commit yes or no.

Common Belief: All pods must have a priority class to run.

Reality: Pods without a priority class get a default priority of zero and can run but are treated as lowest priority.

Why it matters: Assuming all pods need priority classes can cause confusion and misconfiguration.

Quick: Does preemption allow a low priority pod to evict a high priority pod? Commit yes or no.

Common Belief: Any pod can evict any other pod regardless of priority.

Reality: Only higher priority pods can preempt (evict) lower priority pods, not the other way around.

Why it matters: Misunderstanding preemption can lead to wrong assumptions about pod stability and scheduling.

Quick: Does cluster autoscaling automatically consider pod priority when adding nodes? Commit yes or no.

Common Belief: Cluster autoscalers always respect pod priority when scaling.

Reality: Autoscalers react to unschedulable pods regardless of priority unless specially configured.

Why it matters: This can cause unnecessary scaling triggered by low priority pods, increasing costs.

Expert Zone

1

Priority values are arbitrary integers but should be spaced to allow future priorities between existing ones.

2

Priority classes can be marked as global default, which assigns priority to pods without explicit class, affecting all workloads.

3

Preemption can cause cascading evictions if many high priority pods appear simultaneously, requiring careful cluster planning.

When NOT to use

Priority classes are not a substitute for resource requests and limits or pod disruption budgets. For guaranteed resource allocation, use resource quotas and limits. For controlling pod disruptions during maintenance, use pod disruption budgets instead.

Production Patterns

In production, teams create multiple priority classes for system-critical, business-critical, and batch workloads. They combine priority classes with taints and tolerations to isolate critical pods on dedicated nodes. Preemption is monitored carefully to avoid cascading evictions. Autoscaling is tuned to ignore low priority pods to control costs.

Connections

Operating System Process Scheduling

Priority classes in Kubernetes are similar to process priorities in OS schedulers.

Understanding OS process priorities helps grasp how Kubernetes decides which pods to run or evict first.

Project Management Task Prioritization

Both assign importance levels to tasks or workloads to decide execution order.

Knowing how project managers prioritize tasks clarifies why Kubernetes needs priority classes to manage workload order.

Emergency Room Triage in Healthcare

Priority classes mimic triage systems that treat patients based on urgency.

Recognizing this connection highlights the importance of prioritizing critical workloads to maintain system health.

Common Pitfalls

#1 Assigning the same priority value to all workloads, ignoring their importance differences.

Wrong approach:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: default
value: 100000
globalDefault: true
description: "Default priority class"

Correct approach:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
description: "High priority for critical workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: low-priority
value: 100000
description: "Low priority for batch jobs"

Root cause:

Not differentiating priority values prevents Kubernetes from making meaningful eviction decisions.

#2 Expecting pods without a priority class to have high priority by default.

Wrong approach:

apiVersion: v1
kind: Pod
metadata:
name: critical-pod
spec:
containers:
- name: app
image: myapp:latest

Correct approach:

apiVersion: v1
kind: Pod
metadata:
name: critical-pod
spec:
priorityClassName: high-priority
containers:
- name: app
image: myapp:latest

Root cause:

Pods without priorityClassName get the lowest priority, which may cause unexpected eviction.

#3 Assuming preemption allows any pod to evict others regardless of priority.

Wrong approach:

Scheduling a low priority pod that evicts a high priority pod manually.

Correct approach:

Only higher priority pods trigger preemption automatically; low priority pods cannot evict higher priority pods.

Root cause:

Misunderstanding preemption rules leads to incorrect expectations about pod stability.

Key Takeaways

Priority classes let Kubernetes know which workloads are more important to keep running during resource shortages.

Higher numeric priority values mean higher importance and protect pods from eviction.

Preemption allows high priority pods to evict lower priority pods to get scheduled when resources are tight.

Pods without a priority class get the lowest priority by default and can be evicted first.

In production, multiple priority classes and careful tuning help balance resource use and workload importance.