OOMKilled containers in Kubernetes - Step-by-Step Execution

Process Flow - OOMKilled containers

Container starts running

↓

Container uses memory

↓

Memory usage > limit?

No→Continue running

Yes↓

Kernel kills container (OOMKilled)

↓

Container status set to OOMKilled

↓

Pod restarts or fails

This flow shows how a container runs, uses memory, and if it exceeds its memory limit, the system kills it with OOMKilled status.

Execution Sample

Kubernetes

kubectl get pods
kubectl describe pod mypod
kubectl logs mypod
kubectl get events

Commands to check pod status, details, logs, and events to diagnose OOMKilled containers.

Process Table

Step	Action	Memory Usage (MiB)	Memory Limit (MiB)	Result	Container Status
1	Container starts	0	100	Running	Running
2	Memory usage rises	50	100	Running	Running
3	Memory usage rises	90	100	Running	Running
4	Memory usage rises	110	100	Exceeded limit	OOMKilled
5	Kernel kills container	N/A	100	Container killed	OOMKilled
6	Pod restarts or fails	N/A	100	Restarting or Failed	Running

💡 Container memory usage exceeded limit, kernel killed container with OOMKilled status.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
Memory Usage (MiB)	0	50	90	110	N/A
Container Status	Running	Running	Running	OOMKilled	Running

Key Moments - 3 Insights

Why does the container status change to OOMKilled even though the pod is still listed?

The container is killed by the system because it used more memory than allowed. The pod still exists but the container inside it has stopped with OOMKilled status, as shown in step 4 and 5 of the execution table.

What does 'Memory Limit' mean and why is it important?

Memory Limit is the maximum memory the container can use. If the container uses more than this, the system kills it to protect other containers. This is why in step 4, exceeding 100 MiB causes OOMKilled.

Can the pod restart automatically after OOMKilled?

Yes, depending on the pod's restart policy, it may restart the container after OOMKilled, as shown in step 6. But if it keeps exceeding memory, it may keep failing.

Visual Quiz - 3 Questions

Look at the execution table, what is the container status at step 3?

A Pending

B Running

C OOMKilled

D Failed

💡 Hint

Check the 'Container Status' column at step 3 in the execution table.

At which step does the container exceed its memory limit?

A Step 4

B Step 2

C Step 3

D Step 5

💡 Hint

Look at the 'Memory Usage' and 'Memory Limit' columns to find when usage surpasses limit.

If the memory limit was increased to 120 MiB, what would change in the execution table?

A Container would be killed at step 3

B Container would be OOMKilled at step 4

C Container would continue running past step 4

D No change in container status

💡 Hint

Compare memory usage at step 4 (110 MiB) with new limit 120 MiB.

Concept Snapshot

OOMKilled containers happen when a container uses more memory than its limit.
The system kills the container to protect others.
Check pod status with 'kubectl describe pod' and events.
Adjust memory limits to prevent OOMKilled.
Pod may restart depending on restart policy.

Full Transcript

When a container runs in Kubernetes, it has a memory limit set. As it uses memory, if it stays below the limit, it runs fine. But if it uses more memory than allowed, the system kills it to protect other containers. This kill is called OOMKilled. You can see this status by describing the pod or checking events. The pod may restart the container depending on its restart policy. To avoid OOMKilled, increase memory limits or optimize container memory use.

Practice

1. What does it mean when a Kubernetes container status shows OOMKilled?

easy

A. The container was deleted manually by the user.

B. The container was restarted due to a network failure.

C. The container completed its task successfully.

D. The container was stopped because it used more memory than allowed.

Solution

1. Step 1: Understand OOMKilled meaning

   OOMKilled means Out Of Memory Killed, which happens when a container uses more memory than its limit.

2. Step 2: Relate to container status

   When a container exceeds its memory limit, Kubernetes stops it to protect the system.

3. Final Answer:

   The container was stopped because it used more memory than allowed. -> Option D

4. Quick Check:

   OOMKilled = Memory limit exceeded [OK]

Hint: OOMKilled means container used too much memory [OK]

Common Mistakes:

• Confusing OOMKilled with network errors
• Thinking OOMKilled means container finished normally
• Assuming OOMKilled is a manual stop

2. Which kubectl command helps you check why a pod's container was OOMKilled?

easy

A. kubectl get pods --all-namespaces

B. kubectl logs <pod-name>

C. kubectl describe pod <pod-name>

D. kubectl exec <pod-name> -- top

Solution

1. Step 1: Identify command to get pod details

   kubectl describe pod <pod-name> shows detailed pod info including container status and reasons for restarts.

2. Step 2: Confirm OOMKilled reason visibility

   This command shows events and container states, including if a container was OOMKilled.

3. Final Answer:

   kubectl describe pod <pod-name> -> Option C

4. Quick Check:

   Describe pod shows OOMKilled reason [OK]

Hint: Use 'kubectl describe pod' to see OOMKilled details [OK]

Common Mistakes:

• Using 'kubectl get pods' which lacks detailed reason
• Checking logs which may not show OOMKilled cause
• Running exec commands which don't show pod status

3. Given this pod description snippet, what caused the container to stop?

State: Terminated
Reason: OOMKilled
Exit Code: 137

medium

A. The container was killed because it used too much memory.

B. The container ran out of CPU resources.

C. The container was terminated due to a manual stop.

D. The container crashed due to a software error.

Solution

1. Step 1: Analyze the Reason field

   The Reason is 'OOMKilled', which means the container was killed because it exceeded memory limits.

2. Step 2: Understand Exit Code 137

   Exit code 137 means the process was killed by signal 9 (SIGKILL), typical for OOMKilled events.

3. Final Answer:

   The container was killed because it used too much memory. -> Option A

4. Quick Check:

   Reason OOMKilled + Exit 137 = Memory kill [OK]

Hint: Exit code 137 with OOMKilled means memory limit exceeded [OK]

Common Mistakes:

• Confusing CPU limits with memory limits
• Assuming manual stop without checking reason
• Thinking software error caused termination

4. You see a pod's container repeatedly OOMKilled. Which change fixes this issue?

medium

A. Increase the container's memory limit in the pod spec.

B. Decrease the container's CPU limit in the pod spec.

C. Remove the memory limit from the pod spec.

D. Restart the pod manually every time it OOMKills.

Solution

1. Step 1: Identify cause of repeated OOMKilled

   Repeated OOMKilled means the container needs more memory than allowed.

2. Step 2: Choose proper fix

   Increasing the memory limit lets the container use more memory and prevents OOMKilled.

3. Final Answer:

   Increase the container's memory limit in the pod spec. -> Option A

4. Quick Check:

   More memory limit stops OOMKilled [OK]

Hint: Raise memory limit to stop OOMKilled repeats [OK]

Common Mistakes:

• Lowering CPU limit which doesn't affect memory
• Removing memory limit causing instability
• Relying on manual restarts instead of fixing limits

5. A pod's container is OOMKilled even though its memory limit is set to 512Mi. You want to prevent this without increasing the limit. What is the best approach?

hard

A. Increase the CPU limit to speed up processing.

B. Optimize the application to use less memory inside the container.

C. Remove the memory limit to avoid OOMKilled.

D. Add more replicas of the pod to distribute load.

Solution

1. Step 1: Understand memory limit and OOMKilled

   The container hits the 512Mi limit and is killed. Increasing limit is not an option.

2. Step 2: Find alternative to increasing memory

   Optimizing the app to use less memory reduces usage below the limit, preventing OOMKilled.

3. Step 3: Evaluate other options

   Removing limit risks node stability, increasing CPU doesn't reduce memory, adding replicas doesn't fix memory per container.

4. Final Answer:

   Optimize the application to use less memory inside the container. -> Option B

5. Quick Check:

   Lower memory use avoids OOMKilled without raising limit [OK]

Hint: Reduce app memory use to avoid OOMKilled without raising limit [OK]

Common Mistakes:

• Removing memory limits causing node crashes
• Increasing CPU expecting memory fix
• Adding replicas without fixing memory use