OOMKilled containers in Kubernetes - Commands & Configuration

Introduction

Sometimes, containers in Kubernetes stop working because they use too much memory. This is called OOMKilled, which means the system killed the container to protect the server. Understanding why this happens helps keep your apps running smoothly.

When a container suddenly stops and shows OOMKilled status in Kubernetes.

When you want to prevent your app from using too much memory and crashing.

When you need to check if your memory limits are set correctly for your containers.

When troubleshooting why a pod restarts frequently without clear errors.

When optimizing resource use to avoid wasting server memory.

Config File - pod-memory-limit.yaml

pod-memory-limit.yaml

apiVersion: v1
kind: Pod
metadata:
  name: memory-demo
spec:
  containers:
  - name: memory-demo-container
    image: busybox
    command: ["sh", "-c", "sleep 3600"]
    resources:
      limits:
        memory: "100Mi"
      requests:
        memory: "50Mi"

This YAML file creates a pod named memory-demo with one container.

The resources section sets memory limits and requests:

• requests.memory: The amount of memory Kubernetes reserves for the container.
• limits.memory: The maximum memory the container can use before it is killed.

If the container uses more than 100Mi of memory, Kubernetes will kill it with OOMKilled.

Commands

This command creates the pod with memory limits set. It tells Kubernetes to start the container with the specified memory rules.

Terminal

kubectl apply -f pod-memory-limit.yaml

Expected Output

pod/memory-demo created

Check the status of the pod to see if it is running or if it was killed due to memory issues.

Terminal

kubectl get pods

Expected Output

NAME           READY   STATUS    RESTARTS   AGE

memory-demo    1/1     Running   0          10s

Run a command inside the container to use more memory than the limit, causing OOMKilled. This simulates a memory overload.

Terminal

kubectl exec memory-demo -- sh -c "stress --vm 1 --vm-bytes 150M --vm-hang 0"

Expected Output

No output (command runs silently)

Check the reason why the container was terminated. It should show 'OOMKilled' if the container was killed due to memory overuse.

Terminal

kubectl get pod memory-demo -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

Expected Output

OOMKilled

Get detailed information about the pod, including events that show if the container was killed because it used too much memory.

Terminal

kubectl describe pod memory-demo

Expected Output

Name:         memory-demo

Namespace:    default

Containers:

  memory-demo-container:

    Container ID:   docker://abcdef123456

    Image:          busybox

    State:          Waiting

      Reason:       CrashLoopBackOff

    Last State:     Terminated

      Reason:       OOMKilled

      Exit Code:    137

    Ready:          False

    Restart Count:  3

Events:

  Type     Reason     Age                    From               Message

  ----     ------     ----                   ----               -------

  Warning  OOMKilled  2m                     kubelet            Container memory limit exceeded

Key Concept

If a container uses more memory than its limit, Kubernetes kills it with OOMKilled to protect the system.

Common Mistakes

Not setting memory limits on containers.

Without limits, containers can use unlimited memory, causing the node to become unstable or crash.

Always set memory requests and limits in your pod specs to control resource use.

Setting memory limits too low for the app's needs.

The container will be killed frequently with OOMKilled, causing downtime.

Monitor your app's memory use and set limits high enough to avoid killing but low enough to protect the node.

Ignoring OOMKilled status and not checking pod events.

You miss the root cause of crashes and cannot fix memory issues.

Use kubectl describe and check container termination reasons to diagnose OOMKilled.

Summary

Set memory requests and limits in pod specs to control container memory use.

Use kubectl commands to create pods, check status, and diagnose OOMKilled events.

If a container exceeds its memory limit, Kubernetes kills it to protect the system.

Practice

1. What does it mean when a Kubernetes container status shows OOMKilled?

easy

A. The container was deleted manually by the user.

B. The container was restarted due to a network failure.

C. The container completed its task successfully.

D. The container was stopped because it used more memory than allowed.

Solution

1. Step 1: Understand OOMKilled meaning

   OOMKilled means Out Of Memory Killed, which happens when a container uses more memory than its limit.

2. Step 2: Relate to container status

   When a container exceeds its memory limit, Kubernetes stops it to protect the system.

3. Final Answer:

   The container was stopped because it used more memory than allowed. -> Option D

4. Quick Check:

   OOMKilled = Memory limit exceeded [OK]

Hint: OOMKilled means container used too much memory [OK]

Common Mistakes:

• Confusing OOMKilled with network errors
• Thinking OOMKilled means container finished normally
• Assuming OOMKilled is a manual stop

2. Which kubectl command helps you check why a pod's container was OOMKilled?

easy

A. kubectl get pods --all-namespaces

B. kubectl logs <pod-name>

C. kubectl describe pod <pod-name>

D. kubectl exec <pod-name> -- top

Solution

1. Step 1: Identify command to get pod details

   kubectl describe pod <pod-name> shows detailed pod info including container status and reasons for restarts.

2. Step 2: Confirm OOMKilled reason visibility

   This command shows events and container states, including if a container was OOMKilled.

3. Final Answer:

   kubectl describe pod <pod-name> -> Option C

4. Quick Check:

   Describe pod shows OOMKilled reason [OK]

Hint: Use 'kubectl describe pod' to see OOMKilled details [OK]

Common Mistakes:

• Using 'kubectl get pods' which lacks detailed reason
• Checking logs which may not show OOMKilled cause
• Running exec commands which don't show pod status

3. Given this pod description snippet, what caused the container to stop?

State: Terminated
Reason: OOMKilled
Exit Code: 137

medium

A. The container was killed because it used too much memory.

B. The container ran out of CPU resources.

C. The container was terminated due to a manual stop.

D. The container crashed due to a software error.

Solution

1. Step 1: Analyze the Reason field

   The Reason is 'OOMKilled', which means the container was killed because it exceeded memory limits.

2. Step 2: Understand Exit Code 137

   Exit code 137 means the process was killed by signal 9 (SIGKILL), typical for OOMKilled events.

3. Final Answer:

   The container was killed because it used too much memory. -> Option A

4. Quick Check:

   Reason OOMKilled + Exit 137 = Memory kill [OK]

Hint: Exit code 137 with OOMKilled means memory limit exceeded [OK]

Common Mistakes:

• Confusing CPU limits with memory limits
• Assuming manual stop without checking reason
• Thinking software error caused termination

4. You see a pod's container repeatedly OOMKilled. Which change fixes this issue?

medium

A. Increase the container's memory limit in the pod spec.

B. Decrease the container's CPU limit in the pod spec.

C. Remove the memory limit from the pod spec.

D. Restart the pod manually every time it OOMKills.

Solution

1. Step 1: Identify cause of repeated OOMKilled

   Repeated OOMKilled means the container needs more memory than allowed.

2. Step 2: Choose proper fix

   Increasing the memory limit lets the container use more memory and prevents OOMKilled.

3. Final Answer:

   Increase the container's memory limit in the pod spec. -> Option A

4. Quick Check:

   More memory limit stops OOMKilled [OK]

Hint: Raise memory limit to stop OOMKilled repeats [OK]

Common Mistakes:

• Lowering CPU limit which doesn't affect memory
• Removing memory limit causing instability
• Relying on manual restarts instead of fixing limits

5. A pod's container is OOMKilled even though its memory limit is set to 512Mi. You want to prevent this without increasing the limit. What is the best approach?

hard

A. Increase the CPU limit to speed up processing.

B. Optimize the application to use less memory inside the container.

C. Remove the memory limit to avoid OOMKilled.

D. Add more replicas of the pod to distribute load.

Solution

1. Step 1: Understand memory limit and OOMKilled

   The container hits the 512Mi limit and is killed. Increasing limit is not an option.

2. Step 2: Find alternative to increasing memory

   Optimizing the app to use less memory reduces usage below the limit, preventing OOMKilled.

3. Step 3: Evaluate other options

   Removing limit risks node stability, increasing CPU doesn't reduce memory, adding replicas doesn't fix memory per container.

4. Final Answer:

   Optimize the application to use less memory inside the container. -> Option B

5. Quick Check:

   Lower memory use avoids OOMKilled without raising limit [OK]

Hint: Reduce app memory use to avoid OOMKilled without raising limit [OK]

Common Mistakes:

• Removing memory limits causing node crashes
• Increasing CPU expecting memory fix
• Adding replicas without fixing memory use