Node components (kubelet, kube-proxy, container runtime) in Kubernetes - Deep Dive

Overview - Node components (kubelet, kube-proxy, container runtime)

What is it?

Node components in Kubernetes are the essential parts that run on each worker machine (node) to make sure containers work properly. The kubelet manages the containers on the node, the kube-proxy handles network communication, and the container runtime runs the actual containers. Together, they keep the node connected to the cluster and running applications smoothly.

Why it matters

Without these components, a Kubernetes node would be just a regular machine with no way to run or manage containers as part of the cluster. They solve the problem of coordinating container execution, networking, and health on each node, enabling Kubernetes to manage applications at scale. Without them, you would have to manually manage containers and networking on every machine, which is slow and error-prone.

Where it fits

Before learning node components, you should understand what Kubernetes is and the basic concepts of containers and pods. After this, you can learn about cluster architecture, control plane components, and how nodes communicate with the master to form a full Kubernetes cluster.

Mental Model

Core Idea

Node components are the local workers on each machine that run containers, manage networking, and report status to keep the Kubernetes cluster healthy and functional.

Think of it like...

Imagine a busy restaurant kitchen: the kubelet is the chef who cooks the dishes (containers), the kube-proxy is the waiter who delivers orders and connects the kitchen to the dining area (networking), and the container runtime is the stove that actually cooks the food (runs containers).

┌─────────────┐       ┌───────────────┐       ┌──────────────────┐
│   Kubelet   │──────▶│ Container     │       │   Kube-proxy     │
│ (Chef)      │       │ Runtime       │       │ (Waiter)         │
│ Manages     │       │ (Stove)       │       │ Handles Network  │
│ Containers  │       │ Runs Containers│       │ Traffic          │
└─────────────┘       └───────────────┘       └──────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a Kubernetes Node?

Concept: Introduce the idea of a node as a worker machine in Kubernetes.

A Kubernetes node is a physical or virtual machine where containers run. It is part of the cluster and communicates with the control plane. Each node has components that manage containers and networking.

Result

You understand that nodes are the machines that do the actual work in a Kubernetes cluster.

Knowing what a node is helps you see where the work happens in Kubernetes, not just in the control plane.

2

Foundation

Role of Container Runtime

Concept: Explain what a container runtime is and why it is needed on each node.

The container runtime is software that runs containers on a node. Examples include containerd and CRI-O. It downloads container images, starts and stops containers, and manages their lifecycle.

Result

You know that without a container runtime, containers cannot run on a node.

Understanding the container runtime clarifies how Kubernetes actually runs your applications inside containers.

3

Intermediate

Kubelet: Node’s Container Manager

🤔Before reading on: do you think kubelet directly runs containers or just monitors them? Commit to your answer.

Concept: Introduce kubelet as the agent that manages containers on the node and communicates with the control plane.

The kubelet watches for pod specifications sent by the control plane and ensures the containers described are running and healthy. It uses the container runtime to start and stop containers and reports node and pod status back to the control plane.

Result

You understand kubelet is the node’s main controller for running containers as Kubernetes expects.

Knowing kubelet’s role helps you understand how Kubernetes enforces desired state on each node.

4

Intermediate

Kube-proxy: Networking on the Node

🤔Before reading on: do you think kube-proxy handles all network traffic or just some? Commit to your answer.

Concept: Explain kube-proxy’s role in managing network rules and routing traffic to containers.

Kube-proxy runs on each node and manages network rules that allow pods to communicate inside the cluster and with the outside world. It handles service IPs and load balancing by updating network rules using iptables or IPVS.

Result

You see how kube-proxy enables communication between pods and services on the node and cluster.

Understanding kube-proxy reveals how Kubernetes manages complex networking without manual setup.

5

Intermediate

How Node Components Work Together

Concept: Show the interaction between kubelet, container runtime, and kube-proxy on a node.

When the control plane schedules a pod on a node, kubelet receives the pod spec and asks the container runtime to start containers. Kube-proxy sets up network rules so the pod can communicate. Kubelet monitors container health and reports back to the control plane.

Result

You understand the coordinated workflow that keeps pods running and connected on a node.

Seeing the teamwork between components helps you troubleshoot and optimize node behavior.

6

Advanced

Container Runtime Interface (CRI) Explained

🤔Before reading on: do you think Kubernetes supports only one container runtime or multiple? Commit to your answer.

Concept: Explain the CRI as a standard interface that allows Kubernetes to work with different container runtimes.

Kubernetes uses the Container Runtime Interface (CRI) to communicate with container runtimes. This abstraction lets Kubernetes support multiple runtimes like containerd, CRI-O, or Docker without changing kubelet code.

Result

You understand Kubernetes can work with various container runtimes thanks to CRI.

Knowing about CRI explains Kubernetes’ flexibility and future-proof design.

7

Expert

Kubelet’s Node Status and Health Checks

🤔Before reading on: do you think kubelet only reports pod status or also node health? Commit to your answer.

Concept: Dive into how kubelet monitors node health and reports it to the control plane for cluster decisions.

Kubelet performs node status checks like disk pressure, memory, and network availability. It reports these conditions to the control plane, which can then decide to evict pods or mark nodes as unhealthy. This helps maintain cluster stability.

Result

You see how kubelet’s health reporting influences cluster-wide decisions and pod scheduling.

Understanding kubelet’s health checks reveals how Kubernetes maintains resilience and self-healing.

Under the Hood

Kubelet runs as a daemon on each node, continuously watching the API server for pod specs. It uses the CRI to communicate with the container runtime, which pulls images and runs containers. Kube-proxy listens for service changes and updates network rules using iptables or IPVS to route traffic. All components report status back to the control plane via the API server, enabling centralized management.

Why designed this way?

This design separates concerns: kubelet manages container lifecycle, container runtime handles container execution, and kube-proxy manages networking. Using CRI allows Kubernetes to support multiple runtimes, future-proofing the system. Decentralizing node management reduces load on the control plane and improves scalability.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Control Plane │◀─────▶│    Kubelet    │◀─────▶│ Container     │
│ (API Server)  │       │ (Node Agent)  │       │ Runtime (CRI) │
└───────────────┘       └───────────────┘       └───────────────┘
                             │
                             ▼
                      ┌───────────────┐
                      │  Kube-proxy   │
                      │ (Network Mgmt)│
                      └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does kubelet run containers directly or use another component? Commit to your answer.

Common Belief: Kubelet runs containers directly on the node without help.

Reality: Kubelet does not run containers itself; it uses the container runtime via the CRI to start and manage containers.

Why it matters: Thinking kubelet runs containers directly can lead to confusion when troubleshooting container runtime issues.

Quick: Does kube-proxy handle all network traffic including external internet access? Commit to your answer.

Common Belief: Kube-proxy manages all network traffic including internet access for pods.

Reality: Kube-proxy manages cluster internal networking and service routing but does not handle external internet access directly; that is managed by other components like ingress controllers or cloud networking.

Why it matters: Misunderstanding kube-proxy’s role can cause network configuration errors and failed pod communication.

Quick: Can Kubernetes only use Docker as the container runtime? Commit to your answer.

Common Belief: Kubernetes only supports Docker as the container runtime.

Reality: Kubernetes supports multiple container runtimes through the CRI, including containerd, CRI-O, and Docker (deprecated in newer versions).

Why it matters: Believing only Docker works limits understanding of Kubernetes flexibility and can cause issues with newer Kubernetes versions.

Quick: Does kubelet only report pod status or also node health? Commit to your answer.

Common Belief: Kubelet only reports the status of pods running on the node.

Reality: Kubelet also monitors and reports node health conditions like disk pressure and memory availability to the control plane.

Why it matters: Ignoring node health reporting can lead to poor cluster stability and unexpected pod evictions.

Expert Zone

1

Kubelet’s sync loop frequency affects how quickly pod state changes are detected and acted upon, impacting cluster responsiveness.

2

Kube-proxy can run in different modes (iptables or IPVS), each with trade-offs in performance and complexity.

3

Container runtimes differ in features like image caching and security, influencing node performance and security posture.

When NOT to use

Node components are essential for Kubernetes nodes; however, in serverless or managed Kubernetes services, direct interaction with these components is limited or abstracted. For lightweight edge devices, alternative runtimes or minimal Kubernetes distributions may be better suited.

Production Patterns

In production, kubelet is often configured with custom resource limits and eviction policies to maintain node stability. Kube-proxy may be replaced or supplemented by advanced CNI plugins for better networking. Container runtimes are chosen based on security and performance needs, with containerd being the current standard.

Connections

Operating System Services

Node components rely on OS-level features like cgroups and namespaces to isolate containers.

Understanding OS services helps grasp how container runtimes create isolated environments for applications.

Load Balancing

Kube-proxy implements basic load balancing for services within the cluster.

Knowing load balancing principles clarifies how kube-proxy distributes traffic efficiently among pods.

Restaurant Kitchen Workflow

The coordination between kubelet, container runtime, and kube-proxy mirrors how chefs, stoves, and waiters work together in a kitchen.

Seeing this connection helps understand the importance of clear roles and communication in complex systems.

Common Pitfalls

#1 Confusing kubelet with container runtime responsibilities.

Wrong approach:

Trying to debug container start failures by only checking kubelet logs without inspecting the container runtime.

Correct approach:

Check both kubelet and container runtime logs because kubelet delegates container management to the runtime.

Root cause:

Misunderstanding that kubelet does not directly run containers but uses the container runtime.

#2 Assuming kube-proxy handles all networking including external access.

Wrong approach:

Configuring firewall rules expecting kube-proxy to route internet traffic for pods.

Correct approach:

Use ingress controllers or cloud provider networking features for external access; kube-proxy manages internal cluster networking.

Root cause:

Misconception about the scope of kube-proxy’s networking responsibilities.

#3 Using deprecated Docker runtime in new Kubernetes versions.

Wrong approach:

Installing Docker as the container runtime on nodes in Kubernetes 1.24+ clusters.

Correct approach:

Use containerd or CRI-O as the container runtime to comply with current Kubernetes standards.

Root cause:

Not keeping up with Kubernetes runtime deprecation and CRI adoption.

Key Takeaways

Node components are the essential local agents that run containers, manage networking, and report status in Kubernetes nodes.

Kubelet manages container lifecycle by communicating with the container runtime through a standard interface called CRI.

Kube-proxy handles internal cluster networking by updating network rules to route traffic between pods and services.

The container runtime is the software that actually runs containers on the node, and Kubernetes supports multiple runtimes via CRI.

Understanding how these components work together helps maintain, troubleshoot, and optimize Kubernetes clusters effectively.