EKS networking with VPC CNI in AWS - Deep Dive

Overview - EKS networking with VPC CNI

What is it?

EKS networking with VPC CNI is how Amazon's Kubernetes service connects its containers to the network using the Virtual Private Cloud Container Network Interface. It allows each container to get its own IP address from the VPC, making communication simple and secure. This setup helps containers talk to each other and to other resources in the cloud as if they were regular machines on the network.

Why it matters

Without VPC CNI, containers would have to share IP addresses or use complex network tricks, making communication slower and less secure. VPC CNI solves this by giving each container a real IP address in the cloud network, making it easier to manage, monitor, and secure. This means apps run smoother, scale better, and connect safely to other services.

Where it fits

Before learning this, you should understand basic cloud networking concepts like IP addresses, subnets, and VPCs. After this, you can explore advanced Kubernetes networking, security policies, and multi-cluster communication.

Mental Model

Core Idea

VPC CNI lets each container in EKS behave like a real network device with its own IP inside the cloud network.

Think of it like...

Imagine a big apartment building (the VPC) where each apartment (container) has its own mailbox (IP address) directly accessible from the street (network). This makes sending and receiving mail (data) straightforward without sharing or confusion.

┌─────────────────────────────┐
│          VPC Network         │
│ ┌───────────────┐           │
│ │  Subnet A     │           │
│ │ ┌───────────┐ │           │
│ │ │ Container │ │           │
│ │ │  IP: 10.x │ │           │
│ │ └───────────┘ │           │
│ └───────────────┘           │
│ ┌───────────────┐           │
│ │  Subnet B     │           │
│ │ ┌───────────┐ │           │
│ │ │ Container │ │           │
│ │ │  IP: 10.y │ │           │
│ │ └───────────┘ │           │
│ └───────────────┘           │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Basics of VPC and IP Addressing

Concept: Understanding what a VPC and IP addresses are in cloud networking.

A Virtual Private Cloud (VPC) is like a private network in the cloud where you can place your resources. Each resource gets an IP address, which is like its phone number on the network. Subnets divide the VPC into smaller parts, each with its own range of IP addresses.

Result

You know how cloud resources get unique IP addresses inside a private network.

Knowing how IP addresses and VPCs work is essential because EKS networking builds on these concepts to assign IPs to containers.

2

Foundation

What is Kubernetes Pod Networking?

Concept: Learning how Kubernetes connects containers inside pods to networks.

In Kubernetes, containers run inside pods. Each pod needs a network address to send and receive data. Kubernetes uses networking plugins to give pods IP addresses and connect them to the network so they can talk to each other and outside services.

Result

You understand that pods need IP addresses and a way to communicate on the network.

Recognizing that pods are networked units helps you see why a networking plugin like VPC CNI is needed.

3

Intermediate

How VPC CNI Assigns IPs to Pods

🤔Before reading on: do you think pods get IPs from Kubernetes or directly from the VPC? Commit to your answer.

Concept: VPC CNI assigns IP addresses to pods directly from the VPC subnet range.

Unlike other plugins that create separate networks, VPC CNI gives each pod an IP from the VPC subnet. This means pods appear as normal devices on the cloud network. The plugin manages IP assignment and routing so pods can communicate without extra translation.

Result

Pods have real VPC IP addresses, making network communication straightforward.

Understanding that pods get real VPC IPs explains why VPC CNI simplifies networking and improves security.

4

Intermediate

Role of ENIs in VPC CNI Networking

🤔Before reading on: do you think pods share network interfaces or get their own? Commit to your answer.

Concept: VPC CNI uses Elastic Network Interfaces (ENIs) attached to worker nodes to provide IPs to pods.

Each worker node has one or more ENIs, which are like network cards with multiple IP addresses. VPC CNI assigns pod IPs from these ENIs. This means pods share the node's ENIs but have unique IPs, allowing efficient use of network resources.

Result

Pods use IPs from ENIs on nodes, enabling real VPC networking.

Knowing ENIs are the bridge between pods and the VPC clarifies how IP management works under the hood.

5

Intermediate

IP Address Management and Limits

🤔Before reading on: do you think IP addresses for pods are unlimited? Commit to your answer.

Concept: There is a limit to how many IPs can be assigned to pods based on ENI and subnet sizes.

Each ENI supports a fixed number of IP addresses depending on the instance type. The subnet size also limits total IPs. VPC CNI manages IP allocation carefully to avoid running out. If limits are reached, new pods may fail to get IPs and not start.

Result

You understand IP limits affect how many pods can run per node and subnet.

Recognizing IP limits helps plan cluster size and subnet design to avoid networking issues.

6

Advanced

Customizing VPC CNI for Performance

🤔Before reading on: do you think VPC CNI settings can be tuned? Commit to your answer.

Concept: VPC CNI has configurable options to optimize IP usage and network performance.

You can adjust settings like how many IPs are pre-allocated, whether to use prefix delegation for more IPs, and enable features like trunk ENIs for better scaling. These tweaks help balance resource use and pod density.

Result

Clusters can run more pods efficiently with tuned VPC CNI settings.

Knowing you can customize VPC CNI empowers better cluster scaling and cost control.

7

Expert

VPC CNI Interaction with Kubernetes Network Policies

🤔Before reading on: do you think VPC CNI enforces Kubernetes network policies by default? Commit to your answer.

Concept: VPC CNI integrates with Kubernetes network policies but requires additional setup for enforcement.

By default, VPC CNI handles IP assignment and routing but does not enforce network policies. To control pod communication, you must use additional tools like Calico or AWS Network Policies. Understanding this separation helps design secure clusters.

Result

You know that IP networking and security policies are related but separate concerns.

Understanding this separation prevents security gaps and helps design layered network controls.

Under the Hood

VPC CNI works by attaching multiple IP addresses to the Elastic Network Interfaces (ENIs) of each worker node. When a pod starts, the CNI plugin assigns it an available IP from the ENI's pool. The Linux kernel routes pod traffic through the node's ENI, making pods appear as native devices on the VPC network. This avoids network address translation (NAT), reducing latency and complexity.

Why designed this way?

AWS designed VPC CNI to leverage existing VPC networking for simplicity, security, and performance. Alternatives like overlay networks add complexity and overhead. Using ENIs and VPC IPs directly fits AWS's managed infrastructure model and allows seamless integration with other AWS services. This design trades off IP address limits for better network transparency and control.

┌─────────────────────────────┐
│        Worker Node          │
│ ┌───────────────┐           │
│ │   ENI (eth0)  │◄────────┐ │
│ │  IPs: 10.0.x  │         │ │
│ └───────────────┘         │ │
│ ┌───────────────┐         │ │
│ │   Pod 1       │         │ │
│ │ IP: 10.0.x.10 │─────────┼─┤
│ └───────────────┘         │ │
│ ┌───────────────┐         │ │
│ │   Pod 2       │         │ │
│ │ IP: 10.0.x.11 │─────────┼─┤
│ └───────────────┘         │ │
└─────────────────────────────┘
           │
           ▼
┌─────────────────────────────┐
│          VPC Network         │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do pods get IPs from a separate Kubernetes network or directly from the VPC? Commit to your answer.

Common Belief: Pods get IP addresses from a separate Kubernetes network, isolated from the VPC.

Reality: With VPC CNI, pods get IP addresses directly from the VPC subnet, making them first-class network citizens.

Why it matters: Believing pods are isolated can lead to wrong assumptions about connectivity and security, causing misconfigurations.

Quick: Do you think VPC CNI automatically enforces Kubernetes network policies? Commit to your answer.

Common Belief: VPC CNI enforces all Kubernetes network policies by default.

Reality: VPC CNI handles IP assignment and routing but does not enforce network policies; additional tools are needed.

Why it matters: Assuming automatic policy enforcement can leave clusters vulnerable to unwanted traffic.

Quick: Can you run unlimited pods per node with VPC CNI? Commit to your answer.

Common Belief: You can run as many pods as you want on a node without IP limits.

Reality: IP addresses are limited by ENI capacity and subnet size, limiting pod density per node.

Why it matters: Ignoring IP limits can cause pod startup failures and cluster instability.

Quick: Does VPC CNI use network address translation (NAT) for pod traffic? Commit to your answer.

Common Belief: VPC CNI uses NAT to translate pod IPs to node IPs for external communication.

Reality: VPC CNI assigns real VPC IPs to pods, avoiding NAT and improving performance.

Why it matters: Misunderstanding this can lead to unnecessary complexity and performance tuning errors.

Expert Zone

1

VPC CNI's IP allocation strategy can cause IP exhaustion in large clusters if subnet sizes are not planned carefully.

2

Prefix delegation allows nodes to request larger blocks of IPs, improving scalability but requiring subnet design changes.

3

Trunk ENIs enable attaching multiple ENIs per node, increasing pod density but adding complexity in IP management.

When NOT to use

VPC CNI is not ideal when you need cross-cloud or multi-cluster overlay networking, or when IP address scarcity is severe. Alternatives like Calico with overlay networks or Cilium with eBPF may be better for advanced network policies or non-AWS environments.

Production Patterns

In production, teams often combine VPC CNI with Calico for network policy enforcement, use prefix delegation to scale pod density, and monitor IP usage closely. They design subnets with extra IP capacity and automate ENI management to avoid outages.

Connections

Overlay Networking

Alternative approach to container networking

Understanding VPC CNI helps contrast overlay networks that create separate virtual networks, highlighting trade-offs in complexity and performance.

IP Address Management (IPAM)

Builds on IP allocation principles

Knowing how VPC CNI manages IPs deepens understanding of IPAM systems that allocate and track IPs in large networks.

Urban Planning

Similar principles of resource allocation and addressing

Just like city planners allocate addresses and roads to buildings, VPC CNI allocates IPs and routes to pods, showing how network design mirrors real-world infrastructure planning.

Common Pitfalls

#1 Running too many pods per node without considering IP limits

Wrong approach:

kubectl run pod1 --image=nginx
kubectl run pod2 --image=nginx
... (dozens more without subnet planning)

Correct approach:

Plan subnet size and ENI capacity before scaling pods; use prefix delegation or trunk ENIs to increase IP availability.

Root cause:

Misunderstanding that each pod needs a unique IP from the VPC subnet leads to IP exhaustion.

#2 Assuming VPC CNI enforces network policies by itself

Wrong approach:

Relying on VPC CNI alone for pod isolation without additional network policy tools.

Correct approach:

Deploy Calico or AWS Network Policies alongside VPC CNI to enforce pod communication rules.

Root cause:

Confusing IP assignment with security enforcement causes gaps in cluster protection.

#3 Ignoring ENI limits on instance types

Wrong approach:

Using small instance types expecting high pod density without ENI capacity checks.

Correct approach:

Select instance types with sufficient ENI and IP address capacity for expected pod counts.

Root cause:

Lack of awareness about AWS ENI and IP address limits per instance type.

Key Takeaways

VPC CNI assigns real VPC IP addresses to Kubernetes pods, making them first-class network devices.

Each pod’s IP comes from the node’s ENIs, linking pod networking directly to AWS infrastructure.

IP address limits on ENIs and subnets constrain how many pods can run per node and cluster.

VPC CNI handles IP assignment and routing but does not enforce network security policies by itself.

Proper subnet planning, instance selection, and additional network policy tools are essential for scalable and secure EKS clusters.

Practice

1. What is the main purpose of the VPC CNI plugin in Amazon EKS?

easy

A. To assign VPC IP addresses directly to Kubernetes pods

B. To provide a web interface for managing EKS clusters

C. To store container images in Amazon S3

D. To monitor CPU usage of EKS nodes

Solution

1. Step 1: Understand VPC CNI role in EKS

   The VPC CNI plugin allows pods to get IP addresses from the VPC subnet, enabling direct networking.

2. Step 2: Compare options with VPC CNI function

   Options A, B, and D describe unrelated features like UI, storage, and monitoring, not networking.

3. Final Answer:

   To assign VPC IP addresses directly to Kubernetes pods -> Option A

4. Quick Check:

   VPC CNI assigns IPs to pods = C [OK]

Hint: VPC CNI = pod IPs from VPC subnet [OK]

Common Mistakes:

• Confusing VPC CNI with monitoring tools
• Thinking VPC CNI manages container images
• Assuming VPC CNI provides UI features

2. Which command correctly enables the AWS VPC CNI plugin on an existing EKS cluster named my-cluster using eksctl?

easy

A. eksctl create cluster --name my-cluster --enable-vpc-cni

B. eksctl utils update-cni --cluster my-cluster

C. aws eks enable-cni --cluster-name my-cluster

D. kubectl apply -f vpc-cni.yaml

Solution

1. Step 1: Identify eksctl command to update CNI

   The correct eksctl command to update or enable the VPC CNI plugin on an existing cluster is eksctl utils update-cni --cluster my-cluster.

2. Step 2: Eliminate incorrect commands

   eksctl create cluster --name my-cluster --enable-vpc-cni creates a new cluster, not update CNI. aws eks enable-cni --cluster-name my-cluster is invalid AWS CLI syntax. kubectl apply -f vpc-cni.yaml applies a manifest but does not enable CNI via eksctl.

3. Final Answer:

   eksctl utils update-cni --cluster my-cluster -> Option B

4. Quick Check:

   eksctl update-cni command = A [OK]

Hint: Use eksctl utils update-cni for existing clusters [OK]

Common Mistakes:

• Using create cluster instead of update-cni
• Trying invalid AWS CLI commands
• Applying manifests without eksctl update

3. Given the following pod spec snippet using AWS VPC CNI, what IP address range will the pod likely receive?

{
  "metadata": {"name": "mypod"},
  "spec": {
    "containers": [{"name": "app", "image": "nginx"}],
    "nodeSelector": {"eks.amazonaws.com/nodegroup": "group1"}
  }
}

Assuming the VPC subnet CIDR is 10.0.0.0/16 and the node group subnet is 10.0.1.0/24.

medium

A. An IP from 10.0.1.0/24 subnet

B. An IP from 192.168.0.0/16 subnet

C. An IP from 172.16.0.0/12 subnet

D. An IP from 127.0.0.1 loopback

Solution

1. Step 1: Understand VPC CNI IP assignment

   VPC CNI assigns pod IPs from the subnet of the node where the pod runs. Here, node group subnet is 10.0.1.0/24.

2. Step 2: Match pod IP range with node subnet

   Since the pod runs on a node in 10.0.1.0/24 subnet, pod IP will be from that subnet, not from unrelated ranges.

3. Final Answer:

   An IP from 10.0.1.0/24 subnet -> Option A

4. Quick Check:

   Pod IP = node subnet CIDR = B [OK]

Hint: Pod IPs come from node subnet CIDR [OK]

Common Mistakes:

• Assuming pod IPs come from unrelated subnets
• Confusing pod IP with loopback or private ranges
• Ignoring node group subnet CIDR

4. You deployed the VPC CNI plugin but pods fail to get IP addresses and remain in Pending state. What is the most likely cause?

medium

A. The EKS cluster control plane is stopped

B. The pod image is missing from the container registry

C. The VPC subnet has no available IP addresses left

D. The Kubernetes API server is overloaded

Solution

1. Step 1: Identify reason pods stay Pending with VPC CNI

   If pods cannot get IPs, a common cause is no free IPs in the VPC subnet used by nodes.

2. Step 2: Exclude unrelated causes

   Pod image missing causes image pull errors, not IP issues. Control plane stopped or API overload cause cluster-wide failures, not just IP assignment.

3. Final Answer:

   The VPC subnet has no available IP addresses left -> Option C

4. Quick Check:

   Pods Pending due to no IPs = A [OK]

Hint: No free IPs in subnet causes pod Pending [OK]

Common Mistakes:

• Blaming image pull errors for IP assignment issues
• Assuming control plane is stopped without checking
• Ignoring subnet IP exhaustion

5. You want to optimize your EKS cluster networking to support thousands of pods per node using the VPC CNI plugin. Which configuration change should you apply?

hard

A. Reduce the pod security policies to allow more pods

B. Increase the node instance type to the largest available

C. Disable the VPC CNI plugin and use Calico instead

D. Enable the 'WARM_IP_TARGET' setting to pre-allocate more IP addresses per node

Solution

1. Step 1: Understand how to scale pod IPs with VPC CNI

   The VPC CNI plugin uses environment variables like 'WARM_IP_TARGET' to keep extra IPs ready per node, allowing more pods to launch quickly.

2. Step 2: Evaluate other options

   Increasing instance size helps but does not directly increase IPs. Disabling VPC CNI changes networking model. Pod security policies do not affect IP allocation.

3. Final Answer:

   Enable the 'WARM_IP_TARGET' setting to pre-allocate more IP addresses per node -> Option D

4. Quick Check:

   WARM_IP_TARGET pre-allocates IPs = D [OK]

Hint: Use WARM_IP_TARGET to pre-allocate pod IPs [OK]

Common Mistakes:

• Thinking instance size alone increases IPs
• Confusing network plugins instead of tuning VPC CNI
• Misunderstanding pod security policies impact