EKS networking with VPC CNI in AWS - Deep Dive

Overview - EKS networking with VPC CNI

What is it?

EKS networking with VPC CNI is how Amazon's Kubernetes service connects its containers to the network using the Virtual Private Cloud Container Network Interface. It allows each container to get its own IP address from the VPC, making communication simple and secure. This setup helps containers talk to each other and to other resources in the cloud as if they were regular machines on the network.

Why it matters

Without VPC CNI, containers would have to share IP addresses or use complex network tricks, making communication slower and less secure. VPC CNI solves this by giving each container a real IP address in the cloud network, making it easier to manage, monitor, and secure. This means apps run smoother, scale better, and connect safely to other services.

Where it fits

Before learning this, you should understand basic cloud networking concepts like IP addresses, subnets, and VPCs. After this, you can explore advanced Kubernetes networking, security policies, and multi-cluster communication.

Mental Model

Core Idea

VPC CNI lets each container in EKS behave like a real network device with its own IP inside the cloud network.

Think of it like...

Imagine a big apartment building (the VPC) where each apartment (container) has its own mailbox (IP address) directly accessible from the street (network). This makes sending and receiving mail (data) straightforward without sharing or confusion.

┌─────────────────────────────┐
│          VPC Network         │
│ ┌───────────────┐           │
│ │  Subnet A     │           │
│ │ ┌───────────┐ │           │
│ │ │ Container │ │           │
│ │ │  IP: 10.x │ │           │
│ │ └───────────┘ │           │
│ └───────────────┘           │
│ ┌───────────────┐           │
│ │  Subnet B     │           │
│ │ ┌───────────┐ │           │
│ │ │ Container │ │           │
│ │ │  IP: 10.y │ │           │
│ │ └───────────┘ │           │
│ └───────────────┘           │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Basics of VPC and IP Addressing

Concept: Understanding what a VPC and IP addresses are in cloud networking.

A Virtual Private Cloud (VPC) is like a private network in the cloud where you can place your resources. Each resource gets an IP address, which is like its phone number on the network. Subnets divide the VPC into smaller parts, each with its own range of IP addresses.

Result

You know how cloud resources get unique IP addresses inside a private network.

Knowing how IP addresses and VPCs work is essential because EKS networking builds on these concepts to assign IPs to containers.

2

Foundation

What is Kubernetes Pod Networking?

Concept: Learning how Kubernetes connects containers inside pods to networks.

In Kubernetes, containers run inside pods. Each pod needs a network address to send and receive data. Kubernetes uses networking plugins to give pods IP addresses and connect them to the network so they can talk to each other and outside services.

Result

You understand that pods need IP addresses and a way to communicate on the network.

Recognizing that pods are networked units helps you see why a networking plugin like VPC CNI is needed.

3

Intermediate

How VPC CNI Assigns IPs to Pods

🤔Before reading on: do you think pods get IPs from Kubernetes or directly from the VPC? Commit to your answer.

Concept: VPC CNI assigns IP addresses to pods directly from the VPC subnet range.

Unlike other plugins that create separate networks, VPC CNI gives each pod an IP from the VPC subnet. This means pods appear as normal devices on the cloud network. The plugin manages IP assignment and routing so pods can communicate without extra translation.

Result

Pods have real VPC IP addresses, making network communication straightforward.

Understanding that pods get real VPC IPs explains why VPC CNI simplifies networking and improves security.

4

Intermediate

Role of ENIs in VPC CNI Networking

🤔Before reading on: do you think pods share network interfaces or get their own? Commit to your answer.

Concept: VPC CNI uses Elastic Network Interfaces (ENIs) attached to worker nodes to provide IPs to pods.

Each worker node has one or more ENIs, which are like network cards with multiple IP addresses. VPC CNI assigns pod IPs from these ENIs. This means pods share the node's ENIs but have unique IPs, allowing efficient use of network resources.

Result

Pods use IPs from ENIs on nodes, enabling real VPC networking.

Knowing ENIs are the bridge between pods and the VPC clarifies how IP management works under the hood.

5

Intermediate

IP Address Management and Limits

🤔Before reading on: do you think IP addresses for pods are unlimited? Commit to your answer.

Concept: There is a limit to how many IPs can be assigned to pods based on ENI and subnet sizes.

Each ENI supports a fixed number of IP addresses depending on the instance type. The subnet size also limits total IPs. VPC CNI manages IP allocation carefully to avoid running out. If limits are reached, new pods may fail to get IPs and not start.

Result

You understand IP limits affect how many pods can run per node and subnet.

Recognizing IP limits helps plan cluster size and subnet design to avoid networking issues.

6

Advanced

Customizing VPC CNI for Performance

🤔Before reading on: do you think VPC CNI settings can be tuned? Commit to your answer.

Concept: VPC CNI has configurable options to optimize IP usage and network performance.

You can adjust settings like how many IPs are pre-allocated, whether to use prefix delegation for more IPs, and enable features like trunk ENIs for better scaling. These tweaks help balance resource use and pod density.

Result

Clusters can run more pods efficiently with tuned VPC CNI settings.

Knowing you can customize VPC CNI empowers better cluster scaling and cost control.

7

Expert

VPC CNI Interaction with Kubernetes Network Policies

🤔Before reading on: do you think VPC CNI enforces Kubernetes network policies by default? Commit to your answer.

Concept: VPC CNI integrates with Kubernetes network policies but requires additional setup for enforcement.

By default, VPC CNI handles IP assignment and routing but does not enforce network policies. To control pod communication, you must use additional tools like Calico or AWS Network Policies. Understanding this separation helps design secure clusters.

Result

You know that IP networking and security policies are related but separate concerns.

Understanding this separation prevents security gaps and helps design layered network controls.

Under the Hood

VPC CNI works by attaching multiple IP addresses to the Elastic Network Interfaces (ENIs) of each worker node. When a pod starts, the CNI plugin assigns it an available IP from the ENI's pool. The Linux kernel routes pod traffic through the node's ENI, making pods appear as native devices on the VPC network. This avoids network address translation (NAT), reducing latency and complexity.

Why designed this way?

AWS designed VPC CNI to leverage existing VPC networking for simplicity, security, and performance. Alternatives like overlay networks add complexity and overhead. Using ENIs and VPC IPs directly fits AWS's managed infrastructure model and allows seamless integration with other AWS services. This design trades off IP address limits for better network transparency and control.

┌─────────────────────────────┐
│        Worker Node          │
│ ┌───────────────┐           │
│ │   ENI (eth0)  │◄────────┐ │
│ │  IPs: 10.0.x  │         │ │
│ └───────────────┘         │ │
│ ┌───────────────┐         │ │
│ │   Pod 1       │         │ │
│ │ IP: 10.0.x.10 │─────────┼─┤
│ └───────────────┘         │ │
│ ┌───────────────┐         │ │
│ │   Pod 2       │         │ │
│ │ IP: 10.0.x.11 │─────────┼─┤
│ └───────────────┘         │ │
└─────────────────────────────┘
           │
           ▼
┌─────────────────────────────┐
│          VPC Network         │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do pods get IPs from a separate Kubernetes network or directly from the VPC? Commit to your answer.

Common Belief: Pods get IP addresses from a separate Kubernetes network, isolated from the VPC.

Reality: With VPC CNI, pods get IP addresses directly from the VPC subnet, making them first-class network citizens.

Why it matters: Believing pods are isolated can lead to wrong assumptions about connectivity and security, causing misconfigurations.

Quick: Do you think VPC CNI automatically enforces Kubernetes network policies? Commit to your answer.

Common Belief: VPC CNI enforces all Kubernetes network policies by default.

Reality: VPC CNI handles IP assignment and routing but does not enforce network policies; additional tools are needed.

Why it matters: Assuming automatic policy enforcement can leave clusters vulnerable to unwanted traffic.

Quick: Can you run unlimited pods per node with VPC CNI? Commit to your answer.

Common Belief: You can run as many pods as you want on a node without IP limits.

Reality: IP addresses are limited by ENI capacity and subnet size, limiting pod density per node.

Why it matters: Ignoring IP limits can cause pod startup failures and cluster instability.

Quick: Does VPC CNI use network address translation (NAT) for pod traffic? Commit to your answer.

Common Belief: VPC CNI uses NAT to translate pod IPs to node IPs for external communication.

Reality: VPC CNI assigns real VPC IPs to pods, avoiding NAT and improving performance.

Why it matters: Misunderstanding this can lead to unnecessary complexity and performance tuning errors.

Expert Zone

1

VPC CNI's IP allocation strategy can cause IP exhaustion in large clusters if subnet sizes are not planned carefully.

2

Prefix delegation allows nodes to request larger blocks of IPs, improving scalability but requiring subnet design changes.

3

Trunk ENIs enable attaching multiple ENIs per node, increasing pod density but adding complexity in IP management.

When NOT to use

VPC CNI is not ideal when you need cross-cloud or multi-cluster overlay networking, or when IP address scarcity is severe. Alternatives like Calico with overlay networks or Cilium with eBPF may be better for advanced network policies or non-AWS environments.

Production Patterns

In production, teams often combine VPC CNI with Calico for network policy enforcement, use prefix delegation to scale pod density, and monitor IP usage closely. They design subnets with extra IP capacity and automate ENI management to avoid outages.

Connections

Overlay Networking

Alternative approach to container networking

Understanding VPC CNI helps contrast overlay networks that create separate virtual networks, highlighting trade-offs in complexity and performance.

IP Address Management (IPAM)

Builds on IP allocation principles

Knowing how VPC CNI manages IPs deepens understanding of IPAM systems that allocate and track IPs in large networks.

Urban Planning

Similar principles of resource allocation and addressing

Just like city planners allocate addresses and roads to buildings, VPC CNI allocates IPs and routes to pods, showing how network design mirrors real-world infrastructure planning.

Common Pitfalls

#1 Running too many pods per node without considering IP limits

Wrong approach:

kubectl run pod1 --image=nginx
kubectl run pod2 --image=nginx
... (dozens more without subnet planning)

Correct approach:

Plan subnet size and ENI capacity before scaling pods; use prefix delegation or trunk ENIs to increase IP availability.

Root cause:

Misunderstanding that each pod needs a unique IP from the VPC subnet leads to IP exhaustion.

#2 Assuming VPC CNI enforces network policies by itself

Wrong approach:

Relying on VPC CNI alone for pod isolation without additional network policy tools.

Correct approach:

Deploy Calico or AWS Network Policies alongside VPC CNI to enforce pod communication rules.

Root cause:

Confusing IP assignment with security enforcement causes gaps in cluster protection.

#3 Ignoring ENI limits on instance types

Wrong approach:

Using small instance types expecting high pod density without ENI capacity checks.

Correct approach:

Select instance types with sufficient ENI and IP address capacity for expected pod counts.

Root cause:

Lack of awareness about AWS ENI and IP address limits per instance type.

Key Takeaways

VPC CNI assigns real VPC IP addresses to Kubernetes pods, making them first-class network devices.

Each pod’s IP comes from the node’s ENIs, linking pod networking directly to AWS infrastructure.

IP address limits on ENIs and subnets constrain how many pods can run per node and cluster.

VPC CNI handles IP assignment and routing but does not enforce network security policies by itself.

Proper subnet planning, instance selection, and additional network policy tools are essential for scalable and secure EKS clusters.