A/B testing with Ingress in Kubernetes - Deep Dive

Overview - A/B testing with Ingress

What is it?

A/B testing with Ingress is a way to send different portions of user traffic to two or more versions of an application running in Kubernetes. It uses the Ingress resource to control how requests are routed to different backend services. This helps teams compare versions by observing user behavior and performance in real time.

Why it matters

Without A/B testing, teams must guess which version of an app works better or rely on slow, manual feedback. A/B testing with Ingress lets teams safely test new features or changes on a small group of users before full rollout. This reduces risk, improves user experience, and speeds up learning from real user data.

Where it fits

Before learning this, you should understand Kubernetes basics, especially Services and Ingress concepts. After mastering A/B testing with Ingress, you can explore advanced traffic management with service meshes or automated canary deployments.

Mental Model

Core Idea

A/B testing with Ingress splits incoming user requests between multiple app versions by routing traffic based on rules, enabling side-by-side comparison in production.

Think of it like...

Imagine a restaurant with two chefs cooking the same dish but with slight recipe changes. The waiter (Ingress) decides which chef's dish to serve to each customer, so the restaurant owner can see which version customers like better.

┌─────────────┐
│  User       │
└─────┬───────┘
      │ HTTP Request
      ▼
┌─────────────┐
│  Ingress    │
│  Controller │
└─────┬───────┘
  Traffic Split Rules
      │
 ┌────┴─────┐
 │          │
▼          ▼
Service A  Service B
(Version1) (Version2)

Build-Up - 6 Steps

1

Foundation

Understanding Kubernetes Ingress Basics

Concept: Learn what an Ingress is and how it routes external traffic to services inside Kubernetes.

Ingress is a Kubernetes resource that manages external access to services in a cluster, usually HTTP. It defines rules to route requests based on hostnames or paths to backend services. An Ingress Controller implements these rules and handles the actual traffic routing.

Result

You can expose multiple services under one IP or hostname and route requests to them based on rules.

Knowing how Ingress works is essential because A/B testing relies on controlling traffic routing at this entry point.

2

Foundation

Basics of A/B Testing Concept

Concept: Understand what A/B testing means and why it splits user traffic between two versions.

A/B testing is a method to compare two versions (A and B) of a feature or app by sending a portion of users to each. It helps decide which version performs better by measuring user reactions or metrics.

Result

You grasp the purpose of sending different users to different app versions simultaneously.

Understanding the goal of A/B testing helps you see why traffic routing control is needed.

3

Intermediate

Configuring Traffic Splitting in Ingress

🤔Before reading on: do you think Ingress can split traffic by percentage natively or needs extra tools? Commit to your answer.

Concept: Learn how to configure Ingress to split traffic between services, often using annotations or specific Ingress Controllers.

Standard Kubernetes Ingress does not support traffic splitting by percentage out of the box. However, some Ingress Controllers like NGINX or Istio support annotations or custom resources to split traffic. For example, NGINX Ingress can use 'nginx.ingress.kubernetes.io/canary' annotations to route a percentage of traffic to a canary service.

Result

You can set rules so that, for example, 90% of traffic goes to Service A and 10% to Service B.

Knowing that traffic splitting depends on the Ingress Controller helps avoid confusion and guides tool choice.

4

Intermediate

Using Headers and Cookies for User Targeting

🤔Before reading on: do you think traffic splitting is always random or can be based on user attributes? Commit to your answer.

Concept: Learn how to route traffic based on HTTP headers or cookies to target specific users or groups consistently.

Ingress Controllers can route traffic based on headers or cookies. For example, you can send users with a specific cookie value to version B, ensuring the same user sees the same version on repeat visits. This is useful for consistent A/B testing experiences.

Result

Users get a stable experience with one version, improving test reliability.

Understanding user targeting prevents skewed test results caused by users switching versions mid-session.

5

Advanced

Combining Ingress with Service Mesh for Advanced Control

🤔Before reading on: do you think Ingress alone can handle complex traffic rules like retries or weighted splits? Commit to your answer.

Concept: Explore how service meshes like Istio extend Ingress capabilities for fine-grained traffic control in A/B testing.

Service meshes add a data plane that intercepts traffic inside the cluster, allowing advanced routing like weighted splits, retries, timeouts, and mirroring. Istio’s VirtualService resource works with Ingress Gateway to split traffic by percentage with high precision and observability.

Result

You can implement robust A/B tests with detailed control and monitoring.

Knowing when to use service meshes helps scale A/B testing beyond simple scenarios.

6

Expert

Handling State and Session Consistency in A/B Tests

🤔Before reading on: do you think stateless routing is enough for all A/B tests? Commit to your answer.

Concept: Understand challenges of user sessions and state when routing traffic in A/B testing and how to solve them.

If users switch between versions mid-session, it can cause confusion or errors. Techniques like sticky sessions, cookie-based routing, or shared session stores ensure users stay on one version. Also, backend compatibility and database migrations must be managed carefully to avoid data corruption.

Result

A/B tests run smoothly without user experience breaks or data issues.

Recognizing state challenges prevents subtle bugs that can invalidate test results or harm users.

Under the Hood

Ingress Controllers watch Kubernetes Ingress resources and configure underlying proxies (like NGINX or Envoy) to route HTTP traffic. For A/B testing, they use rules or annotations to modify routing behavior, sometimes integrating with service mesh proxies for advanced features. Traffic splitting often relies on load balancing algorithms or header inspection to decide the backend service for each request.

Why designed this way?

Kubernetes Ingress was designed as a simple, standard way to expose HTTP services. Traffic splitting was not included initially to keep the API simple and generic. Advanced routing was left to specialized controllers or service meshes to allow flexibility and innovation without bloating core Kubernetes.

┌───────────────┐
│  User Request │
└───────┬───────┘
        │
┌───────▼────────┐
│ Ingress Resource│
│  (Rules/Hosts) │
└───────┬────────┘
        │
┌───────▼────────┐
│Ingress Controller│
│ (NGINX, Envoy)  │
└───────┬────────┘
        │
┌───────▼────────┐
│ Proxy Load     │
│ Balancer Logic │
│ (Split, Header │
│  Inspection)   │
└───────┬────────┘
        │
┌───────▼────────┐   ┌─────────────┐
│ Service A Pod  │   │ Service B Pod│
│ (Version A)   │   │ (Version B)  │
└───────────────┘   └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Kubernetes Ingress natively support traffic splitting by percentage? Commit to yes or no.

Common Belief: Kubernetes Ingress can split traffic by percentage out of the box.

Reality: Standard Kubernetes Ingress does not support traffic splitting by percentage; this feature depends on the Ingress Controller or additional tools.

Why it matters: Assuming native support leads to failed attempts and confusion when traffic does not split as expected.

Quick: Is routing by cookie or header always random? Commit to yes or no.

Common Belief: Traffic splitting is always random and cannot target specific users consistently.

Reality: Ingress Controllers can route traffic based on cookies or headers, enabling consistent user experience in A/B tests.

Why it matters: Ignoring this causes unstable tests where users see different versions, invalidating results.

Quick: Can you run complex A/B tests with only Ingress and no other tools? Commit to yes or no.

Common Belief: Ingress alone is enough for all A/B testing needs.

Reality: For advanced traffic control like retries, mirroring, or weighted splits, service meshes or specialized tools are needed.

Why it matters: Trying to do everything with Ingress leads to brittle setups and limits test sophistication.

Quick: Does switching users between versions mid-session have no impact? Commit to yes or no.

Common Belief: Users can switch between app versions anytime without issues.

Reality: Switching versions mid-session can cause errors or confusing user experience; session consistency is important.

Why it matters: Ignoring session consistency risks broken features and unreliable test data.

Expert Zone

1

Some Ingress Controllers support canary releases with gradual traffic shifting, but the implementation details and annotation syntax vary widely.

2

Using cookies for routing requires careful handling of cookie expiration and security to avoid users losing their assigned version.

3

Combining Ingress with service mesh allows observability tools to track A/B test metrics automatically, which is often overlooked.

When NOT to use

Avoid using Ingress-based A/B testing for highly stateful applications or when you need complex traffic policies like circuit breaking or retries. Instead, use service meshes like Istio or Linkerd that provide richer traffic management and observability.

Production Patterns

In production, teams often start with simple percentage-based canary deployments using NGINX Ingress annotations, then migrate to service mesh-based routing for more control. They also integrate monitoring tools to measure user behavior and automate rollbacks if issues arise.

Connections

Canary Deployments

A/B testing with Ingress builds on canary deployment principles by splitting traffic to test versions gradually.

Understanding canary deployments helps grasp how traffic splitting reduces risk during rollouts.

Load Balancing

Ingress traffic splitting is a specialized form of load balancing that directs requests based on rules rather than just distributing evenly.

Knowing load balancing fundamentals clarifies how traffic routing decisions are made.

Behavioral Psychology

A/B testing applies behavioral psychology by observing how different user groups respond to changes.

Recognizing this connection highlights the importance of consistent user experience and unbiased sampling.

Common Pitfalls

#1 Trying to split traffic using standard Kubernetes Ingress without a supporting controller.

Wrong approach:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ab-test-ingress
spec:
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service-a
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: service-b
port:
number: 80

Correct approach:

Use an Ingress Controller that supports canary or traffic splitting, e.g., NGINX Ingress with annotations:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ab-test-ingress
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service-a
port:
number: 80

Root cause:

Misunderstanding that Ingress resource alone controls traffic routing without controller support.

#2 Routing users randomly without sticky sessions or cookies.

Wrong approach:

Ingress configured to split traffic purely by percentage without user identification, causing users to see different versions on refresh.

Correct approach:

Configure routing based on cookies or headers to ensure users consistently hit the same version, e.g., using 'nginx.ingress.kubernetes.io/canary-by-cookie' annotation.

Root cause:

Not accounting for session consistency leads to unreliable test results and poor user experience.

#3 Using Ingress for complex traffic policies like retries or mirroring.

Wrong approach:

Trying to configure retries or traffic mirroring in standard Ingress annotations, which are unsupported.

Correct approach:

Use a service mesh like Istio that supports VirtualService resources for advanced traffic control.

Root cause:

Assuming Ingress can handle all traffic management features without additional tools.

Key Takeaways

A/B testing with Ingress lets you split user traffic between app versions to compare performance safely in production.

Standard Kubernetes Ingress does not support traffic splitting by percentage; this depends on the Ingress Controller or additional tools.

Routing based on cookies or headers ensures users have a consistent experience during A/B tests, improving result reliability.

For advanced traffic control like retries or mirroring, combine Ingress with service meshes such as Istio.

Managing user session consistency and backend compatibility is critical to avoid errors and misleading test data.

Practice

1. What is the main purpose of using A/B testing with Kubernetes Ingress?

easy

A. To monitor CPU usage of pods

B. To increase the number of pods automatically

C. To split user traffic between different versions of an application

D. To backup Kubernetes cluster data

Solution

1. Step 1: Understand A/B testing concept in Kubernetes

   A/B testing with Ingress is used to route traffic between different app versions to test new features safely.

2. Step 2: Identify the purpose of Ingress in traffic management

   Ingress controls how external traffic reaches services, enabling traffic splitting for A/B testing.

3. Final Answer:

   To split user traffic between different versions of an application -> Option C

4. Quick Check:

   A/B testing = traffic split [OK]

Hint: A/B testing means splitting traffic between app versions [OK]

Common Mistakes:

• Confusing A/B testing with scaling pods
• Thinking Ingress is for monitoring only
• Assuming Ingress handles backups

2. Which annotation is commonly used in Kubernetes Ingress to split traffic by percentage for A/B testing?

easy

A. nginx.ingress.kubernetes.io/canary-weight

B. nginx.ingress.kubernetes.io/rewrite-target

C. nginx.ingress.kubernetes.io/ssl-redirect

D. nginx.ingress.kubernetes.io/proxy-body-size

Solution

1. Step 1: Identify annotations for traffic splitting

   The annotation 'nginx.ingress.kubernetes.io/canary-weight' is used to specify the percentage of traffic sent to the canary version.

2. Step 2: Differentiate from other annotations

   Other annotations like rewrite-target or ssl-redirect serve different purposes unrelated to traffic splitting.

3. Final Answer:

   nginx.ingress.kubernetes.io/canary-weight -> Option A

4. Quick Check:

   Canary weight = traffic percentage [OK]

Hint: Look for 'canary-weight' to split traffic by percent [OK]

Common Mistakes:

• Using rewrite-target for traffic splitting
• Confusing SSL redirect with traffic control
• Ignoring canary annotations

3. Given this Ingress snippet for A/B testing, what percentage of traffic goes to the canary service?

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: stable-service
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: canary-service
            port:
              number: 80

medium

A. 70%

B. 30%

C. 50%

D. 100%

Solution

1. Step 1: Read the canary-weight annotation

   The annotation 'nginx.ingress.kubernetes.io/canary-weight' is set to "30", meaning 30% of traffic goes to canary.

2. Step 2: Understand traffic split logic

   Traffic is split by percentage; 30% to canary-service, remaining 70% to stable-service.

3. Final Answer:

   30% -> Option B

4. Quick Check:

   Canary weight = 30% traffic [OK]

Hint: Check canary-weight value for traffic percent [OK]

Common Mistakes:

• Assuming canary gets 70% instead of 30%
• Confusing service names with traffic percentages
• Ignoring canary annotation

4. You configured this Ingress for A/B testing but all traffic goes to the stable service only. What is the likely error?

metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: stable-service
            port:
              number: 80

medium

A. Missing canary backend path in spec

B. Incorrect canary-weight value format

C. Host name is invalid

D. Stable service port is wrong

Solution

1. Step 1: Check Ingress spec for canary backend

   The spec only has one backend for stable-service; no path defined for canary-service.

2. Step 2: Understand traffic routing requirements

   For canary traffic to work, a separate path with canary annotations and backend service must be defined.

3. Final Answer:

   Missing canary backend path in spec -> Option A

4. Quick Check:

   Canary needs separate backend path [OK]

Hint: Ensure canary backend path exists in Ingress spec [OK]

Common Mistakes:

• Assuming canary-weight alone routes traffic
• Ignoring missing canary backend path
• Blaming host or port without checking paths

5. You want to route 20% of users with header X-User-Type: beta to the canary service and the rest to stable. Which Ingress annotation setup achieves this?

hard

A. Use nginx.ingress.kubernetes.io/canary-by-cookie: "beta"

B. Use nginx.ingress.kubernetes.io/canary-weight: "20" only

C. Use nginx.ingress.kubernetes.io/canary: "false"

D. Use nginx.ingress.kubernetes.io/canary-by-header: "X-User-Type" and nginx.ingress.kubernetes.io/canary-by-header-value: "beta"

Solution

1. Step 1: Identify header-based routing annotations

   To route traffic based on header, use 'canary-by-header' and 'canary-by-header-value' annotations.

2. Step 2: Understand difference from weight-based routing

   Weight-based routing splits traffic by percentage regardless of headers; header-based routing targets specific users.

3. Final Answer:

   Use nginx.ingress.kubernetes.io/canary-by-header: "X-User-Type" and nginx.ingress.kubernetes.io/canary-by-header-value: "beta" -> Option D

4. Quick Check:

   Header-based routing uses canary-by-header annotations [OK]

Hint: Use canary-by-header and canary-by-header-value for header routing [OK]

Common Mistakes:

• Using only canary-weight for header routing
• Confusing cookie-based routing with header routing
• Setting canary to false disables routing