Horizontal Pod Autoscaler in Microservices - System Design Exercise

Design: Horizontal Pod Autoscaler (HPA) System

Design the autoscaling control loop and its integration with Kubernetes. Out of scope: detailed Kubernetes cluster management, pod scheduling, and application-level scaling logic.

Functional Requirements

FR1: Automatically scale the number of pod replicas in a Kubernetes cluster based on observed metrics.

FR2: Support scaling based on CPU utilization and custom metrics like request rate or memory usage.

FR3: Ensure minimum and maximum pod replica limits are respected.

FR4: Provide near real-time scaling decisions with latency under 30 seconds.

FR5: Maintain system availability during scaling operations.

FR6: Expose metrics and scaling status for monitoring.

Non-Functional Requirements

NFR1: Handle up to 10,000 pods across multiple namespaces.

NFR2: Scaling decisions must be made every 15 seconds or less.

NFR3: System availability target of 99.9% uptime.

NFR4: Scaling actions should avoid thrashing (rapid scale up/down).

NFR5: Integrate with Kubernetes API and metrics server.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Metrics Collector (e.g., Metrics Server or Prometheus Adapter)

Autoscaler Controller (control loop logic)

Kubernetes API Server integration

Scaling Decision Engine

Rate Limiter or Stabilizer to prevent thrashing

Monitoring and Alerting system

Design Patterns

Control Loop Pattern for continuous monitoring and action

Observer Pattern for metrics collection

Circuit Breaker or Rate Limiting to avoid thrashing

Leader Election for high availability of autoscaler

Event-driven architecture for reacting to metric changes

Reference Architecture

                    +---------------------+
                    |  Metrics Server /    |
                    |  Custom Metrics API  |
                    +----------+----------+
                               |
                               v
+----------------+      +---------------------+      +-------------------+
| Kubernetes API |<---->| Horizontal Pod      |<---->| Kubernetes Cluster |
| Server         |      | Autoscaler Controller|      | (Pods, Nodes)      |
+----------------+      +---------------------+      +-------------------+
                               ^
                               |
                    +---------------------+
                    | Monitoring & Logging |
                    +---------------------+

Components

Metrics Server / Custom Metrics API

Kubernetes Metrics Server, Prometheus Adapter

Collects resource usage metrics (CPU, memory) and custom metrics from pods and nodes.

Horizontal Pod Autoscaler Controller

Kubernetes Controller written in Go

Runs control loop to fetch metrics, calculate desired replicas, and update Kubernetes API.

Kubernetes API Server

Kubernetes Core Component

Exposes API to read and update pod replica counts and other cluster state.

Kubernetes Cluster (Pods and Nodes)

Containerized microservices running in pods

Hosts the application workloads that are scaled by the autoscaler.

Monitoring & Logging

Prometheus, Grafana, ELK Stack

Tracks autoscaler performance, scaling events, and system health.

Request Flow

1. 1. Metrics Server collects CPU and custom metrics from pods and nodes.

2. 2. Horizontal Pod Autoscaler Controller queries Metrics Server periodically (every 15 seconds).

3. 3. Controller calculates desired number of replicas based on target utilization and current metrics.

4. 4. Controller checks minimum and maximum replica constraints.

5. 5. Controller updates the Kubernetes API Server with new replica count if scaling is needed.

6. 6. Kubernetes API Server triggers pod creation or deletion in the cluster.

7. 7. Monitoring system records scaling events and metrics for visibility.

Database Schema

Not applicable as Kubernetes stores state in etcd. Key entities: HorizontalPodAutoscaler resource with fields: target metrics, minReplicas, maxReplicas, currentReplicas, desiredReplicas, lastScaleTime.

Scaling Discussion

Bottlenecks

Metrics Server overload when collecting metrics from thousands of pods.

Autoscaler Controller becoming a single point of failure.

API Server rate limits when many scaling requests happen simultaneously.

Thrashing due to rapid scale up/down cycles.

Latency in metrics collection causing delayed scaling decisions.

Solutions

Use scalable metrics backends like Prometheus with efficient scraping and aggregation.

Implement leader election among multiple autoscaler controller instances for high availability.

Batch scaling requests and use exponential backoff to avoid API rate limits.

Add stabilization windows and cooldown periods to prevent thrashing.

Optimize metrics collection intervals and use predictive scaling techniques.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying metrics. Use 20 minutes to design the control loop and components. Reserve 10 minutes to discuss scaling challenges and trade-offs. Use last 5 minutes for questions and summary.

Explain the control loop concept and how metrics drive scaling decisions.

Discuss integration with Kubernetes API and metrics sources.

Highlight how to prevent thrashing with stabilization techniques.

Mention high availability via leader election for the autoscaler controller.

Address scaling bottlenecks and realistic latency targets.