Blue-green deployment in Microservices - System Design Exercise

Design: Blue-Green Deployment System

Design the deployment strategy and infrastructure for blue-green deployment of microservices. Exclude CI/CD pipeline details and code-level rollback mechanisms.

Functional Requirements

FR1: Deploy new versions of microservices with zero downtime

FR2: Allow quick rollback to previous version if issues occur

FR3: Minimize risk of deployment failures affecting users

FR4: Support automated traffic switching between versions

FR5: Monitor health of both blue and green environments

Non-Functional Requirements

NFR1: Handle up to 10,000 concurrent users during deployment

NFR2: API response latency p99 under 200ms during deployment

NFR3: Availability target of 99.9% uptime including deployment windows

NFR4: Deployment process should complete within 5 minutes

NFR5: Support multiple microservices independently deployed

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Load balancer or API gateway for traffic routing

Two identical production environments (blue and green)

Health check and monitoring system

Deployment automation tool

Service registry and discovery

Design Patterns

Canary deployment as an alternative

Feature toggles for gradual rollout

Circuit breaker for fault tolerance

Immutable infrastructure for environment consistency

Reference Architecture

          +-------------------+
          |    Users/Clients  |
          +---------+---------+
                    |
                    v
          +---------+---------+
          |  Load Balancer /  |
          |   API Gateway    |
          +----+--------+----+
               |        |
       +-------+        +-------+
       |                        |
+------+-------+        +-------+------+
|   Blue Env   |        |   Green Env  |
| (Current)    |        | (New Version)|
+--------------+        +--------------+
       |                        |
+------+-------+        +-------+------+
| Microservices|        | Microservices|
+--------------+        +--------------+
       |                        |
+------+-------+        +-------+------+
|  Database(s) |        |  Database(s) |
+--------------+        +--------------+

Components

Load Balancer / API Gateway

Nginx, Envoy, or AWS ALB

Route user traffic to either blue or green environment based on deployment state

Blue Environment

Kubernetes cluster or VM instances

Current stable version of microservices serving live traffic

Green Environment

Kubernetes cluster or VM instances

New version of microservices deployed and tested before switching traffic

Health Check and Monitoring

Prometheus, Grafana, or custom scripts

Continuously verify readiness and performance of both environments

Deployment Automation

Jenkins, ArgoCD, or Spinnaker

Automate deployment, testing, and traffic switching processes

Database

Relational or NoSQL DB with versioning support

Store persistent data accessible by both environments, with migration strategy

Request Flow

1. User sends request to Load Balancer/API Gateway.

2. Load Balancer routes request to Blue environment (current live version).

3. Deploy new microservice version to Green environment without affecting Blue.

4. Run automated tests and health checks on Green environment.

5. If Green passes health checks, switch Load Balancer traffic from Blue to Green.

6. Users now receive responses from Green environment.

7. Monitor Green environment closely for errors or performance issues.

8. If issues detected, rollback by switching traffic back to Blue environment.

9. Once Green is stable, Blue environment can be updated for next deployment.

Database Schema

Entities remain consistent across blue and green environments. Database schema changes require backward-compatible migrations. Use versioned migration scripts to update schema without downtime. Both environments connect to the same database instance or cluster to ensure data consistency.

Scaling Discussion

Bottlenecks

Load balancer capacity limits during traffic switch

Database schema changes causing downtime or data inconsistency

Health check delays slowing deployment process

Resource duplication doubling infrastructure costs

Rollback complexity if multiple microservices fail simultaneously

Solutions

Use scalable load balancers with auto-scaling and connection draining

Implement zero-downtime database migrations with feature toggles

Optimize health checks for fast and reliable readiness signals

Use container orchestration to efficiently share resources

Automate rollback procedures and isolate failures per microservice

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain how blue-green deployment reduces downtime and risk

Describe traffic routing and environment switching clearly

Discuss database migration challenges and solutions

Highlight monitoring and rollback strategies

Mention alternatives like canary deployments and feature toggles