Rollback strategies in Microservices - Architecture Diagram

System Overview - Rollback strategies

This system demonstrates how to safely rollback a microservices deployment when a new version causes issues. Key requirements include minimizing downtime, preserving data integrity, and allowing quick recovery to the previous stable version.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  +-------------------+
  |                   |
Service V2          Service V1
  |                   |
  v                   v
Database <---------- Cache
  |
  v
Message Queue
  |
  v
Monitoring & Alerting
  |
  v
Rollback Controller

Components

User

client

Sends requests to the system

Load Balancer

load_balancer

Distributes incoming traffic between service versions

API Gateway

api_gateway

Routes requests to appropriate microservice versions

Service V2

service

New version of the microservice handling requests

Service V1

service

Previous stable version of the microservice

Database

database

Stores persistent data for services

Cache

cache

Speeds up data retrieval for services

Message Queue

queue

Handles asynchronous communication and rollback commands

Monitoring & Alerting

monitoring

Detects issues and triggers rollback if needed

Rollback Controller

controller

Manages rollback process and switches traffic back to stable version

Request Flow - 12 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Service V2

Service V2 → Cache

Cache → Service V2

Service V2 → Database

Database → Service V2

Service V2 → Cache

Service V2 → User

Monitoring & Alerting → Rollback Controller

Rollback Controller → Load Balancer

Rollback Controller → Message Queue

Failure Scenario

Component Fails:Service V2

Impact:New service version causes errors or slow responses, degrading user experience

Mitigation:Monitoring detects failures and triggers Rollback Controller to switch traffic back to stable Service V1 version, ensuring system stability

Architecture Quiz - 3 Questions

Which component is responsible for directing user requests to the correct microservice version?

A API Gateway

B Load Balancer

C Rollback Controller

D Message Queue

Design Principle

This architecture uses traffic routing and monitoring to enable quick rollback of microservice versions. It isolates new versions behind an API Gateway and Load Balancer, allowing seamless switch back to stable versions without downtime. Monitoring triggers automated rollback commands, ensuring system reliability and user experience.

Practice

1. What is the main purpose of a rollback strategy in microservices?

easy

A. To quickly undo a bad deployment and restore the previous stable state

B. To add new features to the system without downtime

C. To permanently delete old versions of services

D. To monitor system performance continuously

Solution

1. Step 1: Understand rollback purpose

   Rollback strategies are designed to revert changes that cause issues, restoring stability.

2. Step 2: Identify correct purpose in options

   Only To quickly undo a bad deployment and restore the previous stable state describes undoing a bad deployment to restore a stable state.

3. Final Answer:

   To quickly undo a bad deployment and restore the previous stable state -> Option A

4. Quick Check:

   Rollback purpose = Undo bad deployment [OK]

Hint: Rollback means undo bad changes fast [OK]

Common Mistakes:

• Confusing rollback with feature deployment
• Thinking rollback deletes old versions permanently
• Mixing rollback with monitoring

2. Which of the following is a correct description of the blue-green deployment rollback method?

easy

A. Switch traffic back to the old environment if the new one fails

B. Gradually increase traffic to the new version while monitoring

C. Manually fix database schema errors after deployment

D. Deploy new code directly to production without testing

Solution

1. Step 1: Recall blue-green deployment basics

   Blue-green uses two identical environments; one active, one idle for new version.

2. Step 2: Identify rollback action

   If new version fails, traffic switches back to old environment instantly.

3. Final Answer:

   Switch traffic back to the old environment if the new one fails -> Option A

4. Quick Check:

   Blue-green rollback = Switch traffic back [OK]

Hint: Blue-green rollback switches traffic instantly [OK]

Common Mistakes:

• Confusing blue-green with canary deployment
• Thinking rollback fixes database manually
• Ignoring traffic switching concept

3. Consider this simplified code snippet for a canary deployment rollback trigger:

if error_rate > 0.05:
    rollback_canary()

What happens when the error rate exceeds 5% during canary deployment?

medium

A. The system continues deployment without changes

B. The error rate is ignored and logged only

C. The rollback_canary function is called to revert changes

D. The deployment is paused but not rolled back

Solution

1. Step 1: Analyze the condition in code

   The code checks if error_rate is greater than 0.05 (5%).

2. Step 2: Understand the action on condition true

   If true, rollback_canary() is called to revert the canary deployment.

3. Final Answer:

   The rollback_canary function is called to revert changes -> Option C

4. Quick Check:

   Error rate > 5% triggers rollback [OK]

Hint: Error rate > threshold triggers rollback function [OK]

Common Mistakes:

• Ignoring the rollback call in the code
• Assuming deployment pauses without rollback
• Confusing logging with rollback action

4. A microservice deployment uses database migration with rollback scripts. The rollback script fails due to a syntax error. What is the best immediate action?

medium

A. Ignore the failure and continue deployment

B. Restart the service without rollback

C. Delete the database and start fresh

D. Manually fix the rollback script and retry rollback

Solution

1. Step 1: Identify rollback script failure impact

   A syntax error in rollback script prevents safe undo of migration changes.

2. Step 2: Choose safe recovery action

   Fixing the script manually and retrying rollback ensures data integrity and system stability.

3. Final Answer:

   Manually fix the rollback script and retry rollback -> Option D

4. Quick Check:

   Fix rollback script error before retrying [OK]

Hint: Fix rollback script errors before retrying rollback [OK]

Common Mistakes:

• Ignoring rollback failure and proceeding
• Deleting database without backup
• Restarting service without fixing rollback

5. You have a microservices system using canary deployments with automated rollback on failure. Suddenly, a rollback triggers repeatedly due to a false positive error spike caused by monitoring noise. What is the best architectural improvement to reduce unnecessary rollbacks?

hard

A. Disable rollback automation and rely on manual checks

B. Implement a cooldown period before allowing another rollback

C. Remove monitoring to avoid false alarms

D. Rollback immediately on any error spike without delay

Solution

1. Step 1: Understand problem cause

   False positive error spikes cause repeated rollbacks due to noisy monitoring data.

2. Step 2: Identify architectural fix

   Adding a cooldown period prevents rapid repeated rollbacks, allowing noise to settle before next rollback.

3. Final Answer:

   Implement a cooldown period before allowing another rollback -> Option B

4. Quick Check:

   Cooldown period reduces rollback noise impact [OK]

Hint: Cooldown period prevents rollback storms from noise [OK]

Common Mistakes:

• Disabling automation loses rollback benefits
• Removing monitoring hides real issues
• Rolling back immediately causes instability