Rollback strategies in Microservices - System Design Exercise

Design: Microservices Rollback Strategies

Design focuses on rollback strategies for microservice deployments including deployment orchestration, data consistency, and monitoring. Does not cover CI/CD pipeline design or detailed microservice implementation.

Functional Requirements

FR1: Support safe rollback of microservice deployments in case of failures

FR2: Minimize downtime during rollback

FR3: Ensure data consistency and integrity after rollback

FR4: Allow rollback of single or multiple microservices independently

FR5: Provide monitoring and alerting for rollback triggers

Non-Functional Requirements

NFR1: Handle up to 100 microservices in the system

NFR2: Rollback latency should be under 5 minutes

NFR3: Availability target of 99.9% during rollback operations

NFR4: Support rollback in both stateless and stateful microservices

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Deployment orchestrator (e.g., Kubernetes, Spinnaker)

Service registry and discovery

Versioned container images or artifacts

Database migration and rollback tools

Monitoring and alerting system

Feature flags or toggles

Design Patterns

Blue-Green Deployment

Canary Deployment

Rolling Updates with Rollback

Database Migration Rollback

Circuit Breaker Pattern

Feature Flags for quick disable

Reference Architecture

          +---------------------+
          |  Deployment System  |
          | (Kubernetes, Spinnaker) |
          +----------+----------+
                     |
          +----------v----------+
          |   Service Mesh /    |
          |  Service Registry   |
          +----------+----------+
                     |
   +-----------------+-----------------+
   |                 |                 |
+--v--+           +--v--+           +--v--+
|MS 1 |           |MS 2 |           |MS N |
+--+--+           +--+--+           +--+--+
   |                 |                 |
+--v-----------------v-----------------v--+
|           Shared Databases / Storage      |
+------------------------------------------+

Monitoring & Alerting System connected to Deployment System and Services

Components

Deployment System

Kubernetes, Spinnaker

Orchestrates deployments and rollbacks of microservices

Service Mesh / Registry

Istio, Consul

Manages service discovery and traffic routing for version control

Microservices

Containerized services (Docker)

Business logic units that can be independently deployed and rolled back

Shared Databases / Storage

Relational/NoSQL databases

Stores persistent data with migration and rollback support

Monitoring & Alerting System

Prometheus, Grafana, Alertmanager

Detects failures and triggers rollback actions

Feature Flags

LaunchDarkly, Unleash

Enables quick disabling of features without full rollback

Request Flow

1. 1. Deployment System initiates a new version deployment of a microservice.

2. 2. Service Mesh routes a small percentage of traffic to the new version (canary).

3. 3. Monitoring System observes service health and performance metrics.

4. 4. If issues detected, Deployment System triggers rollback to previous stable version.

5. 5. Service Mesh redirects traffic back to the stable version.

6. 6. Database migrations are rolled back if needed using migration tools.

7. 7. Feature flags can be toggled to disable problematic features quickly.

8. 8. Monitoring confirms system stability post-rollback.

Database Schema

Entities: - MicroserviceVersion: id, service_name, version, deployment_time, status - DeploymentRecord: id, microservice_version_id, start_time, end_time, result - RollbackRecord: id, deployment_record_id, rollback_time, reason Relationships: - MicroserviceVersion 1:N DeploymentRecord - DeploymentRecord 1:1 RollbackRecord (optional)

Scaling Discussion

Bottlenecks

Deployment system overwhelmed by simultaneous rollbacks

Database rollback complexity with large data volumes

Monitoring delays causing slow rollback detection

Service mesh routing overhead with many versions

Feature flag management complexity at scale

Solutions

Implement deployment throttling and prioritization for rollbacks

Use incremental and backward-compatible database migrations

Optimize monitoring with real-time alerting and anomaly detection

Use lightweight service mesh proxies and version-aware routing

Automate feature flag lifecycle and cleanup

Interview Tips

Time: Spend 10 minutes understanding rollback requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain different deployment strategies and their rollback implications

Highlight importance of data consistency during rollback

Discuss monitoring and alerting integration for fast rollback triggers

Describe how feature flags complement rollback strategies

Address scaling challenges and mitigation techniques

Practice

1. What is the main purpose of a rollback strategy in microservices?

easy

A. To quickly undo a bad deployment and restore the previous stable state

B. To add new features to the system without downtime

C. To permanently delete old versions of services

D. To monitor system performance continuously

Solution

1. Step 1: Understand rollback purpose

   Rollback strategies are designed to revert changes that cause issues, restoring stability.

2. Step 2: Identify correct purpose in options

   Only To quickly undo a bad deployment and restore the previous stable state describes undoing a bad deployment to restore a stable state.

3. Final Answer:

   To quickly undo a bad deployment and restore the previous stable state -> Option A

4. Quick Check:

   Rollback purpose = Undo bad deployment [OK]

Hint: Rollback means undo bad changes fast [OK]

Common Mistakes:

• Confusing rollback with feature deployment
• Thinking rollback deletes old versions permanently
• Mixing rollback with monitoring

2. Which of the following is a correct description of the blue-green deployment rollback method?

easy

A. Switch traffic back to the old environment if the new one fails

B. Gradually increase traffic to the new version while monitoring

C. Manually fix database schema errors after deployment

D. Deploy new code directly to production without testing

Solution

1. Step 1: Recall blue-green deployment basics

   Blue-green uses two identical environments; one active, one idle for new version.

2. Step 2: Identify rollback action

   If new version fails, traffic switches back to old environment instantly.

3. Final Answer:

   Switch traffic back to the old environment if the new one fails -> Option A

4. Quick Check:

   Blue-green rollback = Switch traffic back [OK]

Hint: Blue-green rollback switches traffic instantly [OK]

Common Mistakes:

• Confusing blue-green with canary deployment
• Thinking rollback fixes database manually
• Ignoring traffic switching concept

3. Consider this simplified code snippet for a canary deployment rollback trigger:

if error_rate > 0.05:
    rollback_canary()

What happens when the error rate exceeds 5% during canary deployment?

medium

A. The system continues deployment without changes

B. The error rate is ignored and logged only

C. The rollback_canary function is called to revert changes

D. The deployment is paused but not rolled back

Solution

1. Step 1: Analyze the condition in code

   The code checks if error_rate is greater than 0.05 (5%).

2. Step 2: Understand the action on condition true

   If true, rollback_canary() is called to revert the canary deployment.

3. Final Answer:

   The rollback_canary function is called to revert changes -> Option C

4. Quick Check:

   Error rate > 5% triggers rollback [OK]

Hint: Error rate > threshold triggers rollback function [OK]

Common Mistakes:

• Ignoring the rollback call in the code
• Assuming deployment pauses without rollback
• Confusing logging with rollback action

4. A microservice deployment uses database migration with rollback scripts. The rollback script fails due to a syntax error. What is the best immediate action?

medium

A. Ignore the failure and continue deployment

B. Restart the service without rollback

C. Delete the database and start fresh

D. Manually fix the rollback script and retry rollback

Solution

1. Step 1: Identify rollback script failure impact

   A syntax error in rollback script prevents safe undo of migration changes.

2. Step 2: Choose safe recovery action

   Fixing the script manually and retrying rollback ensures data integrity and system stability.

3. Final Answer:

   Manually fix the rollback script and retry rollback -> Option D

4. Quick Check:

   Fix rollback script error before retrying [OK]

Hint: Fix rollback script errors before retrying rollback [OK]

Common Mistakes:

• Ignoring rollback failure and proceeding
• Deleting database without backup
• Restarting service without fixing rollback

5. You have a microservices system using canary deployments with automated rollback on failure. Suddenly, a rollback triggers repeatedly due to a false positive error spike caused by monitoring noise. What is the best architectural improvement to reduce unnecessary rollbacks?

hard

A. Disable rollback automation and rely on manual checks

B. Implement a cooldown period before allowing another rollback

C. Remove monitoring to avoid false alarms

D. Rollback immediately on any error spike without delay

Solution

1. Step 1: Understand problem cause

   False positive error spikes cause repeated rollbacks due to noisy monitoring data.

2. Step 2: Identify architectural fix

   Adding a cooldown period prevents rapid repeated rollbacks, allowing noise to settle before next rollback.

3. Final Answer:

   Implement a cooldown period before allowing another rollback -> Option B

4. Quick Check:

   Cooldown period reduces rollback noise impact [OK]

Hint: Cooldown period prevents rollback storms from noise [OK]

Common Mistakes:

• Disabling automation loses rollback benefits
• Removing monitoring hides real issues
• Rolling back immediately causes instability