Bird
Raised Fist0

Design a Code Deployment System - Blue-Green, Canary, Feature Flags, Rollback Strategy

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Code Deployment System - Blue-Green, Canary, Feature Flags, Rollback Strategy
Design a scalable, reliable code deployment system supporting blue-green, canary releases, feature flags, and rollback strategies at internet scale
hardModern Infrastructure DesignFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed SystemsContinuous Delivery Concepts
⚙️
Functional Requirements
  • Support blue-green deployments enabling zero-downtime switch between two identical environments
  • Implement canary deployments to gradually roll out new versions to a subset of users
  • Provide feature flag management to enable/disable features dynamically without redeployment
  • Support automated rollback strategies triggered by failure detection or manual intervention
  • Integrate with CI/CD pipelines for automated deployment triggers
  • Provide deployment status monitoring and audit logging
📊
Non-Functional Requirements
  • Handle deployment requests for services with up to 2 billion daily active users
  • Ensure deployment orchestration latency under 100ms P99 for triggering deployment steps
  • Achieve 99.99% availability for deployment control plane
  • Support rollback within 1 minute of failure detection
  • Scale to handle peak deployment QPS of 10,000 deployment operations per minute
❓
Clarifying Questions to Ask
  • What is the maximum number of concurrent deployments expected?
  • Are deployments per microservice or monolithic applications?
  • What is the rollback granularity - per service, per instance, or per feature flag?
  • Is the system expected to support multi-region deployments?
  • What metrics or signals trigger automated rollback?
  • Should feature flags support gradual rollout percentages or only binary on/off?
Out of Scope
✗ Designing the underlying build system or artifact repository
✗ Service mesh or network-level traffic routing mechanisms
✗ Detailed security policies or authentication mechanisms
✗ Infrastructure provisioning (e.g., Kubernetes cluster management)
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user triggers on average 40 deployment-related events per day (e.g., feature flag checks, canary traffic routing decisions)
  • Average event size is 100 bytes
  • Peak deployment operations: 10,000 per minute
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec (feature flag checks and deployment events)
peak write qps10,000 deployments/min × 1 deployment event = 167 writes/sec (deployment control plane)
read qpsAssuming 10 reads per write for monitoring and status queries: 925,926 × 10 = 9,259,260 reads/sec
💾
Storage Estimate
daily storage925,926 writes/sec × 100 bytes × 86400 sec = 8 TB/day
retention period30 days
total storage8 TB/day × 30 days = 240 TB
with replicationReplication factor 3 → 240 TB × 3 = 720 TB total storage
🖥️
Server Estimate
deployment control planeTo handle 167 writes/sec and 10,000 deployments/min, 10 servers with autoscaling using Kubernetes
feature flag serviceTo handle 925K writes/sec and 9.2M reads/sec, use 100 Cassandra nodes for writes and 200 Redis nodes for caching reads
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |   CI/CD Pipeline     |
          +----------+-----------+
                     |
                     v
          +----------------------+
          | Deployment Controller |
          +----------+-----------+
                     |
         +-----------+------------+
         |                        |
 +-------v-------+        +-------v-------+
 | Blue Environment|      | Green Environment|
 +-------+-------+        +-------+-------+
         |                        |
         v                        v
 +----------------+       +----------------+
 | Canary Manager  |       | Feature Flag   |
 | (Traffic Router)|       | Service (Redis)|
 +--------+-------+       +--------+-------+
          |                         |
          v                         v
 +----------------+       +----------------+
 | Monitoring &   |       | Audit Logging  |
 | Alerting (Prom)|       | (Kafka + ES)   |
 +----------------+       +----------------+
🧩
Components
Deployment Controller
Orchestrates deployment workflows including blue-green switches, canary rollouts, and rollbacks
Why thisKubernetes Operators provide declarative deployment orchestration with native support for rollout strategies and rollback hooks
Why notCustom scripts lack scalability and observability at 10,000 deployments/min and cannot guarantee atomic rollbacks
Feature Flag Service (Redis + Cassandra)
Stores feature flag states and supports dynamic toggling with low latency
Why thisRedis provides sub-millisecond reads for feature flag checks; Cassandra offers scalable, highly available persistent storage for flag configurations
Why notPostgreSQL cannot handle 925K writes/sec with low latency and requires complex manual sharding
Canary Manager (Envoy + Traffic Router)
Routes a configurable percentage of user traffic to canary versions during rollout
Why thisEnvoy supports dynamic traffic shifting with fine-grained control and integrates with service mesh for observability
Why notDNS-based routing is too slow and coarse-grained for rapid canary traffic adjustments
Monitoring & Alerting (Prometheus + Alertmanager)
Collects deployment metrics and triggers alerts for rollback conditions
Why thisPrometheus supports high cardinality metrics and Alertmanager enables automated rollback triggers based on SLIs/SLOs
Why notCloudWatch lacks fine-grained custom metric support and has higher latency for alerting
Audit Logging (Kafka + Elasticsearch)
Streams deployment events and stores logs for compliance and debugging
Why thisKafka provides durable, ordered event streaming at scale; Elasticsearch enables fast querying and visualization
Why notRelational DBs cannot handle the write throughput and query latency at scale
🔄
Data Flow
1
CI/CD Pipeline → Deployment Controller
Trigger deployment with version metadata and strategy
Deployment Controller receives declarative deployment specs
2
Deployment Controller → Blue/Green Environments
Deploy new version to inactive environment
Blue-green ensures zero downtime by switching traffic atomically
3
Deployment Controller → Canary Manager
Configure traffic routing percentages for canary rollout
Envoy dynamically shifts traffic to canary instances based on config
4
Canary Manager → Monitoring & Alerting
Send metrics on canary performance and errors
Prometheus collects metrics to evaluate canary health
5
Monitoring & Alerting → Deployment Controller
Trigger automated rollback if thresholds breached
Alertmanager triggers rollback hooks in Deployment Controller
6
Deployment Controller → Feature Flag Service
Update feature flags dynamically during rollout
Feature flags enable gradual feature exposure without redeploy
7
Deployment Controller → Audit Logging
Log deployment events and status changes
Kafka streams events to Elasticsearch for querying and compliance
🗄️
deployments
PK: deployment_id
deployment_id (UUID)service_name (text)version (text)status (text)strategy (text)timestamp (timestamp)initiated_by (text)
Stores deployment metadata and status updates
🗄️
feature_flags
PK: flag_id
flag_id (UUID)feature_name (text)enabled (boolean)rollout_percentage (int)environment (text)version (text)updated_at (timestamp)
Stores feature flag states per environment and version
🔌
POST
CreateDeployment
Request
{
  "service_name": "string",
  "version": "string",
  "strategy": "string",
  "initiated_by": "string"
}
Response
{
  "deployment_id": "string",
  "status": "string"
}
Starts a new deployment with specified strategy
🔌
PUT
UpdateFeatureFlag
Request
{
  "enabled": "boolean",
  "rollout_percentage": "int"
}
Response
{
  "flag_id": "string",
  "status": "string"
}
Updates feature flag state dynamically
🔌
GET
GetDeploymentStatus
Request
{}
Response
{
  "deployment_id": "string",
  "status": "string",
  "progress": "string"
}
Fetches current status and progress of a deployment

Practice

(1/5)
1. In the API Gateway request flow, at Step 3 the Authentication component verifies the JWT token. What happens if the token is expired at this step?
easy
A. The request is forwarded to backend services with a warning header
B. The API Gateway immediately rejects the request with a 401 Unauthorized response
C. The Gateway refreshes the token automatically and retries the request
D. The request is routed to a fallback service for manual authentication

Solution

  1. Step 1: Trace Authentication at Step 3

    The Authentication component validates the JWT token; if expired, it must reject the request immediately to prevent unauthorized access.

  2. Step 2: Evaluate other options

    The request is forwarded to backend services with a warning header risks unauthorized access. The Gateway refreshes the token automatically and retries the request is incorrect because token refresh requires client interaction. The request is routed to a fallback service for manual authentication adds unnecessary complexity and latency.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Expired JWT -> immediate 401 rejection at Gateway [check]
Hint: Expired JWT -> reject immediately at gateway [check]
Common Mistakes:
2. Without the automated rollback system, what exactly breaks when a misconfiguration causes a route leak?
medium
A. The system silently drops all BGP updates, causing network partition
B. Route leak detection alerts flood the monitoring system but traffic continues normally
C. The system automatically quarantines all routes, causing a complete service outage
D. Alice's traffic is routed incorrectly; operators must manually detect and fix the leak, causing prolonged outages

Solution

  1. Step 1: Trace failure without rollback

    Misconfigurations propagate incorrect routes; without rollback, manual detection delays fix.

  2. Step 2: Eliminate other failure modes

    Silent drops (A) don't occur; automatic quarantine (C) would cause total outage, not typical; alert floods (D) don't prevent traffic misrouting.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Manual fix delays cause prolonged incorrect routing [check]
Hint: No rollback means manual fix delays outages [check]
Common Mistakes:
3. For a system requiring guaranteed message processing with minimal latency, which retry strategy is best suited when transient failures occur?
medium
A. Manual retries triggered by operators after failure alerts
B. Immediate retry with exponential backoff and jitter
C. Scheduled retries using a delayed queue with fixed intervals
D. Infinite immediate retries until success or manual intervention

Solution

  1. Step 1: Analyze retry strategies for latency and reliability

    Scheduled retries with delayed queues (B) balance retry timing and system load, reducing hot loops.

  2. Step 2: Evaluate other options

    Immediate retries (A) risk overload; manual retries (C) add latency; infinite retries (D) cause resource exhaustion.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Delayed queue retries balance latency and failure handling [check]
Hint: Delayed retries avoid overload, unlike immediate or manual retries [OK]
Common Mistakes:
4. If the Kafka broker cluster loses ZooKeeper coordination, what exactly breaks in the message delivery process?
medium
A. Messages are lost immediately as brokers cannot write to logs
B. Producers can still write, but partition leadership cannot be elected, causing delivery stalls
C. Consumers receive duplicate messages due to offset corruption
D. The entire cluster shuts down automatically to prevent data loss

Solution

  1. Step 1: Role of ZooKeeper in Kafka

    ZooKeeper manages partition leadership and consumer offsets; losing it affects coordination but not immediate message loss.

  2. Step 2: Failure impact on delivery

    Without ZooKeeper, offset metadata can become inconsistent, causing consumers to re-read messages and receive duplicates (C); producers may still write (B is partially true but leadership election stalls eventually); cluster does not shut down automatically (D).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    ZooKeeper loss -> offset inconsistency -> duplicate consumer messages [check]
Hint: ZooKeeper loss corrupts offsets causing duplicates, not immediate loss [check]
Common Mistakes:
5. For a global DNS service aiming to minimize latency for users worldwide, which architecture is more appropriate and why?
medium
A. A single centralized authoritative DNS server with a large cache to serve all queries
B. A recursive DNS server that forwards all queries to root servers directly without caching
C. A distributed network of authoritative DNS servers geographically located near users with DNS Anycast routing
D. A peer-to-peer DNS system where clients query each other for domain resolution

Solution

  1. Step 1: Analyze latency requirements and architecture options

    A single centralized server creates latency for distant users and is a single point of failure. Peer-to-peer systems lack reliability and consistency guarantees. Recursive servers forwarding to root servers without caching cause excessive load and latency.

  2. Step 2: Identify benefits of distributed authoritative servers with Anycast

    Geographically distributed authoritative servers with Anycast routing direct users to the nearest server, minimizing latency and improving availability.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Distributed authoritative servers + Anycast reduce latency globally [check]
Hint: Anycast + distributed authoritative servers minimize global latency [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard