Bird
Raised Fist0

Design a Code Deployment System - Blue-Green, Canary, Feature Flags, Rollback Strategy

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Code Deployment System with Blue-Green, Canary, Feature Flags, and Rollback
45 min · senior
senior

You are tasked with designing a code deployment system for a startup with 50 million daily active users (DAU). The system must support blue-green deployments, canary releases, feature flags, and robust rollback strategies. The deployment system must ensure zero downtime, support encrypted artifact storage and transmission, and maintain a P99 deployment latency under 200ms for deployment state queries. The system should also integrate with CI/CD pipelines and provide audit logs for compliance.

Constraints
  • All deployment artifacts and metadata must be encrypted at rest and in transit using AES-256 and TLS 1.3.
  • The system must support zero downtime deployments with instant rollback capability.
  • P99 latency for deployment status queries must be under 200ms to support real-time monitoring dashboards.
  • Must support gradual traffic shifting for canary deployments with configurable percentages.
  • Feature flags must be dynamically configurable without redeploying services.
  • Audit logs must be immutable and queryable for compliance and debugging.
📋
Guided Steps
1/5
Step 1
Artifact Storage and Encryption
How will you store deployment artifacts securely and ensure encrypted transmission and storage?
💡 Hint
Consider a scalable object storage with built-in encryption and integration with key management services.
Key Decision
Use AWS S3 with server-side encryption (SSE-KMS) for artifact storage and TLS 1.3 for all artifact transfers to ensure AES-256 encryption at rest and in transit.
Interviewer Expectation
Mid-level: Names S3 and TLS for encryption. Senior: Explains SSE-KMS for key rotation and auditability, rejects self-managed storage due to complexity and lack of compliance guarantees.
⚠️Common Wrong Answer
Storing artifacts on local file servers without encryption - leads to data breaches and non-compliance at scale.
Step 2
Deployment Orchestration and Traffic Shifting
How will you orchestrate blue-green and canary deployments with traffic shifting and rollback capabilities?
💡 Hint
Think about a deployment orchestrator that integrates with service discovery and load balancers to control traffic percentages.
Key Decision
Use Kubernetes with Argo Rollouts for orchestrating blue-green and canary deployments, leveraging Istio or Envoy for traffic shifting and rollback control.
Interviewer Expectation
Mid-level: Mentions Kubernetes and canary deployments. Senior: Details Argo Rollouts integration with Istio virtual services for precise traffic control and instant rollback, rejects manual load balancer config due to error-prone nature at scale.
⚠️Common Wrong Answer
Manual DNS switching for blue-green - causes downtime and inconsistent user experience under high traffic.
Step 3
Feature Flag Management
How will you implement feature flags that can be updated dynamically without redeploying services?
💡 Hint
Consider a distributed configuration system with low-latency reads and push-based updates to clients.
Key Decision
Use LaunchDarkly or an open-source alternative like Unleash backed by Redis Pub/Sub for low-latency feature flag evaluation and dynamic updates via streaming APIs.
Interviewer Expectation
Mid-level: Names feature flag service. Senior: Explains Redis Pub/Sub for near real-time flag updates, rejects polling due to latency and load at 50M DAU scale.
⚠️Common Wrong Answer
Hardcoding flags in config files requiring redeployment - leads to slow rollout and increased risk.
Step 4
Deployment State Query and Monitoring
How will you design the deployment status API to meet P99 latency under 200ms for 50M DAU scale?
💡 Hint
Think about caching deployment state and using a fast NoSQL store optimized for read-heavy workloads.
Key Decision
Use DynamoDB with DAX (DynamoDB Accelerator) caching layer to serve deployment status queries with P99 < 200ms latency, combined with CloudWatch metrics for monitoring.
Interviewer Expectation
Mid-level: Suggests DynamoDB for storage. Senior: Explains DAX caching reduces read latency, rejects relational DB due to high read QPS and latency at scale.
⚠️Common Wrong Answer
Using relational DB without caching - causes high latency and DB overload under heavy query load.
Step 5
Audit Logging and Compliance
How will you implement immutable, queryable audit logs for deployments and feature flag changes?
💡 Hint
Consider an append-only log storage with indexing and encryption for compliance and forensic analysis.
Key Decision
Use AWS CloudTrail integrated with Amazon OpenSearch Service for immutable, encrypted audit logs with full-text search and query capabilities.
Interviewer Expectation
Mid-level: Mentions audit logs exist. Senior: Details CloudTrail immutability, encryption, and OpenSearch indexing for fast queries, rejects flat file logs due to lack of durability and searchability.
⚠️Common Wrong Answer
Storing logs in plain text files on servers - risks tampering and slow forensic investigations.
🔌
API Design Challenge
Sketch 3 core API endpoints for the deployment system that handle deployment initiation, status querying, and feature flag updates.
Hints
  • How does a client initiate a blue-green or canary deployment via API?
  • What parameters and response structure are needed for querying deployment status efficiently?
  • How can feature flags be updated dynamically through an API without service restarts?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Encrypted artifact storage with key management to meet compliance.
  • Orchestration supporting blue-green and canary with traffic shifting and rollback.
  • Dynamic feature flag system with low-latency updates.
  • Deployment status API with P99 latency under 200ms.
Good to Have - 30%
  • Immutable and queryable audit logs for compliance and debugging.
  • Integration with CI/CD pipelines for automation.
  • Use of service mesh (Istio/Envoy) for traffic control.
Bonus - 10%
  • Detailed rollback strategy with automated health checks.
  • Support for multi-region deployments with global traffic management.
Anti-patterns to Avoid
Using local file servers for artifact storage - leads to poor scalability and security risks due to lack of encryption and centralized management.
Manual traffic switching via DNS - causes downtime and inconsistent user experience under high traffic due to DNS caching delays and propagation issues.
Polling-based feature flag updates - causes high latency and load at 50M DAU scale, resulting in excessive network traffic and delayed flag propagation.
Relational databases without caching for deployment status - leads to high latency and database overload under heavy query load, failing to meet P99 latency requirements.
Plain text or mutable logs for audit - risks tampering and slow forensic investigations due to lack of immutability and indexing.
✅
Model Answer
▼ Reveal

The system uses AWS S3 with SSE-KMS for encrypted artifact storage and TLS 1.3 for secure transmission. Kubernetes with Argo Rollouts and Istio/Envoy service mesh orchestrates blue-green and canary deployments with precise traffic shifting and instant rollback. Feature flags are managed dynamically using LaunchDarkly or Unleash backed by Redis Pub/Sub for low-latency updates. Deployment status queries are served by DynamoDB with DAX caching to meet P99 latency under 200ms. Audit logs are immutable, encrypted, and queryable using AWS CloudTrail integrated with Amazon OpenSearch. The system integrates with CI/CD pipelines via REST APIs and webhook triggers.

Key Decisions
  • Use AWS S3 with SSE-KMS to ensure AES-256 encryption at rest and TLS 1.3 for in-transit encryption.
  • Adopt Kubernetes with Argo Rollouts and Istio for orchestrated blue-green and canary deployments with traffic shifting and rollback.
  • Implement feature flags with LaunchDarkly or Unleash backed by Redis Pub/Sub for dynamic, low-latency updates without redeployment.
  • Serve deployment status queries from DynamoDB with DAX caching to achieve P99 latency under 200ms at 50M DAU scale.
  • Store immutable audit logs using AWS CloudTrail integrated with Amazon OpenSearch for encrypted, queryable compliance logs.
  • Enforce TLS 1.3 for all network communication to secure data in transit.
  • Integrate with CI/CD pipelines via webhook triggers and REST APIs for automated deployment workflows.
  • Use Envoy sidecars for traffic routing and instant rollback without downtime.
  • AWS S3 with SSE-KMS for encrypted artifact storage ensuring compliance and key rotation.
  • Kubernetes with Argo Rollouts and Istio for orchestrated blue-green and canary deployments with traffic control.
  • LaunchDarkly or Unleash backed by Redis Pub/Sub for dynamic feature flag management at scale.
  • DynamoDB with DAX caching layer to achieve P99 < 200ms latency on deployment status queries.
  • AWS CloudTrail with Amazon OpenSearch for immutable, encrypted, and queryable audit logs.
  • TLS 1.3 enforced for all network communication to secure data in transit.
  • Integration with CI/CD pipelines via webhook triggers and REST APIs for deployment automation.
  • Use of Envoy sidecars for traffic routing and rollback without service downtime.

Practice

(1/5)
1. In a message delivery system, at step 4 the Delivery Worker checks the recipient's presence TTL in Redis. What happens if the TTL has expired at this step?
easy
A. The Delivery Worker queues the message for offline delivery and marks it as pending
B. The Delivery Worker immediately marks the message as delivered
C. The Delivery Worker deletes the message from the queue assuming recipient is offline permanently
D. The Delivery Worker retries presence check indefinitely until TTL refreshes

Solution

  1. Step 1: Understand Delivery Worker role at step 4

    Delivery Worker uses Redis presence TTL to detect if recipient is online. If TTL expired, recipient is offline.

  2. Step 2: Determine correct offline message handling

    Marking message delivered immediately (A) is incorrect because recipient hasn't received it. Deleting message (C) causes data loss. Retrying indefinitely (D) blocks processing. Correct behavior is to queue for offline delivery and mark pending (B).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Expired TTL -> queue offline delivery, mark pending [check]
Hint: Expired presence TTL means offline, queue message pending [OK]
Common Mistakes:
2. A file synchronization system must efficiently detect and upload only changed parts of large files to minimize bandwidth. Which component is primarily responsible for identifying these changed chunks before upload?
easy
A. The server-side storage engine that compares full file versions to detect differences
B. The client-side chunking module that splits files and computes hashes for delta detection
C. The network layer that compresses data during transmission
D. The metadata database that stores user permissions and file ownership

Solution

  1. Step 1: Identify chunking responsibility

    The client-side chunking module splits files into chunks and computes hashes to detect which chunks have changed, enabling delta uploads.

  2. Step 2: Eliminate other components

    The server-side storage engine stores chunks but does not perform delta detection; network layer compresses data but doesn't detect changes; metadata DB manages permissions, unrelated to chunk detection.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Client chunking detects changed parts before upload [check]
Hint: Client chunking hashes detect changes before upload [check]
Common Mistakes:
3. A global e-commerce platform wants to route users to the nearest data center to reduce latency. Which component primarily handles this geographic traffic distribution?
easy
A. GeoDNS service that resolves domain names to IPs based on user location
B. A centralized load balancer that routes traffic based on server health checks
C. A CDN that caches static content globally without DNS involvement
D. An application-level proxy that inspects HTTP headers to route requests

Solution

  1. Step 1: Identify GeoDNS role

    GeoDNS resolves DNS queries differently based on the client's geographic location, enabling routing to the nearest data center.

  2. Step 2: Eliminate other options

    A centralized load balancer (A) is a single point and doesn't handle geo-based DNS resolution. CDN (C) caches content but doesn't route DNS queries. Application-level proxy (D) routes after DNS resolution, not at DNS level.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GeoDNS routes based on client location via DNS resolution [check]
Hint: GeoDNS routes users by location at DNS level [check]
Common Mistakes:
4. For a mobile app with limited bandwidth and unpredictable network latency, which API design is better to minimize data usage and why?
medium
A. Use REST with multiple endpoints to fetch only needed data per screen
B. Use REST with a single endpoint returning all data to avoid multiple round-trips
C. Use GraphQL subscriptions to push all data updates continuously
D. Use GraphQL to request exactly the fields needed in one query, reducing over-fetching

Solution

  1. Step 1: Analyze bandwidth and latency constraints

    Mobile apps benefit from minimizing data transfer and round-trips due to limited bandwidth and latency.

  2. Step 2: Compare REST and GraphQL trade-offs

    REST multiple endpoints cause multiple round-trips; single endpoint REST over-fetches; GraphQL queries exactly needed fields in one request.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    GraphQL reduces over-fetching and round-trips for mobile [check]
Hint: GraphQL single query cuts data and round-trips [check]
Common Mistakes:
5. At 10x scale, which component in your 45-minute system design answer is most likely to fail first, and why?
hard
A. The caching layer because it always becomes stale at scale
B. The load balancer because it cannot distribute traffic evenly at high scale
C. The database because it cannot handle increased query volume without sharding
D. The stateful session management because it binds user sessions to specific servers limiting horizontal scaling

Solution

  1. Step 1: Identify stateful session management bottleneck

    Stateful sessions bind users to servers, preventing easy horizontal scaling and causing failure under 10x load.

  2. Step 2: Exclude other components that scale better or have mitigation

    Databases can shard; load balancers scale horizontally; caches can be invalidated or refreshed.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stateful sessions limit scaling -> first failure point [check]
Hint: Stateful sessions block scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard