Bird
Raised Fist0

Design a Code Deployment System - Blue-Green, Canary, Feature Flags, Rollback Strategy

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
Deployment Controller

Receives deployment command

Starts blue-green deployment with feature flags enabled

Protocol:gRPC
BeforeNo active deployment in progress
AfterDeployment Controller triggers feature flag update and canary rollout
Feature Flag Service

Updates feature flags

Writes new flags to Redis cache and Cassandra for durability

Protocol:Redis SET / Cassandra Write
BeforeOld feature flag version active
AfterNew feature flags available in cache and DB
Canary Manager

Configures Envoy and Traffic Router

Routes small % of traffic to new version (green)

Protocol:Envoy Config API
BeforeAll traffic routed to blue environment
AfterTraffic split: majority blue, minority green
Monitoring & Alerting

Collects metrics and evaluates health

Monitors canary performance and triggers alerts if needed

Protocol:Prometheus scrape / Alertmanager alerts
BeforeNo alerts, metrics normal
AfterMetrics show canary health; alerts raised if failures
Audit Logging

Logs deployment and routing events

Sends audit events to Kafka, indexed in Elasticsearch

Protocol:Kafka Produce
BeforeNo audit logs for current deployment
AfterAudit logs available for compliance and rollback analysis
Feature Flag Service

Reads feature flags

Reads feature flags from Redis cache and Cassandra DB for routing decisions

Protocol:Redis GET / Cassandra Query
BeforeNew feature flags available in cache and DB
AfterFeature flags served to routing components for deployment
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────┐
│ Deployment Controller │
└─────────────┬─────────┘
              │ gRPC Deploy Cmd
              ▼
┌─────────────────────────────┐
│ Feature Flag Service         │
│ (Redis + Cassandra)          │
└─────────────┬───────────────┘
              │ Redis GET / Cassandra Query
              ▼
┌───────────────────────┐
│ Canary Manager        │
│ (Envoy + Traffic Rtr) │
└─────────────┬─────────┘
              │ Envoy Config Update
              ▼
┌───────────────┐
│ Envoy Proxy   │
└───────┬───────┘
        │ HTTP Traffic
        ▼
┌───────────────────────┐
│ Monitoring & Alerting  │
│ (Prometheus + Alertmgr)│
└─────────────┬─────────┘
              │ Metrics / Alerts
              ▼
┌───────────────┐
│ Audit Logging  │
│ (Kafka + ES)   │
└───────────────┘
Design Rationale

This architecture supports controlled code deployments using blue-green and canary strategies. The Deployment Controller initiates deployments and updates feature flags stored in Redis and Cassandra for fast reads and durability. The Canary Manager configures Envoy proxies and traffic routers to split traffic between old and new versions, enabling gradual rollout. Monitoring & Alerting collects metrics via Prometheus and triggers alerts on anomalies, ensuring canary health is tracked. Audit Logging captures deployment and routing events through Kafka and Elasticsearch for traceability and rollback support. This design balances fast feature flag access, safe traffic routing, and observability for reliable deployments.

Primary Focus

✓

Practice

(1/5)
1. A high-throughput messaging system must decouple 1 million writes per second from fan-out delivery to thousands of clients. Which component best handles this decoupling to ensure durability and scalability?
easy
A. Use a distributed log system like Kafka partitioned by conversation ID
B. Use a relational database with row-level locking to queue messages
C. Use Redis as an in-memory queue with TTL for message expiration
D. Use direct synchronous HTTP calls from sender to all recipients

Solution

  1. Step 1: Identify the decoupling requirement

    Relational DB with row-level locking (Use a relational database with row-level locking to queue messages) cannot handle 1M writes/sec due to locking and durability overhead. Redis (Use Redis as an in-memory queue with TTL for message expiration) lacks durability guarantees and TTL causes message loss risk. Direct synchronous calls (Use direct synchronous HTTP calls from sender to all recipients) block sender and don't scale.

  2. Step 2: Evaluate Kafka's suitability

    Kafka partitions by conversation ID ensure ordering and durability, enabling scalable fan-out with consumer groups.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka partitions -> durable, scalable decoupling [check]
Hint: Distributed logs decouple writes and fan-out reliably [check]
Common Mistakes:
2. In a time-series ingestion pipeline, at step 4 the Aggregation Worker reads raw event batches from Kafka and writes summarized metrics to the time-series database. What happens if the Aggregation Worker crashes after reading but before writing?
easy
A. The raw events remain in Kafka and will be reprocessed after worker restart
B. The Aggregation Worker automatically retries writing without re-reading events
C. The summarized metrics are partially written causing data corruption in the database
D. The raw events are lost because Kafka deletes them immediately after consumption

Solution

  1. Step 1: Understand Kafka's offset commit behavior

    Kafka retains messages until consumers commit offsets. If the worker crashes before committing, messages remain available for reprocessing.

  2. Step 2: Analyze crash timing

    Since the worker crashed before writing summaries and committing offsets, raw events remain in Kafka and will be reprocessed after restart.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka retains uncommitted messages for reprocessing [check]
Hint: Kafka commits offsets after processing to avoid data loss [check]
Common Mistakes:
3. For a video calling system targeting mobile clients with intermittent connectivity, which architecture best balances latency and reliability for media delivery?
medium
A. Use a multipoint control unit (MCU) media server that mixes all streams into one
B. Use a fully peer-to-peer WebRTC connection with no media server relay
C. Use a selective forwarding unit (SFU) media server to relay streams only when needed
D. Use HTTP Live Streaming (HLS) for media delivery to ensure reliability

Solution

  1. Step 1: Evaluate architectures for mobile intermittent connectivity

    Pure peer-to-peer (A) suffers from connectivity issues; MCU (C) adds latency and CPU overhead; HLS (D) is high latency and not real-time.

  2. Step 2: SFU (B) relays streams selectively, reducing latency and handling network variability well.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    SFU balances latency and reliability for mobile intermittent networks [check]
Hint: SFU relays selectively, best for mobile intermittent connectivity [check]
Common Mistakes:
4. For managing user playlists with frequent updates and offline sync, which storage architecture is best suited given the need for eventual consistency and offline support?
medium
A. Conflict-free replicated data type (CRDT) based distributed store
B. Centralized NoSQL store with eventual consistency but no offline sync
C. Strongly consistent relational database with synchronous replication
D. Local device-only storage with manual sync to server

Solution

  1. Step 1: Identify requirements

    Playlists require offline edits and eventual sync with conflict resolution.

  2. Step 2: Evaluate architectures

    Relational DB (A) blocks offline edits; centralized NoSQL (C) lacks offline sync; local-only (D) risks data loss and complex merges. CRDTs (B) enable conflict-free merges and offline edits.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    CRDTs enable offline edits with automatic conflict resolution [check]
Hint: CRDTs handle offline sync and conflicts automatically [check]
Common Mistakes:
5. At 10x scale, which component in a full-text search system is most likely to become the bottleneck first, and why?
hard
A. The inverted index storage nodes, because random disk seeks increase dramatically with index size
B. The distributed log system (Kafka), because partition count limits throughput and causes uneven load
C. The indexing workers, because they must process and merge a growing volume of update events in real-time
D. The query router, because it must maintain stateful TCP connections to all index shards

Solution

  1. Step 1: Analyze query router bottleneck

    Query routers are typically stateless or use connection pooling; they scale horizontally well, so (A) is less likely.

  2. Step 2: Evaluate inverted index storage nodes

    As index size grows 10x, random disk seeks increase non-linearly, causing latency spikes and throughput degradation, making (B) the primary bottleneck.

  3. Step 3: Consider indexing workers and Kafka

    Indexing workers and Kafka scale with partitioning and parallelism, so (C) and (D) are secondary bottlenecks.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Disk seek latency grows fastest, limiting index node throughput [check]
Hint: Disk seeks grow fastest, hitting index storage nodes first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard