Bird
Raised Fist0

Design a CI/CD Pipeline System - Build, Test, Deploy, Rollback, Artifact Registry

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a CI/CD Pipeline System - Build, Test, Deploy, Rollback, Artifact Registry
Design a scalable, reliable, and efficient CI/CD pipeline supporting build, test, deploy, rollback, and artifact management at enterprise scale.
hardModern Infrastructure DesignFreq: high
Prerequisites:ContainerizationDeployment StrategiesDistributed SystemsBuild Systems
⚙️
Functional Requirements
  • Automated build system that compiles source code and produces artifacts (e.g., Docker images, binaries).
  • Automated test execution including unit, integration, and end-to-end tests with pass/fail reporting.
  • Deployment orchestration supporting multiple environments (dev, staging, production) with canary and blue-green deployment strategies.
  • Rollback mechanism to revert to previous stable versions on deployment failures.
  • Artifact registry to store, version, and retrieve build artifacts securely and efficiently.
  • Pipeline orchestration with dependency management and parallel execution of build/test/deploy stages.
📊
Non-Functional Requirements
  • Support at least 10,000 concurrent pipelines triggered per minute across the organization.
  • Pipeline execution latency P99 under 5 minutes for typical builds and tests.
  • System availability of 99.99% to ensure minimal downtime for deployments.
  • Artifact storage durability of 99.999999999% (11 nines) to prevent data loss.
  • Scalable to handle 1 million artifact uploads and 10 million artifact downloads per day.
❓
Clarifying Questions to Ask
  • What is the expected average and peak number of pipeline triggers per minute?
  • What types of deployment strategies must be supported (e.g., canary, blue-green, rolling)?
  • Are pipelines triggered only by code commits or also manually or scheduled?
  • What is the maximum acceptable pipeline execution time for critical deployments?
  • What artifact types and sizes are expected to be stored and served?
  • Is multi-tenancy required to isolate pipelines and artifacts per team or project?
Out of Scope
✗ Source code version control system design (e.g., Git).
✗ Detailed container orchestration platform internals (e.g., Kubernetes cluster design).
✗ Security scanning and vulnerability analysis of artifacts.
📌
Assumptions
  • 10,000 pipeline triggers per minute on average (600,000 per hour).
  • Each pipeline produces 3 artifacts on average, each 200 MB in size.
  • Each artifact is downloaded 10 times on average for deployments and testing.
  • Retention period for artifacts is 30 days with replication factor 3.
⚡
QPS Calculation
write qps10,000 pipelines/min × 3 artifacts/pipeline / 60 ≈ 500 writes/sec
peak write qpsAssuming 3× peak factor: 500 × 3 = 1,500 writes/sec
read qps10,000 pipelines/min × 3 artifacts × 10 downloads / 60 ≈ 5,000 reads/sec
💾
Storage Estimate
daily storage10,000 pipelines/min × 3 artifacts × 200 MB × 60 min × 24 hr = 864 TB/day
retention storage864 TB/day × 30 days = 25.9 PB
with replication25.9 PB × 3 (replication) = 77.7 PB total storage
🖥️
Server Estimate
build serversAssuming average build time 5 min, 10,000 pipelines/min → 50,000 concurrent builds. With 8 cores/server handling 4 builds concurrently, need 12,500 build servers.
artifact storage nodesAssuming 10 TB usable storage per node, need 7,770 nodes for 77.7 PB replicated storage.
🏗️
Architecture Diagram
Architecture Diagram
                +----------------------+
                |   Developer / Trigger |
                +----------+-----------+
                           |
                           v
                +----------+-----------+
                |   API Gateway (Envoy) |
                +----------+-----------+
                           |
                           v
                +----------+-----------+
                | Pipeline Orchestrator |
                +----------+-----------+
                  |           |          |
        +---------+           |          +---------+
        |                     |                    |
        v                     v                    v
 +------+-----+        +------+-----+        +-----+-------+
 | Build System|        | Test System |        | Deploy System|
 +------+-----+        +------+-----+        +-----+-------+
        |                     |                    |
        v                     v                    v
 +------+---------------------------------------------+-----+
 |               Artifact Registry (e.g., Harbor)           |
 +-----------------------------------------------------------+
                           |
                           v
                +----------+-----------+
                | Rollback Manager (State DB) |
                +----------------------+
🧩
Components
API Gateway (Envoy)
Handles all incoming pipeline trigger requests, authentication, and routing to orchestrator.
Why thisEnvoy provides high-performance L7 proxy with advanced routing, TLS termination, and observability.
Why notNginx lacks native service mesh integration and dynamic configuration at scale.
Pipeline Orchestrator (Argo Workflows)
Manages pipeline execution, dependencies, retries, and parallelism across build, test, deploy stages.
Why thisArgo Workflows natively supports Kubernetes-native pipelines with DAG execution and scalability.
Why notJenkins lacks native Kubernetes integration and struggles with large-scale parallelism.
Build System (BuildKit + Kubernetes)
Executes containerized builds efficiently with caching and parallelism on Kubernetes clusters.
Why thisBuildKit supports incremental builds and caching, Kubernetes provides scalable container orchestration.
Why notTraditional VM-based build farms have slower startup and poor resource utilization.
Artifact Registry (Harbor)
Stores, versions, and serves container images and build artifacts with security scanning and replication.
Why thisHarbor supports multi-tenancy, replication, and integrates with Kubernetes and Docker registries.
Why notDocker Hub lacks enterprise-grade replication and fine-grained access control.
Deploy System (Spinnaker)
Orchestrates deployments with support for canary, blue-green, and rolling updates across clusters.
Why thisSpinnaker provides multi-cloud deployment strategies with built-in rollback and monitoring.
Why notCustom scripts lack standardization and a rich feature set for complex deployment strategies.

Practice

(1/5)
1. In the Redlock algorithm request flow, at Step 4 the client attempts to acquire locks on all Redis masters. What happens if the client successfully acquires locks on only 3 out of 5 masters within the timeout?
easy
A. The client proceeds with the lock assuming majority success
B. The client releases any acquired locks and retries later
C. The client holds partial locks and waits for the rest to acquire
D. The client escalates to a centralized lock coordinator

Solution

  1. Step 1: Trace client lock acquisition at Step 4

    Redlock requires majority of Redis masters to be locked within timeout to consider lock acquired.

  2. Step 2: Handle partial lock acquisition

    If fewer than majority locks acquired, client must release any partial locks to avoid deadlocks and retry.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Partial lock acquisition -> release all and retry [check]
Hint: Partial locks -> release all, no partial ownership [check]
Common Mistakes:
2. A web application requires real-time bidirectional communication with low latency and minimal overhead for interactive gaming. Which technology best fits this requirement?
easy
A. Use WebSockets because they provide full-duplex communication with low latency suitable for interactive use.
B. Use Server-Sent Events because they support unidirectional streaming from server to client efficiently.
C. Use Long Polling because it is simpler to implement and works well for real-time bidirectional communication.
D. Use HTTP/2 multiplexing as it inherently supports bidirectional streaming without additional protocols.

Solution

  1. Step 1: Identify communication direction and latency needs

    Interactive gaming requires low latency and bidirectional communication, which Server-Sent Events and Long Polling do not efficiently provide.

  2. Step 2: Evaluate protocol capabilities

    WebSockets provide full-duplex communication with minimal overhead, ideal for real-time interactive applications. HTTP/2 multiplexing is not widely supported for bidirectional streaming in browsers and adds complexity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    WebSockets enable low-latency bidirectional streams [check]
Hint: Bidirectional + low latency -> WebSockets [check]
Common Mistakes:
3. A system must handle 1 million writes per second and fan-out to millions of users asynchronously. Which component best handles decoupling write load from delivery?
easy
A. Use direct HTTP calls from sender to all receivers to guarantee delivery order
B. Use a relational database with sharded writes to handle all writes and fan-out synchronously
C. Use an in-memory cache like Redis to store all messages and fan-out immediately
D. Use a distributed message queue like Kafka to buffer writes and enable asynchronous fan-out

Solution

  1. Step 1: Identify the need for decoupling writes from fan-out

    Handling 1 million writes per second with synchronous fan-out is not feasible on a single machine or direct calls.

  2. Step 2: Evaluate components for asynchronous buffering

    Kafka provides durable, partitioned logs that buffer writes and allow scalable asynchronous delivery. Relational DBs can't handle this write load synchronously. Redis lacks durability guarantees for this scale. Direct HTTP calls block and don't scale.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka buffers writes asynchronously, enabling high write throughput [check]
Hint: Use durable queues to decouple high write load [check]
Common Mistakes:
4. For propagating trace context across asynchronous messaging systems, which approach best preserves trace continuity without coupling services tightly?
medium
A. Embed trace context in message metadata headers supported by the messaging system
B. Use HTTP headers to propagate trace context in message payloads directly
C. Store trace context in a centralized database and have consumers query it on message receipt
D. Generate new trace IDs at each service and correlate offline during analysis

Solution

  1. Step 1: Understand asynchronous trace context propagation

    Embedding trace context in message metadata headers leverages messaging system features to propagate context without modifying payload or tight coupling.

  2. Step 2: Evaluate other options

    HTTP headers don't exist in messaging protocols. Centralized DB lookup adds latency and coupling. Generating new trace IDs breaks trace continuity and requires complex offline correlation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Message metadata headers preserve trace context asynchronously [check]
Hint: Use message metadata headers, not payload or DB lookup [OK]
Common Mistakes:
5. At 10x scale, which component of the vector database is most likely to become the first bottleneck, and why?
hard
A. The persistent vector storage, because disk I/O cannot keep up with write volume
B. The embedding generation service, because model inference latency grows linearly with requests
C. The query API servers, because network bandwidth saturates with large result sets
D. The HNSW index maintenance, because graph updates are CPU-intensive and hard to parallelize

Solution

  1. Step 1: Analyze scaling challenges of components

    Persistent storage (The persistent vector storage, because disk I/O cannot keep up with write volume) can be scaled horizontally with sharding; network bandwidth (The query API servers, because network bandwidth saturates with large result sets) can be optimized with compression.

  2. Step 2: Evaluate HNSW index maintenance

    HNSW graph updates require CPU-intensive link adjustments and are difficult to parallelize, causing CPU bottlenecks at scale.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    HNSW index maintenance CPU-bound and hard to shard, first to bottleneck [check]
Hint: HNSW index updates CPU-bound, hard to parallelize at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard