Deployment pipelines (dev, staging, prod) in Jenkins - Deep Dive

Overview - Deployment pipelines (dev, staging, prod)

What is it?

Deployment pipelines are automated sequences that move software through different environments: development, staging, and production. Each environment serves a purpose to test and verify the software before it reaches real users. Jenkins is a tool that helps create and manage these pipelines automatically. This ensures software is delivered quickly and safely.

Why it matters

Without deployment pipelines, software updates would be slow, error-prone, and risky. Developers might accidentally release broken code to users, causing frustration and loss of trust. Pipelines automate testing and deployment steps, catching problems early and making releases predictable and reliable. This saves time, reduces mistakes, and improves user experience.

Where it fits

Before learning deployment pipelines, you should understand basic software development and version control. After mastering pipelines, you can explore advanced topics like continuous delivery, infrastructure as code, and automated rollback strategies.

Mental Model

Core Idea

A deployment pipeline is a step-by-step automated path that software follows from development to production, ensuring quality and safety at each stage.

Think of it like...

Think of a deployment pipeline like a car assembly line. Each station (dev, staging, prod) checks and adds parts to the car, making sure it works perfectly before it reaches the customer.

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│ Development │───▶│  Staging    │───▶│ Production  │
│  (Build &   │    │ (Test & QA) │    │ (Live Users)│
│  Unit Test) │    │             │    │             │
└─────────────┘    └─────────────┘    └─────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding deployment environments

Concept: Learn what development, staging, and production environments are and why they exist.

Development is where developers write and test code quickly. Staging is a copy of production used to test the software in conditions close to real use. Production is the live environment where users interact with the software.

Result

You can explain the purpose of each environment and why software moves through them.

Knowing the role of each environment helps you understand why we don’t deploy directly to production.

2

Foundation

Basics of Jenkins pipelines

Concept: Introduce Jenkins pipelines as scripts that automate building, testing, and deploying software.

Jenkins pipelines use a simple script language to define steps like compiling code, running tests, and deploying to servers. These scripts run automatically when code changes.

Result

You can create a basic Jenkins pipeline that builds and tests code automatically.

Understanding Jenkins pipelines is key to automating repetitive tasks and reducing human error.

3

Intermediate

Creating multi-stage pipelines

🤔Before reading on: do you think a single Jenkins pipeline can handle all environments or do you need separate pipelines for each?

Concept: Learn how to define multiple stages in one Jenkins pipeline for dev, staging, and prod.

In Jenkinsfile, you can define stages like 'Build', 'Test', 'Deploy to Dev', 'Deploy to Staging', and 'Deploy to Prod'. Each stage runs steps specific to that environment, such as different servers or tests.

Result

You can write a Jenkinsfile that moves software through all environments automatically.

Knowing how to structure multi-stage pipelines lets you control the flow and quality checks between environments.

4

Intermediate

Implementing environment-specific tests

🤔Before reading on: do you think tests should be the same in all environments or vary depending on the stage?

Concept: Understand that different environments require different levels of testing.

In development, run fast unit tests. In staging, run integration and user acceptance tests. Production may have monitoring instead of tests. Jenkins pipelines can run different test suites per stage.

Result

Your pipeline runs appropriate tests at each stage, catching issues early without slowing down development.

Tailoring tests per environment balances speed and quality effectively.

5

Advanced

Using approvals and manual gates

🤔Before reading on: do you think deployment to production should be fully automatic or require human approval?

Concept: Learn how to add manual approval steps in Jenkins pipelines before deploying to sensitive environments.

Jenkins supports input steps that pause the pipeline and wait for a human to approve. This is common before production deployment to ensure final checks.

Result

Your pipeline can pause and wait for approval, preventing accidental production releases.

Understanding manual gates helps balance automation with control and risk management.

6

Expert

Handling pipeline failures and rollbacks

🤔Before reading on: do you think a failed deployment should stop the pipeline or try to fix itself automatically?

Concept: Explore strategies for detecting failures and automatically rolling back changes in Jenkins pipelines.

You can configure Jenkins to detect failed deployments and trigger rollback scripts to restore previous stable versions. This requires careful scripting and monitoring integration.

Result

Your pipeline can recover from errors automatically, minimizing downtime and user impact.

Knowing how to handle failures and rollbacks is critical for reliable production systems.

Under the Hood

Jenkins pipelines run as code on Jenkins servers. Each stage executes commands on agents or nodes, which can be different machines or containers. The pipeline tracks status and logs for each step. It uses a domain-specific language (DSL) based on Groovy to define stages, steps, and conditions. When triggered, Jenkins schedules jobs, allocates resources, and manages environment variables to pass data between stages.

Why designed this way?

Jenkins pipelines were designed to automate complex workflows reliably and flexibly. Using code (Jenkinsfile) allows version control and repeatability. The stage-based model fits the natural flow of software delivery. Groovy DSL was chosen for its scripting power and Java compatibility. Alternatives like GUI-only tools lacked flexibility and traceability.

┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│ Jenkinsfile │──────▶│ Jenkins Core│──────▶│ Agent/Node  │
│ (Pipeline   │       │ (Scheduler, │       │ (Executes   │
│  Script)    │       │  Controller)│       │  Steps)     │
└─────────────┘       └─────────────┘       └─────────────┘
        │                    │                    │
        ▼                    ▼                    ▼
  Defines stages       Runs jobs and       Executes commands
  and steps           manages resources   on build machines

Myth Busters - 4 Common Misconceptions

Quick: do you think deployment pipelines always deploy code automatically to production without any checks? Commit yes or no.

Common Belief: Deployment pipelines push code automatically to production as soon as code is committed.

Reality: Most pipelines include manual approval steps or automated tests before production deployment to prevent errors.

Why it matters: Assuming automatic production deployment can lead to risky releases and downtime if errors slip through.

Quick: do you think staging is just a copy of production with no real testing? Commit yes or no.

Common Belief: Staging is just a backup of production and doesn’t need special tests.

Reality: Staging is a critical environment where integration and acceptance tests run to catch issues before production.

Why it matters: Ignoring staging testing can cause bugs to reach users, harming reputation and causing costly fixes.

Quick: do you think Jenkins pipelines require complex programming skills to create? Commit yes or no.

Common Belief: Only expert programmers can write Jenkins pipelines because they are complex scripts.

Reality: Jenkins pipelines use a simple, readable syntax and many examples exist; beginners can start small and grow skills.

Why it matters: Believing pipelines are too hard can discourage teams from automating, leading to manual errors and slow releases.

Quick: do you think rollback is automatic in all deployment pipelines? Commit yes or no.

Common Belief: If a deployment fails, the pipeline automatically rolls back to the previous version.

Reality: Rollback must be explicitly scripted and tested; many pipelines stop on failure without rollback.

Why it matters: Assuming automatic rollback can cause prolonged outages if manual intervention is needed but not planned.

Expert Zone

1

Pipeline stages can share environment variables and credentials securely using Jenkins credentials plugin, avoiding hardcoding sensitive data.

2

Parallel stages can speed up pipelines by running tests or deployments simultaneously, but require careful resource management.

3

Declarative pipelines provide structure and error handling, but scripted pipelines offer more flexibility for complex logic.

When NOT to use

Deployment pipelines are not suitable for very simple projects with infrequent releases; manual deployment may suffice. For highly dynamic infrastructure, consider infrastructure as code tools like Terraform combined with pipelines. Also, if your team lacks automation culture, start small before full pipelines.

Production Patterns

In production, pipelines often integrate with container registries, Kubernetes clusters, and monitoring tools. Blue-green or canary deployments are common patterns to reduce risk. Pipelines include notifications to teams on success or failure and integrate with issue trackers for traceability.

Connections

Continuous Integration

Deployment pipelines build on continuous integration by adding deployment and testing stages after code integration.

Understanding CI helps grasp how pipelines automate the entire software delivery process, not just building code.

Assembly Line Manufacturing

Deployment pipelines follow the same stepwise quality checks and handoffs as assembly lines in factories.

Seeing pipelines as assembly lines clarifies why each environment adds value and prevents defects.

Project Management Workflows

Pipelines mirror project workflows where tasks move through stages with approvals and reviews.

Knowing project workflows helps understand pipeline gates and manual approvals as quality controls.

Common Pitfalls

#1 Deploying directly to production without testing in staging.

Wrong approach:

pipeline {
stages {
stage('Build') {
steps { echo 'Building...' }
}
stage('Deploy to Prod') {
steps { echo 'Deploying to production...' }
}
}
}

Correct approach:

pipeline {
stages {
stage('Build') {
steps { echo 'Building...' }
}
stage('Deploy to Staging') {
steps { echo 'Deploying to staging...' }
}
stage('Test in Staging') {
steps { echo 'Running tests...' }
}
stage('Deploy to Prod') {
steps { input 'Approve deployment?'; echo 'Deploying to production...' }
}
}
}

Root cause:

Misunderstanding the purpose of staging and skipping critical testing steps.

#2 Hardcoding passwords and secrets in pipeline scripts.

Wrong approach:

pipeline {
environment {
DB_PASSWORD = 'mypassword123'
}
stages {
stage('Deploy') {
steps { echo 'Using password $DB_PASSWORD' }
}
}
}

Correct approach:

pipeline {
environment {
DB_PASSWORD = credentials('db-password-id')
}
stages {
stage('Deploy') {
steps { echo 'Using password securely' }
}
}
}

Root cause:

Lack of knowledge about Jenkins credentials management and security best practices.

#3 Ignoring pipeline failures and continuing deployment.

Wrong approach:

pipeline {
stages {
stage('Test') {
steps {
sh 'run-tests.sh || true'
}
}
stage('Deploy') {
steps { echo 'Deploying despite test failures' }
}
}
}

Correct approach:

pipeline {
stages {
stage('Test') {
steps {
sh 'run-tests.sh'
}
}
stage('Deploy') {
steps { echo 'Deploying only if tests pass' }
}
}
}

Root cause:

Trying to force deployment without respecting test results, risking broken releases.

Key Takeaways

Deployment pipelines automate moving software through development, staging, and production to ensure quality and safety.

Jenkins pipelines use code to define stages and steps, making automation repeatable and version-controlled.

Different environments serve distinct purposes: fast feedback in development, thorough testing in staging, and stable operation in production.

Manual approvals and environment-specific tests balance automation with control and risk management.

Handling failures and rollbacks in pipelines is essential to maintain reliable and resilient production systems.

Practice

1. What is the main purpose of having separate dev, staging, and prod stages in a Jenkins deployment pipeline?

easy

A. To deploy only to production without checks

B. To run the same tests multiple times unnecessarily

C. To test and verify code in different environments before final release

D. To slow down the deployment process

Solution

1. Step 1: Understand pipeline stages

   Each stage (dev, staging, prod) represents a different environment for testing and validation.

2. Step 2: Purpose of separate environments

   Separate environments help catch issues early and ensure stability before production deployment.

3. Final Answer:

   To test and verify code in different environments before final release -> Option C

4. Quick Check:

   Separate stages = safer deployment [OK]

Hint: Remember: dev, staging, prod = test, verify, release [OK]

Common Mistakes:

• Thinking all stages do the same thing
• Assuming staging is optional
• Believing deployment skips testing

2. Which of the following is the correct syntax to define a stage named "Staging" in a Jenkins declarative pipeline?

easy

A. stage 'Staging' steps { echo 'Deploying to staging' }

B. stage: Staging { steps: { echo 'Deploying to staging' } }

C. stage { name: 'Staging' steps { echo 'Deploying to staging' } }

D. stage('Staging') { steps { echo 'Deploying to staging' } }

Solution

1. Step 1: Recall Jenkins declarative syntax

   Stages are defined with stage('Name') { steps { ... } } blocks.

2. Step 2: Check each option

   stage('Staging') { steps { echo 'Deploying to staging' } } matches correct syntax; others have invalid syntax or missing braces.

3. Final Answer:

   stage('Staging') { steps { echo 'Deploying to staging' } } -> Option D

4. Quick Check:

   Correct stage syntax = stage('Staging') { steps { echo 'Deploying to staging' } } [OK]

Hint: Use parentheses and braces exactly as in official docs [OK]

Common Mistakes:

• Missing braces or parentheses
• Using colons instead of braces
• Incorrect keyword order

3. Given this Jenkins pipeline snippet, what will be the output if the staging stage fails?

pipeline {
  agent any
  stages {
    stage('Dev') {
      steps { echo 'Deploying to dev' }
    }
    stage('Staging') {
      steps { error 'Staging deployment failed' }
    }
    stage('Prod') {
      steps { echo 'Deploying to prod' }
    }
  }
}

medium

A. Deploying to dev\nError: Staging deployment failed\nProd stage skipped

B. Deploying to dev\nError: Staging deployment failed\nProd stage runs

C. Only Prod stage runs

D. Pipeline completes successfully without errors

Solution

1. Step 1: Understand pipeline flow

   Pipeline runs stages in order: Dev, Staging, Prod.

2. Step 2: Effect of error in Staging stage

   Error in Staging stops pipeline; Prod stage does not run.

3. Final Answer:

   Deploying to dev\nError: Staging deployment failed\nProd stage skipped -> Option A

4. Quick Check:

   Error in stage stops later stages [OK]

Hint: Error in a stage stops pipeline unless handled [OK]

Common Mistakes:

• Assuming Prod runs after Staging error
• Ignoring error messages
• Thinking pipeline completes despite errors

4. You have this Jenkins pipeline snippet:

pipeline {
  agent any
  stages {
    stage('Dev') {
      steps { echo 'Deploying to dev' }
    }
    stage('Staging') {
      steps { sh 'deploy.sh' }
    }
    stage('Prod') {
      steps { echo 'Deploying to prod' }
    }
  }
}

The deploy.sh script is missing execute permission. What will happen when the pipeline runs the Staging stage?

medium

A. The Staging stage will succeed without running the script

B. The Staging stage will fail with a permission denied error

C. The pipeline will skip the Staging stage and continue to Prod

D. The pipeline will automatically fix permissions and run the script

Solution

1. Step 1: Understand shell script execution in Jenkins

   Jenkins runs shell commands as is; missing execute permission causes failure.

2. Step 2: Effect of missing execute permission

   Shell returns 'permission denied' error; stage fails and pipeline stops.

3. Final Answer:

   The Staging stage will fail with a permission denied error -> Option B

4. Quick Check:

   Missing execute permission = stage failure [OK]

Hint: Scripts must be executable before Jenkins runs them [OK]

Common Mistakes:

• Assuming Jenkins fixes permissions automatically
• Thinking stage will skip on error
• Believing script runs without execute permission

5. You want to create a Jenkins pipeline that deploys to dev automatically, but requires manual approval before deploying to staging and prod. Which pipeline snippet correctly implements this?

hard

A. pipeline {
agent any
stages {
stage('Dev') {
steps { echo 'Deploying to dev' }
}
stage('Staging') {
steps {
input 'Approve Staging?'
echo 'Deploying to staging'
}
}
stage('Prod') {
steps {
input 'Approve Prod?'
echo 'Deploying to prod'
}
}
}
}

B. pipeline {
agent any
stages {
stage('Dev') {
steps { echo 'Deploying to dev' }
}
stage('Staging') {
steps { echo 'Deploying to staging' }
}
stage('Prod') {
steps { echo 'Deploying to prod' }
}
}
}

C. pipeline {
agent any
stages {
stage('Dev') {
steps {
input 'Approve Dev?'
echo 'Deploying to dev'
}
}
stage('Staging') {
steps { echo 'Deploying to staging' }
}
stage('Prod') {
steps { echo 'Deploying to prod' }
}
}
}

D. pipeline {
agent any
stages {
stage('Dev') {
steps { echo 'Deploying to dev' }
}
stage('Staging') {
steps {
input 'Approve Prod?'
echo 'Deploying to staging'
}
}
stage('Prod') {
steps {
input 'Approve Staging?'
echo 'Deploying to prod'
}
}
}
}

Solution

1. Step 1: Identify manual approval usage

   Jenkins input step pauses pipeline for manual approval.

2. Step 2: Check stages with input

   pipeline { agent any stages { stage('Dev') { steps { echo 'Deploying to dev' } } stage('Staging') { steps { input 'Approve Staging?' echo 'Deploying to staging' } } stage('Prod') { steps { input 'Approve Prod?' echo 'Deploying to prod' } } } } uses input before staging and prod deployments, dev runs automatically.

3. Final Answer:

   Pipeline with input steps before staging and prod deployments -> Option A

4. Quick Check:

   Manual approval = input step before stage [OK]

Hint: Use input step to pause for approval in Jenkins [OK]

Common Mistakes:

• Adding input to dev instead of staging/prod
• Skipping input steps entirely
• Mixing approval prompts between stages