Steps within stages in Jenkins - Deep Dive

Overview - Steps within stages

What is it?

In Jenkins pipelines, a stage is a major phase of the build process, and steps are the individual tasks or commands executed inside these stages. Steps are the smallest units of work that Jenkins runs, like running a shell command, checking out code, or sending notifications. They define exactly what happens during each stage of your pipeline. This helps organize and control the flow of your automation.

Why it matters

Without steps inside stages, Jenkins wouldn't know what specific actions to perform during each phase of your build or deployment. This would make automation unclear and unreliable, causing delays and errors in software delivery. Steps allow precise control and visibility, making your pipeline predictable and easy to maintain. They turn broad phases into actionable tasks.

Where it fits

Before learning about steps within stages, you should understand what Jenkins pipelines and stages are. After mastering steps, you can learn about advanced pipeline features like parallel execution, post actions, and shared libraries. This knowledge fits into the broader journey of automating software builds and deployments with Jenkins.

Mental Model

Core Idea

Steps are the individual tasks inside a stage that tell Jenkins exactly what to do during that phase of the pipeline.

Think of it like...

Think of a stage as a recipe chapter, and steps as the individual cooking instructions you follow one by one to complete that chapter.

Pipeline
└─ Stage 1: Build
   ├─ Step 1: Checkout code
   ├─ Step 2: Compile
   └─ Step 3: Run tests
└─ Stage 2: Deploy
   ├─ Step 1: Upload artifacts
   └─ Step 2: Restart service

Build-Up - 7 Steps

1

Foundation

Understanding Jenkins pipeline stages

Concept: Stages divide the pipeline into major phases for clarity and control.

A Jenkins pipeline is split into stages like Build, Test, and Deploy. Each stage groups related work. For example, the Build stage might compile code and run tests. Stages help you see progress and organize your pipeline.

Result

You get a clear pipeline structure with named phases visible in Jenkins UI.

Knowing stages helps you organize your pipeline logically before adding detailed tasks.

2

Foundation

What are steps in Jenkins pipelines

Concept: Steps are the smallest units of work Jenkins runs inside stages.

Steps are commands or actions like running shell scripts, checking out code, or sending messages. They tell Jenkins exactly what to do during a stage. Without steps, stages would be empty containers.

Result

You understand that steps are the actual work Jenkins performs.

Recognizing steps as the building blocks of stages clarifies how pipelines execute tasks.

3

Intermediate

Writing multiple steps inside a stage

🤔Before reading on: do you think Jenkins runs steps inside a stage one after another or all at once? Commit to your answer.

Concept: Steps inside a stage run sequentially in the order they are written.

Inside a stage block, you write multiple steps in sequence. For example: stage('Build') { steps { echo 'Checking out code' sh 'make build' echo 'Build complete' } } Jenkins runs these steps one by one, waiting for each to finish before starting the next.

Result

Steps execute in order, so you can control the flow precisely.

Understanding sequential execution prevents confusion about task order and dependencies.

4

Intermediate

Using different step types inside stages

🤔Before reading on: do you think steps can only run shell commands or can they do other things too? Commit to your answer.

Concept: Steps can be many types: shell commands, Jenkins-specific commands, or plugin actions.

Steps include: - sh: run shell commands - echo: print messages - checkout: get source code - junit: publish test results Example: steps { checkout scm sh 'make test' junit 'reports/*.xml' } This mix lets you automate complex workflows.

Result

You can combine different step types to perform varied tasks in one stage.

Knowing step variety expands your ability to automate beyond just running scripts.

5

Intermediate

Declarative vs scripted steps syntax

🤔Before reading on: do you think steps syntax is the same in declarative and scripted pipelines? Commit to your answer.

Concept: Steps syntax differs between declarative and scripted pipelines but serve the same purpose.

In declarative pipelines, steps go inside a steps block: stage('Test') { steps { sh 'run-tests.sh' } } In scripted pipelines, steps are just commands inside the stage closure: stage('Test') { sh 'run-tests.sh' } Both run the same commands but have different structure rules.

Result

You can write steps correctly depending on pipeline style.

Understanding syntax differences avoids errors and confusion when switching pipeline types.

6

Advanced

Controlling step execution with conditions

🤔Before reading on: do you think steps inside a stage always run or can you skip some based on conditions? Commit to your answer.

Concept: You can control whether steps run using conditions like when or scripted if statements.

In declarative pipelines, use 'when' to control stages or steps: stage('Deploy') { when { branch 'main' } steps { sh 'deploy.sh' } } In scripted pipelines, use if statements: if (env.BRANCH_NAME == 'main') { sh 'deploy.sh' } This lets you run steps only when needed.

Result

Steps run selectively, making pipelines efficient and safe.

Knowing conditional execution helps avoid unnecessary or harmful actions.

7

Expert

Parallel steps inside stages and pitfalls

🤔Before reading on: do you think steps inside a stage can run in parallel by default? Commit to your answer.

Concept: Jenkins supports running steps in parallel inside stages, but it requires special syntax and care.

To run steps in parallel, use the parallel block: stage('Test') { parallel { unit: { sh 'run-unit-tests.sh' }, integration: { sh 'run-integration-tests.sh' } } } Be careful: parallel steps run simultaneously, so shared resources or order-dependent tasks can cause failures.

Result

You can speed up pipelines but must manage concurrency risks.

Understanding parallel execution unlocks faster pipelines but requires careful resource management.

Under the Hood

Jenkins parses the pipeline script and breaks it into stages and steps. Each step corresponds to a command or plugin action Jenkins executes on the build agent. Jenkins manages step execution order, environment, and output collection. For parallel steps, Jenkins creates separate threads or processes to run them simultaneously, coordinating results and status.

Why designed this way?

Steps inside stages provide a clear, modular way to define pipeline tasks. This design separates concerns: stages organize phases, steps define actions. It allows flexibility to mix simple commands and complex plugin calls. Parallelism support was added to speed up builds while maintaining control.

Pipeline Script
└─ Parse
   ├─ Stage 1
   │  ├─ Step 1 (run command)
   │  ├─ Step 2 (plugin call)
   │  └─ Step 3
   └─ Stage 2
      └─ Parallel Steps
         ├─ Step A (thread 1)
         └─ Step B (thread 2)

Myth Busters - 4 Common Misconceptions

Quick: Do steps inside a stage run in parallel by default? Commit yes or no.

Common Belief: Steps inside a stage run all at the same time to speed up the pipeline.

Reality: By default, steps inside a stage run one after another, sequentially.

Why it matters: Assuming parallel execution causes unexpected failures when steps depend on each other.

Quick: Can you put any Jenkins command anywhere inside a stage? Commit yes or no.

Common Belief: You can put any Jenkins command or plugin call anywhere inside a stage without restrictions.

Reality: Some commands must be inside the steps block in declarative pipelines; placing them elsewhere causes errors.

Why it matters: Misplacing commands leads to pipeline syntax errors and failed builds.

Quick: Do conditional statements inside steps affect the whole stage automatically? Commit yes or no.

Common Belief: If you put a condition inside steps, the entire stage will be skipped if the condition is false.

Reality: Conditions inside steps only skip those steps; the stage still runs unless the stage-level condition is used.

Why it matters: Confusing step and stage conditions can cause unexpected partial executions.

Quick: Is it safe to run any commands in parallel steps without extra care? Commit yes or no.

Common Belief: Running steps in parallel is always safe and faster, so you should parallelize everything.

Reality: Parallel steps can cause race conditions or resource conflicts if not designed carefully.

Why it matters: Ignoring concurrency issues leads to flaky builds and hard-to-debug failures.

Expert Zone

1

Some steps implicitly change the environment for following steps, so order matters beyond just execution sequence.

2

Parallel steps share the same workspace by default, which can cause file conflicts unless isolated.

3

Certain plugins provide custom steps that behave differently in scripted vs declarative pipelines, requiring careful usage.

When NOT to use

Avoid complex step logic inside stages when you need reusable code; instead, use shared libraries or separate scripts. For very dynamic workflows, consider scripted pipelines or external orchestration tools.

Production Patterns

In production, teams use steps to integrate testing, code analysis, and deployment commands inside stages. Parallel steps speed up test suites. Conditional steps prevent deployments on feature branches. Steps often call shared library functions for consistency.

Connections

Task orchestration

Steps inside stages are a form of task orchestration within a pipeline.

Understanding steps helps grasp how complex workflows are broken into manageable tasks in automation systems.

Software build lifecycle

Stages and steps map directly to phases and tasks in the software build lifecycle.

Knowing steps clarifies how automation mirrors real-world software development processes.

Project management workflows

Steps within stages resemble breaking down project phases into individual tasks.

Recognizing this connection helps apply pipeline design principles to managing any complex process.

Common Pitfalls

#1 Writing steps outside the steps block in declarative pipelines

Wrong approach:

stage('Build') {
echo 'Starting build'
sh 'make build'
}

Correct approach:

stage('Build') {
steps {
echo 'Starting build'
sh 'make build'
}
}

Root cause:

Misunderstanding declarative pipeline syntax requires steps to be inside the steps block.

#2 Assuming steps run in parallel without using parallel syntax

Wrong approach:

stage('Test') {
steps {
sh 'test1.sh'
sh 'test2.sh'
}
}

Correct approach:

stage('Test') {
parallel {
test1: { sh 'test1.sh' },
test2: { sh 'test2.sh' }
}
}

Root cause:

Not knowing that parallel execution requires explicit syntax.

#3 Running steps that depend on each other in parallel without isolation

Wrong approach:

stage('Deploy') {
parallel {
step1: { sh 'deploy-part1.sh' },
step2: { sh 'deploy-part2.sh' }
}
}

Correct approach:

stage('Deploy') {
steps {
sh 'deploy-part1.sh'
sh 'deploy-part2.sh'
}
}

Root cause:

Ignoring dependencies and shared resources when parallelizing steps.

Key Takeaways

Steps are the smallest units of work inside Jenkins pipeline stages that define exactly what Jenkins does.

Steps inside a stage run sequentially by default, controlling the precise order of tasks.

Different types of steps let you run shell commands, use plugins, and perform complex automation.

Parallel execution of steps speeds up pipelines but requires careful design to avoid conflicts.

Understanding steps syntax and placement is essential to writing correct and maintainable Jenkins pipelines.