Idempotent pipeline steps in Jenkins - Commands & Configuration

Introduction

When running automated pipelines, repeating the same step should not cause errors or change results unexpectedly. Idempotent steps ensure that running a step multiple times leads to the same outcome, making pipelines reliable and safe to rerun.

When you want to safely rerun a build step without causing duplicate deployments or errors.

When a pipeline step creates resources like files or servers and you want to avoid recreating them if they already exist.

When you want to make your pipeline stable so that failures can be retried without side effects.

When multiple team members trigger the same pipeline and you want consistent results.

When automating cleanup steps that should not fail if the target is already removed.

Config File - Jenkinsfile

Jenkinsfile

pipeline {
  agent any
  stages {
    stage('Create Directory') {
      steps {
        script {
          def dirExists = fileExists('output')
          if (!dirExists) {
            sh 'mkdir output'
          } else {
            echo 'Directory already exists, skipping creation.'
          }
        }
      }
    }
    stage('Write File') {
      steps {
        script {
          def fileExists = fileExists('output/result.txt')
          if (!fileExists) {
            sh 'echo Hello World > output/result.txt'
          } else {
            echo 'File already exists, skipping write.'
          }
        }
      }
    }
  }
}

This Jenkinsfile defines two stages that check if a directory or file exists before creating or writing them. This prevents errors if the pipeline runs multiple times.

fileExists checks for presence, if conditions control execution, and echo informs about skipping steps.

Commands

This command updates or creates the Jenkins pipeline job from a YAML configuration file. It sets up the pipeline to run the idempotent steps.

Terminal

jenkins-jobs --conf jenkins.ini update my-pipeline.yaml

Expected Output

Job 'my-pipeline' updated successfully.

This command triggers the pipeline build and waits for it to finish, showing the console output to verify the steps run correctly.

Terminal

jenkins-cli build my-pipeline -s

Expected Output

[Pipeline] Start of Pipeline

[Pipeline] stage

[Pipeline] { (Create Directory)

[Pipeline] script

[Pipeline] sh

+ mkdir output

[Pipeline] echo

Directory already exists, skipping creation.

[Pipeline] }

[Pipeline] stage

[Pipeline] { (Write File)

[Pipeline] script

[Pipeline] sh

+ echo Hello World > output/result.txt

[Pipeline] echo

File already exists, skipping write.

[Pipeline] }

[Pipeline] End of Pipeline

Finished: SUCCESS

→

-s - Waits for the build to complete and shows the console output

Rerunning the pipeline to confirm idempotency: the steps detect existing files and skip creation, avoiding errors or duplicate work.

Terminal

jenkins-cli build my-pipeline -s

Expected Output

[Pipeline] Start of Pipeline

[Pipeline] stage

[Pipeline] { (Create Directory)

[Pipeline] script

[Pipeline] echo

Directory already exists, skipping creation.

[Pipeline] }

[Pipeline] stage

[Pipeline] { (Write File)

[Pipeline] script

[Pipeline] echo

File already exists, skipping write.

[Pipeline] }

[Pipeline] End of Pipeline

Finished: SUCCESS

→

-s - Waits for the build to complete and shows the console output

Key Concept

If you remember nothing else from this pattern, remember: always check if a resource exists before creating or modifying it in pipeline steps to keep them safe to run multiple times.

Common Mistakes

Running commands that create files or directories without checking if they already exist.

This causes errors or overwrites data when the pipeline runs again, breaking idempotency.

Use conditional checks like fileExists in Jenkins pipelines to skip creation if the resource is already present.

Assuming pipeline steps are idempotent without verifying their behavior.

Some shell commands or scripts may not be safe to rerun and can cause unexpected side effects.

Explicitly add checks and handle existing resources to ensure safe repeated execution.

Summary

Use conditional checks in Jenkins pipeline steps to verify if files or directories exist before creating them.

This prevents errors and makes pipeline steps safe to rerun multiple times without side effects.

Testing the pipeline by running it multiple times confirms that idempotency is working as expected.