Idempotent pipeline steps in Jenkins - Step-by-Step Execution

Process Flow - Idempotent pipeline steps

Start Pipeline Step

↓

Check if Step Already Done?

Yes→Skip Step

No↓

Execute Step Actions

↓

Mark Step as Done

↓

End Step

The pipeline step first checks if it has already run successfully. If yes, it skips running again. If no, it runs and marks itself done to avoid repeats.

Execution Sample

Jenkins

stage('Build') {
  steps {
    script {
      if (!fileExists('build.done')) {
        sh 'make build'
        writeFile file: 'build.done', text: 'done'
      } else {
        echo 'Build step skipped'
      }
    }
  }
}

This Jenkins pipeline stage runs the build only if a marker file does not exist, ensuring idempotency.

Process Table

Step	Check Condition	Condition Result	Action Taken	Output
1	Does 'build.done' file exist?	No	Run 'make build' and create 'build.done'	Build runs, 'build.done' created
2	Does 'build.done' file exist?	Yes	Skip build step	Output: 'Build step skipped'
3	Pipeline ends	-	-	Pipeline completes without re-running build

💡 Step is skipped when 'build.done' file exists, ensuring the build runs only once.

Status Tracker

Variable	Start	After Step 1	After Step 2	Final
build.done file	Does not exist	Created	Exists	Exists

Key Moments - 3 Insights

Why does the pipeline check for 'build.done' before running the build?

To avoid running the build multiple times unnecessarily. The execution_table row 1 shows the build runs only if 'build.done' is missing.

What happens if the 'build.done' file is accidentally deleted after the first run?

The pipeline will run the build again because the check in execution_table row 1 will be false, causing the build to execute again.

Is the build step guaranteed to run only once ever?

No, it runs once per pipeline workspace state. If the marker file is removed or workspace cleaned, the build runs again as shown in execution_table rows 1 and 2.

Visual Quiz - 3 Questions

Look at the execution_table, what action is taken when the 'build.done' file exists?

A Delete 'build.done' file

B Run 'make build' again

C Skip the build step

D Restart the pipeline

💡 Hint

Refer to execution_table row 2 under 'Action Taken'

At which step does the pipeline create the 'build.done' file?

A Step 3

B Step 1

C Step 2

D It never creates it

💡 Hint

Check execution_table row 1 'Action Taken' column

If the 'build.done' file is deleted after step 1, what will happen on the next pipeline run?

A The build step will run again

B The build step will be skipped

C The pipeline will fail

D The pipeline will exit immediately

💡 Hint

See key_moments explanation about file deletion and execution_table row 1 condition

Concept Snapshot

Idempotent pipeline steps check if their work is already done before running.
Use a marker (like a file) to track completion.
If marker exists, skip the step to save time.
If not, run the step and create the marker.
This avoids repeating costly or unsafe actions.

Full Transcript

In Jenkins pipelines, idempotent steps avoid repeating work by checking a marker like a file. If the marker exists, the step is skipped. If not, the step runs and creates the marker. This ensures the step runs only once per workspace state, saving time and preventing errors. The example shows a build step that runs only if 'build.done' file is missing. If the file exists, the build is skipped. If the file is deleted, the build runs again. This pattern helps pipelines be safe and efficient.

Practice

1. What does it mean for a Jenkins pipeline step to be idempotent?

easy

A. It always creates new resources every time it runs

B. It can run multiple times without changing the result or causing errors

C. It runs only once and then stops the pipeline

D. It requires manual approval before running again

Solution

1. Step 1: Understand idempotency in pipelines

   Idempotent means running a step many times does not change the outcome after the first run.

2. Step 2: Compare options with definition

   Only It can run multiple times without changing the result or causing errors describes this behavior correctly; others describe non-idempotent or manual steps.

3. Final Answer:

   It can run multiple times without changing the result or causing errors -> Option B

4. Quick Check:

   Idempotent = safe to repeat without side effects [OK]

Hint: Idempotent means safe to run repeatedly without side effects [OK]

Common Mistakes:

• Thinking idempotent means running only once
• Confusing idempotent with manual approval
• Assuming idempotent always creates new resources

2. Which of the following Jenkins pipeline step snippets correctly checks if a file exists before creating it to ensure idempotency?

easy

A. if (!fileExists('output.txt')) { writeFile file: 'output.txt', text: 'data' }

B. writeFile file: 'output.txt', text: 'data'

C. sh 'touch output.txt'

D. deleteFile 'output.txt'; writeFile file: 'output.txt', text: 'data'

Solution

1. Step 1: Identify idempotent file creation

   Checking if the file exists before creating it avoids duplicates or errors.

2. Step 2: Analyze each option

   if (!fileExists('output.txt')) { writeFile file: 'output.txt', text: 'data' } uses fileExists to check before writing, making it idempotent. Others either always write or delete first, which is not idempotent.

3. Final Answer:

   if (!fileExists('output.txt')) { writeFile file: 'output.txt', text: 'data' } -> Option A

4. Quick Check:

   Check before create = idempotent step [OK]

Hint: Use fileExists() to check before creating files [OK]

Common Mistakes:

• Not checking file existence before writing
• Deleting files before writing causing side effects
• Assuming shell touch is idempotent without checks

3. Given this Jenkins pipeline snippet, what will be the output if the file build.log already exists?

if (!fileExists('build.log')) {
  writeFile file: 'build.log', text: 'Build started'
} else {
  echo 'File already exists'
}

medium

A. File already exists

B. File build.log deleted

C. Error: fileExists not found

D. Build started

Solution

1. Step 1: Understand the condition logic

   The code checks if 'build.log' does not exist, then writes to it; else it echoes a message.

2. Step 2: Apply the condition with existing file

   Since 'build.log' exists, the else branch runs, printing 'File already exists'.

3. Final Answer:

   File already exists -> Option A

4. Quick Check:

   Existing file triggers else echo [OK]

Hint: If file exists, else branch runs printing message [OK]

Common Mistakes:

• Assuming writeFile runs even if file exists
• Confusing fileExists with file creation
• Expecting an error when file exists

4. You have this Jenkins pipeline step:

sh 'mkdir /app/data'
sh 'echo "Hello" > /app/data/greeting.txt'

Why might this step fail when run multiple times, and how can you fix it to be idempotent?

medium

A. echo command overwrites file; fix by appending with >>

B. No fix needed; commands are already idempotent

C. mkdir needs sudo; fix by running as root

D. mkdir fails if directory exists; fix by adding -p flag: mkdir -p /app/data

Solution

1. Step 1: Identify why mkdir fails on repeats

   Without -p, mkdir errors if directory exists, breaking idempotency.

2. Step 2: Fix mkdir command for idempotency

   Adding -p makes mkdir create directory only if missing, no error if exists.

3. Final Answer:

   mkdir fails if directory exists; fix by adding -p flag: mkdir -p /app/data -> Option D

4. Quick Check:

   mkdir -p avoids error if directory exists [OK]

Hint: Use mkdir -p to avoid errors if directory exists [OK]

Common Mistakes:

• Ignoring mkdir error on existing directory
• Thinking echo causes failure
• Assuming commands are idempotent without checks

5. You want to create a Jenkins pipeline step that installs a package only if it is not already installed, making the step idempotent. Which scripted pipeline snippet achieves this correctly?

hard

A. sh 'apt-get remove curl; apt-get install -y curl'

B. sh 'apt-get install -y curl'

C. sh 'dpkg -s curl || apt-get install -y curl'

D. sh 'apt-get update && apt-get install curl'

Solution

1. Step 1: Understand idempotent package installation

   Installing only if missing avoids repeated installs and errors.

2. Step 2: Analyze each option for idempotency

   sh 'dpkg -s curl || apt-get install -y curl' checks if package is installed with dpkg -s, installs only if missing. Others always install or remove, not idempotent.

3. Final Answer:

   sh 'dpkg -s curl || apt-get install -y curl' -> Option C

4. Quick Check:

   Check before install = idempotent package step [OK]

Hint: Check package before install to keep step idempotent [OK]

Common Mistakes:

• Always running install causing unnecessary work
• Removing package before install causing downtime
• Skipping package existence check