Milestone step for concurrency in Jenkins - Deep Dive

Overview - Milestone step for concurrency

What is it?

The Milestone step in Jenkins is a way to control the flow of concurrent builds in a pipeline. It acts like a checkpoint that allows only the latest build to proceed past it, canceling older builds waiting at the same point. This helps avoid wasted work and resource conflicts when multiple builds run simultaneously. It is especially useful in pipelines where only the most recent changes should be deployed or tested.

Why it matters

Without the Milestone step, Jenkins pipelines can waste time and resources running outdated builds that are no longer relevant. This can slow down delivery and cause conflicts in shared environments. The Milestone step ensures that only the newest build continues, improving efficiency and reducing errors in continuous integration and delivery. It helps teams deliver faster and with more confidence.

Where it fits

Before learning the Milestone step, you should understand Jenkins pipelines and how concurrency works in them. After mastering it, you can explore advanced pipeline controls like locks, throttling, and parallel stages to manage resources and build order more precisely.

Mental Model

Core Idea

The Milestone step acts as a gate that only lets the newest build pass, canceling older builds waiting at the same checkpoint.

Think of it like...

Imagine a single-lane bridge where only the latest car is allowed to cross, and any older cars waiting must turn back to avoid traffic jams.

┌─────────────┐
│ Build #1    │
│ Waiting at  │
│ Milestone   │
├─────────────┤
│ Build #2    │
│ Latest,     │
│ Allowed to  │
│ Proceed     │
└─────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Jenkins Pipeline Basics

Concept: Learn what a Jenkins pipeline is and how it defines automated build steps.

A Jenkins pipeline is a script that defines the steps to build, test, and deploy your software. It runs these steps in order, and you can write it using a simple syntax called Groovy. Pipelines help automate repetitive tasks so you don't have to do them manually.

Result

You can write and run a basic pipeline that executes commands automatically.

Knowing pipelines is essential because the Milestone step is a special command used inside them to control build flow.

2

Foundation

Basics of Concurrency in Jenkins Pipelines

Concept: Understand how Jenkins can run multiple builds of the same pipeline at the same time.

When many changes happen quickly, Jenkins may start several builds simultaneously. These builds run in parallel or overlap in time. Without control, this can cause conflicts or waste resources because older builds might still run even if newer ones are available.

Result

You see multiple builds running concurrently in Jenkins for the same pipeline.

Recognizing concurrency challenges prepares you to use tools like the Milestone step to manage build order.

3

Intermediate

Introducing the Milestone Step

🤔Before reading on: do you think the Milestone step cancels all previous builds or just pauses them? Commit to your answer.

Concept: The Milestone step marks a point in the pipeline where only the newest build can continue, canceling older builds waiting there.

Add the 'milestone' command in your pipeline script at a key point. When multiple builds reach this point, Jenkins lets only the latest build pass and cancels the others. This prevents outdated builds from continuing and wasting resources.

Result

Older builds stop at the milestone and do not proceed; only the newest build continues.

Understanding that Milestone cancels older builds rather than just pausing them helps prevent resource waste and ensures pipeline relevance.

4

Intermediate

Using Milestone in Declarative Pipelines

🤔Before reading on: do you think Milestone can be used in declarative pipelines or only scripted ones? Commit to your answer.

Concept: Milestone can be used in declarative pipelines inside script blocks to control concurrency.

In a declarative pipeline, you add a 'script' block and call milestone inside it. For example: pipeline { agent any stages { stage('Build') { steps { script { milestone 1 } echo 'Building...' } } } } This ensures only the latest build passes milestone 1.

Result

Declarative pipelines can use milestone to control concurrency effectively.

Knowing how to integrate milestone in declarative pipelines expands your ability to manage builds in modern Jenkins scripts.

5

Advanced

Combining Milestone with Other Concurrency Controls

🤔Before reading on: do you think Milestone alone is enough to manage all concurrency issues? Commit to your answer.

Concept: Milestone works well with other steps like 'lock' and 'throttle' to manage complex concurrency scenarios.

Milestone cancels older builds but does not prevent multiple builds from starting. To avoid resource conflicts, combine milestone with 'lock' to reserve resources or 'throttle' to limit parallel builds. This layered approach ensures efficient and safe pipeline execution.

Result

Pipelines run only the latest builds and avoid resource conflicts using combined concurrency controls.

Understanding that milestone is one tool among many helps design robust pipelines that handle concurrency gracefully.

6

Expert

Internal Behavior and Edge Cases of Milestone

🤔Before reading on: do you think Milestone cancels builds immediately or waits for the milestone step to be reached? Commit to your answer.

Concept: Milestone cancels builds only when they reach the milestone step, which can lead to some builds running longer than expected if the milestone is late in the pipeline.

Jenkins tracks builds by milestone number. When a newer build reaches a milestone, Jenkins cancels older builds that have not yet passed that milestone. If the milestone is near the end, older builds may consume resources longer. Also, milestone numbers must be unique and increasing to work correctly.

Result

You see that placing milestones early in pipelines improves efficiency by canceling outdated builds sooner.

Knowing when and how milestone cancels builds helps optimize pipeline design to save time and resources.

Under the Hood

Jenkins maintains a global record of the highest milestone number reached by the latest build for each pipeline. When a build reaches a milestone step, Jenkins compares its milestone number to the global record. If the build's milestone number is less than the highest recorded, Jenkins cancels that build. This ensures only the newest build proceeds beyond each milestone.

Why designed this way?

The Milestone step was designed to solve the problem of wasted work in concurrent builds without requiring complex locking or queuing. It uses a simple numeric checkpoint system to efficiently cancel outdated builds. Alternatives like locking all builds would reduce concurrency and slow pipelines, so milestone balances concurrency with relevance.

┌───────────────┐
│ Build #1      │
│ Reaches Milestone 1 ──┐
│ (milestone=1) │      │
└───────────────┘      │
                       ▼
               ┌─────────────────┐
               │ Global Milestone │
               │ Record = 1      │
               └─────────────────┘
                       ▲
┌───────────────┐      │
│ Build #2      │      │
│ Reaches Milestone 1 ─┘
│ (milestone=1) │
└───────────────┘

Build #2 is newer, so Jenkins updates global record to 1 and cancels Build #1 if it hasn't passed milestone.

Myth Busters - 4 Common Misconceptions

Quick: Does the Milestone step pause older builds or cancel them? Commit to your answer.

Common Belief: Milestone pauses older builds at the checkpoint until the latest build finishes.

Reality: Milestone cancels older builds that reach the same milestone after a newer build has passed it.

Why it matters: Believing milestone only pauses builds can lead to resource waste and confusion when old builds continue running unnecessarily.

Quick: Can you use the same milestone number multiple times in a pipeline? Commit to your answer.

Common Belief: You can reuse the same milestone number anywhere in the pipeline without issues.

Reality: Milestone numbers should be unique and increasing to work correctly; reusing numbers can cause unexpected cancellations or no effect.

Why it matters: Misusing milestone numbers can cause builds to cancel incorrectly or never cancel, defeating the purpose of concurrency control.

Quick: Does Milestone prevent multiple builds from starting simultaneously? Commit to your answer.

Common Belief: Milestone stops multiple builds from starting at the same time.

Reality: Milestone only cancels builds when they reach the milestone step; it does not prevent builds from starting concurrently.

Why it matters: Expecting milestone to limit build start can cause resource conflicts if other controls like locks are not used.

Quick: Is placing the milestone step late in the pipeline always best? Commit to your answer.

Common Belief: Placing milestone late in the pipeline is best to allow builds to do most work before cancellation.

Reality: Placing milestone early cancels outdated builds sooner, saving resources and time.

Why it matters: Late milestones waste resources by letting old builds run long before cancellation.

Expert Zone

1

Milestone cancellation only triggers when builds reach the milestone step, so builds stuck before it are not canceled.

2

Milestone numbers must be strictly increasing to avoid confusing Jenkins about build order.

3

Milestone does not guarantee exclusive resource access; combining with locks is necessary for resource safety.

When NOT to use

Do not use Milestone when you need to serialize all builds strictly or manage resource locks; use 'lock' or 'throttle' plugins instead. Also avoid if you want to keep all build history regardless of concurrency.

Production Patterns

In production, teams place early milestones after checkout or build steps to cancel outdated builds quickly. They combine milestone with locks to prevent resource conflicts and use milestone numbers to represent logical pipeline phases.

Connections

Semaphore in Operating Systems

Both control access to shared resources and manage concurrency.

Understanding semaphores helps grasp how milestone controls build flow by allowing only one 'latest' process to proceed.

Version Control Branching

Milestone ensures only the latest code changes proceed, similar to how branches manage code versions.

Knowing how branches represent code states clarifies why only the newest build should continue in pipelines.

Traffic Light Systems

Milestone acts like a traffic light that stops older cars and lets only the newest car go.

This connection shows how controlling flow prevents jams and collisions, just like in Jenkins pipelines.

Common Pitfalls

#1 Using the same milestone number multiple times in a pipeline.

Wrong approach:

milestone 1
// some steps
milestone 1
// more steps

Correct approach:

milestone 1
// some steps
milestone 2
// more steps

Root cause:

Misunderstanding that milestone numbers must be unique and increasing to track build progress correctly.

#2 Placing milestone step too late in the pipeline.

Wrong approach:

pipeline {
stages {
stage('Build') {
steps {
echo 'Building...'
}
}
stage('Test') {
steps {
milestone 1
echo 'Testing...'
}
}
}
}

Correct approach:

pipeline {
stages {
stage('Build') {
steps {
milestone 1
echo 'Building...'
}
}
stage('Test') {
steps {
echo 'Testing...'
}
}
}
}

Root cause:

Not realizing that early milestones cancel outdated builds sooner, saving resources.

#3 Expecting milestone to prevent builds from starting concurrently.

Wrong approach:

pipeline {
stages {
stage('Build') {
steps {
milestone 1
echo 'Building...'
}
}
}
}
// No other concurrency control

Correct approach:

pipeline {
options { disableConcurrentBuilds() }
stages {
stage('Build') {
steps {
milestone 1
echo 'Building...'
}
}
}
}

Root cause:

Confusing milestone's cancellation behavior with concurrency prevention at build start.

Key Takeaways

The Milestone step in Jenkins pipelines controls concurrency by allowing only the newest build to proceed past a checkpoint, canceling older builds.

Milestone helps save time and resources by preventing outdated builds from continuing unnecessarily.

Milestone numbers must be unique and increasing to work correctly and avoid unexpected cancellations.

Placing milestones early in the pipeline improves efficiency by canceling old builds sooner.

Milestone does not prevent builds from starting concurrently; combine it with other controls like locks for full concurrency management.