Copying artifacts between jobs in Jenkins - Deep Dive

Overview - Copying artifacts between jobs

What is it?

Copying artifacts between jobs in Jenkins means taking files produced by one job and making them available to another job. Artifacts are usually build outputs like compiled code, reports, or packages. This process helps jobs share results without rebuilding everything. It is done using plugins or built-in Jenkins features.

Why it matters

Without copying artifacts, each job would have to redo all work, wasting time and resources. Sharing artifacts speeds up pipelines, reduces errors, and enables modular job design. It makes continuous integration and delivery smoother and more efficient, saving developers from repetitive tasks.

Where it fits

Learners should first understand Jenkins jobs, builds, and artifacts basics. After mastering artifact copying, they can explore advanced pipeline scripting, multi-branch pipelines, and distributed builds. This topic fits in the middle of Jenkins pipeline learning.

Mental Model

Core Idea

Copying artifacts between jobs is like passing a finished product from one worker to another so the next step can build on it without starting over.

Think of it like...

Imagine a bakery where one baker makes dough and passes it to another baker who bakes the bread. Instead of each baker starting from scratch, they share the dough to save time and effort.

Job A (Build) ──> [Artifact: compiled files] ──> Job B (Test/Deploy)

┌─────────┐       ┌───────────────┐       ┌───────────────┐
│ Job A   │──────▶│ Artifact Store│──────▶│ Job B         │
│ (Build) │       │ (Shared Files)│       │ (Test/Deploy) │
└─────────┘       └───────────────┘       └───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Jenkins Artifacts

Concept: Learn what artifacts are and how Jenkins stores them after a job runs.

Artifacts are files saved from a Jenkins job, like compiled programs or reports. Jenkins keeps these files so you can use them later. You define which files to save using the 'Archive the artifacts' post-build action in a job configuration.

Result

Jenkins stores specified files after a job finishes, making them available for download or use by other jobs.

Knowing what artifacts are and how Jenkins saves them is the base for sharing build outputs between jobs.

2

Foundation

Basic Job Configuration for Artifacts

Concept: How to configure a Jenkins job to archive artifacts properly.

In a Jenkins freestyle job, go to 'Post-build Actions' and select 'Archive the artifacts'. Enter the file patterns to save, like '**/target/*.jar'. Run the job to see artifacts saved in the build's page.

Result

Artifacts appear in the job's build page and can be downloaded or referenced.

Configuring artifact archiving correctly ensures the files you want to share are actually saved.

3

Intermediate

Using Copy Artifact Plugin

🤔Before reading on: do you think Jenkins can copy artifacts between jobs without plugins? Commit to your answer.

Concept: Introduce the Copy Artifact plugin to transfer artifacts from one job to another.

Install the 'Copy Artifact' plugin in Jenkins. In the downstream job, add a 'Copy artifacts from another project' build step. Specify the source job name and which build's artifacts to copy (e.g., latest successful). You can also filter files to copy.

Result

The downstream job downloads artifacts from the specified upstream job and uses them during its run.

Using a plugin simplifies artifact sharing and adds control over which files and builds to copy.

4

Intermediate

Copying Artifacts in Pipeline Scripts

🤔Before reading on: do you think pipeline syntax differs much from freestyle steps for copying artifacts? Commit to your answer.

Concept: Learn how to copy artifacts using Jenkins pipeline code for more automation and flexibility.

In a Jenkins pipeline, use the 'copyArtifacts' step inside a stage. Example: pipeline { agent any stages { stage('Copy Artifacts') { steps { copyArtifacts(projectName: 'UpstreamJob', selector: lastSuccessful()) } } } } This copies artifacts from the last successful build of 'UpstreamJob'.

Result

Pipeline job downloads artifacts automatically during execution, enabling complex workflows.

Pipeline syntax integrates artifact copying into code, making jobs more reproducible and easier to maintain.

5

Advanced

Handling Artifact Dependencies and Versions

🤔Before reading on: do you think copying artifacts always uses the latest build? Commit to your answer.

Concept: Manage which build's artifacts to copy and handle versioning to avoid mismatches.

The Copy Artifact plugin and pipeline steps allow selecting builds by criteria: latest successful, specific build number, or triggered build. Use parameters or environment variables to specify versions. This avoids using outdated or incompatible artifacts.

Result

Jobs use the correct artifact versions, preventing errors from mismatched files.

Controlling artifact versions is crucial for reliable pipelines and avoiding hard-to-debug failures.

6

Expert

Optimizing Artifact Sharing in Distributed Builds

🤔Before reading on: do you think copying artifacts is always fast and reliable in distributed Jenkins setups? Commit to your answer.

Concept: Understand challenges and solutions when copying artifacts across multiple Jenkins agents or nodes.

In distributed Jenkins, artifacts may reside on different agents. Copying artifacts involves network transfer and storage considerations. Use shared storage (like NFS or cloud storage) or artifact repositories (Nexus, Artifactory) to centralize artifacts. Jenkins can then pull artifacts efficiently. Also, clean up old artifacts to save space.

Result

Artifact sharing is faster, more reliable, and scalable across many agents and jobs.

Knowing infrastructure limits and using artifact repositories prevents bottlenecks and failures in large Jenkins environments.

Under the Hood

Jenkins stores artifacts as files linked to specific builds on the master or agent nodes. When copying artifacts, Jenkins accesses these stored files and transfers them to the workspace of the downstream job. Plugins like Copy Artifact use Jenkins APIs to locate and fetch the correct files based on build selectors. In pipelines, the copyArtifacts step wraps this logic in code. Underneath, network file transfer or shared storage is used depending on setup.

Why designed this way?

Jenkins separates jobs to keep builds independent and reproducible. Artifacts are stored per build to track exact outputs. Copying artifacts rather than rebuilding saves time and resources. Plugins and pipeline steps provide flexible, scriptable ways to share files without tightly coupling jobs. This modular design supports complex workflows and scaling.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Job A Build   │──────▶│ Artifact Store│──────▶│ Job B Workspace│
│ (Produces     │       │ (Files saved) │       │ (Copies files) │
│ artifacts)    │       │               │       │               │
└───────────────┘       └───────────────┘       └───────────────┘
       │                      ▲                        │
       │                      │                        │
       └───────────── API/Plugin/Step ────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does copying artifacts automatically rebuild the source job? Commit yes or no.

Common Belief: Copying artifacts triggers the source job to rebuild and produce fresh files.

Reality: Copying artifacts only transfers existing files from a completed build; it does not trigger any build.

Why it matters: Assuming copying triggers builds can cause confusion and unexpected pipeline delays.

Quick: Can you copy artifacts between jobs without any plugin or pipeline step? Commit yes or no.

Common Belief: Jenkins can copy artifacts between jobs natively without extra plugins or scripting.

Reality: Jenkins requires the Copy Artifact plugin or pipeline steps to copy artifacts between jobs; it is not automatic.

Why it matters: Not knowing this leads to failed attempts and wasted time troubleshooting missing files.

Quick: Does copying artifacts always get the latest build's files? Commit yes or no.

Common Belief: Copying artifacts always fetches the latest build's artifacts by default.

Reality: You can specify which build's artifacts to copy; defaults vary and can be configured to avoid errors.

Why it matters: Using wrong build artifacts can cause tests or deployments to fail unexpectedly.

Quick: Is copying artifacts between jobs always fast and reliable regardless of Jenkins setup? Commit yes or no.

Common Belief: Copying artifacts is always quick and works the same on any Jenkins setup.

Reality: In distributed Jenkins setups, copying artifacts can be slow or fail if storage or network is not configured properly.

Why it matters: Ignoring infrastructure limits causes flaky pipelines and wasted debugging effort.

Expert Zone

1

Copy Artifact plugin supports build selectors like 'Permalink' and 'Triggered build' which help tightly couple jobs in complex pipelines.

2

Pipeline 'stash' and 'unstash' steps provide an alternative for copying files within the same pipeline run but do not persist artifacts between separate jobs.

3

Artifact retention policies and cleanup strategies are critical to prevent disk space exhaustion on Jenkins masters and agents.

When NOT to use

Avoid copying artifacts for very large files or binary blobs; instead, use dedicated artifact repositories like Nexus or Artifactory. For ephemeral pipeline stages, use 'stash/unstash' to share files within the same pipeline run. Also, do not rely on copying artifacts if jobs run on isolated agents without shared storage unless you configure network transfer properly.

Production Patterns

In production, teams use artifact repositories to store and version artifacts centrally. Jenkins jobs publish artifacts there, and downstream jobs download from the repository. Copy Artifact plugin is used mainly for small files or legacy freestyle jobs. Pipelines use scripted copying combined with parameters to select exact artifact versions, enabling reproducible builds and deployments.

Connections

Artifact Repositories

Builds on

Understanding artifact copying helps grasp why centralized artifact repositories are essential for scaling and managing build outputs.

Continuous Integration Pipelines

Supports

Copying artifacts is a key enabler for chaining jobs in CI pipelines, allowing modular and efficient workflows.

Supply Chain Management (Logistics)

Analogous process

Just like supply chains move goods between factories and stores, artifact copying moves build outputs between jobs, highlighting the importance of version control and delivery timing.

Common Pitfalls

#1 Trying to copy artifacts without installing the Copy Artifact plugin.

Wrong approach:

In downstream job, add 'Copy artifacts from another project' step without plugin installed.

Correct approach:

First install the 'Copy Artifact' plugin from Jenkins plugin manager, then configure the copy step.

Root cause:

Assuming Jenkins has all features by default without checking required plugins.

#2 Copying artifacts without specifying the correct build selector, leading to missing or wrong files.

Wrong approach:

copyArtifacts(projectName: 'UpstreamJob') // no selector specified

Correct approach:

copyArtifacts(projectName: 'UpstreamJob', selector: lastSuccessful())

Root cause:

Not understanding that Jenkins needs to know which build's artifacts to copy.

#3 Copying artifacts between jobs running on different agents without shared storage or network setup.

Wrong approach:

Relying on local file paths without configuring shared storage or artifact transfer.

Correct approach:

Use shared storage solutions or artifact repositories accessible by all agents.

Root cause:

Ignoring Jenkins distributed architecture and assuming local files are always accessible.

Key Takeaways

Artifacts are files saved from a Jenkins job that can be shared with other jobs to avoid rebuilding.

Copying artifacts requires proper configuration, often using the Copy Artifact plugin or pipeline steps.

Selecting the correct build's artifacts is crucial to prevent using outdated or incompatible files.

In distributed Jenkins setups, shared storage or artifact repositories improve reliability and performance.

Understanding artifact copying enables building efficient, modular, and scalable Jenkins pipelines.