Multiple SCM repositories in Jenkins - Commands & Configuration

Introduction

Sometimes your project needs code from more than one place. Jenkins can get code from multiple source code repositories in one job. This helps when your app depends on different projects stored separately.

When your build needs code from two or more Git repositories at the same time.

When you want to combine a shared library repository with your main app repository in one Jenkins job.

When you have frontend and backend code in separate repositories but want to build them together.

When you want to pull configuration files from a different repository than your application code.

When you want to test changes that span multiple repositories in one pipeline.

Config File - Jenkinsfile

Jenkinsfile

pipeline {
  agent any
  stages {
    stage('Checkout Multiple Repos') {
      steps {
        script {
          // Checkout main app repository
          checkout([$class: 'GitSCM', branches: [[name: '*/main']],
            userRemoteConfigs: [[url: 'https://github.com/example-org/main-app.git']]])

          // Checkout shared library repository into 'libs' folder
          dir('libs') {
            checkout([$class: 'GitSCM', branches: [[name: '*/main']],
              userRemoteConfigs: [[url: 'https://github.com/example-org/shared-libs.git']]])
          }
        }
      }
    }
    stage('Build') {
      steps {
        echo 'Building the project using code from multiple repositories'
      }
    }
  }
}

This Jenkinsfile defines a pipeline with two stages. In the first stage, it checks out two Git repositories: the main app repository into the workspace root, and a shared library repository into a subfolder named 'libs'. This allows the build stage to use code from both places.

The checkout step uses the Git plugin to pull code. The dir('libs') block changes the folder before checking out the second repo.

Commands

This command updates or creates a Jenkins job named 'my-multi-scm-job' using a job definition file. It applies the configuration so Jenkins knows to run the pipeline with multiple SCM checkouts.

Terminal

jenkins-jobs --conf jenkins.ini update my-multi-scm-job.yaml

Expected Output

Job 'my-multi-scm-job' updated successfully.

This command triggers the Jenkins job named 'my-multi-scm-job' to start the pipeline. The '-s' flag waits for the build to finish and shows the console output.

Terminal

java -jar jenkins-cli.jar -s http://localhost:8080 build my-multi-scm-job -s

Expected Output

[Pipeline] Start of Pipeline

[Pipeline] stage

[Pipeline] { (Checkout Multiple Repos)

Cloning repository https://github.com/example-org/main-app.git

Cloning repository https://github.com/example-org/shared-libs.git

[Pipeline] dir

[Pipeline] checkout

[Pipeline] }

[Pipeline] stage

[Pipeline] { (Build)

Building the project using code from multiple repositories

[Pipeline] }

[Pipeline] End of Pipeline

Finished: SUCCESS

→

-s - Waits for the build to complete and shows the console output

This command fetches the console output of the last build of the job to verify the checkout and build steps ran correctly.

Terminal

java -jar jenkins-cli.jar -s http://localhost:8080 console my-multi-scm-job

Expected Output

[Pipeline] Start of Pipeline

[Pipeline] stage

[Pipeline] { (Checkout Multiple Repos)

Cloning repository https://github.com/example-org/main-app.git

Cloning repository https://github.com/example-org/shared-libs.git

[Pipeline] dir

[Pipeline] checkout

[Pipeline] }

[Pipeline] stage

[Pipeline] { (Build)

Building the project using code from multiple repositories

[Pipeline] }

[Pipeline] End of Pipeline

Finished: SUCCESS

Key Concept

If you remember nothing else from this pattern, remember: Jenkins can check out multiple source code repositories into different folders in one job using multiple checkout steps.

Common Mistakes

Trying to check out multiple repositories without specifying different folders for each.

This causes the second checkout to overwrite the first repository's files, leading to build failures.

Use the 'dir' step to change directories before each checkout so repositories do not overwrite each other.

Using multiple 'checkout' steps outside a 'script' block in declarative pipeline.

Declarative pipelines require multiple SCM checkouts inside a 'script' block to run Groovy code.

Wrap multiple checkout steps inside a 'script' block within the stage.

Summary

Use multiple 'checkout' steps inside a 'script' block to pull code from different repositories.

Use 'dir' to place each repository in its own folder to avoid overwriting files.

Trigger the Jenkins job and check console output to verify all repositories were checked out successfully.