Docker compose in pipelines in Jenkins - Deep Dive

Overview - Docker compose in pipelines

What is it?

Docker Compose is a tool that helps you run multiple Docker containers together using a simple file. In pipelines, like Jenkins, Docker Compose lets you start, stop, and manage groups of containers automatically during your build and test steps. This means you can create complex environments with databases, web servers, and apps all working together without manual setup. It simplifies testing and deployment by automating container orchestration inside your pipeline.

Why it matters

Without Docker Compose in pipelines, you would have to start and manage each container manually, which is slow and error-prone. This slows down development and testing, making it hard to catch bugs early or deploy consistent environments. Docker Compose automates this, saving time and reducing mistakes, so teams can deliver software faster and with more confidence.

Where it fits

Before learning Docker Compose in pipelines, you should understand basic Docker concepts like containers and images, and how Jenkins pipelines work. After mastering this, you can explore advanced pipeline features like parallel stages, container orchestration with Kubernetes, or infrastructure as code tools that integrate with pipelines.

Mental Model

Core Idea

Docker Compose in pipelines is like a remote control that starts and stops a whole set of connected containers automatically during your build and test process.

Think of it like...

Imagine you have a home theater system with multiple devices: TV, speakers, and a game console. Instead of turning each on separately, you press one button on a remote that powers them all up in the right order. Docker Compose in pipelines acts like that remote, managing all containers together with one command.

┌─────────────────────────────┐
│ Jenkins Pipeline            │
│  ┌───────────────────────┐ │
│  │ Docker Compose Command │ │
│  └─────────┬─────────────┘ │
│            │               │
│  ┌─────────▼─────────────┐ │
│  │ Multiple Containers   │ │
│  │ ┌─────┐ ┌─────┐ ┌────┐│ │
│  │ │App  │ │DB   │ │API ││ │
│  │ └─────┘ └─────┘ └────┘│ │
│  └───────────────────────┘ │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Docker Compose Basics

Concept: Learn what Docker Compose is and how it defines multi-container setups using a YAML file.

Docker Compose uses a file called docker-compose.yml to describe multiple containers, their images, networks, and volumes. For example, you can define a web app container and a database container that work together. Running 'docker-compose up' starts all containers as one group.

Result

You can start multiple containers with one command, and they connect as defined in the file.

Knowing that Docker Compose groups containers simplifies managing complex environments compared to running containers one by one.

2

Foundation

Basics of Jenkins Pipelines

Concept: Understand how Jenkins pipelines automate steps like building, testing, and deploying software.

A Jenkins pipeline is a script that defines stages and steps to run commands automatically. It can run shell commands, build code, or run tests. Pipelines help automate repetitive tasks and ensure consistent processes.

Result

You can automate your software workflow with a script that Jenkins runs.

Seeing pipelines as automated workflows helps you understand how Docker Compose commands fit as steps inside them.

3

Intermediate

Running Docker Compose in Jenkins Pipeline

🤔Before reading on: do you think you can run 'docker-compose up' directly inside a Jenkins pipeline shell step? Commit to your answer.

Concept: Learn how to execute Docker Compose commands inside Jenkins pipeline steps to start and stop containers during builds.

Inside a Jenkins pipeline, you can add a shell step that runs 'docker-compose up -d' to start containers in detached mode. Later, you can run 'docker-compose down' to stop and clean up. This lets your pipeline create the environment it needs for testing or deployment.

Result

Containers start and stop automatically as part of the pipeline run.

Understanding that pipelines can control container lifecycles lets you automate environment setup and teardown seamlessly.

4

Intermediate

Managing Docker Compose Files in Pipelines

🤔Before reading on: do you think the docker-compose.yml file must be on the Jenkins server or can it be in the project repository? Commit to your answer.

Concept: Learn where to store and how to reference the docker-compose.yml file in your pipeline for consistent environment setup.

Typically, the docker-compose.yml file lives in your project repository. The pipeline checks out the code, so the file is available. Then the pipeline runs 'docker-compose' commands in the directory with the file. This ensures the environment matches the code version.

Result

The pipeline uses the exact compose file tied to the code being built.

Knowing the compose file travels with your code guarantees environment consistency across builds and teams.

5

Intermediate

Handling Pipeline Cleanup with Docker Compose

Concept: Learn how to clean up containers after pipeline steps to avoid leftover resources.

After tests or builds, run 'docker-compose down' to stop containers and remove networks and volumes created. This prevents resource leaks on the Jenkins agent and keeps the system clean for future runs.

Result

No leftover containers or networks remain after the pipeline finishes.

Cleaning up prevents resource exhaustion and flaky builds caused by leftover containers.

6

Advanced

Using Docker Compose with Jenkins Agents

🤔Before reading on: do you think Docker Compose commands run the same on any Jenkins agent? Commit to your answer.

Concept: Understand how Jenkins agents must have Docker and Docker Compose installed and configured to run these commands.

Jenkins agents (machines running pipeline steps) need Docker and Docker Compose installed. They must have permissions to run Docker commands. If agents are containers themselves, you may need special setup like Docker-in-Docker or mounting the Docker socket.

Result

Docker Compose commands run successfully on the agent machines during pipeline execution.

Knowing agent requirements avoids confusing errors and ensures your pipeline environment works as expected.

7

Expert

Optimizing Docker Compose in Pipelines for Speed

🤔Before reading on: do you think running 'docker-compose up' always rebuilds images and slows pipelines? Commit to your answer.

Concept: Learn techniques to speed up pipelines by caching images, using detached mode, and controlling rebuilds.

Use 'docker-compose up -d' to start containers without blocking. Avoid rebuilding images every run by pre-building and pushing images to a registry. Use 'docker-compose pull' to update images only when needed. Also, use pipeline caching to save downloaded images between runs.

Result

Pipelines run faster with less waiting for container startup and image downloads.

Understanding how to control image builds and container startup reduces pipeline time and resource use.

Under the Hood

Docker Compose reads the YAML file to create a network and volumes, then starts containers in the specified order. It uses Docker Engine APIs to manage containers. In pipelines, Jenkins runs shell commands that invoke Docker Compose, which communicates with Docker daemon on the agent machine. The pipeline script controls when containers start and stop, enabling automated environment lifecycle management.

Why designed this way?

Docker Compose was created to simplify managing multi-container Docker apps without scripting complex Docker commands. Integrating it into pipelines leverages this simplicity to automate environment setup in CI/CD workflows. This design avoids reinventing orchestration inside pipelines and uses existing Docker tools, making it flexible and widely compatible.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Jenkins       │       │ Docker Compose│       │ Docker Engine │
│ Pipeline      │──────▶│ CLI reads     │──────▶│ Manages       │
│ Script        │       │ docker-compose│       │ Containers    │
│ (shell steps) │       │ .yml file     │       │ and Networks  │
└───────────────┘       └───────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does 'docker-compose up' always rebuild images before starting containers? Commit to yes or no.

Common Belief: Running 'docker-compose up' always rebuilds the images from scratch.

Reality: 'docker-compose up' only rebuilds images if you add the '--build' flag or if images are missing. Otherwise, it uses existing images.

Why it matters: Assuming it always rebuilds can lead to unnecessarily long pipeline times and confusion about why builds are slow.

Quick: Can Jenkins pipeline steps run Docker Compose commands without Docker installed on the agent? Commit to yes or no.

Common Belief: You can run Docker Compose commands in Jenkins pipelines even if Docker is not installed on the agent.

Reality: Docker and Docker Compose must be installed and properly configured on the Jenkins agent to run these commands.

Why it matters: Not having Docker installed causes pipeline failures that can be hard to diagnose.

Quick: Does 'docker-compose down' remove your source code or project files? Commit to yes or no.

Common Belief: 'docker-compose down' deletes all project files and source code along with containers.

Reality: 'docker-compose down' only stops containers and removes networks and volumes created by Compose; it does not touch your source code or project files.

Why it matters: Misunderstanding this can cause fear of cleaning up environments and lead to cluttered systems.

Quick: Does running Docker Compose inside a Jenkins pipeline guarantee the same environment on all agents? Commit to yes or no.

Common Belief: Using Docker Compose in pipelines always creates identical environments on any Jenkins agent.

Reality: Environment consistency depends on agent setup, Docker versions, and image versions; Compose alone does not guarantee identical environments.

Why it matters: Ignoring agent differences can cause hard-to-debug inconsistencies in builds and tests.

Expert Zone

1

Docker Compose networks are isolated per project by default, which can cause issues if multiple pipelines run simultaneously on the same agent using the same project name.

2

Using 'depends_on' in docker-compose.yml controls container start order but does not wait for services to be 'ready'; you may need health checks or wait scripts in pipelines.

3

Running Docker Compose inside Jenkins containers requires careful handling of Docker socket mounting or using Docker-in-Docker, which can introduce security and stability concerns.

When NOT to use

Avoid using Docker Compose in pipelines when your application requires complex orchestration, scaling, or production-like environments; instead, use Kubernetes or dedicated container orchestration platforms. For simple single-container tasks, plain Docker commands may be simpler and faster.

Production Patterns

In production pipelines, teams often pre-build and push images to registries, then use 'docker-compose pull' in pipelines to fetch images quickly. Pipelines include cleanup steps to prevent resource leaks. Some use parallel stages to run tests against different Compose environments simultaneously.

Connections

Infrastructure as Code (IaC)

Docker Compose files are a form of IaC that define environment setup declaratively.

Understanding Docker Compose helps grasp how IaC tools like Terraform or Ansible define and automate infrastructure consistently.

Continuous Integration/Continuous Deployment (CI/CD)

Docker Compose in pipelines is a practical tool to implement CI/CD by automating environment setup for builds and tests.

Knowing how Compose integrates with pipelines clarifies how CI/CD automates software delivery with reliable environments.

Orchestration in Theater Production

Both involve coordinating multiple components to work together smoothly at the right time.

Seeing orchestration in theater helps appreciate the complexity and importance of managing container lifecycles in pipelines.

Common Pitfalls

#1 Leaving containers running after pipeline finishes, causing resource exhaustion.

Wrong approach:

pipeline {
stages {
stage('Start') {
steps {
sh 'docker-compose up -d'
}
}
stage('Test') {
steps {
sh './run-tests.sh'
}
}
}
}

Correct approach:

pipeline {
stages {
stage('Start') {
steps {
sh 'docker-compose up -d'
}
}
stage('Test') {
steps {
sh './run-tests.sh'
}
}
}
post {
always {
sh 'docker-compose down'
}
}
}

Root cause:

Forgetting to add cleanup steps in the pipeline causes containers to persist and consume resources.

#2 Running Docker Compose commands on Jenkins agents without Docker installed.

Wrong approach:

pipeline {
stages {
stage('Start') {
steps {
sh 'docker-compose up -d'
}
}
}
}

Correct approach:

Ensure Jenkins agent has Docker and Docker Compose installed and configured before running pipeline steps that use them.

Root cause:

Assuming Jenkins agents have required tools by default leads to command failures.

#3 Storing docker-compose.yml outside the project repository causing version mismatch.

Wrong approach:

pipeline {
stages {
stage('Start') {
steps {
dir('/some/other/location') {
sh 'docker-compose up -d'
}
}
}
}
}

Correct approach:

pipeline {
stages {
stage('Start') {
steps {
checkout scm
sh 'docker-compose up -d'
}
}
}
}

Root cause:

Not keeping compose files with code causes environment drift and inconsistent builds.

Key Takeaways

Docker Compose lets you define and run multiple containers together easily using a simple YAML file.

Integrating Docker Compose commands inside Jenkins pipelines automates environment setup and teardown during builds and tests.

Proper agent setup with Docker and Compose installed is essential for pipeline success.

Cleaning up containers after pipeline steps prevents resource leaks and flaky builds.

Optimizing image builds and container startup speeds up pipelines and saves resources.