Docker layer caching in CI - Commands & Configuration

Introduction

Building Docker images can take a long time if every step runs from scratch. Docker layer caching saves time by reusing parts of the image that did not change. This is very helpful in Continuous Integration (CI) pipelines where images are built often.

When your CI pipeline builds Docker images on every code push and you want to speed up the build.

When your Dockerfile has many steps and only a few change between builds.

When you want to reduce resource use and cost by avoiding full rebuilds in CI.

When you want faster feedback from your CI system after code changes.

When you use a remote CI system that supports caching Docker layers.

Config File - Dockerfile

Dockerfile

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . ./
CMD ["python", "app.py"]

This Dockerfile starts from a Python base image. It sets the working directory to /app. It copies only the requirements.txt file first and installs dependencies. This step is cached if requirements.txt does not change. Then it copies the rest of the app code. Finally, it runs the app with python app.py.

This order helps Docker cache the dependency installation layer, speeding up builds when only app code changes.

Commands

Builds the Docker image locally with the tag 'my-app:latest'. Docker will cache each step as a layer for future builds.

Terminal

docker build -t my-app:latest .

Expected Output

Sending build context to Docker daemon  12.29kB

Step 1/6 : FROM python:3.11-slim

 ---> 123abc456def

Step 2/6 : WORKDIR /app

 ---> Using cache

 ---> 789def012abc

Step 3/6 : COPY requirements.txt ./

 ---> Using cache

 ---> 345ghi678jkl

Step 4/6 : RUN pip install --no-cache-dir -r requirements.txt

 ---> Running in 9a8b7c6d5e4f

Collecting flask

  Downloading Flask-2.2.2-py3-none-any.whl (96 kB)

Installing collected packages: flask

Successfully installed flask-2.2.2

Removing intermediate container 9a8b7c6d5e4f

 ---> 567mno890pqr

Step 5/6 : COPY . ./

 ---> 234stu567vwx

Step 6/6 : CMD ["python", "app.py"]

 ---> Running in 1a2b3c4d5e6f

Removing intermediate container 1a2b3c4d5e6f

 ---> 890yz123abc

Successfully built 890yz123abc

Successfully tagged my-app:latest

Run the build again after no changes to requirements.txt. Docker reuses cached layers for unchanged steps, speeding up the build.

Terminal

docker build -t my-app:latest .

Expected Output

Using cache

 ---> 123abc456def

Using cache

 ---> 789def012abc

Using cache

 ---> 345ghi678jkl

Using cache

 ---> 567mno890pqr

Using cache

 ---> 234stu567vwx

Using cache

 ---> 890yz123abc

Successfully built 890yz123abc

Successfully tagged my-app:latest

In CI, this command uses the previously built image as a cache source to speed up the build on a new machine or environment.

Terminal

docker build --cache-from my-app:latest -t my-app:latest .

Expected Output

Using cache

 ---> 123abc456def

Using cache

 ---> 789def012abc

Using cache

 ---> 345ghi678jkl

Using cache

 ---> 567mno890pqr

Using cache

 ---> 234stu567vwx

Using cache

 ---> 890yz123abc

Successfully built 890yz123abc

Successfully tagged my-app:latest

→

--cache-from - Specifies an existing image to use as a cache source

Pushes the built image to a remote registry so it can be used as a cache source in CI pipelines on other machines.

Terminal

docker push my-app:latest

Expected Output

The push refers to repository [docker.io/library/my-app]

123abc456def: Pushed

789def012abc: Pushed

345ghi678jkl: Pushed

567mno890pqr: Pushed

234stu567vwx: Pushed

890yz123abc: Pushed

latest: digest: sha256:abcdef1234567890 size: 1573

Key Concept

If you remember nothing else from this pattern, remember: Docker caches each build step as a layer and reuses unchanged layers to speed up image builds.

Common Mistakes

Copying all source files before installing dependencies in the Dockerfile.

This causes Docker to rerun the dependency installation step every time any source file changes, slowing down builds.

Copy only the dependency files (like requirements.txt) first, run installation, then copy the rest of the source code.

Not pushing the built image to a registry for cache sharing in CI.

Without pushing, CI runners on different machines cannot reuse cached layers, causing full rebuilds every time.

Push the image to a remote registry after building so CI jobs can pull it as a cache source.

Not using the --cache-from flag in CI builds.

Docker will not use the remote cache image, so builds will be slower and start from scratch.

Use --cache-from with the remote image tag to enable layer caching in CI.

Summary

Write Dockerfiles to copy dependency files and install dependencies before copying app code to maximize cache reuse.

Use docker build commands locally and in CI to build images and reuse cached layers.

Push images to a remote registry and use --cache-from in CI to share cache layers across build machines.