Docker layer caching in CI - Commands & Configuration

Introduction

Building Docker images can take a long time if every step runs from scratch. Docker layer caching saves time by reusing parts of the image that did not change. This is very helpful in Continuous Integration (CI) pipelines where images are built often.

When your CI pipeline builds Docker images on every code push and you want to speed up the build.

When your Dockerfile has many steps and only a few change between builds.

When you want to reduce resource use and cost by avoiding full rebuilds in CI.

When you want faster feedback from your CI system after code changes.

When you use a remote CI system that supports caching Docker layers.

Config File - Dockerfile

Dockerfile

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . ./
CMD ["python", "app.py"]

This Dockerfile starts from a Python base image. It sets the working directory to /app. It copies only the requirements.txt file first and installs dependencies. This step is cached if requirements.txt does not change. Then it copies the rest of the app code. Finally, it runs the app with python app.py.

This order helps Docker cache the dependency installation layer, speeding up builds when only app code changes.

Commands

Builds the Docker image locally with the tag 'my-app:latest'. Docker will cache each step as a layer for future builds.

Terminal

docker build -t my-app:latest .

Expected Output

Sending build context to Docker daemon  12.29kB

Step 1/6 : FROM python:3.11-slim

 ---> 123abc456def

Step 2/6 : WORKDIR /app

 ---> Using cache

 ---> 789def012abc

Step 3/6 : COPY requirements.txt ./

 ---> Using cache

 ---> 345ghi678jkl

Step 4/6 : RUN pip install --no-cache-dir -r requirements.txt

 ---> Running in 9a8b7c6d5e4f

Collecting flask

  Downloading Flask-2.2.2-py3-none-any.whl (96 kB)

Installing collected packages: flask

Successfully installed flask-2.2.2

Removing intermediate container 9a8b7c6d5e4f

 ---> 567mno890pqr

Step 5/6 : COPY . ./

 ---> 234stu567vwx

Step 6/6 : CMD ["python", "app.py"]

 ---> Running in 1a2b3c4d5e6f

Removing intermediate container 1a2b3c4d5e6f

 ---> 890yz123abc

Successfully built 890yz123abc

Successfully tagged my-app:latest

Run the build again after no changes to requirements.txt. Docker reuses cached layers for unchanged steps, speeding up the build.

Terminal

docker build -t my-app:latest .

Expected Output

Using cache

 ---> 123abc456def

Using cache

 ---> 789def012abc

Using cache

 ---> 345ghi678jkl

Using cache

 ---> 567mno890pqr

Using cache

 ---> 234stu567vwx

Using cache

 ---> 890yz123abc

Successfully built 890yz123abc

Successfully tagged my-app:latest

In CI, this command uses the previously built image as a cache source to speed up the build on a new machine or environment.

Terminal

docker build --cache-from my-app:latest -t my-app:latest .

Expected Output

Using cache

 ---> 123abc456def

Using cache

 ---> 789def012abc

Using cache

 ---> 345ghi678jkl

Using cache

 ---> 567mno890pqr

Using cache

 ---> 234stu567vwx

Using cache

 ---> 890yz123abc

Successfully built 890yz123abc

Successfully tagged my-app:latest

→

--cache-from - Specifies an existing image to use as a cache source

Pushes the built image to a remote registry so it can be used as a cache source in CI pipelines on other machines.

Terminal

docker push my-app:latest

Expected Output

The push refers to repository [docker.io/library/my-app]

123abc456def: Pushed

789def012abc: Pushed

345ghi678jkl: Pushed

567mno890pqr: Pushed

234stu567vwx: Pushed

890yz123abc: Pushed

latest: digest: sha256:abcdef1234567890 size: 1573

Key Concept

If you remember nothing else from this pattern, remember: Docker caches each build step as a layer and reuses unchanged layers to speed up image builds.

Common Mistakes

Copying all source files before installing dependencies in the Dockerfile.

This causes Docker to rerun the dependency installation step every time any source file changes, slowing down builds.

Copy only the dependency files (like requirements.txt) first, run installation, then copy the rest of the source code.

Not pushing the built image to a registry for cache sharing in CI.

Without pushing, CI runners on different machines cannot reuse cached layers, causing full rebuilds every time.

Push the image to a remote registry after building so CI jobs can pull it as a cache source.

Not using the --cache-from flag in CI builds.

Docker will not use the remote cache image, so builds will be slower and start from scratch.

Use --cache-from with the remote image tag to enable layer caching in CI.

Summary

Write Dockerfiles to copy dependency files and install dependencies before copying app code to maximize cache reuse.

Use docker build commands locally and in CI to build images and reuse cached layers.

Push images to a remote registry and use --cache-from in CI to share cache layers across build machines.

Practice

1. What is the main purpose of Docker layer caching in Continuous Integration (CI)?

easy

A. To speed up Docker image builds by reusing unchanged layers

B. To increase the size of Docker images

C. To delete unused Docker images automatically

D. To run Docker containers faster

Solution

1. Step 1: Understand Docker layers

   Docker images are built in layers, each representing a step in the Dockerfile.

2. Step 2: Purpose of caching layers in CI

   Reusing unchanged layers avoids rebuilding them, which speeds up the build process in CI pipelines.

3. Final Answer:

   To speed up Docker image builds by reusing unchanged layers -> Option A

4. Quick Check:

   Docker layer caching = speed up builds [OK]

Hint: Caching reuses unchanged steps to build faster [OK]

Common Mistakes:

• Thinking caching deletes images
• Confusing caching with container runtime speed
• Assuming caching increases image size

2. Which Docker build command option is used to specify a cache source in a CI pipeline?

easy

A. --rm

B. --build-arg

C. --no-cache

D. --cache-from

Solution

1. Step 1: Identify cache-related options

   Docker build supports options like --cache-from to specify images to use as cache sources.

2. Step 2: Match option to cache source usage

   --cache-from tells Docker to use layers from the specified image as cache, which is essential in CI for caching.

3. Final Answer:

   --cache-from -> Option D

4. Quick Check:

   Cache source option = --cache-from [OK]

Hint: Use --cache-from to specify cache images in CI builds [OK]

Common Mistakes:

• Using --no-cache disables caching
• Confusing --build-arg with caching
• Thinking --rm affects caching

3. Given this Docker build command in a CI pipeline:

docker build --cache-from myrepo/myimage:latest -t myrepo/myimage:latest .

What happens if the base image layers have not changed since the last build?

medium

A. Docker rebuilds all layers from scratch

B. Docker uses cached layers from myrepo/myimage:latest to speed up the build

C. Docker ignores the cache and pulls a new base image

D. Docker fails with an error about missing cache

Solution

1. Step 1: Understand --cache-from behavior

   The --cache-from option allows Docker to reuse layers from the specified image if they match the current build steps.

2. Step 2: Effect when base image layers are unchanged

   If the base image layers are unchanged, Docker will reuse those cached layers, speeding up the build process.

3. Final Answer:

   Docker uses cached layers from myrepo/myimage:latest to speed up the build -> Option B

4. Quick Check:

   Unchanged layers reused = faster build [OK]

Hint: Unchanged layers reuse cache with --cache-from [OK]

Common Mistakes:

• Assuming Docker always rebuilds all layers
• Thinking cache-from pulls new images
• Expecting errors if cache is missing

4. A CI pipeline uses Docker layer caching with --cache-from, but caching is not working and builds are slow. What is the most likely cause?

medium

A. The cache image is not pushed to or pulled from the registry

B. The base image is too large

C. The Dockerfile has syntax errors

D. The Docker daemon is not running

Solution

1. Step 1: Check cache image availability

   For caching to work in CI, the cache image must be pushed to a registry and pulled before building.

2. Step 2: Identify common caching failure cause

   If the cache image is missing or not pulled, Docker cannot reuse layers, causing slow builds.

3. Final Answer:

   The cache image is not pushed to or pulled from the registry -> Option A

4. Quick Check:

   Cache image missing = no caching [OK]

Hint: Ensure cache image is pushed and pulled in CI [OK]

Common Mistakes:

• Blaming Dockerfile syntax for caching issues
• Ignoring cache image push/pull steps
• Assuming base image size affects caching directly

5. In a CI pipeline, you want to maximize Docker layer caching efficiency across multiple builds. Which approach is best?

hard

A. Use --no-cache to force fresh builds every time

B. Build images locally on the CI server without pushing

C. Push the built image to a registry and use --cache-from with that image in subsequent builds

D. Delete all images after each build to save space

Solution

1. Step 1: Understand persistent caching in CI

   To reuse cache between builds, the cache must be stored persistently, usually by pushing images to a registry.

2. Step 2: Use cache-from with pushed images

   Using --cache-from with the pushed image allows Docker to reuse layers from previous builds, speeding up future builds.

3. Final Answer:

   Push the built image to a registry and use --cache-from with that image in subsequent builds -> Option C

4. Quick Check:

   Persistent cache storage + --cache-from = efficient caching [OK]

Hint: Push images and use --cache-from for persistent caching [OK]

Common Mistakes:

• Using --no-cache disables caching
• Deleting images removes cache
• Building locally without pushing loses cache between runs