Docker layer caching in CI - Step-by-Step Execution

Process Flow - Docker layer caching in CI

Start CI Build

↓

Check Cache for Layers

↓

Reuse Layer

↓

Assemble Image

↓

Push Image & Cache

↓

End CI Build

The CI build checks if Docker image layers exist in cache. If yes, it reuses them; if not, it builds new layers, then assembles and pushes the image and cache.

Execution Sample

Docker

docker build --cache-from=myrepo/myimage:latest -t myrepo/myimage:ci .

This command builds a Docker image using cached layers from a previous image to speed up the build in CI.

Process Table

Step	Action	Cache Check	Layer Built/Reused	Result
1	Start build	N/A	N/A	Build begins
2	Check base image layer	Hit	Reused	Base layer reused from cache
3	Check dependencies layer	Hit	Reused	Dependencies layer reused from cache
4	Check application code layer	Miss	Built	New application code layer built
5	Assemble final image	N/A	N/A	Image assembled with reused and new layers
6	Push image and cache	N/A	N/A	Image and updated cache pushed to registry
7	Build complete	N/A	N/A	CI build finished successfully

💡 Build ends after pushing updated image and cache to registry

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
Base Layer	Not checked	Reused	Reused	Reused	Reused
Dependencies Layer	Not checked	Not checked	Reused	Reused	Reused
App Code Layer	Not checked	Not checked	Not checked	Built	Built
Image State	Empty	Partial (base)	Partial (deps)	Partial (app)	Complete

Key Moments - 3 Insights

Why is the application code layer rebuilt even though other layers are reused?

Because the application code changed, cache for that layer missed, so it must be rebuilt (see execution_table step 4).

What happens if the cache is not available at all?

All layers will be built from scratch, making the build slower. This is not shown here but would mean all cache checks miss.

How does using --cache-from help in CI builds?

It tells Docker to use layers from a previous image as cache source, speeding up build by reusing unchanged layers (see execution_table steps 2 and 3).

Visual Quiz - 3 Questions

Look at the execution table, at which step is the application code layer built?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Check the 'Layer Built/Reused' column for 'Built' status related to application code.

According to the variable tracker, what is the state of the dependencies layer after step 3?

A Reused

B Not checked

C Built

D Missing

💡 Hint

Look at the 'Dependencies Layer' row under 'After Step 3' column.

If the cache was missing for all layers, how would the execution table change?

A All 'Cache Check' would be 'Hit'

B All 'Layer Built/Reused' would be 'Reused'

C All 'Cache Check' would be 'Miss' and layers built

D No change, cache is ignored

💡 Hint

Consider what happens when cache is not found for layers in the build process.

Concept Snapshot

Docker layer caching in CI:
- Docker images build in layers.
- CI uses --cache-from to reuse layers.
- Cache hit: layer reused, speeds build.
- Cache miss: layer rebuilt.
- Push updated image to update cache.

Full Transcript

In Docker layer caching during CI builds, the process starts by checking if each image layer exists in the cache. If a layer is found (cache hit), it is reused to save time. If not found (cache miss), the layer is rebuilt. After all layers are processed, the final image is assembled and pushed to the registry along with the updated cache. This speeds up CI builds by avoiding rebuilding unchanged layers. The example command uses --cache-from to specify the cache source image. The execution table shows steps where base and dependencies layers are reused, but the application code layer is rebuilt due to changes. Variables track the state of each layer through the build. Key points include understanding why some layers rebuild and how cache-from helps reuse layers. The quiz tests understanding of when layers build or reuse and the effect of cache misses.

Practice

1. What is the main purpose of Docker layer caching in Continuous Integration (CI)?

easy

A. To speed up Docker image builds by reusing unchanged layers

B. To increase the size of Docker images

C. To delete unused Docker images automatically

D. To run Docker containers faster

Solution

1. Step 1: Understand Docker layers

   Docker images are built in layers, each representing a step in the Dockerfile.

2. Step 2: Purpose of caching layers in CI

   Reusing unchanged layers avoids rebuilding them, which speeds up the build process in CI pipelines.

3. Final Answer:

   To speed up Docker image builds by reusing unchanged layers -> Option A

4. Quick Check:

   Docker layer caching = speed up builds [OK]

Hint: Caching reuses unchanged steps to build faster [OK]

Common Mistakes:

• Thinking caching deletes images
• Confusing caching with container runtime speed
• Assuming caching increases image size

2. Which Docker build command option is used to specify a cache source in a CI pipeline?

easy

A. --rm

B. --build-arg

C. --no-cache

D. --cache-from

Solution

1. Step 1: Identify cache-related options

   Docker build supports options like --cache-from to specify images to use as cache sources.

2. Step 2: Match option to cache source usage

   --cache-from tells Docker to use layers from the specified image as cache, which is essential in CI for caching.

3. Final Answer:

   --cache-from -> Option D

4. Quick Check:

   Cache source option = --cache-from [OK]

Hint: Use --cache-from to specify cache images in CI builds [OK]

Common Mistakes:

• Using --no-cache disables caching
• Confusing --build-arg with caching
• Thinking --rm affects caching

3. Given this Docker build command in a CI pipeline:

docker build --cache-from myrepo/myimage:latest -t myrepo/myimage:latest .

What happens if the base image layers have not changed since the last build?

medium

A. Docker rebuilds all layers from scratch

B. Docker uses cached layers from myrepo/myimage:latest to speed up the build

C. Docker ignores the cache and pulls a new base image

D. Docker fails with an error about missing cache

Solution

1. Step 1: Understand --cache-from behavior

   The --cache-from option allows Docker to reuse layers from the specified image if they match the current build steps.

2. Step 2: Effect when base image layers are unchanged

   If the base image layers are unchanged, Docker will reuse those cached layers, speeding up the build process.

3. Final Answer:

   Docker uses cached layers from myrepo/myimage:latest to speed up the build -> Option B

4. Quick Check:

   Unchanged layers reused = faster build [OK]

Hint: Unchanged layers reuse cache with --cache-from [OK]

Common Mistakes:

• Assuming Docker always rebuilds all layers
• Thinking cache-from pulls new images
• Expecting errors if cache is missing

4. A CI pipeline uses Docker layer caching with --cache-from, but caching is not working and builds are slow. What is the most likely cause?

medium

A. The cache image is not pushed to or pulled from the registry

B. The base image is too large

C. The Dockerfile has syntax errors

D. The Docker daemon is not running

Solution

1. Step 1: Check cache image availability

   For caching to work in CI, the cache image must be pushed to a registry and pulled before building.

2. Step 2: Identify common caching failure cause

   If the cache image is missing or not pulled, Docker cannot reuse layers, causing slow builds.

3. Final Answer:

   The cache image is not pushed to or pulled from the registry -> Option A

4. Quick Check:

   Cache image missing = no caching [OK]

Hint: Ensure cache image is pushed and pulled in CI [OK]

Common Mistakes:

• Blaming Dockerfile syntax for caching issues
• Ignoring cache image push/pull steps
• Assuming base image size affects caching directly

5. In a CI pipeline, you want to maximize Docker layer caching efficiency across multiple builds. Which approach is best?

hard

A. Use --no-cache to force fresh builds every time

B. Build images locally on the CI server without pushing

C. Push the built image to a registry and use --cache-from with that image in subsequent builds

D. Delete all images after each build to save space

Solution

1. Step 1: Understand persistent caching in CI

   To reuse cache between builds, the cache must be stored persistently, usually by pushing images to a registry.

2. Step 2: Use cache-from with pushed images

   Using --cache-from with the pushed image allows Docker to reuse layers from previous builds, speeding up future builds.

3. Final Answer:

   Push the built image to a registry and use --cache-from with that image in subsequent builds -> Option C

4. Quick Check:

   Persistent cache storage + --cache-from = efficient caching [OK]

Hint: Push images and use --cache-from for persistent caching [OK]

Common Mistakes:

• Using --no-cache disables caching
• Deleting images removes cache
• Building locally without pushing loses cache between runs