Multi-stage builds concept in Docker - Deep Dive

Overview - Multi-stage builds concept

What is it?

Multi-stage builds in Docker let you use multiple steps in one Dockerfile to create smaller, cleaner images. Each step can use a different base image and only the final step's content is kept in the final image. This helps separate building your app from running it, making images lighter and faster to deploy.

Why it matters

Without multi-stage builds, Docker images often include unnecessary tools and files used only during building, making them large and slow to transfer. Multi-stage builds solve this by letting you discard build tools and keep only what your app needs to run. This saves bandwidth, storage, and speeds up deployment, which is crucial for fast and efficient software delivery.

Where it fits

Before learning multi-stage builds, you should understand basic Docker concepts like Dockerfiles, images, and containers. After mastering multi-stage builds, you can explore advanced Docker optimizations, CI/CD pipelines, and container security best practices.

Mental Model

Core Idea

Multi-stage builds let you create a Docker image in steps, keeping only the final needed parts to make the image smaller and cleaner.

Think of it like...

It's like cooking a meal in stages: you prepare ingredients in one kitchen, then only bring the cooked dish to the dining table, leaving the messy kitchen behind.

┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│ Stage 1: Build│ --> │ Stage 2: Test │ --> │ Stage 3: Final│
│ (compile code)│     │ (run tests)   │     │ (runtime only)│
└───────────────┘     └───────────────┘     └───────────────┘
          │                   │                    │
          └───── artifacts ───┴───── artifacts ────┘
                   (copied)             (final image)

Build-Up - 7 Steps

1

Foundation

Understanding Dockerfile Basics

Concept: Learn what a Dockerfile is and how it builds a Docker image step-by-step.

A Dockerfile is a text file with instructions to build a Docker image. Each instruction creates a layer. For example, FROM sets the base image, RUN executes commands, and COPY adds files. When you run 'docker build', Docker follows these steps to create an image.

Result

You can create a Docker image that packages your app and its environment.

Knowing how Dockerfiles work is essential because multi-stage builds are just an advanced way to write Dockerfiles with multiple build steps.

2

Foundation

Why Image Size Matters

Concept: Understand why smaller Docker images improve speed and efficiency.

Large images take longer to download, use more disk space, and slow deployment. For example, an image with build tools and source code is bigger than one with only the app runtime. Smaller images save bandwidth and start faster.

Result

You realize that reducing image size improves your app's delivery and performance.

Recognizing the impact of image size motivates using techniques like multi-stage builds to keep images lean.

3

Intermediate

Introducing Multi-stage Build Syntax

🤔Before reading on: do you think a Dockerfile can have multiple FROM instructions? Commit to yes or no.

Concept: Learn that multi-stage builds use multiple FROM lines to create separate build stages in one Dockerfile.

In a multi-stage Dockerfile, you write multiple FROM instructions, each starting a new stage. You can name stages with 'AS' to refer to them later. For example: FROM golang:1.20 AS builder RUN go build -o app . FROM alpine:3.18 COPY --from=builder /app /app CMD ["/app"] This builds the app in the first stage, then copies only the final binary to a smaller image.

Result

You can write Dockerfiles that build and package your app in separate steps, reducing final image size.

Understanding multiple FROM instructions unlocks the power to separate build and runtime environments cleanly.

4

Intermediate

Copying Artifacts Between Stages

🤔Before reading on: do you think files from one stage are automatically available in the next? Commit yes or no.

Concept: Learn how to explicitly copy files from one build stage to another using COPY --from.

Files created in one stage are isolated and not shared automatically. To use build outputs in later stages, you use COPY --from= to copy only needed files. This keeps the final image clean without build tools or source code.

Result

You can control exactly what goes into the final image, avoiding unnecessary files.

Knowing that stages are isolated prevents accidental bloating of images and enforces clean separation.

5

Intermediate

Reducing Final Image Size Dramatically

Concept: See how multi-stage builds can shrink images by removing build dependencies.

By building your app in a full-featured image (like golang) and copying only the compiled binary to a minimal image (like alpine), you remove compilers and libraries from the final image. This can reduce image size from hundreds of MBs to just a few MBs.

Result

Your final Docker image is much smaller, faster to download, and more secure.

Understanding this size reduction shows why multi-stage builds are a best practice for production images.

6

Advanced

Optimizing Build Cache with Multi-stage

🤔Before reading on: do you think multi-stage builds affect Docker's layer caching? Commit yes or no.

Concept: Learn how multi-stage builds interact with Docker's cache to speed up rebuilds.

Docker caches each build step to avoid repeating work. Multi-stage builds can reuse cache for early stages if unchanged. Naming stages and ordering instructions carefully helps maximize cache hits, speeding up builds. For example, separating dependencies installation from code copying helps cache dependencies longer.

Result

Builds become faster and more efficient, saving developer time.

Knowing how caching works with multi-stage builds helps write Dockerfiles that build quickly and reliably.

7

Expert

Advanced Tricks and Pitfalls in Multi-stage

🤔Before reading on: do you think multi-stage builds can cause subtle bugs if not managed carefully? Commit yes or no.

Concept: Explore complex scenarios like multi-platform builds, secret handling, and common mistakes in multi-stage Dockerfiles.

Multi-stage builds support building images for different CPU architectures by using build arguments and platform flags. Secrets like API keys should not be copied into final images; use build-time secrets instead. Common mistakes include copying too much data, forgetting to clean temporary files, or misnaming stages causing build failures.

Result

You can write robust, secure, and efficient multi-stage Dockerfiles for complex real-world projects.

Understanding these advanced details prevents costly errors and unlocks professional-grade Docker usage.

Under the Hood

Docker builds images layer by layer. Each FROM starts a new build stage with its own filesystem. When you use COPY --from, Docker copies files from one stage's filesystem to another. Only the final stage's layers form the final image. Intermediate stages are discarded unless tagged. This separation isolates build tools from runtime, reducing image size.

Why designed this way?

Originally, Dockerfiles had one stage, causing large images with build tools included. Multi-stage builds were introduced to solve this by allowing multiple isolated build environments in one file. This design balances simplicity (one Dockerfile) with flexibility (multiple stages), avoiding complex scripts or multiple Dockerfiles.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Stage 1: Build│──────▶│ Stage 2: Final│──────▶│ Final Image   │
│ (full tools)  │       │ (minimal base)│       │ (small size)  │
└───────────────┘       └───────────────┘       └───────────────┘
       │                        ▲
       │ COPY --from=Stage1     │
       └────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think all files from earlier stages are included in the final image automatically? Commit yes or no.

Common Belief: All files created in any build stage end up in the final Docker image.

Reality: Only files explicitly copied into the final stage using COPY --from are included; other files are discarded.

Why it matters: Assuming all files are included leads to unexpectedly large images and potential security risks from leftover build files.

Quick: Can you use environment variables set in one stage directly in another? Commit yes or no.

Common Belief: Environment variables set in one build stage are available in all subsequent stages.

Reality: Each stage is isolated; environment variables do not carry over between stages.

Why it matters: Expecting environment variables to persist causes build failures or unexpected behavior.

Quick: Do you think multi-stage builds always make builds slower? Commit yes or no.

Common Belief: Using multiple stages makes Docker builds slower because it does more work.

Reality: Multi-stage builds can be faster by improving cache usage and avoiding unnecessary steps in the final image.

Why it matters: Avoiding multi-stage builds due to this misconception misses out on build speed and size benefits.

Quick: Is it safe to copy secret files from build stages into the final image? Commit yes or no.

Common Belief: You can safely copy secret files or credentials from build stages into the final image.

Reality: Secrets copied into final images remain there permanently, risking exposure; build-time secrets should be handled differently.

Why it matters: Leaking secrets in images can cause serious security breaches in production.

Expert Zone

1

Multi-stage builds can be combined with build arguments and conditional logic to create highly flexible Dockerfiles that adapt to different environments.

2

The order of stages and instructions affects Docker's layer caching, so careful arrangement can drastically speed up iterative builds.

3

Using minimal base images in the final stage improves security by reducing attack surface but may require adding necessary runtime libraries manually.

When NOT to use

Multi-stage builds are not ideal when your build process requires complex orchestration outside Docker or when using external build systems like Bazel. In such cases, separate build pipelines or specialized tools may be better.

Production Patterns

In production, multi-stage builds are used to compile code in a full SDK image, run tests in a separate stage, and produce a minimal runtime image for deployment. This pattern ensures fast, secure, and reliable containerized applications.

Connections

Continuous Integration/Continuous Deployment (CI/CD)

Multi-stage builds integrate with CI/CD pipelines to automate building, testing, and deploying optimized images.

Understanding multi-stage builds helps design efficient pipelines that produce small, secure images ready for deployment.

Software Build Systems

Multi-stage builds mirror traditional build systems that separate compile and package steps.

Recognizing this connection clarifies why separating build and runtime environments improves software delivery.

Cooking Recipes

Like multi-stage builds, cooking recipes separate preparation and serving steps to deliver a clean final dish.

This cross-domain insight shows how breaking complex tasks into stages improves quality and efficiency.

Common Pitfalls

#1 Including build tools in the final image, making it unnecessarily large.

Wrong approach:

FROM golang:1.20
RUN go build -o app .
CMD ["./app"]

Correct approach:

FROM golang:1.20 AS builder
RUN go build -o app .
FROM alpine:3.18
COPY --from=builder /app /app
CMD ["/app"]

Root cause:

Not separating build and runtime stages causes all build dependencies to remain in the final image.

#2 Assuming files from one stage are automatically available in the next.

Wrong approach:

FROM node:18
RUN npm build
FROM alpine
RUN ls /app # expecting build files here

Correct approach:

FROM node:18 AS builder
RUN npm build
FROM alpine
COPY --from=builder /app /app
RUN ls /app # now files are present

Root cause:

Misunderstanding that each stage has its own isolated filesystem.

#3 Copying secret keys into the final image, risking exposure.

Wrong approach:

FROM builder
COPY secret.key /app/secret.key
FROM alpine
COPY --from=builder /app /app

Correct approach:

Use Docker build secrets or environment variables at build time without copying them into the image.

Root cause:

Lack of awareness about secure secret handling in Docker builds.

Key Takeaways

Multi-stage builds let you write one Dockerfile with multiple steps to produce smaller, cleaner images.

Each build stage is isolated; only files you explicitly copy to the final stage are included in the final image.

Separating build and runtime environments reduces image size, improves security, and speeds up deployment.

Understanding Docker's layer caching with multi-stage builds helps optimize build speed and efficiency.

Advanced use of multi-stage builds includes handling secrets securely and building for multiple platforms.