Environment variables in Compose in Docker - Deep Dive

Overview - Environment variables in Compose

What is it?

Environment variables in Docker Compose are key-value pairs used to configure containers at runtime. They allow you to pass settings like passwords, ports, or feature flags into your containers without hardcoding them. This makes your Docker Compose files flexible and reusable across different environments. You define them in the Compose file or in separate files and they get injected into the container's environment.

Why it matters

Without environment variables, you would have to change your container images or Compose files for every environment, risking mistakes and losing flexibility. Environment variables let you keep your configuration separate from your code, making deployments safer and easier. This separation is crucial for managing different setups like development, testing, and production without rewriting your files.

Where it fits

Before learning environment variables in Compose, you should understand basic Docker concepts like containers and images, and how Docker Compose orchestrates multiple containers. After this, you can learn about secrets management, configuration files, and advanced Compose features like overrides and profiles to handle complex deployments.

Mental Model

Core Idea

Environment variables in Compose act like adjustable dials that configure your containers without changing their internal setup.

Think of it like...

It's like setting the temperature on a thermostat instead of rebuilding the heater every time you want it warmer or cooler.

┌───────────────────────────────┐
│ Docker Compose File            │
│ ┌───────────────────────────┐ │
│ │ environment:              │ │
│ │   - VAR1=value1           │ │
│ │   - VAR2=value2           │ │
│ └───────────────────────────┘ │
│               │               │
│               ▼               │
│ ┌───────────────────────────┐ │
│ │ Container Environment     │ │
│ │ ┌───────────────────────┐ │ │
│ │ │ VAR1=value1           │ │ │
│ │ │ VAR2=value2           │ │ │
│ │ └───────────────────────┘ │ │
│ └───────────────────────────┘ │
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What are Environment Variables

Concept: Introduce environment variables as simple key-value pairs used to configure software behavior.

Environment variables are like labels with values that programs read to adjust how they run. For example, a program might check an environment variable called PORT to decide which network port to listen on. They are set outside the program, so you can change settings without touching the program code.

Result

You understand that environment variables are external settings that programs can use to change behavior dynamically.

Knowing that environment variables separate configuration from code helps you keep your software flexible and easier to manage.

2

Foundation

Basics of Docker Compose Files

Concept: Explain the structure of a Docker Compose file and how it defines multiple containers.

Docker Compose files are YAML files that describe how to run one or more containers together. Each container is a service with settings like the image to use, ports to expose, and environment variables to pass. This file lets you start all containers with one command.

Result

You can read and write simple Compose files that start containers with basic settings.

Understanding Compose files is essential because environment variables are configured inside these files to customize containers.

3

Intermediate

Setting Environment Variables in Compose

🤔Before reading on: do you think environment variables in Compose can be set only inline or also from files? Commit to your answer.

Concept: Show how to define environment variables directly in the Compose file and from external files.

You can set environment variables in Compose in two main ways: 1. Inline in the service definition: services: app: environment: - DEBUG=true - PORT=8080 2. Using an external .env file: Create a file named .env with: DEBUG=true PORT=8080 Compose automatically loads this file and injects variables. You can also specify env_file in the Compose file to load other files.

Result

You can configure containers flexibly by choosing where to store environment variables.

Knowing multiple ways to set environment variables lets you organize configuration for different environments and keep secrets out of your Compose files.

4

Intermediate

Variable Substitution in Compose Files

🤔Before reading on: do you think Compose variables can only come from .env files or also from your shell environment? Commit to your answer.

Concept: Explain how Compose supports variable substitution from the shell environment or .env files inside the Compose YAML.

Compose allows you to write variables inside your Compose file like this: services: app: image: "myapp:${TAG}" Here, ${TAG} is replaced by the value of TAG from your shell environment or the .env file. This lets you reuse the same Compose file but change images, ports, or other settings by changing environment variables outside the file.

Result

You can write Compose files that adapt dynamically to your environment without editing the file itself.

Understanding variable substitution unlocks powerful reuse and customization of Compose files across different deployment scenarios.

5

Intermediate

Overriding Environment Variables per Environment

🤔Before reading on: do you think Compose supports multiple .env files or only one? Commit to your answer.

Concept: Teach how to override environment variables for different environments using multiple .env files or Compose overrides.

By default, Compose loads a single .env file. To manage different environments (like dev, test, prod), you can: - Use multiple Compose files with overrides: docker-compose.yml docker-compose.prod.yml - Pass environment variables from the shell before running Compose: TAG=prod docker-compose up - Use env_file to load different files per service. This lets you keep environment-specific settings separate and avoid mistakes.

Result

You can deploy the same Compose setup with different configurations safely and easily.

Knowing how to override environment variables prevents configuration drift and errors between environments.

6

Advanced

Security Considerations for Environment Variables

🤔Before reading on: do you think environment variables are secure for storing secrets? Commit to your answer.

Concept: Discuss the risks of storing sensitive data in environment variables and better alternatives.

Environment variables are visible to anyone who can inspect the container or host process environment. This means secrets like passwords or API keys stored as environment variables can leak. Better approaches include: - Using Docker secrets (supported in Docker Swarm) - Using external secret managers (HashiCorp Vault, AWS Secrets Manager) - Mounting secrets as files inside containers Avoid committing secrets to version control or .env files shared publicly.

Result

You understand the limits of environment variables for secrets and how to protect sensitive data.

Knowing environment variables' security limits helps you design safer deployments and avoid costly leaks.

7

Expert

How Compose Merges Environment Variables Internally

🤔Before reading on: do you think Compose merges environment variables from multiple sources by overriding or combining? Commit to your answer.

Concept: Reveal the internal merging order Compose uses when environment variables come from multiple places.

Docker Compose merges environment variables from these sources in this order: 1. Variables from the shell environment where Compose runs 2. Variables from the .env file in the Compose directory 3. Variables from env_file entries 4. Variables set in the Compose file under environment: Variables defined later override earlier ones. This means environment variables defined inline in the Compose file override variables from env_file entries, which override .env file variables, which override shell environment variables. Understanding this helps debug why a variable has a certain value.

Result

You can predict exactly which environment variable value will be used when multiple sources exist.

Knowing Compose's merge order prevents confusion and bugs caused by unexpected variable values in complex setups.

Under the Hood

Docker Compose reads environment variables from multiple sources and combines them before starting containers. It parses the Compose YAML, loads .env files, reads env_file entries, and merges these with the shell environment variables. Then it injects the final set into the container's environment at runtime. Containers access these variables as part of their process environment, just like any program running on a computer.

Why designed this way?

This design balances flexibility and simplicity. By supporting multiple sources and a clear override order, Compose lets users organize configuration cleanly for different environments and use cases. It avoids hardcoding values and supports dynamic substitution, which was not possible with static Dockerfiles alone. Alternatives like embedding config inside images were less flexible and required rebuilding images for every change.

┌───────────────┐
│ Shell Env     │
│ (user vars)   │
└──────┬────────┘
       │
┌──────▼────────┐
│ .env File     │
│ (default vars)│
└──────┬────────┘
       │
┌──────▼────────┐
│ env_file(s)   │
│ (service vars)│
└──────┬────────┘
       │
┌──────▼────────┐
│ Compose File  │
│ environment:  │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Merged Env    │
│ for Container │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: do you think environment variables set in the Compose file always override shell environment variables? Commit to yes or no.

Common Belief: Environment variables defined inside the Compose file always take precedence over shell environment variables.

Reality: Inline environment variables in the Compose file override variables from env_file entries, which override .env file variables, which override shell environment variables.

Why it matters: Assuming Compose file variables override shell variables can cause unexpected behavior when debugging why a container uses a different value.

Quick: do you think environment variables are secure enough to store passwords in Compose files? Commit to yes or no.

Common Belief: It's safe to store passwords and secrets as environment variables in Compose files or .env files.

Reality: Environment variables are visible to anyone with access to the container or host, making them insecure for secrets.

Why it matters: Storing secrets in environment variables risks accidental leaks, leading to security breaches.

Quick: do you think Compose automatically reloads environment variables if you change the .env file while containers run? Commit to yes or no.

Common Belief: Changing the .env file while containers are running updates the environment variables inside those containers automatically.

Reality: Environment variables are set only when containers start; changing .env files later has no effect until containers restart.

Why it matters: Expecting live updates can cause confusion and bugs when changes don't apply as expected.

Quick: do you think environment variables set with env_file override those set inline in the Compose file? Commit to yes or no.

Common Belief: Variables from env_file always override inline environment variables in the Compose file.

Reality: Inline environment variables in the Compose file override variables from env_file entries.

Why it matters: Misunderstanding override order can cause configuration errors and hard-to-find bugs.

Expert Zone

1

Compose does not support variable expansion inside env_file files, only in the Compose YAML, which can surprise users expecting full substitution.

2

When using multiple Compose files with overrides, environment variables merge differently depending on the order of files passed to the command.

3

Docker Compose v2 introduced stricter parsing of environment variables and .env files, rejecting invalid lines, which can break older setups silently.

When NOT to use

Avoid using environment variables for highly sensitive secrets in production; instead, use Docker secrets or external secret managers. Also, for very complex configuration, consider dedicated configuration management tools or service meshes that provide dynamic config injection.

Production Patterns

In production, teams often use multiple Compose files with environment-specific overrides and external env_files stored securely. They combine Compose with CI/CD pipelines that inject environment variables dynamically. Secrets are managed outside Compose using Docker secrets or vault integrations, and environment variables are used mainly for non-sensitive configuration.

Connections

12-Factor App Configuration

Builds-on

Understanding environment variables in Compose helps implement the 12-Factor App principle of storing config in the environment, enabling portable and scalable apps.

Unix Shell Environment Variables

Same pattern

Knowing how environment variables work in Unix shells clarifies how Compose inherits and overrides variables, since Compose relies on the host shell environment.

Thermostat Control Systems (Engineering)

Analogous control mechanism

Just like a thermostat adjusts temperature without rebuilding the heating system, environment variables adjust container behavior without changing images, showing a universal control pattern.

Common Pitfalls

#1 Storing secrets directly in Compose files or .env files committed to version control.

Wrong approach:

services:
db:
environment:
- POSTGRES_PASSWORD=mysecretpassword

# .env file
POSTGRES_PASSWORD=mysecretpassword

Correct approach:

Use Docker secrets or external secret managers instead:

services:
db:
secrets:
- db_password

secrets:
db_password:
file: ./db_password.txt

Root cause:

Misunderstanding that environment variables are secure storage leads to accidental secret exposure.

#2 Expecting environment variable changes in .env files to update running containers automatically.

Wrong approach:

Edit .env file and expect containers to pick up changes without restart.

Correct approach:

After changing .env, run:
docker-compose down
and then
docker-compose up -d

To restart containers with new variables.

Root cause:

Not knowing that environment variables are set only at container start causes confusion about live updates.

#3 Defining environment variables with incorrect YAML syntax causing Compose parsing errors.

Wrong approach:

services:
app:
environment: DEBUG=true PORT=8080

Correct approach:

services:
app:
environment:
- DEBUG=true
- PORT=8080

Root cause:

Lack of understanding YAML list syntax leads to invalid Compose files.

Key Takeaways

Environment variables in Docker Compose let you configure containers flexibly without changing images or code.

Compose supports multiple ways to set environment variables: inline, env_file, .env files, and shell environment, with a clear override order.

Environment variables are not secure for secrets; use dedicated secret management tools for sensitive data.

Variable substitution in Compose files enables dynamic, reusable configurations across different environments.

Understanding how Compose merges and overrides environment variables helps prevent bugs and unexpected behavior.