Container filesystem is ephemeral in Docker - Deep Dive

Overview - Container filesystem is ephemeral

What is it?

A container filesystem is a temporary storage space created when a container starts. It holds all the files and changes made while the container runs. However, once the container stops or is removed, this filesystem and its changes disappear. This means any data saved inside the container is lost unless stored outside.

Why it matters

This concept exists because containers are designed to be lightweight and fast to start or stop. Without ephemeral filesystems, containers would keep growing in size and become harder to manage. Without this, developers would struggle to keep environments consistent and clean, leading to errors and wasted resources.

Where it fits

Before learning this, you should understand what containers are and how they run applications. After this, you can learn about persistent storage options like volumes and bind mounts that keep data safe beyond container life.

Mental Model

Core Idea

A container’s filesystem is like a temporary workspace that disappears when you leave, so anything you want to keep must be saved elsewhere.

Think of it like...

Imagine writing notes on a whiteboard during a meeting. When the meeting ends, the whiteboard is wiped clean. If you want to keep your notes, you must write them down on paper before leaving.

┌─────────────────────────────┐
│       Container Start       │
│  ┌───────────────────────┐  │
│  │ Temporary Filesystem   │  │
│  │ (Ephemeral Workspace)  │  │
│  └───────────────────────┘  │
│           Running           │
│  Changes saved here only    │
│                             │
│       Container Stop        │
│  ┌───────────────────────┐  │
│  │ Filesystem disappears  │  │
│  └───────────────────────┘  │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a container filesystem

Concept: Introduce the container filesystem as the storage area inside a running container.

When you start a container, Docker creates a special filesystem for it. This filesystem holds the operating system files, your application files, and any changes made while the container runs. It looks like a normal disk but exists only while the container is active.

Result

You get a fresh, isolated filesystem for each container that holds all necessary files for the container to work.

Understanding that containers have their own isolated filesystems helps explain why they behave like separate mini-computers.

2

Foundation

Ephemeral means temporary

Concept: Explain the meaning of ephemeral in the context of container filesystems.

Ephemeral means something that lasts only a short time. For containers, this means the filesystem exists only while the container runs. When the container stops or is deleted, the filesystem and all changes inside it vanish.

Result

Any files created or changed inside the container are lost after it stops unless saved outside.

Knowing that container filesystems are temporary prevents surprises when data disappears after container removal.

3

Intermediate

Why container filesystems reset on stop

🤔Before reading on: do you think container filesystems keep changes after stopping? Commit to yes or no.

Concept: Explain the design choice behind ephemeral filesystems for containers.

Containers are designed to be fast and consistent. Resetting the filesystem on stop means every time you start a container, it begins fresh. This avoids leftover files causing bugs or inconsistencies. It also keeps containers lightweight and easy to manage.

Result

Containers always start with a clean state, making deployments predictable and repeatable.

Understanding this design helps you appreciate why containers are reliable and easy to scale.

4

Intermediate

How data loss happens inside containers

🤔Before reading on: do you think data inside a container is safe after it stops? Commit to yes or no.

Concept: Show how data created inside a container is lost when the container stops or is removed.

If you create or modify files inside a container's filesystem, those changes live only inside that container. When you stop or delete the container, Docker deletes the container’s filesystem. This means all your changes vanish unless saved outside.

Result

Data created inside containers without external storage is temporary and lost on container removal.

Knowing this prevents accidental data loss and encourages using persistent storage.

5

Intermediate

Using volumes to keep data safe

Concept: Introduce Docker volumes as a way to store data outside the ephemeral container filesystem.

Docker volumes are special storage areas outside the container’s filesystem. You can attach volumes to containers so data saved there stays even if the container stops or is deleted. This is how you keep important data safe beyond container life.

Result

Data stored in volumes persists independently of container lifecycle.

Understanding volumes is key to managing data safely in containerized applications.

6

Advanced

Union filesystem layers in containers

🤔Before reading on: do you think container filesystems are a single disk or layered? Commit to your answer.

Concept: Explain how container filesystems use layered union filesystems to build the final view.

Containers use a layered filesystem where the base image is read-only, and a top writable layer stores changes. When you modify files, changes go to this top layer. When the container stops, this writable layer is discarded, making the filesystem ephemeral.

Result

The container filesystem is a combination of base image layers plus a temporary writable layer that disappears on container removal.

Knowing about layers explains why containers start fast and how changes are isolated.

7

Expert

Ephemeral filesystem impact on CI/CD pipelines

🤔Before reading on: do you think ephemeral filesystems help or hinder continuous integration? Commit to your answer.

Concept: Explore how ephemeral filesystems improve reliability and speed in automated build and test pipelines.

In CI/CD, containers start fresh for each build or test run. The ephemeral filesystem ensures no leftover data from previous runs affects results. This guarantees clean environments and reproducible builds. However, it requires careful use of volumes or artifact storage to keep important data.

Result

Ephemeral filesystems enable reliable, repeatable automation but require explicit data persistence strategies.

Understanding this helps design robust pipelines that avoid flaky tests and data pollution.

Under the Hood

Docker uses a layered union filesystem like OverlayFS. The container’s base image layers are read-only and stacked. When the container runs, Docker adds a top writable layer where all file changes happen. This writable layer lives only as long as the container does. When the container stops or is removed, Docker deletes this writable layer, making the filesystem ephemeral.

Why designed this way?

This design was chosen to keep containers lightweight, fast to start, and consistent. Using layers avoids copying the entire image for each container, saving space and time. Making the top layer ephemeral ensures containers start clean every time, preventing state-related bugs and simplifying management.

┌───────────────────────────────┐
│        Container Filesystem    │
│ ┌───────────────┐             │
│ │ Writable Layer│  ← Changes │
│ ├───────────────┤             │
│ │ Read-only     │             │
│ │ Image Layers  │             │
│ └───────────────┘             │
│                               │
│ Container runs → writable layer active
│ Container stops → writable layer deleted
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does data inside a container always persist after stopping? Commit yes or no.

Common Belief: Data saved inside a container stays there even after the container stops.

Reality: Data inside the container’s filesystem is lost when the container stops or is removed unless stored in volumes or external storage.

Why it matters: Assuming data persists can cause unexpected data loss and application failures.

Quick: Can you share data between containers by writing to their filesystems? Commit yes or no.

Common Belief: Containers can share data by writing to their own filesystems directly.

Reality: Each container has its own isolated ephemeral filesystem; data sharing requires volumes or network storage.

Why it matters: Misunderstanding this leads to failed communication and data inconsistency between containers.

Quick: Is the container filesystem a full copy of the image? Commit yes or no.

Common Belief: Each container has a full copy of the image on disk.

Reality: Containers share image layers and only add a small writable layer on top, saving space and time.

Why it matters: Knowing this helps optimize storage and understand container startup speed.

Quick: Does using volumes make the container filesystem non-ephemeral? Commit yes or no.

Common Belief: Attaching volumes changes the container filesystem to be permanent.

Reality: Volumes store data outside the container filesystem; the container filesystem remains ephemeral.

Why it matters: Confusing this can cause wrong assumptions about where data is stored and lost.

Expert Zone

1

The ephemeral writable layer can cause subtle bugs if applications expect files to persist between container restarts without volumes.

2

Some container runtimes optimize ephemeral filesystems differently, affecting performance and storage usage.

3

Ephemeral filesystems simplify rollback and scaling but require careful design for stateful applications.

When NOT to use

Ephemeral filesystems are not suitable when you need to keep data permanently, such as databases or user uploads. In those cases, use Docker volumes, bind mounts, or external storage solutions like cloud storage or databases.

Production Patterns

In production, ephemeral filesystems are used for stateless services like web servers or microservices. Persistent data is stored in volumes or external databases. CI/CD pipelines use ephemeral containers to ensure clean test environments. Stateful services run with attached volumes or specialized storage.

Connections

Immutable Infrastructure

Ephemeral container filesystems embody the principle of immutable infrastructure by ensuring containers start fresh every time.

Understanding ephemeral filesystems clarifies how immutable infrastructure avoids configuration drift and improves reliability.

Functional Programming

Both ephemeral container filesystems and functional programming emphasize statelessness and avoiding side effects.

Knowing this connection helps appreciate how stateless design leads to more predictable and testable systems.

Temporary Workspace in Software Development

Ephemeral filesystems are like temporary workspaces developers use to try changes without affecting the main codebase.

Recognizing this similarity helps understand why containers encourage experimentation without risk.

Common Pitfalls

#1 Expecting data inside the container to persist after restart

Wrong approach:

docker run -it ubuntu bash
# echo 'hello' > /data.txt
exit
# docker start
# cat /data.txt

Correct approach:

docker volume create mydata
# docker run -v mydata:/data -it ubuntu bash
# echo 'hello' > /data/data.txt
exit
# docker run -v mydata:/data -it ubuntu bash
# cat /data/data.txt

Root cause:

Misunderstanding that container filesystems are temporary and do not save data after container stops.

#2 Trying to share data by copying files between container filesystems

Wrong approach:

docker cp container1:/file.txt container2:/file.txt

Correct approach:

docker volume create sharedvol
# docker run -v sharedvol:/data container1
# docker run -v sharedvol:/data container2

Root cause:

Not realizing container filesystems are isolated and ephemeral, so direct file sharing requires volumes.

#3 Using container filesystem for database storage

Wrong approach:

docker run -it postgres
# createdb mydb
# exit
# docker rm -f
# docker run -it postgres
# psql mydb

Correct approach:

docker volume create pgdata
# docker run -v pgdata:/var/lib/postgresql/data -it postgres
# createdb mydb
# exit
# docker run -v pgdata:/var/lib/postgresql/data -it postgres
# psql mydb

Root cause:

Ignoring ephemeral nature of container filesystems causes data loss for stateful services.

Key Takeaways

Container filesystems are temporary and disappear when the container stops or is removed.

This ephemeral nature ensures containers start fresh, making them reliable and easy to manage.

To keep data beyond container life, use Docker volumes or external storage.

Understanding ephemeral filesystems helps prevent accidental data loss and design better containerized applications.

Ephemeral filesystems support modern DevOps practices like immutable infrastructure and clean CI/CD pipelines.

Practice

1. What happens to files created inside a Docker container when the container is stopped and restarted?

easy

A. The files are deleted because the container filesystem is ephemeral.

B. The files are saved permanently inside the container.

C. The files are automatically copied to the host machine.

D. The files are backed up to Docker Hub.

Solution

1. Step 1: Understand container filesystem behavior

   Docker containers have an ephemeral filesystem, meaning changes inside the container do not persist after restart.

2. Step 2: Apply this to file creation

   Files created inside the container are lost when the container stops or restarts unless saved externally.

3. Final Answer:

   The files are deleted because the container filesystem is ephemeral. -> Option A

4. Quick Check:

   Container filesystem ephemeral = files lost on restart [OK]

Hint: Remember: container files reset on restart unless saved outside [OK]

Common Mistakes:

• Thinking files inside container persist after restart
• Assuming Docker automatically saves files externally
• Confusing container filesystem with host filesystem

2. Which Docker command correctly mounts a host directory /data to a container directory /app/data to persist data?

easy

A. docker run --env /data:/app/data myimage

B. docker run -p /data:/app/data myimage

C. docker run --link /data:/app/data myimage

D. docker run -v /data:/app/data myimage

Solution

1. Step 1: Identify volume mount syntax

   The correct syntax to mount a host directory is using -v host_path:container_path.

2. Step 2: Match the correct option

   docker run -v /data:/app/data myimage uses -v /data:/app/data, which is the correct way to mount the host directory.

3. Final Answer:

   docker run -v /data:/app/data myimage -> Option D

4. Quick Check:

   Volume mount uses -v host:container [OK]

Hint: Use -v host_path:container_path to mount volumes [OK]

Common Mistakes:

• Using -p instead of -v for volumes
• Confusing --link or --env with volume mounts
• Forgetting to specify both host and container paths

3. Consider this Dockerfile snippet:

FROM alpine
RUN echo "Hello" > /greeting.txt
CMD cat /greeting.txt

If you run this container, then modify /greeting.txt inside the running container to "Hi", then restart the container, what will cat /greeting.txt output?

medium

A. Hi

B. Hello

C. File not found error

D. Empty output

Solution

1. Step 1: Understand Dockerfile build vs container runtime

   The file /greeting.txt is created during image build with content "Hello".

2. Step 2: Recognize ephemeral container filesystem

   Changes made inside a running container (changing to "Hi") are lost after restart because the container filesystem resets.

3. Final Answer:

   Hello -> Option B

4. Quick Check:

   Container restart resets filesystem, original file restored [OK]

Hint: Changes inside container lost on restart; image files stay [OK]

Common Mistakes:

• Assuming changes inside container persist after restart
• Confusing image build files with runtime changes
• Expecting file to be missing after restart

4. You ran a container and created a file inside it. After restarting, the file disappeared. You want to keep the file persistent. Which of these fixes the problem?

medium

A. Restart the container with --restart=always flag.

B. Increase container memory limit.

C. Use a Docker volume to store the file outside the container.

D. Run the container with --rm option.

Solution

1. Step 1: Identify cause of file loss

   Files inside container are lost on restart because filesystem is ephemeral.

2. Step 2: Choose method to persist data

   Using a Docker volume stores data outside container, preserving files across restarts.

3. Final Answer:

   Use a Docker volume to store the file outside the container. -> Option C

4. Quick Check:

   Volumes keep data persistent beyond container life [OK]

Hint: Persist files with volumes, not restart flags or memory [OK]

Common Mistakes:

• Thinking restart flags keep files
• Confusing memory limits with filesystem persistence
• Using --rm removes container on exit, losing files

5. You want to run a container that writes logs to /var/log/app.log inside the container. You want these logs to persist even if the container is removed or recreated. Which Docker run command achieves this?

hard

A. docker run -v /host/logs/app.log:/var/log/app.log myapp

B. docker run --rm myapp

C. docker run -v /host/logs:/var/log/app.log myapp

D. docker run -v /host/logs/app.log:/var/log myapp

Solution

1. Step 1: Understand volume mount for specific file

   To persist a single log file, mount the host file to the container file path.

2. Step 2: Analyze options

   docker run -v /host/logs/app.log:/var/log/app.log myapp mounts /host/logs/app.log to /var/log/app.log inside container, persisting logs exactly.

3. Step 3: Why other options fail

   docker run -v /host/logs/app.log:/var/log myapp tries to mount file to directory path (invalid). docker run -v /host/logs:/var/log/app.log myapp tries to mount directory to file path (invalid). docker run --rm myapp removes container on exit, losing logs.

4. Final Answer:

   docker run -v /host/logs/app.log:/var/log/app.log myapp -> Option A

5. Quick Check:

   Mount host file to container file for persistent logs [OK]

Hint: Mount host file to container file path to persist logs [OK]

Common Mistakes:

• Mounting directory to file path (invalid)
• Using --rm which deletes container and files
• Mounting directory when only file persistence needed