Container filesystem is ephemeral in Docker - Deep Dive

Overview - Container filesystem is ephemeral

What is it?

A container filesystem is a temporary storage space created when a container starts. It holds all the files and changes made while the container runs. However, once the container stops or is removed, this filesystem and its changes disappear. This means any data saved inside the container is lost unless stored outside.

Why it matters

This concept exists because containers are designed to be lightweight and fast to start or stop. Without ephemeral filesystems, containers would keep growing in size and become harder to manage. Without this, developers would struggle to keep environments consistent and clean, leading to errors and wasted resources.

Where it fits

Before learning this, you should understand what containers are and how they run applications. After this, you can learn about persistent storage options like volumes and bind mounts that keep data safe beyond container life.

Mental Model

Core Idea

A container’s filesystem is like a temporary workspace that disappears when you leave, so anything you want to keep must be saved elsewhere.

Think of it like...

Imagine writing notes on a whiteboard during a meeting. When the meeting ends, the whiteboard is wiped clean. If you want to keep your notes, you must write them down on paper before leaving.

┌─────────────────────────────┐
│       Container Start       │
│  ┌───────────────────────┐  │
│  │ Temporary Filesystem   │  │
│  │ (Ephemeral Workspace)  │  │
│  └───────────────────────┘  │
│           Running           │
│  Changes saved here only    │
│                             │
│       Container Stop        │
│  ┌───────────────────────┐  │
│  │ Filesystem disappears  │  │
│  └───────────────────────┘  │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a container filesystem

Concept: Introduce the container filesystem as the storage area inside a running container.

When you start a container, Docker creates a special filesystem for it. This filesystem holds the operating system files, your application files, and any changes made while the container runs. It looks like a normal disk but exists only while the container is active.

Result

You get a fresh, isolated filesystem for each container that holds all necessary files for the container to work.

Understanding that containers have their own isolated filesystems helps explain why they behave like separate mini-computers.

2

Foundation

Ephemeral means temporary

Concept: Explain the meaning of ephemeral in the context of container filesystems.

Ephemeral means something that lasts only a short time. For containers, this means the filesystem exists only while the container runs. When the container stops or is deleted, the filesystem and all changes inside it vanish.

Result

Any files created or changed inside the container are lost after it stops unless saved outside.

Knowing that container filesystems are temporary prevents surprises when data disappears after container removal.

3

Intermediate

Why container filesystems reset on stop

🤔Before reading on: do you think container filesystems keep changes after stopping? Commit to yes or no.

Concept: Explain the design choice behind ephemeral filesystems for containers.

Containers are designed to be fast and consistent. Resetting the filesystem on stop means every time you start a container, it begins fresh. This avoids leftover files causing bugs or inconsistencies. It also keeps containers lightweight and easy to manage.

Result

Containers always start with a clean state, making deployments predictable and repeatable.

Understanding this design helps you appreciate why containers are reliable and easy to scale.

4

Intermediate

How data loss happens inside containers

🤔Before reading on: do you think data inside a container is safe after it stops? Commit to yes or no.

Concept: Show how data created inside a container is lost when the container stops or is removed.

If you create or modify files inside a container's filesystem, those changes live only inside that container. When you stop or delete the container, Docker deletes the container’s filesystem. This means all your changes vanish unless saved outside.

Result

Data created inside containers without external storage is temporary and lost on container removal.

Knowing this prevents accidental data loss and encourages using persistent storage.

5

Intermediate

Using volumes to keep data safe

Concept: Introduce Docker volumes as a way to store data outside the ephemeral container filesystem.

Docker volumes are special storage areas outside the container’s filesystem. You can attach volumes to containers so data saved there stays even if the container stops or is deleted. This is how you keep important data safe beyond container life.

Result

Data stored in volumes persists independently of container lifecycle.

Understanding volumes is key to managing data safely in containerized applications.

6

Advanced

Union filesystem layers in containers

🤔Before reading on: do you think container filesystems are a single disk or layered? Commit to your answer.

Concept: Explain how container filesystems use layered union filesystems to build the final view.

Containers use a layered filesystem where the base image is read-only, and a top writable layer stores changes. When you modify files, changes go to this top layer. When the container stops, this writable layer is discarded, making the filesystem ephemeral.

Result

The container filesystem is a combination of base image layers plus a temporary writable layer that disappears on container removal.

Knowing about layers explains why containers start fast and how changes are isolated.

7

Expert

Ephemeral filesystem impact on CI/CD pipelines

🤔Before reading on: do you think ephemeral filesystems help or hinder continuous integration? Commit to your answer.

Concept: Explore how ephemeral filesystems improve reliability and speed in automated build and test pipelines.

In CI/CD, containers start fresh for each build or test run. The ephemeral filesystem ensures no leftover data from previous runs affects results. This guarantees clean environments and reproducible builds. However, it requires careful use of volumes or artifact storage to keep important data.

Result

Ephemeral filesystems enable reliable, repeatable automation but require explicit data persistence strategies.

Understanding this helps design robust pipelines that avoid flaky tests and data pollution.

Under the Hood

Docker uses a layered union filesystem like OverlayFS. The container’s base image layers are read-only and stacked. When the container runs, Docker adds a top writable layer where all file changes happen. This writable layer lives only as long as the container does. When the container stops or is removed, Docker deletes this writable layer, making the filesystem ephemeral.

Why designed this way?

This design was chosen to keep containers lightweight, fast to start, and consistent. Using layers avoids copying the entire image for each container, saving space and time. Making the top layer ephemeral ensures containers start clean every time, preventing state-related bugs and simplifying management.

┌───────────────────────────────┐
│        Container Filesystem    │
│ ┌───────────────┐             │
│ │ Writable Layer│  ← Changes │
│ ├───────────────┤             │
│ │ Read-only     │             │
│ │ Image Layers  │             │
│ └───────────────┘             │
│                               │
│ Container runs → writable layer active
│ Container stops → writable layer deleted
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does data inside a container always persist after stopping? Commit yes or no.

Common Belief: Data saved inside a container stays there even after the container stops.

Reality: Data inside the container’s filesystem is lost when the container stops or is removed unless stored in volumes or external storage.

Why it matters: Assuming data persists can cause unexpected data loss and application failures.

Quick: Can you share data between containers by writing to their filesystems? Commit yes or no.

Common Belief: Containers can share data by writing to their own filesystems directly.

Reality: Each container has its own isolated ephemeral filesystem; data sharing requires volumes or network storage.

Why it matters: Misunderstanding this leads to failed communication and data inconsistency between containers.

Quick: Is the container filesystem a full copy of the image? Commit yes or no.

Common Belief: Each container has a full copy of the image on disk.

Reality: Containers share image layers and only add a small writable layer on top, saving space and time.

Why it matters: Knowing this helps optimize storage and understand container startup speed.

Quick: Does using volumes make the container filesystem non-ephemeral? Commit yes or no.

Common Belief: Attaching volumes changes the container filesystem to be permanent.

Reality: Volumes store data outside the container filesystem; the container filesystem remains ephemeral.

Why it matters: Confusing this can cause wrong assumptions about where data is stored and lost.

Expert Zone

1

The ephemeral writable layer can cause subtle bugs if applications expect files to persist between container restarts without volumes.

2

Some container runtimes optimize ephemeral filesystems differently, affecting performance and storage usage.

3

Ephemeral filesystems simplify rollback and scaling but require careful design for stateful applications.

When NOT to use

Ephemeral filesystems are not suitable when you need to keep data permanently, such as databases or user uploads. In those cases, use Docker volumes, bind mounts, or external storage solutions like cloud storage or databases.

Production Patterns

In production, ephemeral filesystems are used for stateless services like web servers or microservices. Persistent data is stored in volumes or external databases. CI/CD pipelines use ephemeral containers to ensure clean test environments. Stateful services run with attached volumes or specialized storage.

Connections

Immutable Infrastructure

Ephemeral container filesystems embody the principle of immutable infrastructure by ensuring containers start fresh every time.

Understanding ephemeral filesystems clarifies how immutable infrastructure avoids configuration drift and improves reliability.

Functional Programming

Both ephemeral container filesystems and functional programming emphasize statelessness and avoiding side effects.

Knowing this connection helps appreciate how stateless design leads to more predictable and testable systems.

Temporary Workspace in Software Development

Ephemeral filesystems are like temporary workspaces developers use to try changes without affecting the main codebase.

Recognizing this similarity helps understand why containers encourage experimentation without risk.

Common Pitfalls

#1 Expecting data inside the container to persist after restart

Wrong approach:

docker run -it ubuntu bash
# echo 'hello' > /data.txt
exit
# docker start
# cat /data.txt

Correct approach:

docker volume create mydata
# docker run -v mydata:/data -it ubuntu bash
# echo 'hello' > /data/data.txt
exit
# docker run -v mydata:/data -it ubuntu bash
# cat /data/data.txt

Root cause:

Misunderstanding that container filesystems are temporary and do not save data after container stops.

#2 Trying to share data by copying files between container filesystems

Wrong approach:

docker cp container1:/file.txt container2:/file.txt

Correct approach:

docker volume create sharedvol
# docker run -v sharedvol:/data container1
# docker run -v sharedvol:/data container2

Root cause:

Not realizing container filesystems are isolated and ephemeral, so direct file sharing requires volumes.

#3 Using container filesystem for database storage

Wrong approach:

docker run -it postgres
# createdb mydb
# exit
# docker rm -f
# docker run -it postgres
# psql mydb

Correct approach:

docker volume create pgdata
# docker run -v pgdata:/var/lib/postgresql/data -it postgres
# createdb mydb
# exit
# docker run -v pgdata:/var/lib/postgresql/data -it postgres
# psql mydb

Root cause:

Ignoring ephemeral nature of container filesystems causes data loss for stateful services.

Key Takeaways

Container filesystems are temporary and disappear when the container stops or is removed.

This ephemeral nature ensures containers start fresh, making them reliable and easy to manage.

To keep data beyond container life, use Docker volumes or external storage.

Understanding ephemeral filesystems helps prevent accidental data loss and design better containerized applications.

Ephemeral filesystems support modern DevOps practices like immutable infrastructure and clean CI/CD pipelines.