Two questions come up in almost every Docker workshop I run. Why does adding RUN rm -rf /tmp/build at the end of a Dockerfile not make the image smaller? And why does a database container become slow the first time it modifies a large file that came with the image? Both answers come from the same place: how the container's root filesystem is assembled from layers.

This post explains it from the bottom up, using overlayfs, the union filesystem behind Docker's default overlay2 storage driver and containerd's overlay snapshotter. You can follow along with a few mount commands on any modern Linux machine, no Docker needed.

Images are stacks of read-only layers

An image is an ordered list of layers. Each layer is a set of filesystem changes: files added, modified or deleted compared to the layer below. A Dockerfile instruction that changes the filesystem (RUN, COPY, ADD) produces a new layer.

Layers are content-addressed and shared. If ten images use the same base, its layers are stored once on disk and downloaded once.

When a container starts, the runtime needs to present all those layers as one normal-looking directory tree, plus a writable place for whatever the container changes. That is the union filesystem's job.

overlayfs in four directories

overlayfs combines directories into one view:

  • lowerdir: one or more read-only directories, the image layers. If several, the leftmost has the highest priority.
  • upperdir: a writable directory where changes go, the container layer.
  • workdir: an empty directory overlayfs uses internally for atomic operations; it must be on the same filesystem as upperdir.
  • merged: the mount point that shows the combined result, which becomes the container's root.

Try it:

mkdir -p /tmp/ovl/{base,app,upper,work,merged}
echo "from base"  > /tmp/ovl/base/os-release
echo "v1"         > /tmp/ovl/base/config.txt
echo "app code"   > /tmp/ovl/app/server.js
echo "v2"         > /tmp/ovl/app/config.txt     # same name, higher layer wins

sudo mount -t overlay overlay \
  -o lowerdir=/tmp/ovl/app:/tmp/ovl/base,upperdir=/tmp/ovl/upper,workdir=/tmp/ovl/work \
  /tmp/ovl/merged

ls /tmp/ovl/merged            # config.txt  os-release  server.js
cat /tmp/ovl/merged/config.txt  # v2

The merged view contains files from both lower layers, and where both contain config.txt, the higher-priority lower layer (app, listed first) wins.

Reading is cheap

Reading a file in the merged view reads it directly from whichever layer holds it. There is no copying, which is why starting a container is fast regardless of image size.

Writing triggers copy-up

The lower layers are read-only. When a process modifies a file that exists only in a lower layer, overlayfs first copies the entire file to upperdir, then applies the change there. This is called copy-up.

echo "changed" | sudo tee -a /tmp/ovl/merged/os-release
ls -l /tmp/ovl/upper          # os-release now exists in upper
cat /tmp/ovl/base/os-release  # unchanged: "from base"

For a small config file this is invisible. For a 2 GB database file shipped inside the image, the first write, even of one byte, copies 2 GB. That is the slow first write people notice, and it also means the container's writable layer suddenly uses 2 GB of disk.

After copy-up, further reads and writes use the upper copy at normal speed.

Deleting creates a whiteout

Files in lower layers cannot be removed. Deleting one in the merged view creates a whiteout in upperdir: a special character device with the same name, which tells overlayfs to hide the lower file.

sudo rm /tmp/ovl/merged/server.js
ls -l /tmp/ovl/upper
# c--------- 1 root root 0, 0 ... server.js     <- whiteout
ls /tmp/ovl/app               # server.js is still there

Deleting a directory and recreating it uses an "opaque" directory marker, which hides everything below.

Why rm in a later layer does not shrink an image

Now the Dockerfile puzzle is easy. Each instruction is a layer. If one RUN downloads a 300 MB toolchain and a later RUN deletes it, the later layer contains only whiteouts. The 300 MB is still in the earlier layer, which is part of the image and is downloaded by everyone who pulls it.

# image still contains the toolchain
RUN curl -fsSLo /tmp/sdk.tar.gz https://downloads.example.com/sdk.tar.gz && tar -xzf /tmp/sdk.tar.gz -C /opt
RUN /opt/sdk/bin/build && rm -rf /opt/sdk /tmp/sdk.tar.gz

Two fixes:

  • Download, use and delete in the same RUN, so the files never exist in any committed layer.
  • Use a multi-stage build, and copy only the result into the final stage, so build tools never reach the image you ship.

The same applies to secrets: a file copied in one layer and deleted in the next is still recoverable from the image. Use build secrets (RUN --mount=type=secret) instead.

What this means for data

Container writable layers are not designed for heavy write workloads:

  • Copy-up makes the first write to large image files expensive.
  • Writes go through overlayfs, which adds overhead compared with writing to a plain filesystem.
  • The writable layer disappears with the container.

That is why databases, queues and anything with significant writes should use volumes or bind mounts. A volume is a plain directory on the host filesystem mounted into the container, bypassing overlayfs entirely.

docker run -d --name pg -v pgdata:/var/lib/postgresql/data postgres:17

Official database images declare their data directory as a volume for exactly this reason.

Looking at a real container

On a Docker host, you can see the directories for a running container:

docker inspect --format '{{json .GraphDriver.Data}}' pg | jq

{
  "LowerDir": "/var/lib/docker/overlay2/5c9e.../diff:/var/lib/docker/overlay2/a13f.../diff:...",
  "MergedDir": "/var/lib/docker/overlay2/7d42.../merged",
  "UpperDir": "/var/lib/docker/overlay2/7d42.../diff",
  "WorkDir": "/var/lib/docker/overlay2/7d42.../work"
}

docker diff pg lists files changed in the upper layer: A added, C changed, D deleted. It is a quick way to see what a container writes outside its volumes, which often reveals logs or caches that should go elsewhere.

Limits and quirks worth knowing

  • Inode and file handle behaviour. Some applications compare inode numbers; after copy-up a file can appear with a different inode on some kernels and configurations.
  • Hard links across layers are copied up individually.
  • Many layers slow down lookups slightly; images with dozens of tiny layers gain nothing.
  • Host filesystem. The backing filesystem for /var/lib/docker must support overlayfs; ext4 and XFS with ftype=1 do.

Key takeaways

  • An image is a stack of read-only layers; a container adds one writable layer on top.
  • overlayfs merges lowerdir layers and an upperdir into one view; reads are direct.
  • Modifying a lower-layer file copies the whole file up first; deleting creates a whiteout.
  • Deleting files in a later Dockerfile layer never reduces image size; clean up in the same step or use multi-stage builds.
  • Put databases and write-heavy data on volumes, not in the container layer.

References