On this page
Container image layering and caching
Optimizing container builds by structuring Dockerfiles to maximize cache hits, reduce build times, and minimize the final image footprint.
Reference Card
- Origin/Prior Art: Union File Systems (UnionFS), Docker build process.
- Related Practices: Immutable builds, Continuous Integration, Multi-stage builds.
- Key Concepts: Union filesystem, Cache invalidation, Layer squashing, Base images.
- Primary Failure Modes: Massive image bloat, Cache misses, Security vulnerabilities in base layers.
Introduction
Container images are not monolithic blobs of data; they are constructed from a stack of read-only layers. Each instruction in a container build file (like a Dockerfile) - such as RUN, COPY, or ADD - creates a new layer that sits on top of the previous one.
Understanding how the build engine caches these layers is critical. A poorly written build file results in massive, multi-gigabyte images that take twenty minutes to build and minutes to pull across the network. A properly structured file produces a tiny, secure image that builds in seconds by reusing cached layers.
Mental Model
Think of building a container like painting an oil painting on multiple sheets of transparent glass, stacked on top of each other.
The first sheet (the base image) contains the background (the operating system). The second sheet contains the trees (system dependencies). The third sheet contains the people (the application code). If you look down through the stack, you see the complete picture.
If you want to change the people (update application code), you only need to wipe and repaint the top sheet of glass. You can reuse the background and the trees instantly. However, if you want to change the background (update the OS), you have to throw away all the sheets above it and repaint everything from scratch.
Operational Pattern
The build engine uses a deterministic cache. When it encounters an instruction, it checks if it has executed that exact instruction on top of that exact parent layer before. If it has, it skips the execution and reuses the cached layer.
Crucially, if a layer’s cache is invalidated, all subsequent layers are automatically invalidated and must be rebuilt.
To maximize caching, operators order instructions from least frequently changed to most frequently changed.
- Base OS:
FROM alpine:3.18(Changes rarely) - System Packages:
RUN apk add --no-cache curl(Changes infrequently) - Application Dependencies:
COPY package.json .followed byRUN npm install(Changes occasionally) - Application Code:
COPY src/ ./src/(Changes constantly, with every commit)
Failure Modes
Accidental Cache Invalidation
The most common mistake is copying the entire repository into the container before installing dependencies.
COPY . .
RUN npm installBecause the COPY . . instruction includes the constantly changing source code files, its cache is invalidated on every single commit. Consequently, the npm install command - which takes minutes to run - is forced to execute from scratch every time, even if package.json hasn’t changed. The COPY command must be split to preserve the cache for the dependency installation.
Image Bloat via Hidden Files
Because layers are read-only, deleting a file in a later layer does not remove it from the final image size; it merely hides it.
RUN curl -O https://example.com/massive-archive.tar.gz
RUN tar -xzf massive-archive.tar.gz
RUN rm massive-archive.tar.gzThe rm command hides the file in the third layer, but the first layer still contains the gigabyte-sized archive, meaning the final image is bloated. File fetching, extraction, and cleanup must happen in a single RUN instruction chained with && to prevent the archive from ever being committed to a layer.
Expanding the Attack Surface
Using a full operating system image (like ubuntu:latest) as a base includes hundreds of unnecessary binaries, compilers, and utilities. This creates a massive attack surface. If an attacker breaches the application, they have access to curl, wget, and Python to download malicious payloads.
Security
Always use minimal base images (like Alpine Linux or Google’s “Distroless” images). Distroless images contain the application runtime but do not even contain a shell (/bin/sh). This makes it exceptionally difficult for an attacker to execute arbitrary commands if they find a vulnerability in the application.
Never embed secrets in build instructions (e.g., ENV API_KEY=12345 or COPY secrets.txt .). Even if you delete the secret in a later layer, it remains permanently accessible to anyone who pulls the image and inspects the underlying layers using tools like docker history.
Operational Guidance
Use multi-stage builds. A multi-stage build uses one large container with compilers and development headers to build the application binary. It then starts a second, completely empty container, and copies only the compiled binary from the first container. This discards the entire build environment, resulting in dramatically smaller and more secure final images.
Pin all package versions. RUN apt-get install python3 is non-deterministic. A build today might produce a completely different image than a build tomorrow. Pin exact versions (e.g., python3=3.10.12-1) to guarantee reproducible builds.
Diagnostics
To diagnose bloated images, use the history command to inspect the exact size of every layer.
docker history my-app:latestLook for unexpectedly large layers. The output will show exactly which Dockerfile instruction generated that layer, allowing you to target the bloat for optimization.
To verify caching behavior, watch the terminal output during a build.
docker build -t my-app .Look for lines labeled CACHED. If steps you expect to be cached are executing fully, inspect the preceding COPY instructions to see what files changed to break the cache.