Images & builds

Build the same Containerfile twice, months apart, and you can get two different images — identical instructions, different bytes — unless you understand what's actually being cached and pinned underneath. This page covers what an OCI image is and how to build one well: layers, the build cache, running as non-root, and why a tag alone doesn't guarantee reproducibility. (The main course page covers the podman build / podman pull commands themselves.)

TL;DR — in 30 seconds:

• An image is an ordered stack of read-only layers; a running container adds one throwaway writable layer on top.
• The build cache is a prefix match: change one instruction and every layer after it rebuilds — order Containerfile instructions least-changing to most-changing.
• Multi-stage builds keep the toolchain out of the shipped image; pin production references to a digest, not a mutable tag.

1. An image is a stack of read-only layers

An OCI image is not a single file — it is an ordered stack of layers, each one a filesystem diff, plus a small JSON manifest that lists them and describes how to run the result (entrypoint, env, exposed ports). This structure is standardized by the OCI Image Format specification, which is why the same image runs unmodified under Podman, Docker, or any other OCI-compliant runtime. Every instruction in a Containerfile that changes the filesystem (RUN, COPY, ADD) produces one new layer on top of the previous one.

At run time, Podman uses overlayfs (the default storage driver) to stack every image layer as read-only, then adds one more writable layer on top for the running container. Nothing a container writes touches the image layers underneath — delete the container and its writable layer goes with it; the image layers are unchanged and shared by every other container built from the same image.

flowchart TB
    subgraph image["Image — read-only layers"]
        L1["Layer 1: base OS files"]
        L2["Layer 2: RUN dnf install ..."]
        L3["Layer 3: COPY package*.json"]
        L4["Layer 4: RUN npm ci"]
        L5["Layer 5: COPY . ."]
    end
    W["Writable layer<br/>(this container only)"]
    L1 --> L2 --> L3 --> L4 --> L5 --> W

Because layers are shared, two images that start with the same base and the same early instructions reuse those layers on disk instead of duplicating them — one reason keeping a small, common base matters.

2. The build cache — and why instruction order matters

podman build caches each layer, keyed on the instruction plus its inputs. On a rebuild, it walks the Containerfile top to bottom and reuses a cached layer as long as nothing about that step changed. The moment one instruction's input changes (different file contents for a COPY, a different command for a RUN), that layer and every layer after it rebuild from scratch — the cache is a prefix match, not a per-instruction match.

This is why dependency installs are split from application code:

# Cache-friendly: deps only reinstall when the manifest changes
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

If COPY . . came first, any source-file edit would invalidate that layer and force npm ci to rerun on every build — correct, but needlessly slow. Ordering instructions from least-frequently-changing to most-frequently-changing keeps rebuilds fast.

3. Multi-stage builds

A single Containerfile can contain multiple FROM stages. Earlier stages hold whatever the build needs (a full SDK, compilers, dev dependencies); the final stage starts fresh from a minimal base and copies in only the finished artifacts with COPY --from=<stage>. Everything else — the SDK, the build cache, dev dependencies — never reaches the shipped image.

FROM registry.access.redhat.com/ubi9/nodejs-20:latest AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM registry.access.redhat.com/ubi9/nodejs-20-minimal:latest
WORKDIR /app
COPY --from=build /app/dist ./dist
CMD ["node", "dist/server.js"]

The result: a smaller final image (less to pull, less attack surface) without giving up a full toolchain during the build itself. podman build --target build . stops at a named stage — handy for debugging the build environment directly.

4. Running as non-root

Every layer inherits the USER in effect when it was built, and the final USER instruction sets who the container runs as by default. Two reasons to set it explicitly rather than leave the base image's default:

• Least privilege. A process compromised while running as root inside the container starts from a much larger blast radius than one running as an unprivileged UID — even under Podman's rootless user-namespace remapping, dropping to a non-root user inside the image is a second, independent layer of defense.
• Portability across UID policies. Some platforms (OpenShift is the well-known example) run containers under an arbitrary, unpredictable UID rather than whatever USER the image declares. An image that chowns its writable paths to a numeric UID (not a username, which may not exist under a reassigned UID) and sets appropriate group permissions survives that remapping; one that assumes it always runs as its declared user does not.

RUN chown -R 1001:0 /app && chmod -R g=u /app
USER 1001

5. Tags vs. digests

A tag (postgres:16-alpine) is a mutable, human-friendly pointer — the maintainer can and does move it to a new build over time. A digest (postgres@sha256:abc123...) is the content hash of the exact image manifest — it can never point anywhere else. podman inspect <image> --format '{{.Digest}}' prints it.

	Tag	Digest
Stability	Moves as the upstream rebuilds	Fixed forever — same bytes every pull
Readability	Human-friendly (16-alpine)	Opaque hash
Use case	Local dev, exploring versions	Production, CI, anywhere reproducibility matters

Pin production and CI references to a digest so "works in staging" can't quietly become a different image in production because a tag moved underneath you.

6. How this connects

• The overlay-layer model is why podman images and podman system df can report disk usage intelligently — shared layers aren't double-counted.
• Non-root by default here compounds with Podman's rootless user-namespace remapping (covered on the main course page) rather than replacing it — they are two independent controls, not alternatives.
• Digest pinning here is the same idea as immutable image references anywhere else in the curriculum (Kubernetes Deployment rollouts, for instance) — a reproducible reference beats a mutable label.

Common gotchas

• Putting COPY . . before the dependency install. Fix: any source edit then invalidates the cache and reinstalls dependencies on every build — copy the manifest and install first, source code last.
• Shipping the build toolchain in the final image. Fix: use a multi-stage build and COPY --from=<stage> to copy only the finished artifact into a minimal final stage.
• Leaving the default USER (often root) in the final image. Fix: set a non-root USER explicitly, and chown to a numeric UID so it survives platforms that reassign the UID at runtime.
• Referencing a mutable tag in production or CI. Fix: pin to a digest — a tag can move to different bytes the moment the maintainer rebuilds it.

Check yourself — two Containerfiles differ only in the order of COPY package*.json / RUN npm ci vs COPY . .. Why does one rebuild faster on a source-only change?

The cache is a prefix match: whichever Containerfile copies the dependency manifest and installs first keeps that layer cached when only source files change. The one that copies everything first invalidates the dependency-install layer (and everything after it) on every source edit.

Check yourself — why pin a production image reference to a digest instead of a tag?

A tag like 16-alpine is a mutable pointer the maintainer can move to a different build; a digest is a content hash of the exact manifest and can never point anywhere else. Pinning to a digest means "works in staging" can't silently become a different image in production.

You can defend this when you can explain why two Containerfiles that install dependencies in a different order rebuild at different speeds, why a multi-stage build ships a smaller image without losing build tooling, and why a production pull should reference a digest rather than a tag.