Image Engineering and Dockerfiles

Introduction

A Docker image is not merely a packaged application. It is a reproducible, content-addressed artifact containing a filesystem, configuration metadata, and the instructions required to create a container. Image engineering is the disciplined practice of designing, building, securing, testing, tagging, storing, and maintaining these artifacts.

In professional DevOps, a Dockerfile is treated as source code. It should be readable, reviewable, deterministic, efficient, secure, and suitable for automated delivery. Poorly designed Dockerfiles can produce unnecessarily large images, expose credentials, invalidate build caches, or create differences between development and production. Well-designed Dockerfiles support repeatable builds and reliable deployment across laptops, CI systems, staging environments, and production clusters.

Learning Objectives

By the end of this chapter, you should be able to:

• Explain image layers, manifests, build contexts, and content-addressable storage.
• Construct Dockerfiles using appropriate instructions and ordering.
• Use build caching and BuildKit effectively.
• Design multi-stage builds for smaller production images.
• Apply tagging, metadata, provenance, and supply-chain practices.
• Diagnose common build failures and optimize image size and security.
• Evaluate a Dockerfile against professional engineering criteria.

1. Images, Layers, and Immutability

A Docker image is composed of a set of filesystem layers and a configuration object. Each layer represents filesystem changes relative to its parent. A container starts from the image's read-only layers and adds a writable container layer. If a container is deleted, changes in that writable layer normally disappear; changes intended to persist should be stored in volumes or external services.

Layers are identified by cryptographic digests. Consequently, an image is addressed by content rather than only by a human-readable name. If the bytes of a layer or configuration change, its digest changes. This property supports integrity checking and deduplication: several images can share identical layers and store them only once on a host or registry.

A Dockerfile instruction such as RUN, COPY, or ADD commonly creates a new layer in the resulting image. ENV, CMD, and similar instructions primarily modify image configuration, although the exact storage representation is implementation-dependent. A later instruction cannot remove bytes from an earlier layer. For example, copying a secret into one layer and deleting it in a later layer does not reliably remove the secret from the image history. The secret remains recoverable from the earlier layer.

This leads to an important engineering principle: prevention is stronger than deletion. Secrets must not enter the build context or an image layer in the first place.

Image references and tags

A typical image reference is:

registry.example.com/team/service:1.4.2

It consists of a registry, repository, and tag. If the registry is omitted, Docker commonly uses a configured default registry. A tag is a mutable human-readable pointer. It is not an immutable identity: service:latest may refer to different digests at different times.

For reproducible deployment, record or deploy by digest, for example:

registry.example.com/team/service@sha256:<digest>

Tags remain valuable for usability, release channels, and automation, but production systems should preserve the resolved digest.

2. The Docker Build Model

When a build is requested, the Docker client or BuildKit receives a build context and a Dockerfile. The context is the set of files available to COPY and ADD. By default it is often the directory supplied to docker build, recursively including its contents except those excluded by .dockerignore.

The context is not the same as the Dockerfile. A Dockerfile may be located elsewhere, while the context is specified independently. This distinction matters because a large context increases upload and scanning time, and accidental inclusion of .git, test data, credentials, or local dependency caches may create security and performance problems.

Example:

docker build -f docker/Dockerfile -t example/api:1.0 .

Here, docker/Dockerfile is the build recipe, while . is the context. A command such as COPY . /app copies files from the context, not arbitrary files from the host filesystem.

The .dockerignore file

A .dockerignore file excludes unnecessary or sensitive paths from the context:

.git
.gitignore
.env
node_modules
__pycache__
*.log
coverage/
tmp/

Patterns should be reviewed carefully. Excluding a file required by COPY causes a build failure or, worse, silently changes the build input. A useful practice is to keep the context close to the application source and to use explicit COPY paths rather than copying the entire context.

3. Dockerfile Instructions and Design Principles

A Dockerfile normally begins with a base image and then declares filesystem changes, configuration, and the default process.

FROM

FROM selects the base image and begins a build stage:

FROM python:3.12-slim

Prefer trusted, maintained, appropriately sized base images. Pinning only a floating tag such as python:3.12-slim improves convenience but does not guarantee that future builds use identical bytes. Stronger reproducibility uses a digest:

FROM python:3.12-slim@sha256:<verified-digest>

Digest pinning creates an operational responsibility: the digest must be deliberately updated when security fixes or approved upgrades are available.

WORKDIR

WORKDIR establishes the working directory for subsequent instructions and the container process:

WORKDIR /app

It is preferable to repeated RUN cd /app commands because its effect is explicit and persists across instructions.

COPY and ADD

Use COPY for ordinary local files. ADD has additional behavior, including archive extraction and some remote-source features, and therefore introduces more implicit behavior. In most application images, COPY is clearer:

COPY package.json package-lock.json ./
RUN npm ci
COPY src/ ./src/

This ordering separates dependency manifests from frequently changing source code. If source files change but dependency manifests do not, the dependency installation layer may remain cached.

RUN

RUN executes commands during the build. Related operations should generally be combined when doing so prevents temporary package indexes or caches from remaining in image layers:

RUN apt-get update \
    && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

The update and installation are intentionally in one instruction. Separating them can allow a stale cached update layer to be reused with a later installation request.

ENV and ARG

ENV defines environment variables available in the built image and at runtime:

ENV PYTHONDONTWRITEBYTECODE=1

ARG defines build-time variables:

ARG APP_VERSION=dev
LABEL org.opencontainers.image.version="$APP_VERSION"

An ARG is not automatically available when the container runs. Neither ARG nor ENV should be used for secrets: values may appear in image history, metadata, build logs, or cache records.

CMD and ENTRYPOINT

CMD supplies default arguments or a default command. ENTRYPOINT defines the principal executable. The exec form is preferred because it avoids an unnecessary shell and allows signals to reach the application directly:

ENTRYPOINT ["/app/bin/server"]
CMD ["--port", "8080"]

The application should run as the foreground process. A container is not made more reliable by running a service manager inside it merely to imitate a virtual machine.

USER, EXPOSE, and metadata

Run as a non-root user when possible:

RUN addgroup --system app && adduser --system --ingroup app app
USER app

EXPOSE documents the intended listening port; it does not publish the port:

EXPOSE 8080

OCI-compatible labels provide machine-readable metadata:

LABEL org.opencontainers.image.title="Example API" \
      org.opencontainers.image.description="HTTP service" \
      org.opencontainers.image.source="https://git.example.com/team/api" \
      org.opencontainers.image.licenses="Apache-2.0"

Useful metadata includes source revision, version, build timestamp, authorship, licenses, and documentation links. Timestamps should be controlled or omitted when byte-for-byte reproducibility is required.

4. Build Cache and BuildKit

A build cache reuses the result of an instruction when its relevant inputs and the preceding build state match. Cache reuse can reduce build time, network traffic, and compute cost. It does not mean that an instruction is always safe to reuse: external package repositories may change while the Dockerfile and copied files remain unchanged.

BuildKit is Docker's modern build engine. It supports improved parallelism, richer progress output, efficient transfer, multi-platform builds, secret and SSH mounts, cache mounts, and remote cache export/import.

A BuildKit cache mount can preserve package-manager data without placing it in the final image layer:

# syntax=docker/dockerfile:1
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    CGO_ENABLED=0 go build -o /out/server ./cmd/server

The cache mount is a build accelerator, not a correctness requirement. The build must still work with an empty cache. Cache contents must not be treated as trusted production inputs.

BuildKit secret mounts prevent a credential from being deliberately copied into a resulting layer:

RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci

A corresponding build might provide the secret through the CLI or CI system. The exact command depends on the Docker version and build environment. Secrets should be short-lived, scoped, audited, and never printed by commands.

Cache ordering strategy

A useful general sequence is:

1. Select a controlled base image.
2. Install stable operating-system dependencies.
3. Copy dependency manifests.
4. Install dependencies.
5. Copy application source.
6. Compile or package the application.
7. Create a minimal runtime stage.

The sequence is not a law; generated files, code generators, and framework behavior may require adaptation. The engineering goal is to place infrequently changing inputs before frequently changing inputs.

5. Multi-Stage Builds

A multi-stage Dockerfile uses several FROM instructions. Intermediate stages can contain compilers, headers, test tools, and source code, while the final stage contains only runtime requirements.

Example for a Go service:

# syntax=docker/dockerfile:1
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/server

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /server
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/server"]

The final image does not inherit the build stage's compiler or source tree. Multi-stage builds reduce attack surface, transfer size, and the probability that development tools are available to an attacker.

Multi-stage builds are not automatically secure. The runtime base must still be maintained, the binary must be tested, certificates and timezone data may be needed, and compatibility must be verified. A scratch or distroless image is inappropriate if the application requires a shell, dynamic libraries, or diagnostic utilities at runtime.

A testing stage can be used without shipping test tools:

FROM build AS test
RUN go test ./...

FROM build AS release
RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server

CI can target the test stage, while the release pipeline targets the final stage.

6. Reproducibility, Tagging, and Provenance

A reproducible build produces equivalent output when supplied with the same declared inputs and controlled environment. Perfect byte identity may be difficult because package repositories, timestamps, compiler versions, and network resources can vary. Reproducibility therefore requires explicit dependency and toolchain control.

Important practices include:

• Pin base images by reviewed digest where appropriate.
• Use lockfiles and deterministic dependency installation.
• Record the source revision in labels or attestations.
• Avoid unpinned remote downloads in RUN commands.
• Control generated timestamps and locale-sensitive behavior.
• Build in a clean, documented environment.
• Verify the resulting digest and scan the image.

A practical tagging scheme may include:

A release process should push a version tag, record the resulting digest, attach provenance, and deploy the digest or an immutable release reference. Promotion between environments should ideally move the same artifact rather than rebuild it with different inputs.

7. Optimization: Size, Speed, and Security

Image optimization is multidimensional. The smallest image is not always the best image if it sacrifices observability, compatibility, or maintainability.

Size optimization

• Select a suitable slim or minimal runtime base.
• Use multi-stage builds.
• Exclude development dependencies from the runtime image.
• Remove package indexes and temporary artifacts in the same layer that creates them.
• Copy only required files.
• Use .dockerignore.
• Avoid embedding documentation, test fixtures, and source maps unless required.

Layer count alone is not a reliable optimization target. A few large layers may be worse than several well-structured small layers, and modern storage engines can share layers efficiently. Optimize total transferred and extracted bytes, build time, cache reuse, and operational clarity.

Security optimization

• Use non-root execution.
• Keep base images and packages patched.
• Scan images and dependencies in CI.
• Do not include credentials, private keys, or tokens.
• Use read-only filesystems and dropped Linux capabilities where the runtime permits.
• Generate a software bill of materials (SBOM).
• Sign images and verify signatures in deployment controls.
• Restrict package installation to approved sources.

A vulnerability scanner reports findings, not automatically exploitable risk. Teams should assess severity, reachability, fix availability, and compensating controls. Conversely, a clean scan is not proof that an image is secure: application logic, configuration, runtime permissions, and supply-chain integrity also matter.

Maintainability optimization

Use a consistent instruction style, meaningful stage names, comments for non-obvious decisions, and automated linting. Avoid clever shell expressions that obscure failures. In shell commands, make failure behavior explicit and check downloaded artifacts with trusted checksums or signatures.

8. A Complete Application Example

The following illustrative Dockerfile builds a small Python web service. It separates dependency installation from source copying and uses a non-root runtime user.

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1
WORKDIR /app

RUN groupadd --system app && useradd --system --gid app app

COPY requirements.txt ./
RUN pip install --no-cache-dir --disable-pip-version-check -r requirements.txt

COPY app/ ./app/
RUN chown -R app:app /app
USER app
EXPOSE 8000
LABEL org.opencontainers.image.title="Example Python service"
CMD ["python", "-m", "app"]

For a larger project, a builder stage could create a wheel in an isolated environment, and the runtime stage could install only the wheel and production dependencies. The correct design depends on whether dependencies contain native extensions, whether a virtual environment is used, and whether the runtime base has compatible system libraries.

9. Applied Case: Reducing a CI Image Pipeline

Background

An illustrative online retailer has a Node.js API. Its original Dockerfile copies the entire repository, installs development dependencies, runs tests and compilation in one stage, and uses a full operating-system base for production. The resulting image is approximately 1.1 GB and takes 12 minutes to build in CI.

Problem

The image contains test tools, source-control metadata, local documentation, and package-manager caches. Developers also report that small source changes trigger a complete dependency installation. A security review identifies a non-root omission and an accidentally copied local environment file.

Analysis

The Dockerfile has three principal design faults. First, the build context is too broad because .dockerignore is absent. Second, dependency manifests are copied together with all source files, so ordinary source edits invalidate the dependency layer. Third, compilation and runtime are not separated.

Approach

The team adds .dockerignore, copies package.json and the lockfile before application code, uses npm ci, and creates separate build and production stages. CI runs tests in the build stage and copies only the compiled output and production dependencies into the final stage. The image runs as a dedicated user. The pipeline records the image digest and generates an SBOM.

Outcome

After validation, the image is approximately 180 MB, and unchanged dependency builds reuse the installation layer. The exact size and build-time improvements depend on the application and registry, but the architectural improvement is stable: build concerns are excluded from the runtime artifact, and inputs are explicit.

Learning points

1. Context control is both a security and performance measure.
2. Cache design begins with instruction and input ordering.
3. Multi-stage builds provide a boundary between compilation and execution.
4. Image size is useful, but digest tracking, patching, testing, and provenance are equally important.

10. Troubleshooting and Governance

A build failure should first be classified. A COPY failure often indicates an incorrect context or .dockerignore rule. A cache miss may result from changed files, instruction order, base-image updates, or build arguments. A runtime failure may indicate that the final stage omitted a shared library, certificate bundle, user permission, or configuration file.

Governance should define approved base images, update ownership, vulnerability response times, registry retention, signing requirements, and exceptions. Organizations should distinguish development images from production images and document who may publish to protected repositories. Image labels and SBOMs support incident response by identifying source revisions and included components.

AI-assisted tools can help draft Dockerfiles, identify likely cache improvements, or summarize scanner findings. However, generated instructions must be reviewed. AI may suggest insecure practices such as embedding secrets, using floating downloads, running as root, or selecting an incompatible minimal base. Human review, automated tests, policy checks, and provenance controls remain necessary.

Best-Practice Checklist

Before publishing an image, verify that:

• The build context excludes secrets, credentials, and irrelevant files.
• Dependencies are installed from lockfiles or otherwise controlled inputs.
• Base images are trusted, maintained, and appropriately pinned.
• Build and runtime stages are separated where useful.
• The process runs as a non-root user.
• No secret appears in Dockerfile instructions, arguments, logs, or layers.
• The image contains only runtime requirements.
• Tests, vulnerability scans, and policy checks pass.
• OCI metadata, source revision, and license information are present.
• The published digest and provenance are recorded.

Chapter Summary

Image engineering treats Docker images as reproducible, versioned, and governed software artifacts. Images consist of content-addressed layers and configuration; because later layers do not erase earlier bytes, secrets must never be copied into an image. The build context determines which files are available, and .dockerignore controls both exposure and transfer cost.

Dockerfile quality depends on explicit instructions, sensible cache ordering, controlled dependencies, appropriate metadata, and non-root execution. BuildKit improves performance and capability through cache mounts, secret mounts, parallelism, and multi-stage builds. Multi-stage designs keep compilers and test tools out of runtime images, although minimal images must still be tested for compatibility and operability.

Tags provide convenient names, but digests provide immutable identity. Professional pipelines combine controlled inputs, image scanning, SBOMs, signatures, provenance, automated tests, and deployment by verified digest. Optimization should balance size, build speed, security, observability, and maintainability rather than pursue size alone.

Key Terms

Image: A packaged filesystem, configuration, and metadata used to create containers.

Layer: An immutable filesystem change represented as part of an image.

Digest: A cryptographic content identifier for an image or layer.

Tag: A mutable human-readable reference to an image.

Manifest: Metadata describing image configuration and layer references.

Dockerfile: A declarative recipe for building an image.

Build context: Files made available to a build operation.

.dockerignore: A pattern file that excludes paths from the build context.

BuildKit: Docker's modern build engine with advanced caching and build features.

Cache: A stored result that can be reused when build inputs match.

Cache mount: Temporary persistent build storage used to accelerate package operations.

Multi-stage build: A Dockerfile design using multiple build stages to separate concerns.

Base image: The image selected by a FROM instruction.

Runtime image: The final image used to execute an application.

OCI label: Standardized metadata attached to an image.

SBOM: A software bill of materials listing components in an artifact.

Provenance: Evidence describing how and from which inputs an artifact was built.

Content-addressable storage: Storage in which objects are identified by their content digest.

Reproducible build: A build designed to produce equivalent results from controlled identical inputs.

Image registry: A service that stores and distributes container images.

Review Questions

Short Answer Questions

1. Why does deleting a secret in a later Dockerfile layer not reliably remove it from an image?
2. Distinguish a Dockerfile from a build context.
3. What is the principal purpose of .dockerignore?
4. Why is the exec form of ENTRYPOINT generally preferred?
5. What is the difference between an image tag and an image digest?

Long Answer Questions

1. Explain Docker image layers and discuss their implications for caching, storage, and secret management.
2. Describe a professional Dockerfile structure for a compiled application, including the role of multi-stage builds.
3. Explain how instruction ordering affects cache reuse and provide an example.
4. Discuss the benefits and limitations of minimal runtime images.
5. Develop a governance policy for image tagging, scanning, metadata, and promotion between environments.

Analytical / Application Questions

1. A Dockerfile copies the entire repository, runs dependency installation, and then compiles the application. Every source edit downloads all dependencies again. Redesign the instruction order and justify your choices.
2. A production image contains a compiler, test framework, source code, and a package-manager cache. Propose a multi-stage design and identify what belongs in the final stage.
3. A team deploys service:latest to production but cannot determine which source revision is running after a rollback. Design a tagging, digest-recording, and provenance process that resolves this problem.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a Docker build context?

A. The set of files available to COPY and ADD during a build

B. The writable layer created when a container starts

C. The registry namespace where an image is stored

D. The runtime environment variables of a container

Correct Answer: A. The set of files available to COPY and ADD during a build

Question 2

Why should dependency manifests often be copied before application source code?

A. To make the final image run as root

B. To improve the chance that dependency installation is reused from cache

C. To prevent the Dockerfile from using multiple stages

D. To publish ports automatically

Correct Answer: B. To improve the chance that dependency installation is reused from cache

Question 3

Which reference provides immutable image identity?

A. latest tag

B. Branch name

C. Content digest

D. Container hostname

Correct Answer: C. Content digest

Question 4

What is the main purpose of a multi-stage build?

A. To expose every build tool at runtime

B. To separate build dependencies from the final runtime image

C. To make all image tags immutable

D. To replace image scanning

Correct Answer: B. To separate build dependencies from the final runtime image

Question 5

Which practice is appropriate for handling a private package-manager credential during a BuildKit build?

A. Copy it into the image and delete it later

B. Store it in an ENV instruction

C. Use a BuildKit secret mount

D. Commit it to the build context

Correct Answer: C. Use a BuildKit secret mount

Question 6

What does EXPOSE 8080 do in a Dockerfile?

A. Publishes port 8080 on the host automatically

B. Documents the intended container listening port

C. Encrypts traffic on port 8080

D. Starts an HTTP server

Correct Answer: B. Documents the intended container listening port

Question 7

Which instruction is normally preferable for copying ordinary local files?

A. COPY

B. ENTRYPOINT

C. EXPOSE

D. ARG

Correct Answer: A. COPY

Question 8

Why should production containers usually avoid running as root?

A. Root prevents image caching

B. Non-root execution can reduce the impact of a compromise

C. Root disables Docker registries

D. Non-root users eliminate all vulnerabilities

Correct Answer: B. Non-root execution can reduce the impact of a compromise

Question 9

Which item should generally be excluded by .dockerignore?

A. A required application entry point

B. The production configuration explicitly needed by COPY

C. Source-control metadata and local secret files

D. A dependency lockfile required for installation

Correct Answer: C. Source-control metadata and local secret files

Question 10

What does an SBOM primarily provide?

A. A list of software components contained in an artifact

B. A command for starting a container

C. A replacement for the Dockerfile

D. A method for publishing a host port

Correct Answer: A. A list of software components contained in an artifact