Building Images with Dockerfiles and BuildKit

Introduction

A container image is a portable, versioned filesystem and metadata package from which containers are created. Building a reliable image is more than writing a few Dockerfile commands: it involves selecting trustworthy inputs, controlling dependencies, designing cache-efficient layers, protecting credentials, and making the result reproducible. Poor image construction can produce large artifacts, slow deployments, security vulnerabilities, and inconsistent behavior between development and production.

Dockerfiles provide a declarative description of an image build. Docker BuildKit is the modern build engine that interprets Dockerfiles and adds features such as improved caching, parallel execution, secret mounts, SSH forwarding, and multiple output formats. This chapter explains how to use these capabilities responsibly.

Learning Objectives

By the end of this chapter, you should be able to:

• Explain Dockerfile instructions and image layers.
• Define and control a build context.
• Write efficient, readable Dockerfiles.
• Use multi-stage builds to separate compilation from runtime.
• Apply build arguments, environment variables, secrets, and cache mounts appropriately.
• Select minimal and trustworthy base images.
• Add health checks and reproducibility controls.
• Diagnose common build and image-quality problems.

1. The Image-Build Model

A Docker image consists conceptually of read-only filesystem layers plus configuration metadata. Most filesystem-changing Dockerfile instructions create a new layer. Containers add a writable layer above the image layers. Layers can be reused when the corresponding build step and its inputs have not changed.

A build has three important inputs:

1. The Dockerfile, which defines instructions.
2. The build context, a set of files made available to the builder.
3. External inputs, such as base images, package repositories, source-control revisions, and downloaded dependencies.

The command docker build -t example:1.0 . uses the current directory as its context. The final dot is significant. Docker sends the context to the builder, subject to .dockerignore. A context that contains virtual environments, dependency caches, Git history, credentials, or compiled artifacts can be unnecessarily large and may accidentally expose sensitive data to build steps.

BuildKit

BuildKit is enabled by default in modern Docker installations and can also be invoked through docker buildx build. It improves builds by constructing a dependency graph rather than blindly executing every instruction in sequence. Independent operations may run concurrently, while reusable results can be imported from local or remote caches.

BuildKit syntax features are selected with a parser directive such as:

# syntax=docker/dockerfile:1

The directive should appear at the beginning of the file. It allows the Dockerfile frontend to provide features such as cache and secret mounts.

2. Dockerfile Instructions and Their Roles

A practical Dockerfile normally uses the following instructions.

A minimal example is:

# syntax=docker/dockerfile:1
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
USER 10001
CMD ["python", "app.py"]

The order is intentional. Dependency metadata is copied before application source, so changing a source file does not necessarily invalidate the dependency-installation layer.

Shell and exec forms

RUN, CMD, and ENTRYPOINT can use shell or exec forms. The exec form is a JSON array:

ENTRYPOINT ["/app/server"]
CMD ["--port", "8000"]

It avoids an additional shell and generally gives clearer signal handling. This matters for graceful shutdown: an application should receive termination signals directly rather than through an unexpected shell process.

3. Build Context and .dockerignore

The context should contain only files required for the build. A typical .dockerignore may contain:

.git
.gitignore
Dockerfile*
.dockerignore
node_modules
__pycache__
*.pyc
.env
coverage
build
dist

Patterns must be adapted to the project. Ignoring Dockerfile itself is optional and does not prevent Docker from using the selected Dockerfile, but it prevents accidental copying into the image.

A small context improves upload time and reduces accidental disclosure. It does not, by itself, guarantee that a file cannot be accessed: files intentionally copied into an image, mounted as secrets, or retrieved from external systems must still be governed carefully.

4. Layers, Ordering, and Cache Optimization

Build cache effectiveness depends on stable inputs and instruction ordering. A useful pattern is:

COPY package.json package-lock.json ./
RUN npm ci
COPY src ./src
RUN npm run build

Dependency installation is invalidated when the lockfile changes, but not whenever an unrelated source file changes.

Package installation should usually update package indexes and install packages in one RUN instruction, followed by cleanup:

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*

The cleanup is in the same layer as installation. Removing files in a later layer does not remove them from the earlier layer stored in the image history.

Build cache is an optimization, not a correctness mechanism. A build must still work when cache is unavailable. When dependencies are mutable, a cached layer can preserve older results; lockfiles, pinned versions, and periodic intentional cache refreshes help control this risk.

Cache mounts

BuildKit can provide a cache directory that is not committed as part of the final image:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

This can accelerate repeated builds while keeping package caches out of the resulting layer. Cache contents should be treated as disposable and potentially untrusted; the build must remain correct if the cache is empty.

5. Multi-Stage Builds

A multi-stage build uses several FROM instructions. A builder stage contains compilers, headers, package managers, and source code. A runtime stage receives only the executable and required runtime files.

FROM golang:1.22 AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/server ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=builder /out/server /server
USER nonroot:nonroot
ENTRYPOINT ["/server"]

The final image does not contain the Go compiler or source tree. Multi-stage builds reduce attack surface and image size, but the runtime stage must contain every required shared library, certificate bundle, timezone file, and configuration dependency. A statically linked binary may have different requirements from a dynamically linked one.

Stages can also represent testing and packaging boundaries:

FROM node:22 AS dependencies
WORKDIR /app
COPY package*.json ./
RUN npm ci

FROM dependencies AS test
COPY . .
RUN npm test

FROM dependencies AS build
COPY . .
RUN npm run build

FROM nginx:1.27-alpine AS runtime
COPY --from=build /app/dist /usr/share/nginx/html

A CI pipeline may target the test stage while production targets runtime. This arrangement makes testing an explicit part of the build graph.

6. Build Arguments and Environment Variables

ARG values are available during image construction:

ARG APP_VERSION=dev
RUN echo "Building ${APP_VERSION}"

They are not automatically available when a container runs. ENV values are stored in the image configuration and are available at runtime:

ENV APP_ENV=production

An argument can be used to select a base image tag:

ARG PYTHON_VERSION=3.12
FROM python:${PYTHON_VERSION}-slim

However, unconstrained arguments can reduce reproducibility and permit unintended inputs. Use controlled values in CI, validate them where feasible, and record the selected version in build metadata.

Neither ARG nor ENV is appropriate for passwords, tokens, or private keys. They may appear in image history, metadata, logs, or exported build records.

7. Secrets and SSH Access with BuildKit

BuildKit secret mounts expose a secret temporarily to a single build step without committing it to the resulting layer:

docker build --secret id=npmrc,src=$HOME/.npmrc -t private-app .

RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

The secret is available only during that command. A build script must not copy it into the filesystem, print it, or embed it in generated artifacts.

For private Git dependencies, BuildKit can forward an SSH agent:

docker build --ssh default -t private-app .

RUN --mount=type=ssh git clone [email protected]:organisation/private-module.git

Access should be narrowly scoped, audited, and limited to trusted build definitions. Secret mounts protect against ordinary layer persistence, but they do not make a malicious Dockerfile safe: a build step can still exfiltrate a mounted secret. Review Dockerfiles and build sources as code.

8. Base-Image Selection and Minimality

A base image establishes operating-system libraries, tooling, defaults, and a supply-chain relationship. Common choices include:

• Full distribution images: broad compatibility and troubleshooting tools, but larger attack surface.
• Slim images: fewer packages and smaller size, but possible missing utilities or libraries.
• Alpine-based images: small footprint, but compatibility issues can arise from its musl C library and package ecosystem.
• Distroless images: very small runtime surface, but limited interactive debugging.
• Scratch: no userspace at all; suitable only when the application is self-contained and its operational needs are understood.

Smallest is not always best. A slightly larger image with predictable libraries, certificates, and supportability may be safer operationally than an extremely minimal image that encourages ad hoc fixes.

Use trusted registries, scan images and dependencies, monitor advisories, and prefer immutable references such as digests when supply-chain assurance is important:

FROM python:3.12-slim@sha256:<approved-digest>

A digest identifies a particular manifest, whereas a tag such as 3.12-slim may point to a different image later. Digest pinning improves reproducibility but creates a maintenance obligation: approved digests must be deliberately updated for security fixes.

9. Health Checks and Runtime Contracts

A health check describes whether a running container is functioning, not merely whether its process exists.

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://127.0.0.1:8000/health || exit 1

The probe must be lightweight, deterministic, and available in the runtime image. If the image lacks wget, use an application-specific probe or include an appropriate tool deliberately. Health checks should not depend on external services unless the operational definition of health explicitly requires that dependency.

A health check is not a substitute for readiness, liveness, startup, and dependency monitoring at the orchestrator level. Its meaning should be documented so that automated restarts do not amplify an outage.

10. Reproducible and Auditable Builds

A reproducible build produces the same intended artifact when given the same declared inputs and build environment. Perfect byte-for-byte identity can be difficult because timestamps, network responses, compiler versions, and package metadata may vary. Practical controls include:

• Pin base images by digest where appropriate.
• Commit dependency lockfiles.
• Pin or constrain toolchain and package versions.
• Avoid downloading unverified latest artifacts.
• Set a consistent timezone and locale when application behavior depends on them.
• Use deterministic compiler options where supported.
• Record source revision, builder identity, and dependency information as provenance.
• Generate and verify software bills of materials (SBOMs).
• Separate source, test, and release stages.

Reproducibility is not identical to security. A reproducible malicious build is still malicious; security also requires provenance, review, vulnerability management, and trustworthy inputs.

11. Worked Example: An Efficient Python Image

Consider a service with requirements.txt, application code, and a WSGI entry point. A practical Dockerfile is:

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS runtime

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1
WORKDIR /app

RUN addgroup --system app && adduser --system --ingroup app app

COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir -r requirements.txt

COPY . .
RUN chown -R app:app /app
USER app

EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=2)"
CMD ["python", "-m", "app"]

The dependency file is copied before source code, improving cache reuse. The application runs as a non-root user. Python bytecode generation is disabled, output is unbuffered for logging, and the health check uses a Python module already present rather than assuming an external command.

For packages requiring compilation, a better design uses a builder stage with a compiler and a runtime stage containing only installed runtime artifacts. The exact mechanism depends on whether dependencies are pure Python, contain native extensions, or require system libraries.

12. Illustrative Case: Reducing a Web-Service Image

Background. A small development team deploys a Node.js web service. Its image is 1.1 GB, builds take 12 minutes, and production containers run as root.

Problem. The Dockerfile copies the entire repository before running npm install, includes development dependencies in production, and leaves package-manager caches in image layers.

Analysis. The image contains the compiler toolchain, test files, .git history, and development modules. Every source change invalidates dependency installation. A review also finds an .env file in the build context.

Approach. The team removes .env and .git through .dockerignore, copies package manifests first, installs with a lockfile, runs tests in a separate stage, builds the application in a builder stage, and copies only production output into a minimal runtime image. Runtime configuration is supplied by the deployment platform rather than baked into the image. The service runs under a dedicated non-root user and exposes a local health endpoint.

Outcome. The resulting image is substantially smaller, rebuilds reuse dependency layers, and credentials are no longer included in the context. The team also establishes a policy for digest updates, vulnerability scanning, SBOM generation, and periodic clean-cache builds.

Learning points. Image size is only one quality measure. The more important improvements are separation of concerns, reduced privilege, controlled inputs, fast feedback, and auditable supply-chain practices.

13. Challenges, Limitations, and Governance

Dockerfile optimization involves trade-offs. Aggressive layer reduction can make debugging harder. Minimal images reduce attack surface but may remove diagnostic tools. Cache reuse improves speed but can conceal stale dependencies. Digest pinning improves stability but requires a controlled update process.

Organizations should establish policies for approved base images, vulnerability thresholds, secret handling, provenance, license review, and image retention. Build logs should avoid credentials and personal data. Third-party packages may introduce licensing obligations; a security scan alone does not determine legal compliance.

AI-assisted tools can suggest Dockerfiles, identify anti-patterns, or generate SBOM analysis, but generated configurations require human review. An AI system may recommend outdated packages, insecure defaults, or commands that leak secrets. Treat generated Dockerfiles as untrusted code and validate them through tests, policy checks, and security review.

Best-Practice Checklist

• Use a small, deliberate build context and a maintained .dockerignore.
• Order instructions to preserve useful cache layers.
• Use lockfiles and controlled dependency versions.
• Keep compilers and development tools out of the runtime image.
• Use multi-stage builds for compiled or packaged applications.
• Never place secrets in ARG, ENV, source code, or copied files.
• Use BuildKit secret and cache mounts appropriately.
• Run as a non-root user whenever feasible.
• Pin important inputs and maintain an update process.
• Add meaningful health checks and test the final image.
• Scan images, generate provenance and SBOM information, and review licenses.
• Verify that the build succeeds without relying on a warm cache.

Chapter Summary

Dockerfiles define image construction, while BuildKit provides efficient execution, caching, secret handling, SSH forwarding, and advanced mounts. The build context should be deliberately limited, and .dockerignore should exclude unnecessary or sensitive files. Instruction ordering is central to cache efficiency: stable dependency metadata should generally precede frequently changing source code.

Multi-stage builds separate compilation and testing from runtime delivery, reducing image size and attack surface. ARG is intended for controlled build-time configuration, whereas ENV persists into runtime; neither should carry secrets. BuildKit secret mounts provide temporary credential access, but the build definition must still be trusted.

Base-image selection requires balancing minimality, compatibility, maintenance, and security. Health checks should represent meaningful service health. Reproducible builds depend on pinned inputs, lockfiles, deterministic practices, provenance, and disciplined updates. High-quality images are therefore not merely small: they are secure, testable, maintainable, auditable, and suitable for dependable deployment.

Key Terms

Dockerfile: Declarative text file describing how to build an image.

Build context: Files made available to the image builder.

BuildKit: Modern Docker build engine supporting advanced execution and caching features.

Layer: Read-only filesystem change associated with an image build step.

Base image: Image selected by a FROM instruction.

Build cache: Stored result that can be reused when build inputs remain compatible.

Multi-stage build: Build using multiple FROM stages with selected artifacts copied between them.

Build argument: Value declared with ARG and available during construction.

Environment variable: Configuration value set with ENV and usually available at runtime.

Secret mount: Temporary BuildKit-mounted credential unavailable in the final layer unless copied deliberately.

Cache mount: Temporary or persistent BuildKit cache directory used to accelerate tools.

Build context: Input file collection sent to the builder.

Digest: Content-based identifier for a particular image manifest.

Tag: Human-readable image reference that may be moved to another image.

Health check: Command used to assess whether a running container is functioning.

SBOM: Software bill of materials listing components in an artifact.

Provenance: Evidence describing how and from which inputs an artifact was built.

Runtime image: Final image used to execute the application.

Reproducible build: Build process designed to produce consistent results from the same declared inputs.

Review Questions

5 Short Answer Questions

1. What is the purpose of a Docker build context?
2. Why should dependency manifests often be copied before application source code?
3. What is the difference between ARG and ENV?
4. What problem does a multi-stage build solve?
5. Why should secrets not be placed in Dockerfile ENV instructions?

5 Long Answer Questions

1. Explain how Dockerfile instruction ordering affects build-cache reuse.
2. Compare full, slim, Alpine, distroless, and scratch base images.
3. Describe how BuildKit secret mounts differ from ordinary environment variables.
4. Explain the relationship between image minimality, security, compatibility, and maintainability.
5. Describe a practical process for creating a reproducible and auditable image build.

3 Analytical / Application Questions

1. A Dockerfile copies the entire repository, runs npm install, and then copies source files again. Analyze its cache and security weaknesses and propose a revised sequence.
2. A compiled service works in its builder image but fails in a scratch runtime image. Identify likely causes and describe a systematic diagnosis.
3. Design a build strategy for a service that requires private package credentials, automated tests, a compiled production artifact, and a non-root runtime container. Justify each stage and security control.

Multiple Choice Questions with Answers

Question 1

Which Dockerfile instruction is primarily used to select the starting image?

A. FROM

B. COPY

C. RUN

D. WORKDIR

Correct Answer: A. FROM

Question 2

What is the principal purpose of a .dockerignore file?

A. To encrypt image layers

B. To limit files sent in the build context

C. To run health checks

D. To select the runtime user

Correct Answer: B. To limit files sent in the build context

Question 3

Which feature allows a build to use a credential without normally storing it in an image layer?

A. ENV

B. ARG

C. BuildKit secret mount

D. EXPOSE

Correct Answer: C. BuildKit secret mount

Question 4

Why are dependency manifest files commonly copied before application source code?

A. To make the container run as root

B. To improve reuse of the dependency-installation cache

C. To publish the application port

D. To disable health checks

Correct Answer: B. To improve reuse of the dependency-installation cache

Question 5

What is the main benefit of a multi-stage build?

A. It automatically encrypts the image

B. It eliminates the need for a base image

C. It separates build tools from the runtime image

D. It guarantees that all dependencies are current

Correct Answer: C. It separates build tools from the runtime image

Question 6

Which statement about ARG is correct?

A. It is intended for build-time values

B. It always persists as a runtime environment variable

C. It securely stores passwords

D. It publishes container ports

Correct Answer: A. It is intended for build-time values

Question 7

What does pinning an image by digest provide?

A. Automatic vulnerability remediation

B. A content-specific image reference

C. A writable filesystem layer

D. A built-in application health endpoint

Correct Answer: B. A content-specific image reference

Question 8

Which instruction documents an intended container port but does not publish it by itself?

A. USER

B. CMD

C. EXPOSE

D. HEALTHCHECK

Correct Answer: C. EXPOSE

Question 9

What is a suitable characteristic of a meaningful health check?

A. It always returns success to prevent restarts

B. It tests whether the relevant service is functioning

C. It requires access to private credentials

D. It installs debugging tools at runtime

Correct Answer: B. It tests whether the relevant service is functioning

Question 10

Which practice best supports reproducible builds?

A. Using unpinned latest dependencies

B. Copying personal credential files into the image

C. Using lockfiles and controlled input versions

D. Relying on a warm local cache

Correct Answer: C. Using lockfiles and controlled input versions