Containers and Kubernetes Delivery

Introduction and Context

Modern delivery platforms must move software from source control to reliable production operation while preserving repeatability, security, observability and speed. Containers address the packaging problem: an application, its runtime, libraries and configuration expectations can be assembled into an image and executed consistently across environments. Kubernetes addresses the orchestration problem: it schedules containers, maintains desired state, exposes services, performs health checks and supports controlled change.

Containers and Kubernetes do not eliminate operational complexity. They relocate it into image construction, registry governance, cluster configuration, networking, identity, resource management and deployment policy. A professional DevOps engineer therefore treats container delivery as a supply chain rather than as a command sequence: source code is transformed into an image, the image is tested and signed, a registry stores it, Kubernetes admits and deploys it, and runtime evidence determines whether the release should continue.

This chapter develops that supply chain from first principles and applies it to a realistic delivery scenario.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain container images, layers, registries and runtime isolation.
2. Build smaller and more secure images using sound Dockerfile practices.
3. Select registry controls for authentication, retention, scanning and provenance.
4. Design Kubernetes Deployments, Services, ConfigMaps, Secrets and health probes.
5. Apply requests, limits, rolling updates and horizontal scaling appropriately.
6. Package applications with Helm and structure environment-specific values.
7. Design CI/CD workflows for secure, observable and reversible delivery.
8. Evaluate progressive delivery, governance risks and operational trade-offs.

1. Containers as a Delivery Abstraction

A container is an isolated process that shares the host operating system kernel while receiving an independent view of files, processes, networking and other resources. Unlike a traditional virtual machine, a container normally does not include a complete guest kernel. This makes containers comparatively fast to start and efficient in resource use, although it also means that kernel vulnerabilities and configuration errors can affect multiple workloads.

A container image is an immutable, versioned filesystem package plus metadata such as an entrypoint, environment defaults and exposed ports. Images are commonly built from layers. Each Dockerfile instruction may create a layer, and unchanged layers can be reused through caching. An image reference usually combines a registry, repository and tag, for example registry.example.com/payments/api:2.4.1.

Tags are human-friendly pointers and can be moved. A digest, such as sha256:..., identifies image content. Production deployment should prefer a digest or an immutable release tag because deploying latest makes it difficult to prove which artifact is running.

The container lifecycle is distinct from the application lifecycle. A container may be stopped and recreated at any time. Persistent data should therefore be stored in managed databases, object storage or explicitly managed volumes rather than in the writable container layer.

Images, Containers and Registries

The relationship among the principal objects is as follows:

• Dockerfile or build definition: instructions for producing an image.
• Image: immutable artifact composed of filesystem layers and metadata.
• Container: a runtime instance of an image.
• Registry: a service that stores and distributes images.
• Runtime: software such as containerd or CRI-O that downloads and executes images.
• Orchestrator: software such as Kubernetes that schedules and manages containers.

A registry is more than a file repository. It should support authentication, authorization, TLS, image retention, vulnerability scanning, audit logs, replication and, where possible, content signing and provenance attestations. Public registries are convenient but require careful evaluation of publisher identity, licensing, maintenance activity and known vulnerabilities.

2. Secure and Efficient Image Construction

A practical Dockerfile

A secure image has a minimal attack surface, deterministic dependencies, no embedded secrets and a predictable process model. A multi-stage build separates compilation tools from the runtime image.

# Build stage
FROM node:22-bookworm-slim AS build
WORKDIR /src
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build && npm prune --omit=dev

# Runtime stage
FROM gcr.io/distroless/nodejs22-debian12:nonroot
WORKDIR /app
COPY --from=build /src/package*.json ./
COPY --from=build /src/node_modules ./node_modules
COPY --from=build /src/dist ./dist
EXPOSE 8080
USER nonroot:nonroot
CMD ["dist/server.js"]

The example illustrates several principles:

1. Pin meaningful inputs. Pinning only a mutable tag does not provide complete reproducibility. Organisations may pin base-image digests and lock dependency versions, while maintaining a controlled update process.
2. Use multi-stage builds. Compilers, package managers and test tools need not be shipped to production.
3. Order instructions for caching. Dependency manifests are copied before application source so that dependency installation can be reused when only source files change.
4. Run as non-root. If the process is compromised, non-root execution reduces the available privileges.
5. Exclude unnecessary files. A .dockerignore should omit version-control metadata, local caches, credentials, build output and test data where appropriate.
6. Use an explicit process. The application should receive signals correctly and terminate gracefully. A shell wrapper that fails to forward signals can make rolling updates slow or unsafe.
7. Do not place secrets in image layers. Removing a secret in a later instruction does not reliably remove it from previous layers.

A Dockerfile is not a security boundary. It should be complemented by runtime policies, image scanning, dependency management, least-privilege identities and cluster controls.

Reproducibility and software supply-chain security

A defensible build records the source revision, dependency lockfile, base-image digest, builder identity, build time and test results. Provenance answers “how was this artifact produced?” while a signature answers “who attests to this artifact?” Tools implementing signed metadata and software-bill-of-materials (SBOM) generation can support verification, vulnerability analysis and incident response.

Scanning is useful but imperfect. A scanner may report a vulnerable library that is not reachable in the deployed code, or miss a newly disclosed issue. Teams should classify findings by severity, exploitability, exposure and available remediation rather than treating a scan result as an automatic risk decision.

Image size and build performance

Small images usually download faster and contain fewer packages, but extreme minimisation may reduce diagnosability or omit required certificates and time-zone data. The correct objective is an appropriate runtime image, not the smallest possible byte count. Build caches and dependency mirrors improve delivery speed, but caches must not allow untrusted content to contaminate a release.

3. Registry Operations and Governance

A registry operating model should define:

A useful promotion model builds an artifact once and promotes the same digest from development to test and production. Rebuilding separately for each environment weakens traceability because supposedly identical releases may differ. Environment-specific configuration belongs outside the image, subject to security controls.

4. Kubernetes Architecture and Desired State

Kubernetes is a control system based on desired state. A user submits objects such as a Deployment; controllers compare the declared state with observed state and take actions to reduce the difference. The control plane includes an API server, a persistent state store, a scheduler and controller processes. Worker nodes run a container runtime, a node agent and networking components.

The principal abstraction is the Pod, the smallest schedulable unit. A Pod contains one or more tightly coupled containers sharing a network namespace and, where configured, storage. Most application workloads use one principal container per Pod, with sidecars only when their lifecycle and resource needs genuinely belong together.

A Deployment manages ReplicaSets and supports rolling replacement of Pods. A Service provides a stable virtual endpoint for a changing set of Pods selected by labels. A ConfigMap stores non-secret configuration; a Secret stores sensitive values but must still be protected through encryption, access control and careful logging. A Namespace provides a logical boundary for names, policies and quotas, but is not equivalent to a strong security boundary in every threat model.

A Deployment resource

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  labels:
    app.kubernetes.io/name: orders-api
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: orders-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: orders-api
    spec:
      serviceAccountName: orders-api
      containers:
        - name: api
          image: registry.example.com/orders/api@sha256:REPLACE_ME
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            initialDelaySeconds: 15
            periodSeconds: 10
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true

The selector must match the Pod template labels. A mismatch prevents the Deployment from managing its Pods correctly. The resource request is used in scheduling decisions; the limit bounds runtime consumption according to the resource. CPU is commonly expressed in millicores, where 1000m represents one CPU unit. Memory quantities are capacity units, not a promise that the application will use the entire limit.

Services and discovery

A ClusterIP Service gives clients a stable DNS name and virtual IP while selecting healthy Pods through labels. A typical service is:

apiVersion: v1
kind: Service
metadata:
  name: orders-api
spec:
  selector:
    app.kubernetes.io/name: orders-api
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

Within a namespace, clients can commonly use orders-api; across namespaces they can use a qualified DNS name. Service discovery decouples consumers from Pod IP addresses, which are ephemeral. NetworkPolicies should restrict which workloads may communicate rather than relying on network reachability by default.

Configuration and secrets

Configuration that varies by environment should be injected through environment variables, mounted files or external configuration services. Secrets should preferably be sourced from a dedicated secret manager and synchronised under a controlled identity. Kubernetes Secret objects are encoded, not automatically confidential; encryption at rest and restrictive RBAC are essential.

5. Health, Reliability and Scaling

Probes

A startup probe gives slow-starting applications time to initialise. A readiness probe determines whether a Pod should receive traffic. A liveness probe detects a process that is running but unable to recover. Confusing these probes can cause outages: a liveness probe that checks a temporarily unavailable dependency may repeatedly restart a healthy application, while a readiness probe that always returns success may route traffic to an unready process.

Health endpoints should be purposeful. Liveness usually tests whether the process can make progress; readiness may test essential local state and carefully selected dependencies. Probe endpoints should not expose credentials or detailed internal diagnostics.

Scheduling and capacity

Requests enable the scheduler to place Pods on nodes with sufficient declared capacity. Limits can protect a node from excessive consumption, but an excessively low memory limit can cause out-of-memory termination. Teams should measure actual usage, define a realistic performance objective and test under representative load.

For a workload with n identical replicas, a rough minimum requested CPU capacity is:

where C_request is the CPU request per Pod and the result is in CPU units. This is a planning approximation, not a complete capacity model: system overhead, daemon workloads, bin-packing, bursts, node failure and scheduling constraints must also be considered.

A Horizontal Pod Autoscaler (HPA) adjusts replicas using metrics such as CPU utilisation. For CPU-based scaling, a simplified conceptual relationship is:

where n_current is the current replica count, u_current is measured average CPU utilisation as a fraction of requested CPU, u_target is the target fraction, and ceil rounds upward. The approximation assumes comparable Pods, valid metrics and a stable enough workload. Kubernetes applies stabilisation and policy controls, so the actual controller behaviour may not equal a single instantaneous calculation.

For example, if four Pods average 80% utilisation and the target is 50%, the conceptual target is ceil(4 × 0.80 ÷ 0.50) = ceil(6.4) = 7 Pods. This does not prove that seven Pods will meet a service-level objective; application throughput, database capacity and latency must be validated.

A PodDisruptionBudget can limit voluntary simultaneous disruption, while topology spread constraints and anti-affinity can reduce the risk of placing all replicas on one failure domain. These controls cannot compensate for insufficient cluster capacity.

6. Helm and Reusable Application Packaging

Helm packages Kubernetes manifests into charts. A chart normally contains Chart.yaml, a values.yaml file and templates. Values separate stable templates from environment-specific choices.

A simple values file might contain:

image:
  repository: registry.example.com/orders/api
  digest: sha256:REPLACE_ME
replicaCount: 3
service:
  port: 80
resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: "1"
    memory: 512Mi

A template can construct an immutable image reference:

image: "{{ .Values.image.repository }}@{{ .Values.image.digest }}"

Helm improves consistency and release history, but templating can obscure the final manifests. Teams should run helm lint, render templates in CI, validate them against cluster schemas and review the rendered output. Avoid putting credentials directly in values files. Use a secrets integration or an encrypted and access-controlled mechanism.

Helm release state and chart provenance require governance. A chart repository should be trusted, dependencies should be pinned and updated, and chart upgrades should be tested for both API compatibility and behavioural effects. A template that renders successfully may still create an unsafe policy or an unavailable workload.

7. CI/CD Pipeline for Container Orchestration

A robust pipeline separates build, verification, promotion and deployment concerns:

1. Source validation: run formatting checks, unit tests and static analysis.
2. Dependency analysis: inspect lockfiles and licence obligations.
3. Image build: build in an isolated environment with controlled inputs.
4. Image tests: execute unit, integration and container smoke tests.
5. Artifact analysis: generate an SBOM, scan for vulnerabilities and check policy.
6. Signing and provenance: attest the source revision, builder and test evidence.
7. Registry push: publish immutable content with traceable metadata.
8. Deployment validation: render Helm, validate policies and perform a dry run.
9. Environment promotion: deploy the same digest to a controlled environment.
10. Progressive release: use a canary or rolling strategy with measured gates.
11. Verification and rollback: inspect metrics, logs and traces; revert or pause if objectives fail.

CI credentials should have minimal scope and short lifetimes. The pipeline identity should be able to push only to intended repositories and deploy only to intended namespaces or clusters. Pull requests should not receive production credentials merely because they run code.

8. Progressive Delivery

A standard rolling update replaces old Pods gradually. It is simple and often sufficient, but traffic is not always divided precisely by version. A canary deployment sends a controlled proportion of traffic to a new version and compares error rate, latency, saturation and business indicators against the stable version. A blue-green deployment maintains two environments and switches traffic between them; it can simplify rollback but may temporarily require more capacity.

Progressive delivery requires explicit gates. For example, a release may pause if the five-minute error rate exceeds an agreed threshold, if p95 latency violates the service-level objective, or if a critical business transaction fails. Thresholds should be based on service behaviour and measurement quality, not arbitrary numbers. Automated rollback is valuable but must avoid oscillation; a short-lived metric spike should not repeatedly alternate versions.

Observability must distinguish versions through labels or trace attributes. Otherwise, a service-wide metric may hide a failing canary. Rollback also requires database compatibility: an application may be reversible only if schema changes support both old and new versions during the transition.

9. Illustrative Case: Online Order Service

Background

An online retailer operates an order API and deploys several times each week. Its initial process built images on a developer laptop, used a mutable latest tag and applied hand-edited manifests to a production cluster.

Problem

A release introduced a dependency vulnerability and a readiness defect. The team could not identify the exact image in production, and the deployment sent traffic to Pods before database connections were ready. A manual rollback restored the previous application image but not a compatible database migration.

Analysis

The failures were systemic rather than purely coding errors. Artifact identity was weak, image construction was not reproducible, deployment configuration was not reviewed, health semantics were incorrect and the database migration lacked backward compatibility. The organisation also lacked a reliable evidence trail for who promoted the release.

Approach and solution

The team adopted the following design:

• A multi-stage Dockerfile builds a non-root runtime image.
• Dependency lockfiles, base-image digests and source revisions are recorded.
• CI creates an SBOM, scans the image and signs the digest.
• The registry prevents release-tag mutation and records promotions.
• Helm templates define the Deployment, Service, probes, resource settings and NetworkPolicy.
• Production values contain the approved image digest, not a mutable tag.
• Readiness checks local application readiness; liveness checks process progress.
• Three replicas are spread across failure domains where capacity permits.
• An HPA uses measured utilisation and is bounded by a tested maximum.
• Database changes follow an expand–migrate–contract approach: additive schema changes are deployed first, application code is then changed, and obsolete fields are removed only after old versions are gone.
• A canary receives limited traffic and is evaluated using version-labelled latency, errors and order-completion metrics.

Outcome

The organisation can identify the exact artifact, reproduce the build process, restrict deployment authority and stop a rollout before all users are affected. Rollback is safer because application and database compatibility are considered together. The process takes more design effort than a manual deployment, but it reduces uncertainty and makes failures diagnosable.

Learning points

1. Kubernetes availability depends on application semantics, not only replica count.
2. Immutable artifacts improve both security and rollback.
3. Health probes must represent distinct operational questions.
4. Progressive delivery is meaningful only when telemetry is version-aware.
5. Database evolution is part of deployment engineering.

10. Limitations, Governance and Emerging Practice

Containers do not provide perfect isolation, and Kubernetes adds a large administrative surface. Misconfigured RBAC, exposed dashboards, excessive service-account permissions, privileged containers and unbounded network access create serious risks. Cluster upgrades, admission policies, node patching and backup restoration require ownership and testing.

Governance should address data classification, residency, audit retention, software licences, vulnerability exceptions and separation of duties. A security exception should name an owner, rationale, compensating controls and expiry date; “accepted forever” is not a meaningful risk treatment.

Supply-chain attacks have increased the importance of provenance, dependency pinning, isolated builders and policy enforcement. Admission controllers can reject unsigned images, images with disallowed registries or workloads lacking resource requests. These controls should fail clearly and provide a safe path for remediation.

AI-assisted code and configuration generation can accelerate authoring, but generated Dockerfiles and manifests require human review. AI may reproduce insecure defaults, invent unsupported fields or expose sensitive prompts and context. Teams should validate generated output against schemas, security policies and runtime tests; responsibility remains with the engineering organisation.

Best practice is therefore contextual rather than absolute: use the simplest platform that satisfies reliability and governance needs, measure actual outcomes, automate repeatable checks and retain a human decision process for risk acceptance.

Chapter Summary

Containers package applications into reproducible runtime artifacts, while Kubernetes maintains desired operational state through Pods, controllers, Services and policies. Secure delivery begins with minimal, non-root, multi-stage images, locked dependencies, immutable references and controlled registries. Kubernetes Deployments provide rolling replacement; Services provide stable discovery; ConfigMaps and Secrets externalise configuration; probes distinguish startup, readiness and liveness; resource requests and limits support scheduling and protection; autoscaling responds to measured demand.

Helm packages reusable Kubernetes resources, but rendered output and dependencies must be validated. A mature CI/CD pipeline builds once, tests, scans, generates provenance, signs and promotes the same artifact. Progressive delivery uses canary, rolling or blue-green techniques with version-aware observability and explicit rollback criteria. Finally, reliable orchestration requires governance of identities, networks, upgrades, data migrations, vulnerabilities and operational evidence.

Key Terms

Container: An isolated process environment sharing the host kernel.

Image: An immutable package of filesystem layers and runtime metadata.

Layer: A reusable filesystem change contributing to an image.

Registry: A service that stores and distributes container artifacts.

Digest: A content-derived identifier for an image or artifact.

Pod: Kubernetes’ smallest schedulable unit.

Deployment: A controller-managed resource for declarative Pod rollouts.

ReplicaSet: A controller that maintains a specified number of matching Pods.

Service: A stable network endpoint selecting Pods by labels.

ConfigMap: Kubernetes storage for non-secret configuration.

Secret: Kubernetes resource for sensitive data, requiring additional protection.

Readiness probe: A check controlling whether a Pod receives traffic.

Liveness probe: A check indicating whether a process should be restarted.

Startup probe: A check that protects slow-starting applications during initialisation.

Resource request: A declared amount used primarily for scheduling.

Resource limit: A configured upper bound on container resource consumption.

Helm: A package manager and templating system for Kubernetes resources.

Canary release: A progressive deployment exposing a new version to limited traffic.

SBOM: A software bill of materials listing components in an artifact.

Provenance: Evidence describing how an artifact was built and tested.

Review Questions

Short Answer Questions

1. Why is deploying an image digest generally safer than deploying the latest tag?
2. Distinguish a readiness probe from a liveness probe.
3. What is the purpose of a Kubernetes Service?
4. State two security advantages of a multi-stage image build.
5. Why should database migrations be considered part of progressive delivery?

Long Answer Questions

1. Explain the complete path from source code to a production Kubernetes workload, including security controls at each stage.
2. Describe how Deployments, ReplicaSets, Pods and Services cooperate during a rolling update.
3. Evaluate the benefits and limitations of Helm for multi-environment application delivery.
4. Discuss how resource requests, limits, health probes and autoscaling interact in a production workload.
5. Compare rolling, canary and blue-green delivery, including the conditions under which each is appropriate.

Analytical / Application Questions

1. A Deployment has four Pods averaging 80% CPU utilisation against a target of 50%. Using the simplified scaling relationship in this chapter, calculate the conceptual target replica count and explain why the result is only a planning approximation.
2. A team reports that a canary has a low error rate but high order-processing latency. Design a release decision using version-labelled telemetry, rollback criteria and investigation steps.
3. Review a proposed policy that permits every workload in a namespace to access every Service and gives all Pods the default service account. Identify the risks and propose layered controls.

Multiple Choice Questions with Answers

Question 1

Which identifier most reliably fixes the exact container image content used in a production deployment?

A. The mutable latest tag

B. A content digest

C. The exposed container port

D. The Kubernetes namespace

Correct Answer: B. A content digest

Question 2

What is the primary purpose of a multi-stage container build?

A. To expose more network ports

B. To combine several production databases

C. To exclude build tools from the runtime image

D. To replace Kubernetes Services

Correct Answer: C. To exclude build tools from the runtime image

Question 3

Which Kubernetes object provides a stable endpoint for a changing set of Pods?

A. Service

B. ConfigMap

C. ReplicaSet

D. Namespace

Correct Answer: A. Service

Question 4

What should a readiness probe primarily determine?

A. Whether the image was signed

B. Whether the Pod should receive traffic

C. Whether the node requires patching

D. Whether a registry tag is immutable

Correct Answer: B. Whether the Pod should receive traffic

Question 5

Four Pods average 80% CPU utilisation and the target is 50%. Using n_target = ceil(n_current × u_current ÷ u_target), what is the conceptual target?

A. 5 Pods

B. 6 Pods

C. 7 Pods

D. 8 Pods

Correct Answer: C. 7 Pods

Question 6

Which statement about Kubernetes Secrets is correct?

A. Base64 encoding alone makes them confidential

B. They never require access control

C. They should be protected with encryption and restrictive access

D. They must be embedded in the container image

Correct Answer: C. They should be protected with encryption and restrictive access

Question 7

What is a principal benefit of promoting the same image digest across environments?

A. It guarantees zero application defects

B. It improves artifact traceability and consistency

C. It removes the need for testing

D. It automatically scales the cluster

Correct Answer: B. It improves artifact traceability and consistency

Question 8

What does Helm primarily provide?

A. Hardware virtualisation

B. Kubernetes packaging and templating

C. A replacement for container runtimes

D. A database transaction engine

Correct Answer: B. Kubernetes packaging and templating

Question 9

Which release method exposes a new version to a controlled portion of traffic before wider release?

A. Canary delivery

B. Image garbage collection

C. Namespace deletion

D. Static linking

Correct Answer: A. Canary delivery

Question 10

Why should a production container normally avoid running as root?

A. Root containers cannot use HTTP

B. Non-root execution reduces potential privileges after compromise

C. Root is required only for health probes

D. Non-root containers automatically encrypt secrets

Correct Answer: B. Non-root execution reduces potential privileges after compromise