Testing, CI/CD and Automation

Introduction

Containerization becomes operationally valuable when it is connected to a repeatable delivery system. Building an image manually on a developer workstation may demonstrate that an application works, but it does not establish that the same result can be reproduced, tested, secured, published and deployed reliably. Continuous integration and continuous delivery provide that discipline.

A Docker-oriented CI/CD pipeline converts a version-control change into an auditable sequence of activities: source retrieval, dependency installation, image construction, automated testing, security inspection, registry publication and deployment. Infrastructure-as-code (IaC) extends the same principle to environments and deployment configuration. Together, these practices reduce configuration drift, shorten feedback cycles and make recovery more predictable.

This chapter explains how to design such pipelines at diploma level, with emphasis on practical implementation, governance and rollback.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish continuous integration, continuous delivery and continuous deployment;
• design tests for containerized applications and images;
• construct a multi-stage Docker build suitable for automation;
• apply quality and security gates to a pipeline;
• publish immutable image artifacts to a registry;
• connect deployment automation with IaC and version control;
• select and implement an appropriate rollback strategy; and
• evaluate operational, security and governance risks in Docker delivery systems.

1. The Delivery Pipeline as a Control System

A pipeline is more than a sequence of shell commands. It is a control system that accepts a change and produces evidence about whether that change is safe to release. Each stage should have a defined input, output, responsibility and failure condition.

A typical flow is:

Figure: A container CI/CD pipeline with a feedback path for rollback or fix-forward.

The stages are commonly implemented as follows:

Continuous integration means that changes are frequently merged and automatically validated. Continuous delivery means that the software is always maintained in a releasable state, although a human may approve production release. Continuous deployment automatically releases changes that pass all required controls. These terms describe delivery practices, not particular tools.

2. Version Control and Artifact Identity

The pipeline should be triggered by version-controlled changes. Application source, Dockerfiles, dependency lock files, pipeline definitions and IaC should be reviewed and stored together where practical. This creates a history from a deployed workload back to its source.

A mutable tag such as latest is unsuitable as the sole production identifier because it can point to different content over time. A stronger identity is an image digest, for example sha256:..., which is calculated from image content. A useful human-readable tag might combine a release version and commit identifier, such as 2.4.1-a81f3c2; the digest remains the authoritative identity.

A robust promotion model builds once and promotes the same image through environments. Rebuilding separately for testing and production can introduce differences in dependencies, timestamps or build context. Configuration should normally be supplied at deployment time rather than baked into the image.

3. Building Images for Automation

A production Dockerfile should be deterministic, minimal and understandable. Multi-stage builds separate compilation tools from the runtime image. The following illustrative example uses a Python application:

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS build
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
COPY src/ ./src/

FROM python:3.12-slim AS runtime
RUN useradd --create-home --uid 10001 appuser
WORKDIR /app
COPY --from=build /install /usr/local
COPY --from=build /build/src ./src
USER 10001
ENV PYTHONDONTWRITEBYTECODE=1
EXPOSE 8080
CMD ["python", "-m", "src.app"]

Important practices include:

• pinning base-image versions, preferably by digest in high-assurance environments;
• using a .dockerignore file to exclude credentials, caches and unnecessary files;
• ordering layers so stable dependency steps can be cached;
• running as a non-root user where feasible;
• avoiding secrets in Dockerfile instructions and image layers;
• defining a health endpoint appropriate to the service; and
• recording build metadata without embedding confidential information.

Build arguments and environment variables have different purposes. An ARG is available during image construction and is not a secure secret store. An ENV becomes part of the image configuration and may be visible to users. Credentials should be injected through a CI secret manager or deployment platform.

BuildKit can improve caching, parallelism and secret handling. However, caching must not bypass necessary validation. A cached dependency layer is useful only when its inputs, especially lock files, are accurately represented in the build context.

4. Testing Containerized Applications

Testing should occur at multiple levels because no single test reveals every class of defect.

4.1 Unit tests

Unit tests check small components in isolation. They should be fast and deterministic, making them suitable for every commit. External services are generally replaced with controlled test doubles, although excessive mocking can hide integration defects.

4.2 Integration tests

Integration tests verify interaction with databases, queues, APIs or filesystems. Containers make ephemeral dependencies practical: a pipeline can start a temporary database, run migrations and tests, then destroy the environment. Test data must be isolated so parallel jobs do not interfere.

4.3 Image and container tests

These tests verify operational assumptions, including:

• the expected process starts with the intended command;
• required ports and files exist;
• the process runs without root privileges;
• environment variables are handled safely;
• the image contains no unexpected development tools; and
• the service responds correctly to a health check.

A container being in the running state does not prove that the application is ready. Readiness checks should represent the ability to serve traffic, while liveness checks should indicate whether a restart may be appropriate. A poorly designed liveness check can cause restart loops during a temporary dependency failure.

4.4 Contract and end-to-end tests

Contract tests verify that an API provider and consumer agree on request and response behavior. End-to-end tests exercise a complete user journey and are valuable but usually slower and more fragile. They should be focused on business-critical paths rather than used as the only test category.

4.5 Test quality and coverage

Coverage is a measurement of exercised code, not proof of correctness. A high percentage can coexist with weak assertions. Teams should combine coverage thresholds with mutation testing, boundary cases, failure-path tests and review of test relevance.

5. Security and Quality Gates

A gate is a rule that determines whether a pipeline may proceed. Gates should be explicit, versioned and proportionate to risk. Typical checks include:

1. dependency scanning for known vulnerabilities;
2. static analysis of source code and configuration;
3. secret detection in source and image layers;
4. image scanning for operating-system and language-package vulnerabilities;
5. Dockerfile linting and policy checks;
6. software bill of materials (SBOM) generation; and
7. signature or provenance verification.

A vulnerability scanner reports findings; it does not automatically determine exploitability or business risk. A practical policy distinguishes severity, reachability, exposure, available fixes and compensating controls. Exceptions should have an owner, justification, expiry date and approving authority.

The pipeline itself is a privileged system. It may access source code, registries, cloud accounts and deployment credentials. Therefore:

• use short-lived, least-privilege credentials;
• protect the main branch and pipeline configuration;
• isolate untrusted pull-request jobs from production secrets;
• log administrative actions without exposing tokens;
• restrict who may approve security exceptions; and
• separate build, release and production privileges where risk warrants it.

Supply-chain security is improved by generating provenance describing how an artifact was built, maintaining an SBOM, signing images and verifying signatures before deployment. These mechanisms support trust decisions but do not replace secure source review or runtime hardening.

6. Registry Publishing and Promotion

A registry stores and distributes image artifacts. A pipeline should authenticate to the registry using a service identity, push only after required gates pass, and retain enough metadata to answer: who built this image, from which source, with which dependencies and under which policy?

A recommended sequence is:

docker buildx build \
  --tag registry.example.com/orders:${GIT_COMMIT} \
  --label org.opencontainers.image.revision=${GIT_COMMIT} \
  --push .

docker inspect registry.example.com/orders:${GIT_COMMIT}

The exact commands vary by CI platform. In a real pipeline, the commit variable must be supplied by the CI system and validated. Production deployment should reference the resulting digest rather than assume that a tag has not changed.

Registry governance includes retention rules, access control, replication, vulnerability re-scanning and protection against deletion of release artifacts. Images should be removed according to a documented lifecycle, but currently deployed and rollback-required versions must be retained.

7. Infrastructure as Code and Deployment Automation

IaC represents infrastructure and deployment configuration in declarative, version-controlled files. Declarative configuration states the desired result; the IaC engine determines the actions needed to approach that result. Examples include Terraform for infrastructure and Kubernetes manifests or Helm charts for application deployment.

Good IaC practice includes:

• peer review and automated validation;
• separate state management with locking and access control;
• explicit environment differences rather than hidden manual changes;
• secret references rather than plaintext secret values;
• plan or preview steps before applying changes; and
• drift detection and documented reconciliation procedures.

A deployment should be observable. At minimum, record the image digest, configuration revision, deployment initiator, time and result. Health checks, logs, metrics and traces should be connected to the release identifier.

A simple pipeline outline is:

stages: [validate, test, build, scan, publish, deploy]

validate:
  script:
    - dockerfile-lint Dockerfile
    - terraform fmt -check
    - terraform validate

test:
  script:
    - pytest --junitxml=test-results.xml
    - docker compose -f compose.test.yml up --abort-on-container-exit --exit-code-from tests

build:
  script:
    - docker buildx build --tag "$IMAGE:$CI_COMMIT_SHA" --load .

scan:
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMAGE:$CI_COMMIT_SHA"

publish:
  script:
    - docker push "$IMAGE:$CI_COMMIT_SHA"
  only: [main]

deploy:
  script:
    - terraform apply -auto-approve -var="image_digest=$IMAGE_DIGEST"
  when: manual

This is illustrative rather than a universally valid configuration. Syntax, authentication and variable names differ between systems. In professional use, the scan must inspect the image actually intended for publication, and the deployment must receive the verified digest produced by the publish job.

8. Release Strategies and Rollback

Deployment strategy should match application architecture, traffic patterns and recovery requirements.

Rollback means returning to a known-good application version and configuration. It is not always safe to reverse database changes. Backward-compatible schema evolution is therefore important: add new fields before using them, support both versions during transition, migrate data carefully and remove obsolete fields only after rollback is no longer required.

Rollback triggers should be objective where possible: failed readiness, elevated error rate, unacceptable latency, resource exhaustion or a critical security alert. A fix-forward decision may be preferable when data migrations are irreversible or the defect is quickly correctable. Both rollback and fix-forward should be rehearsed.

Operational recovery can be discussed using two service objectives. Recovery time objective (RTO) is the maximum acceptable time to restore service. Recovery point objective (RPO) is the maximum acceptable amount of data loss measured in time. These are organizational requirements, not Docker properties. A rollback plan should demonstrate that the chosen process can meet them under realistic conditions.

9. Applied Case: Illustrative Online Retail Service

Background

An online retailer operates an order API, a payment service and a PostgreSQL database. Releases were built manually and deployed with a mutable latest tag. A dependency update caused a production failure, and the team could not immediately identify which image was running.

Problem

The organization needed repeatable builds, evidence-based security checks, controlled promotion and a rapid recovery procedure without exposing payment credentials to build jobs.

Analysis

The main weaknesses were artifact ambiguity, insufficient automated integration testing, excessive pipeline privileges and a database migration that was not rollback-compatible. The team also lacked a reliable connection between a release and operational metrics.

Approach

The team adopted the following design:

1. Every pull request ran unit tests, API contract tests, Dockerfile linting and secret detection.
2. An ephemeral PostgreSQL container supported integration tests.
3. A multi-stage Dockerfile created a non-root runtime image.
4. The image was scanned and an SBOM was generated.
5. Successful builds were pushed with the commit identifier and signed; deployment referenced the digest.
6. Terraform and Kubernetes configuration were reviewed in version control.
7. A canary deployment received 5% of traffic for ten minutes, with automatic checks on error rate and latency.
8. Database changes followed an expand-and-contract pattern.
9. Production credentials were available only to the deployment identity, not to pull-request jobs.
10. The previous digest and configuration remained available for rollback.

Outcome

The retailer could identify the exact source and dependencies of a running image. A failed canary was stopped before full rollout, and the service was returned to the previous image without rebuilding it. The process did not eliminate defects, but it reduced their exposure and improved recovery confidence.

Learning Points

The case demonstrates that CI/CD quality depends on the whole chain. A secure image is insufficient if deployment is manual; automated tests are insufficient if the deployed artifact differs from the tested artifact; and rollback is insufficient if data changes cannot be reversed. Traceability, least privilege, observability and compatible migrations must be designed together.

10. Challenges, Limitations and Professional Governance

Automation can amplify mistakes. A defective pipeline may deploy a defect faster and to more environments. Teams should therefore treat pipeline code as production software, test it, review it and monitor it.

Common limitations include flaky tests, slow integration environments, false-positive scans, unavailable registries, non-reproducible external dependencies and excessive reliance on privileged runners. Parallel execution improves speed but can create race conditions and resource contention. Caching improves performance but can conceal stale inputs if cache keys are poorly designed.

Governance should address software licensing, vulnerability disclosure, audit retention, separation of duties and data protection. Logs may contain personal or payment-related data and should be minimized, access-controlled and retained according to policy. Automated gates should support accountable decisions rather than become unreviewed substitutes for professional judgment.

Artificial intelligence can assist with test generation, log classification and pipeline suggestions, but generated changes require review. AI tools may transmit source code or secrets to external services, so data handling, model governance and human approval are essential.

11. Best-Practice Checklist

Before declaring a container pipeline production-ready, verify that:

• source, Dockerfiles, IaC and pipeline definitions are version-controlled;
• builds are reproducible and produce immutable identities;
• unit, integration and relevant acceptance tests run automatically;
• image, dependency, secret and configuration checks are enforced;
• exceptions are documented and time-limited;
• images are signed or otherwise associated with verifiable provenance;
• registry and deployment credentials follow least privilege;
• the same approved artifact is promoted across environments;
• deployments include health checks and observability;
• rollback and database recovery procedures are tested; and
• release evidence is retained for operational and audit needs.

Chapter Summary

Container CI/CD integrates version control, reproducible image construction, automated testing, security analysis, registry publication and deployment. Multi-stage builds and non-root execution improve image quality, while layered tests expose different classes of defect. Security gates, SBOMs, provenance and image signing strengthen supply-chain assurance, but require risk-aware governance. IaC makes environments reviewable and reduces drift. Immutable image digests support traceability and reliable promotion. Rolling, blue-green and canary strategies provide different balances of capacity, speed and risk. Effective rollback also depends on compatible database migrations, observability and rehearsed recovery procedures.

Key Terms

Continuous integration: Frequent integration of changes with automated validation.

Continuous delivery: Keeping software in a releasable state through automation.

Continuous deployment: Automatically releasing changes that pass required controls.

Image digest: A content-derived identifier for an image artifact.

Container registry: A service that stores and distributes container images.

Multi-stage build: A Docker build using multiple stages to separate build and runtime content.

BuildKit: Docker’s modern build engine supporting efficient builds and advanced features.

Unit test: A test of a small software component in isolation.

Integration test: A test of interactions among components or external services.

Contract test: A test that verifies an agreed interface between services.

Security gate: A rule that must be satisfied before pipeline progression.

SBOM: A software bill of materials listing components and dependencies.

Provenance: Evidence describing how an artifact was produced.

Infrastructure as code: Declarative, version-controlled management of infrastructure.

Configuration drift: Unintended divergence between declared and actual environments.

Canary deployment: Gradual release to a limited portion of users or infrastructure.

Rollback: Returning to a previously known-good version or configuration.

RTO: Maximum acceptable time to restore service.

RPO: Maximum acceptable time-based amount of data loss.

Review Questions

Short Answer Questions

1. What is the difference between continuous delivery and continuous deployment?
2. Why is an image digest more reliable than the latest tag for production identity?
3. State two advantages of a multi-stage Docker build.
4. What is the purpose of an SBOM?
5. Why should database migrations be designed for compatibility with rollback?

Long Answer Questions

1. Explain how unit, integration, image and end-to-end tests complement one another in a container pipeline.
2. Describe a secure process for building, scanning, signing and publishing a Docker image.
3. Explain how IaC, version control and drift detection improve deployment reliability.
4. Compare rolling, blue-green and canary deployment strategies.
5. Discuss the governance risks of giving CI jobs broad access to production credentials.

Analytical / Application Questions

1. A team tests an image tagged build-17 but rebuilds from the same source for production. Analyse the risks and propose a promotion model.
2. A canary release shows increased latency but no increase in error rate. What evidence should be examined before continuing, rolling back or fixing forward?
3. Design a pipeline for a service that requires a database, external API and sensitive deployment credentials. Identify stages, gates, artifact identity and isolation controls.

Multiple Choice Questions with Answers

Question 1

Which artifact identity is most appropriate for an immutable production deployment?

A. The latest tag

B. A registry image digest

C. The developer’s local image name

D. The Dockerfile filename

Correct Answer: B. A registry image digest

Question 2

What is the primary purpose of a multi-stage Docker build?

A. To expose more container ports

B. To eliminate the need for testing

C. To separate build tools from the runtime image

D. To store credentials in a build layer

Correct Answer: C. To separate build tools from the runtime image

Question 3

Which test most directly verifies interaction with a real temporary database?

A. Unit test

B. Integration test

C. Static analysis

D. Dockerfile lint

Correct Answer: B. Integration test

Question 4

What does an SBOM provide?

A. A list of software components and dependencies

B. A replacement for application monitoring

C. A network route to the registry

D. A database rollback command

Correct Answer: A. A list of software components and dependencies

Question 5

Which practice best protects production credentials in pull-request pipelines?

A. Store them in the Dockerfile

B. Use the same administrator token for every job

C. Expose them to all branches

D. Restrict them to trusted deployment jobs using least privilege

Correct Answer: D. Restrict them to trusted deployment jobs using least privilege

Question 6

Which deployment strategy initially exposes a release to a limited audience?

A. Canary

B. Recreate

C. Full replacement

D. Manual compilation

Correct Answer: A. Canary

Question 7

Why should production normally use the same image that passed testing?

A. It makes image names shorter

B. It prevents all runtime failures

C. It avoids differences introduced by rebuilding

D. It removes the need for a registry

Correct Answer: C. It avoids differences introduced by rebuilding

Question 8

What does RTO measure?

A. The acceptable amount of data loss

B. The maximum acceptable service restoration time

C. The number of test cases executed

D. The image compression ratio

Correct Answer: B. The maximum acceptable service restoration time

Question 9

Which statement about vulnerability scanning is correct?

A. A scanner’s finding automatically proves exploitability

B. Scanning removes vulnerabilities from an image

C. Findings should be assessed according to risk and context

D. Scanning replaces secure coding

Correct Answer: C. Findings should be assessed according to risk and context

Question 10

What is configuration drift?

A. A deliberate version tag

B. Unintended difference between declared and actual environments

C. A type of unit test

D. A registry authentication method

Correct Answer: B. Unintended difference between declared and actual environments