9. Containers, CI/CD and Cloud-Native Delivery

Introduction and Context

Backend development does not end when an application passes its tests. A service must be packaged consistently, configured safely, delivered repeatedly, observed in operation, and recovered when deployment introduces a defect. These concerns become particularly significant in systems built with Node.js, Django, or Spring Boot, where applications depend on language runtimes, operating-system libraries, databases, message brokers, environment variables, and external services.

Containerization and continuous delivery address the gap between software development and reliable operation. A container packages an application with the files and runtime assumptions required to execute it. A CI/CD pipeline turns source-code changes into tested, versioned, deployable artifacts. Cloud-native platforms extend these practices through orchestration, automated scaling, health management, infrastructure automation, and progressive release strategies.

The objective is not merely to place an application in Docker or Kubernetes. The deeper objective is to create a predictable delivery system. Predictability depends on immutable artifacts, explicit configuration, controlled environments, observable deployments, secure secrets management, and the ability to reverse a change quickly. This chapter examines these ideas as an integrated engineering discipline.

Learning Objectives

After completing this chapter, a learner should be able to:

• Explain container images, containers, registries, runtimes, and orchestration.
• Design production-oriented Dockerfiles for Node.js, Django, and Spring Boot services.
• Construct CI/CD workflows involving testing, image creation, security checks, promotion, and deployment.
• Distinguish configuration from secrets and apply appropriate environment-management practices.
• Explain Kubernetes objects, declarative operations, services, ingress, probes, and autoscaling.
• Compare rolling, blue-green, and canary release strategies.
• Design rollback, resilience, and disaster-recovery mechanisms.
• Evaluate infrastructure-as-code and major cloud deployment models.
• Identify technical, security, governance, and organizational risks in cloud-native delivery.

From Application Package to Deployable Artifact

A traditional deployment may install a runtime, copy source code, configure operating-system packages, and start a process directly on a server. This approach can produce differences between a developer workstation, a test server, and production. Such differences are often called configuration drift. Containers reduce this problem by building an image from a declared recipe. The image is versioned and promoted through environments without being rebuilt for each environment.

A container image consists of immutable layers containing an application, dependencies, and metadata. A container is a running instance of an image, isolated through operating-system mechanisms. The image is not a virtual machine: containers share the host kernel and generally start faster and use fewer resources. A registry stores and distributes images. A runtime such as containerd or Docker Engine creates and manages running containers.

Containerization does not automatically provide security or reliability. An image may contain vulnerable libraries, excessive privileges, embedded credentials, or unnecessary tools. A sound image is small, reproducible, traceable to source code, and executed as a non-root user whenever possible.

Designing a Production Docker Image

A Dockerfile should express a repeatable build. For a Node.js service, dependency installation should be separated from source copying where practical so that Docker can reuse layers. A multi-stage build can compile TypeScript or frontend assets in one stage and copy only production artifacts into a smaller runtime stage. Django applications may require Python wheels, system libraries, static-file collection, and a WSGI server such as Gunicorn. Spring Boot applications can be packaged as executable JAR files and run on a carefully selected Java runtime image.

A generic multi-stage pattern is conceptually as follows:

```dockerfile

FROM node:20-bookworm AS build

WORKDIR /app

COPY package*.json ./

RUN npm ci

COPY . .

RUN npm run build && npm prune --omit=dev

FROM node:20-bookworm-slim

WORKDIR /app

ENV NODE_ENV=production

COPY --from=build /app/package*.json ./

COPY --from=build /app/node_modules ./node_modules

COPY --from=build /app/dist ./dist

USER node

CMD ["node", "dist/server.js"]

```

The exact base image is a design choice. Minimal images reduce attack surface and transfer time, but overly minimal images can complicate debugging and native-library compatibility. Teams should pin versions, preferably by digest for high-assurance supply chains, scan dependencies and images, maintain a trusted base-image policy, and generate software bills of materials where required.

A container should normally run one principal service process, write logs to standard output and error, receive configuration through its environment or mounted configuration mechanism, and avoid storing durable application data in its writable layer. Databases, uploaded files, and queues require managed persistence or explicitly designed volumes.

Configuration, Secrets, and Environment Promotion

Configuration describes behavior that may differ between environments: database endpoints, feature flags, logging levels, cache addresses, and external API URLs. Secrets are sensitive values such as passwords, tokens, private keys, and signing credentials. Treating every value as a secret creates unnecessary operational complexity, while treating secrets as ordinary configuration exposes the organization to compromise.

The twelve-factor approach, although not a complete deployment methodology, provides useful principles: configuration should be externalized, logs should be treated as event streams, processes should be stateless where feasible, and build, release, and run stages should be separated. A release should combine an immutable artifact with environment-specific configuration rather than rebuild source code for each environment.

Secrets must not be committed to source control, placed in Dockerfiles, printed in pipeline logs, or embedded in client-side bundles. They should be held in a dedicated secret manager or a protected CI/CD facility, accessed through short-lived credentials where possible, rotated, audited, and scoped according to least privilege. Kubernetes Secrets provide an interface for supplying sensitive values, but their existence does not by itself guarantee strong protection; encryption at rest, access control, and external secret-management integration may be necessary.

Environment management is also a governance problem. Development, testing, staging, and production should have clear purposes and promotion rules. Production-like testing is valuable, but test data must be protected and privacy obligations must be respected. Feature flags can decouple code deployment from feature activation, enabling controlled exposure and rapid deactivation without rebuilding the service.

Continuous Integration and Continuous Delivery

Continuous integration (CI) is the practice of frequently integrating changes into a shared repository and validating them automatically. Continuous delivery keeps the software in a releasable state, while continuous deployment automatically releases approved changes to production. These terms are related but not identical. An organization can practice continuous delivery with a human approval gate and still benefit from automated quality controls.

A mature pipeline commonly contains the following logical stages:

1. Source validation: formatting, linting, dependency checks, and static analysis.

2. Build: compilation, packaging, or dependency installation using a reproducible process.

3. Testing: unit, integration, contract, migration, and selected end-to-end tests.

4. Security and supply-chain checks: secret detection, dependency scanning, image scanning, and policy evaluation.

5. Artifact publication: creation of a uniquely tagged image and associated metadata.

6. Deployment: promotion of the same artifact to an environment.

7. Verification: health checks, smoke tests, telemetry inspection, and release approval where applicable.

8. Feedback or rollback: notification, incident response, or automated reversal when defined thresholds are exceeded.

The pipeline should build once and promote the same artifact. Rebuilding separately for staging and production undermines confidence because the artifacts may differ. Image tags such as `latest` are convenient but ambiguous; immutable tags based on a commit identifier or release version provide stronger traceability.

CI/CD quality is measured not only by pipeline speed. Useful delivery-performance measures include deployment frequency, lead time from change to deployment, change-failure rate, and time to restore service. These measures should be used for improvement rather than individual blame. A pipeline that deploys rapidly but frequently causes incidents is not effective delivery.

Testing in the Delivery System

Unit tests provide fast feedback on isolated logic. Integration tests examine boundaries such as databases, queues, and HTTP clients. Contract tests help ensure that independently deployed services agree on interfaces. End-to-end tests provide valuable confidence but are often slower and more fragile. A balanced test strategy places many fast tests early and reserves expensive tests for changes that require them.

Database migrations require special care. A migration should be versioned, repeatable in the intended context, and compatible with the running application during a transition. The expand-and-contract pattern first adds compatible structures, then deploys code that uses them, and only later removes obsolete structures. This approach supports rolling deployments in which old and new application instances coexist.

Kubernetes as a Declarative Delivery Platform

Kubernetes is an orchestration system that maintains a desired state for containerized workloads. Rather than instructing an operator to start a particular process on a particular machine, a user declares objects such as Deployments, Services, ConfigMaps, and Secrets. Controllers compare the desired state with the observed state and take corrective action.

A Pod is the basic scheduling unit and commonly contains one application container, although tightly coupled sidecars may be appropriate. A Deployment manages replicated Pods and supports controlled updates. A Service provides stable networking and load distribution to selected Pods. An Ingress or gateway exposes HTTP routes, often with TLS termination. A ConfigMap supplies non-sensitive configuration, while a Secret supplies sensitive values. A Namespace separates groups of resources and can support access and quota policies.

Kubernetes health probes have distinct purposes. A liveness probe indicates whether a container should be restarted. A readiness probe indicates whether it should receive traffic. A startup probe gives a slow-starting application time to initialize before liveness decisions begin. Poorly designed probes can cause restart loops or route traffic to an unready service. Health endpoints should be lightweight and should distinguish process health from dependency health; an application may be alive even when one optional dependency is unavailable.

A declarative manifest might specify replica count, resource requests and limits, labels, probes, and a rolling-update policy. Resource requests influence scheduling, while limits constrain usage. Without appropriate values, a service may be starved, evicted, or cause noisy-neighbor effects. Names, labels, and selectors must be designed consistently because a selector that matches the wrong Pods can route traffic incorrectly.

Autoscaling and Resilience

The Horizontal Pod Autoscaler adjusts replica count according to metrics such as CPU utilization or application-level indicators. Autoscaling is not a substitute for capacity planning. It requires correct resource requests, sufficient cluster capacity, sensible minimum and maximum values, and protection against oscillation. Queue depth, request latency, and throughput can sometimes be more meaningful than CPU alone.

Resilience is designed at multiple levels. Applications should use timeouts, bounded retries, exponential backoff with jitter, circuit breakers, bulkheads, idempotency, and graceful shutdown. Retries without limits can amplify an outage. A request that creates an order or charges a payment must be protected against duplicate execution through idempotency keys or equivalent mechanisms.

At the platform level, replicas should be distributed across failure domains where practical. Pod disruption budgets can reduce the risk of voluntary maintenance removing all replicas simultaneously. Persistent data requires backup, restoration testing, replication decisions, and a defined recovery point objective (RPO) and recovery time objective (RTO). High availability is not equivalent to disaster recovery: redundancy may handle a failed instance, while disaster recovery addresses larger regional, data, or operational failures.

Release Strategies and Rollback

A rolling release gradually replaces old instances with new ones. It uses resources efficiently and is the default for many Kubernetes Deployments, but old and new versions may coexist. Therefore, APIs, events, database schemas, and session behavior must remain compatible during the transition.

A blue-green release maintains two complete environments. Blue serves production traffic while green receives the new release. After validation, traffic switches to green. The method enables fast rollback by returning traffic to blue, but temporarily requires additional capacity and careful management of stateful components.

A canary release sends a small proportion of traffic to the new version and expands exposure when technical and business indicators remain healthy. Canary analysis may consider error rate, latency percentiles, saturation, conversion, or task completion. Randomly sending ten percent of traffic is not sufficient if the sample is unrepresentative or the affected user group is poorly understood. A canary should have a defined success criterion, observation window, abort threshold, and owner.

Rollback is a prepared operation, not an improvised reaction. Teams should know which previous artifact is safe, how traffic will be redirected, whether database changes are reversible, and how caches and queues will behave. In some cases, rolling forward with a corrective release is safer than rollback, particularly when irreversible schema changes have occurred. Every release should produce a traceable record of code version, image digest, configuration revision, approvals, and deployment outcome.

Infrastructure as Code and Cloud Deployment Models

Infrastructure as code (IaC) represents infrastructure in declarative or programmatic files that can be reviewed, versioned, tested, and applied repeatedly. Tools such as Terraform, Pulumi, CloudFormation, and Ansible support different models and scopes. IaC can provision networks, clusters, databases, identity policies, load balancers, and monitoring resources. A state file, where used, must be protected because it may contain sensitive values and determines what the tool believes exists.

IaC improves repeatability and auditability but does not eliminate risk. An incorrect security-group rule can be deployed at scale. State drift, unreviewed manual changes, provider-version changes, and destructive plans require controls. Pull-request review, policy-as-code, isolated environments, state locking, backups, and approval gates are appropriate safeguards.

Cloud deployment models represent different responsibility boundaries. In infrastructure as a service (IaaS), the customer manages virtual machines, operating systems, and much of the platform. In platform as a service (PaaS), the provider manages more of the runtime and operational substrate. Containers as a service and managed Kubernetes reduce cluster-management effort while retaining application and workload responsibilities. Serverless functions abstract servers further and suit event-driven, variable workloads, although execution limits, cold starts, vendor coupling, and observability must be considered.

The shared-responsibility model is central to cloud governance. A provider may secure the physical facilities and managed control plane, while the customer remains responsible for identity, application vulnerabilities, data classification, access policies, and correct configuration. Cloud-native architecture therefore requires cost governance as well as technical design. Idle resources, excessive log retention, cross-region traffic, oversized databases, and uncontrolled autoscaling can create significant costs.

Illustrative Case Study: Progressive Delivery of a Django Service

Background

A fictional company, LearnSphere, operates a Django REST service for course enrollment. The service uses PostgreSQL, Redis for short-lived caching, and object storage for uploaded documents. Releases were previously performed by copying files to a virtual machine. Incidents resulted from inconsistent Python dependencies, manual environment changes, and migrations executed at uncertain times.

Situation and Analysis

The engineering team identified four principal risks: non-reproducible builds, credentials stored in deployment scripts, insufficient health checks, and an inability to determine whether a release caused increased enrollment failures. The service also performed long-running report generation synchronously, causing request timeouts during peak periods.

Approach and Strategy

The team created a multi-stage Python image, pinned dependencies, ran the application as a non-root user, and moved static assets to object storage. CI executed linting, unit tests, integration tests against an ephemeral PostgreSQL service, dependency scanning, and image scanning. The pipeline published an image tagged with the source commit and recorded its digest.

Secrets moved to a managed secret store. The deployment system injected them at runtime, and access was granted through a workload identity rather than a long-lived cloud key. Kubernetes Deployments used readiness and liveness probes, resource requests, graceful termination, and at least three replicas distributed across worker nodes. A queue-based worker was introduced for reports, making the web process more predictable.

The release process used a canary. Two percent of enrollment traffic was directed to the new version for fifteen minutes. The promotion policy required error rate and p95 latency to remain within agreed thresholds and required no increase in payment-provider failures. If thresholds were exceeded, the routing rule reverted to the previous version. Database changes followed expand-and-contract: a nullable column was added first, application code was updated, data was backfilled, and obsolete behavior was removed in a later release.

Outcome

The process reduced environment differences and made the deployed artifact traceable to a commit. Operators could identify the release associated with an incident and redirect traffic without rebuilding. The asynchronous worker reduced web-request contention. The team also discovered that technical health alone was insufficient: business metrics were needed to detect a payment-provider integration problem that returned valid HTTP responses but produced failed enrollments.

Learning Points

The case demonstrates that containerization, CI/CD, secrets management, observability, schema compatibility, and progressive delivery are interdependent. No single tool solved the problem. Reliability emerged from explicit controls, measurable release criteria, and a rollback path tested before it was needed.

Risks, Governance, and Professional Practice

Common mistakes include using `latest` images, running containers as root, copying secrets into images, relying solely on unit tests, exposing administrative endpoints publicly, omitting resource limits, and treating Kubernetes as a replacement for application design. Another frequent error is deploying a highly available application while leaving the database as a single unprotected instance.

Supply-chain attacks make provenance important. Organizations should control who can publish images, sign or attest artifacts where appropriate, scan direct and transitive dependencies, review base-image updates, and restrict deployment to approved registries. Logs and traces may contain personal or confidential information; retention, redaction, access control, and data-residency requirements must be addressed. Deployment records should support audit without encouraging excessive collection of personal data.

Operational access should follow least privilege and separation of duties. Emergency access must be logged and reviewed, but governance should not make recovery impossible. Teams should conduct incident reviews focused on system improvement, document runbooks, test restoration procedures, and rehearse failure scenarios. Accessibility and inclusion also matter: internal deployment interfaces, alerts, and documentation should be usable by the full engineering team.

Current Developments and Best Practices

Cloud-native delivery is increasingly influenced by platform engineering, in which internal platforms provide standardized workflows for deployment, observability, identity, and service ownership. GitOps extends declarative operations by treating a version-controlled repository as the source of truth for desired environment state. Service meshes can provide traffic policy, mutual TLS, and telemetry, but they add operational complexity and should be adopted for a clearly justified need.

Software supply-chain frameworks increasingly emphasize provenance, signed artifacts, dependency transparency, and policy enforcement before deployment. OpenTelemetry provides a common approach to collecting traces, metrics, and logs, although instrumentation quality remains an application responsibility. Policy-as-code can prevent images with critical vulnerabilities, public storage, or excessive permissions from reaching production.

A practical set of best practices is therefore:

• Build immutable, minimal, traceable images and promote the same artifact.
• Externalize configuration and protect secrets through managed, auditable mechanisms.
• Make pipelines fast enough for frequent use but strong enough to detect meaningful failures.
• Use readiness, liveness, and startup checks appropriately, not mechanically.
• Define service-level objectives and use technical and business telemetry for release decisions.
• Design deployments and database changes for coexistence of old and new versions.
• Automate rollback or traffic reversal and test it regularly.
• Manage infrastructure through reviewed, state-protected code.
• Apply least privilege, vulnerability management, backup testing, and cost controls.
• Prefer the simplest platform that satisfies reliability, scale, compliance, and delivery needs.

Chapter Summary

Containerization packages backend services and their runtime assumptions into reproducible images, but production quality depends on secure construction, immutable versioning, and appropriate persistence. CI/CD connects source changes to automated validation, artifact publication, controlled promotion, verification, and recovery. Configuration should be externalized, while secrets require stronger protection, rotation, auditing, and least-privilege access.

Kubernetes provides declarative orchestration through objects such as Pods, Deployments, Services, probes, and autoscalers. Reliable cloud-native delivery also requires application-level resilience, compatible database evolution, observability, and tested recovery. Rolling, blue-green, and canary strategies offer different balances of cost, risk, speed, and rollback capability. Infrastructure as code improves repeatability and governance, while cloud models differ in their allocation of operational responsibility. Effective delivery is therefore a socio-technical system combining tools, architecture, security, measurement, and professional judgment.

Key Terms

Container image: An immutable, layered package containing application files, dependencies, metadata, and runtime assumptions.

Container: A running, isolated instance of a container image.

Container registry: A service that stores, versions, and distributes container images.

Multi-stage build: A Docker build using separate stages to compile or package software and create a smaller runtime image.

Continuous integration: Frequent integration of code changes with automated build and validation.

Continuous delivery: Maintaining software in a releasable condition through automated delivery processes.

Continuous deployment: Automatically releasing validated changes to a target production environment.

Artifact: A versioned output of a build, such as a container image or executable package.

Configuration: Non-secret values that control application behavior in a particular environment.

Secret: Sensitive information such as passwords, tokens, certificates, or private keys.

Kubernetes Pod: The smallest Kubernetes scheduling unit, containing one or more closely related containers.

Deployment: A Kubernetes object that manages replicated Pods and controlled updates.

Service: A stable Kubernetes networking abstraction that routes traffic to selected Pods.

Readiness probe: A health check determining whether an instance should receive traffic.

Liveness probe: A health check indicating whether an instance should be restarted.

Horizontal autoscaling: Adjusting the number of running instances according to measured demand or resource use.

Blue-green deployment: Operating two environments and switching traffic from the old environment to the new one.

Canary release: Gradually exposing a new version to a limited portion of traffic before wider release.

Infrastructure as code: Managing infrastructure through versioned, reviewable, executable definitions.

Recovery point objective: The maximum acceptable amount of data loss measured in time.

Recovery time objective: The target maximum duration for restoring a service after disruption.

Review Questions

Short Answer Questions

1. How does a container image differ from a virtual machine?

2. Why should production deployments use immutable artifact identifiers rather than only the `latest` tag?

3. Distinguish configuration from secrets and give one example of each.

4. What are the different purposes of Kubernetes readiness and liveness probes?

5. Why is database compatibility important during rolling deployments?

Long Answer Questions

1. Explain the stages of a mature CI/CD pipeline and discuss how each stage contributes to delivery confidence.

2. Describe the design principles of a production Docker image for a Node.js, Django, or Spring Boot service.

3. Explain Kubernetes Deployments, Services, configuration objects, probes, and autoscaling as parts of a delivery platform.

4. Compare rolling, blue-green, and canary deployments with respect to cost, risk, observability, and rollback.

5. Discuss how infrastructure as code, shared responsibility, identity management, and cost governance influence cloud-native operations.

Analytical / Application Questions

1. A new release passes unit tests but causes payment failures for five percent of users. Design a canary analysis and rollback procedure, including technical and business metrics.

2. A team wants to store database passwords in a Dockerfile because the container is private. Analyze the security and operational problems and propose a safer design.

3. During a rolling deployment, the new application requires a database column that the old version cannot tolerate. Diagnose the failure and design an expand-and-contract migration sequence.

22. Multiple Choice Questions with Answers

Question 1

Which practice most directly ensures that staging and production run the same software artifact?

A. Rebuilding the image separately in each environment

B. Promoting one immutable, uniquely identified artifact across environments

C. Using the latest base image during every deployment

D. Copying source files directly to production servers

Correct Answer: B. Promoting one immutable, uniquely identified artifact across environments

Question 2

A Kubernetes Pod is running, but it has not completed initialization and must not receive traffic. Which mechanism is most appropriate?

A. Readiness probe

B. Image registry

C. Horizontal Pod Autoscaler

D. Persistent volume

Correct Answer: A. Readiness probe

Question 3

Why is storing a password in a Dockerfile unsafe?

A. Dockerfiles cannot define environment variables

B. The password may become part of image layers and be retrievable

C. Containers cannot connect to databases using passwords

D. Kubernetes automatically deletes passwords from images

Correct Answer: B. The password may become part of image layers and be retrievable

Question 4

Which release strategy sends a small portion of traffic to a new version before wider exposure?

A. Blue-green release

B. Canary release

C. Full replacement release

D. Database migration

Correct Answer: B. Canary release

Question 5

What is the main purpose of a Kubernetes readiness probe?

A. To determine whether a container image is signed

B. To decide whether a Pod should receive traffic

C. To increase the number of worker nodes

D. To rotate application secrets

Correct Answer: B. To decide whether a Pod should receive traffic

Question 6

A team adds a new nullable database column, deploys code that supports both schemas, and removes the old field later. Which approach is this?

A. Expand-and-contract migration

B. Immutable infrastructure destruction

C. Blue-green database replacement

D. Horizontal autoscaling

Correct Answer: A. Expand-and-contract migration

Question 7

Which statement best describes continuous delivery?

A. Every commit must be released to production without approval

B. Software is kept in a releasable state through automated processes

C. Developers manually copy files to a server after testing

D. Only infrastructure changes are automated

Correct Answer: B. Software is kept in a releasable state through automated processes

Question 8

Why can unlimited retries worsen an outage?

A. Retries always reduce available memory permanently

B. Retries can amplify load on an already failing dependency

C. Retries prevent all requests from being logged

D. Retries disable Kubernetes scheduling

Correct Answer: B. Retries can amplify load on an already failing dependency

Question 9

Which property is most associated with infrastructure as code?

A. Unreviewed manual modification of production servers

B. Versioned and repeatable infrastructure definitions

C. Embedding credentials in application source code

D. Removing the need for access controls

Correct Answer: B. Versioned and repeatable infrastructure definitions

Question 10

A company needs rapid rollback and can temporarily operate two complete environments. Which strategy is most suitable?

A. Blue-green deployment

B. Single-instance deployment

C. Unmanaged rolling copy

D. Manual source replacement

Correct Answer: A. Blue-green deployment