7. Containers, Kubernetes and Cloud-Native Delivery

Introduction and Context

Continuous integration and continuous deployment (CI/CD) become substantially more complex when applications are distributed across containers, Kubernetes clusters, managed cloud services, microservices, and serverless platforms. In a traditional application, a pipeline may compile a binary, run tests, and deploy it to a server. In a cloud-native environment, the pipeline must also build and secure an image, publish it to a registry, create or update deployment manifests, manage configuration and secrets, coordinate dependencies, verify runtime health, and provide a reliable mechanism for rollback.

The central shift is from deploying a machine or application package to reconciling a desired system state. A container image describes an immutable application artifact; Kubernetes resources describe the intended runtime state; and a controller continuously attempts to make the actual state match that intent. This combination supports repeatability and elasticity, but it also introduces new failure modes, security responsibilities, and operational abstractions.

This chapter examines how CI/CD is designed for containerized and cloud-native systems. It connects secure image engineering, registries, Kubernetes delivery, Helm-style packaging, operators, service meshes, microservices, serverless platforms, and GitOps reconciliation into a coherent delivery model.

Learning Objectives

By the end of this chapter, learners should be able to:

• Explain how containers and Kubernetes change CI/CD architecture.
• Build secure, reproducible, and appropriately small container images.
• Design workflows involving registries, promotion, testing, signing, and deployment.
• Compare Kubernetes manifests, Helm-style packaging, operators, and GitOps.
• Select suitable delivery strategies for microservices and serverless systems.
• Evaluate service meshes, progressive delivery, observability, and rollback mechanisms.
• Identify technical, organizational, governance, and security risks in cloud-native delivery.

From Application Packages to Declarative Runtime State

A container image packages application code, runtime libraries, selected operating-system components, and metadata into a portable artifact. It is not a complete virtual machine: the host kernel is shared, and isolation is provided by operating-system mechanisms such as namespaces and control groups. Images are composed of layers, allowing registries and nodes to reuse unchanged content.

The image should be treated as immutable. If a defect is discovered, the preferred response is to build and publish a new image rather than modify a running container. This practice improves traceability because a deployment can be associated with an image digest, source revision, build record, and security evidence.

Kubernetes provides a control plane for scheduling and managing containers. A Deployment, for example, expresses the desired number of replicas, image, update strategy, and pod template. The Kubernetes controllers compare this desired state with the observed state and take corrective action. This is reconciliation rather than a one-time imperative deployment command.

A useful conceptual model is:

Source code → build and test → immutable image → registry → deployment description → reconciliation → observable runtime.

CI primarily produces and validates artifacts. CD makes validated artifacts available in an environment through a controlled mechanism. In a GitOps model, the desired deployment description is stored in version control, and an in-cluster agent applies it. The distinction between artifact creation and environment change is important: a successful build does not automatically imply that production should change.

Secure Container Image Engineering

A reliable image begins with a deterministic build. The Dockerfile or equivalent build definition should use a trusted, version-pinned base image, install only required dependencies, and avoid embedding credentials. Multi-stage builds separate compilation tools from the runtime image. For example, a builder stage may contain a compiler and test framework, while the final stage contains only the compiled application and runtime libraries.

A typical secure image process includes:

1. Select a maintained and minimal base image appropriate to the workload.

2. Pin dependencies or lock files where feasible.

3. Copy dependency manifests before source files to improve build caching without compromising correctness.

4. Run tests and static analysis during the build pipeline.

5. Execute the application as a non-root user.

6. Remove package caches, temporary files, compilers, and credentials from the final layer.

7. Scan the image and its software bill of materials (SBOM).

8. Sign the image or attach verifiable provenance.

9. Push it using a content-addressable digest.

Minimal images reduce attack surface but do not eliminate risk. A small image can still contain a vulnerable library or an insecure application. Image scanning should therefore be combined with dependency analysis, source analysis, runtime controls, and regular rebuilding. Vulnerability severity must be interpreted in context: exploitability, exposure, privileges, compensating controls, and whether the vulnerable component is actually loaded all matter.

The build system should protect against common supply-chain threats. Builds should run with restricted permissions, avoid untrusted scripts where possible, isolate tenants, and record the source revision and builder identity. Reproducible builds are particularly valuable because independent processes can verify that the artifact corresponds to the declared source and dependencies. Provenance metadata answers questions such as who built the image, from which source, using what workflow, and with which inputs.

Registries, Promotion, and Artifact Governance

A container registry stores image manifests and layers and commonly supports access control, replication, retention, scanning, signing, and audit logs. It may be public, private, or integrated with a cloud provider. Registries should be treated as production infrastructure, not merely as file storage.

A robust promotion model builds an image once and promotes the same immutable artifact across environments. Rebuilding separately for testing and production can introduce unverified differences. Human-readable tags such as `release-2025-04` are useful, but deployment records should preferably resolve tags to immutable digests. Mutable tags such as `latest` weaken reproducibility and complicate rollback.

Promotion may be represented by signed metadata, an environment-specific release record, or a Git change to a deployment repository. Admission controls can reject images that are unsigned, originate from an unapproved registry, contain unacceptable vulnerabilities, or lack provenance. Retention policies must balance cost with the need to preserve artifacts for rollback, investigation, and regulatory evidence.

Kubernetes Delivery Architecture

A Kubernetes delivery pipeline normally contains source control, a CI runner, an image builder, security and quality gates, a registry, deployment configuration, a cluster-side delivery mechanism, and observability systems. The pipeline should not rely solely on a command that reports success after submitting a manifest. It should verify that the new revision becomes ready and that application-level indicators remain healthy.

Kubernetes separates several concerns:

• Pod: the smallest deployable unit, usually containing one application container and sometimes tightly coupled sidecars.
• Deployment: manages replicated stateless pods and rolling updates.
• Service: provides stable network access to a changing set of pods.
• Ingress or Gateway: exposes HTTP or other traffic through routing rules.
• ConfigMap and Secret: provide configuration, although secrets require additional protection and should not be treated as ordinary text.
• Namespace: provides a logical boundary for organization and access policies.
• StatefulSet and persistent storage: support workloads requiring stable identity or durable data.

A pipeline should define resource requests and limits, readiness and liveness behavior, security context, network policy, disruption policy, and update strategy. Readiness determines whether a pod should receive traffic; liveness determines whether a container should be restarted. Confusing these probes can cause either premature traffic delivery or unnecessary restarts.

Rolling updates are suitable for many stateless services but require backward compatibility. During a database or API migration, old and new application versions may coexist. Expand-and-contract migration patterns—adding compatible structures first, migrating usage, and removing obsolete structures later—reduce this risk. For high-risk changes, blue-green or canary delivery may be preferable. A canary sends a controlled fraction of traffic to the new version and uses technical and business indicators to decide whether to continue.

Packaging Configuration with Helm-Style Techniques

Raw YAML is explicit but can become repetitive across environments. Helm-style packaging introduces charts, templates, values, and release history. A chart can describe a reusable application unit while environment-specific values select image digests, replica counts, resource limits, domains, and feature flags.

Templating improves reuse but can hide the final configuration. Every chart should therefore be rendered and validated in CI. Tests should check schema validity, required values, security settings, policy compliance, and compatibility with the target cluster version. Chart dependencies must be pinned and reviewed. Excessive templating can make a system difficult to understand; in such cases, plain manifests, Kustomize-like overlays, or a platform-specific composition method may be clearer.

Packaging is not deployment governance. A chart may define how an application is installed, but promotion policy, approvals, evidence, rollback, and ownership still need to be established. Configuration must also be separated from secrets. Secret values should be supplied through a protected secret-management system or encrypted repository mechanism rather than committed in plaintext.

Operators and the Reconciliation Pattern

Kubernetes operators extend the platform with domain-specific knowledge. An operator introduces a custom resource, such as a database cluster or message-broker definition, and a controller that reconciles that resource. Instead of describing every low-level object, a user states a higher-level intention: for example, the desired database version, storage size, replication factor, and backup policy.

Operators can automate provisioning, upgrades, failover, backups, and recovery. Their power also creates risk. A poorly designed operator may perform destructive changes, conceal operational complexity, or create unexpected privileges. Operators require version compatibility testing, clear ownership, backup validation, and carefully designed status reporting. They should expose whether a requested state is progressing, ready, degraded, or blocked rather than merely returning a generic success message.

GitOps and Reconciliation-Based Delivery

GitOps places the desired environment state in a version-controlled repository and uses an agent to reconcile the cluster with that state. A commit becomes an auditable change proposal. Pull requests provide review, policy checks, and separation of duties; the cluster agent performs the actual synchronization.

Two repositories are often separated conceptually: an application repository containing source and CI definitions, and an environment repository containing deployment declarations. CI builds and signs an image, then updates the environment repository with the new digest. A GitOps controller detects the change and applies it. This design prevents a CI runner from requiring broad, long-lived cluster credentials.

GitOps does not mean that every change must be made manually through Git. Emergency procedures may exist, but out-of-band changes should be detected, documented, and either reverted or incorporated into the declared state. Reconciliation must also be observable. Useful indicators include synchronization status, health status, deployment age, drift frequency, failed reconciliation count, and time from approved change to healthy runtime.

Microservices, Service Meshes, and Progressive Delivery

Microservices increase deployment independence but also multiply interfaces, pipelines, versions, ownership boundaries, and failure paths. Each service should have a clear contract, automated tests, health indicators, and an ownership model. Contract testing is valuable because an individual service's unit tests cannot prove compatibility with its consumers.

A service mesh places networking capabilities such as mutual TLS, retries, timeouts, traffic shaping, and telemetry in a platform layer, often through proxies or node-level components. It can simplify consistent policy across services, but it adds latency, operational complexity, certificate management, and another diagnostic surface. Retries require special care: indiscriminate retries can amplify load during an outage. Timeouts, bounded retries, circuit breaking, and idempotent operations should be designed together.

Progressive delivery combines deployment automation with runtime evidence. A canary analysis may examine error rate, latency percentiles, saturation, queue depth, conversion, or domain-specific indicators. Automated rollback should have clear thresholds and should not depend on a single noisy metric. Observability must therefore be designed before progressive delivery is introduced.

Serverless Delivery

Serverless platforms abstract server management and commonly execute functions or containerized workloads in response to events or requests. CI/CD still requires source testing, dependency management, packaging, security scanning, configuration control, and staged release. Additional concerns include cold-start behavior, execution time limits, concurrency, event duplication, permission scope, and vendor-specific runtime behavior.

Serverless deployment can use aliases, versions, traffic splitting, or parallel environments. A safe workflow tests event schemas and failure handling, not merely the function's direct return value. Functions should be idempotent where duplicate events are possible, and dead-letter or replay mechanisms should be tested. Cloud permissions should be narrowly scoped because a compromised function can otherwise access unrelated resources.

Applied Case: A Retail Platform Migrates to GitOps

Background

This illustrative case concerns Northstar Retail, a fictional online retailer whose monolithic application is being decomposed into catalog, order, payment, and notification services. The company operates Kubernetes clusters in two regions and uses a managed container registry.

Situation and Problem

The existing pipeline builds images and executes imperative deployment commands from a shared CI account. Production incidents have occurred because the `latest` tag was reused, configuration differed between regions, and a deployment was considered successful before readiness was achieved. The security team also lacks reliable evidence about image provenance and secret access.

Analysis

The principal problems are not caused by Kubernetes alone. They arise from mutable artifacts, excessive deployment credentials, weak separation between build and release, insufficient health validation, and configuration drift. The company also discovers that its order service cannot tolerate simultaneous old and new database schemas.

Approach and Solution

Northstar adopts multi-stage builds, pinned dependencies, non-root runtime users, SBOM generation, vulnerability scanning, and image signing. CI publishes one image identified by digest. A protected environment repository contains Helm-style values and deployment declarations for each region. A GitOps controller reconciles those declarations, while policy checks reject unsigned images and excessive privileges.

The order service adopts an expand-and-contract database migration. Deployments use readiness probes and rolling-update limits. Payment changes use a canary release with automatic analysis of error rate, latency, and authorization success. A service mesh supplies mutual TLS and traffic policy, but retry budgets are explicitly limited. Secrets are retrieved from a managed secret store, and audit logs connect a release commit to the image digest and cluster revision.

Outcome

The new process makes releases traceable and reduces direct cluster access from CI. Failed canaries stop before full traffic exposure, and configuration differences become visible through repository review. The company does not eliminate incidents, but it improves detection, rollback, and learning because every change has a declared source and observable result.

Learning Points

The case demonstrates that cloud-native delivery is an organizational control system as well as a collection of tools. Immutable artifacts, declarative configuration, least privilege, compatibility-aware migration, and meaningful runtime evidence must work together. Introducing a service mesh or GitOps controller without solving ownership and observability would not by itself produce reliable delivery.

Risks, Governance, and Common Mistakes

A frequent mistake is treating containerization as a security boundary. Containers reduce packaging inconsistency but require host, kernel, runtime, network, identity, and application controls. Another mistake is placing secrets in images, Git history, rendered manifests, or CI logs. Secret rotation, access review, and auditability are essential.

Kubernetes clusters can fail through excessive privileges, unbounded resource consumption, insecure admission settings, exposed dashboards, weak network segmentation, or untested backup and restore procedures. Governance should define approved base images, registry locations, signing requirements, vulnerability exception processes, namespace ownership, data classifications, and incident responsibilities.

Automation can also produce harm at speed. An incorrect reconciliation rule may repeatedly apply a destructive change; a faulty retry policy may overload a dependent service; and an automated rollback may worsen an incident if the previous version is incompatible with an already-migrated database. High-impact actions need safeguards, staged rollout, approval boundaries, and tested recovery procedures.

Regulatory obligations may apply to personal data, financial transactions, healthcare information, software supply chains, and audit records. Teams should know where data is processed, who can access it, how long logs and artifacts are retained, and how evidence is produced. Ethical practice includes honest reporting of deployment risk, avoiding misleading reliability metrics, and preserving meaningful human accountability for consequential changes.

Current Developments and Best Practices

Cloud-native delivery is moving toward stronger software supply-chain frameworks, signed provenance, policy-as-code, confidential workloads, platform engineering, and workload identity. Secure software frameworks increasingly emphasize verifiable build processes rather than relying only on post-build scanning. eBPF-based observability and runtime security can provide deeper visibility, although they require specialist skills and careful privacy controls.

AI-assisted operations can summarize incidents, suggest manifest changes, detect anomalous resource behavior, or generate test cases. Such systems should be treated as advisory unless their actions are tightly constrained. Generated configurations require review because an apparently valid manifest may create excessive privileges, unsafe exposure, or unexpected cost.

Effective practice is less about adopting every cloud-native tool and more about establishing coherent controls:

• Build once, identify artifacts by digest, and preserve provenance.
• Keep images minimal, patched, scanned, signed, and non-root where possible.
• Separate CI artifact production from controlled environment change.
• Store desired state in reviewable, versioned form.
• Validate rendered configuration and enforce policy before admission.
• Design readiness, observability, rollback, and data migration together.
• Use progressive delivery for changes whose risk justifies it.
• Minimize credentials and prefer short-lived workload identity.
• Test restore, failover, reconciliation, and incident procedures.
• Measure delivery outcomes, including deployment frequency, lead time, change failure rate, recovery time, and reliability indicators, without optimizing one metric at the expense of system safety.

Chapter Summary

Cloud-native CI/CD extends ordinary build-and-deploy automation into a system of immutable artifacts, declarative configuration, reconciliation, policy, and runtime evidence. Secure container delivery depends on trusted bases, reproducible builds, least privilege, scanning, SBOMs, signing, and provenance. Registries provide controlled artifact distribution, while Kubernetes manages desired runtime state through resources and controllers. Helm-style packaging improves reuse but requires rendering and policy validation. Operators encode domain-specific reconciliation, and GitOps provides auditable, pull-based environment synchronization.

Microservices require contract management, compatibility-aware data changes, observability, and disciplined ownership. Service meshes can standardize communication security and traffic policy but introduce complexity. Serverless systems require attention to event semantics, permissions, concurrency, and deployment versions. Across all models, successful delivery combines automation with governance, progressive exposure, measurable health, tested rollback, and accountability.

Key Terms

Container image: Immutable layered artifact containing an application and its runtime dependencies.

Container registry: Repository that stores, distributes, and governs container images.

Image digest: Content-derived identifier that refers to an exact artifact.

SBOM: Software bill of materials listing components and dependencies in an artifact.

Provenance: Verifiable information about an artifact’s source, inputs, builder, and process.

Multi-stage build: Image build using separate construction and runtime stages.

Kubernetes: Platform that orchestrates containerized workloads through declarative resources and controllers.

Pod: Kubernetes unit containing one or more co-located containers.

Deployment: Kubernetes resource that manages replicated, updateable stateless pods.

Reconciliation: Continuous process of making observed state match desired state.

Helm chart: Packaged, templated collection of Kubernetes resources and configuration.

Operator: Controller that applies domain-specific operational knowledge through custom resources.

GitOps: Delivery approach in which version-controlled desired state is reconciled into an environment.

Service mesh: Infrastructure layer providing service-to-service communication, security, and telemetry controls.

Canary release: Progressive release exposing a new version to a limited traffic segment.

Readiness probe: Health check indicating whether a workload should receive traffic.

Workload identity: Mechanism allowing software to obtain narrowly scoped identity without long-lived secrets.

Policy as code: Machine-evaluable rules governing configuration, security, or deployment decisions.

Review Questions

Short Answer Questions

1. Why should production deployments use image digests rather than rely only on mutable tags?

2. Distinguish a readiness probe from a liveness probe.

3. What problem does a multi-stage container build address?

4. Explain the role of reconciliation in Kubernetes.

5. Why are contract tests useful in a microservices architecture?

Long Answer Questions

1. Design a secure CI/CD workflow for a Kubernetes application from source commit to production reconciliation.

2. Explain how Helm-style packaging, operators, and GitOps differ and how they may work together.

3. Analyze the security and governance controls required for a private container registry.

4. Compare rolling, blue-green, and canary deployment strategies, including appropriate use cases and risks.

5. Discuss the operational benefits and costs of introducing a service mesh.

Analytical / Application Questions

1. A pipeline builds the same application separately for staging and production, using the `latest` tag. Identify the risks and redesign the artifact-promotion process.

2. A new service version passes unit tests but causes increased latency during a canary. Which evidence, controls, and rollback decisions should the delivery system use?

3. A Kubernetes operator repeatedly recreates a database after an administrator performs an emergency repair. Analyze the GitOps and reconciliation issues and propose a safe operating model.

22. Multiple Choice Questions with Answers

Question 1

Which practice most directly supports reproducible promotion from testing to production?

A. Rebuilding the application separately in each environment

B. Promoting one signed image identified by an immutable digest

C. Using the latest tag so every cluster receives new content

D. Copying container files manually between nodes

Correct Answer: B. Promoting one signed image identified by an immutable digest

Question 2

What is the primary purpose of a Kubernetes readiness probe?

A. To determine whether a container image is signed

B. To restart a process that has exceeded its memory limit

C. To determine whether a pod should receive traffic

D. To create a persistent volume automatically

Correct Answer: C. To determine whether a pod should receive traffic

Question 3

Which image-design choice best reduces unnecessary runtime attack surface?

A. Including compilers and debugging tools in the production image

B. Running the application as root for simpler permissions

C. Using a multi-stage build with only required runtime components

D. Storing registry credentials in an environment variable in the image

Correct Answer: C. Using a multi-stage build with only required runtime components

Question 4

In a GitOps model, what normally represents the desired environment state?

A. An unrecorded command executed by a CI administrator

B. A version-controlled deployment declaration

C. The current memory state of a running pod

D. A mutable image tag alone

Correct Answer: B. A version-controlled deployment declaration

Question 5

Why can indiscriminate retries in a service mesh be dangerous?

A. They prevent all forms of encryption

B. They can amplify load on an already failing dependency

C. They permanently disable readiness probes

D. They convert pods into virtual machines

Correct Answer: B. They can amplify load on an already failing dependency

Question 6

A database schema must support old and new application versions during a rolling update. Which approach is most suitable?

A. Delete the old schema before deploying the new version

B. Use an expand-and-contract migration

C. Disable health checks during migration

D. Deploy a mutable latest image to all pods

Correct Answer: B. Use an expand-and-contract migration

Question 7

What additional concern is especially important for event-driven serverless functions?

A. Whether duplicate events can be safely processed

B. Whether the function has a Kubernetes StatefulSet

C. Whether every invocation uses a dedicated virtual machine

D. Whether container layers are mounted manually

Correct Answer: A. Whether duplicate events can be safely processed

Question 8

What does an operator add to Kubernetes?

A. A domain-specific controller and usually a custom resource

B. A replacement for all source-control systems

C. A guarantee that application code contains no vulnerabilities

D. A requirement that every workload be stateful

Correct Answer: A. A domain-specific controller and usually a custom resource

Question 9

Which control most effectively limits the impact of compromised workload credentials?

A. A shared administrator credential for all services

B. Long-lived credentials stored in the image

C. Short-lived, narrowly scoped workload identity

D. Disabling audit logs to reduce system overhead

Correct Answer: C. Short-lived, narrowly scoped workload identity

Question 10

Why should a canary release use both technical and business indicators where appropriate?

A. Technical metrics alone always measure customer value accurately

B. Business indicators can reveal harm not visible in infrastructure metrics

C. Business metrics remove the need for rollback thresholds

D. Technical indicators cannot be collected from Kubernetes

Correct Answer: B. Business indicators can reveal harm not visible in infrastructure metrics