8. Containers, Kubernetes and Cloud-Native Operations

Introduction and Context

Modern DevOps projects must deliver software that is repeatable, portable, observable, secure, and capable of responding to changing demand. Traditional deployment models often bind an application to a particular operating system, server configuration, network arrangement, or manually maintained runtime. Containers address much of this inconsistency by packaging an application together with its libraries, configuration assumptions, and execution instructions. Kubernetes extends this idea from packaging to operations: it schedules containers, maintains desired state, exposes services, manages configuration, supports controlled release strategies, and provides a foundation for cloud-native systems.

Containers and Kubernetes are not merely tools for replacing virtual machines. They represent an operational model based on declarative configuration, immutable artefacts, automated reconciliation, loosely coupled services, and platform engineering. Their value depends on sound architecture and disciplined practices. A poorly designed monolith placed in a container may remain difficult to operate; a Kubernetes cluster without governance may increase complexity rather than reduce it.

This chapter examines the complete operational path from container image construction to multi-environment delivery. It considers networking, storage, service discovery, autoscaling, Helm, GitOps, security, observability, cost, and organizational responsibilities. The emphasis is applied: students should be able to make informed design choices rather than simply memorize resource definitions.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain container isolation, image construction, registries, and runtime behaviour.

2. Describe Kubernetes architecture and the role of its principal resources.

3. Design practical approaches to networking, service discovery, configuration, and persistent storage.

4. Select suitable autoscaling, deployment, Helm, and GitOps strategies.

5. Analyse operational, security, governance, reliability, and cost concerns in multi-environment clusters.

6. Evaluate cloud-native designs through an applied case and identify appropriate improvements.

From Application Packaging to Desired-State Operations

A container image is an immutable, versioned package containing an application and the filesystem layers required to run it. It normally excludes the host kernel; containers use operating-system-level isolation while sharing the kernel of the host. Namespaces isolate processes, networks, mounts, and other resources, while control groups constrain and measure resource consumption. The result is usually lighter and faster to start than a virtual machine, although containers are not automatically secure or identical across all environments.

A Dockerfile or an equivalent build definition describes how an image is produced. Effective images use a small trusted base, deterministic dependency versions, multi-stage builds, non-root execution, and a clear entry point. The build context should not contain credentials or unnecessary files. Image tags such as `latest` are unsuitable for controlled production delivery because they are mutable; immutable digests or release-specific tags provide stronger traceability.

The image is stored in a registry. A registry may be public, private, self-hosted, or provided by a cloud platform. The software supply chain should connect source revision, build record, test results, image digest, vulnerability report, and deployment record. Signing and verification can strengthen confidence that the image deployed is the image that was built and approved.

Kubernetes introduces a declarative control model. Instead of instructing an operator to start a particular process on a particular server, a user describes a desired state. Controllers compare that state with the actual cluster state and take corrective action. If a Pod fails, a controller may create another one; if a node becomes unavailable, workloads may be rescheduled subject to their constraints. This reconciliation loop is central to Kubernetes operations.

The control plane generally includes the API server, a persistent key-value store for cluster state, a scheduler, and controllers. Worker nodes run a container runtime, a node agent, and networking components. Clients and automation communicate with the API server. Access is governed through authentication, authorization, admission controls, and resource policies.

A Pod is the smallest deployable unit in Kubernetes. It contains one or more tightly coupled containers that share networking and, where configured, storage. Most application containers run in a Deployment, which manages ReplicaSets and supports rolling updates. A StatefulSet is designed for workloads requiring stable identity or ordered behaviour, while a DaemonSet places a Pod on selected nodes, commonly for logging or monitoring agents. Jobs and CronJobs handle finite or scheduled work.

The distinction between a Pod and a container is operationally important. A Pod is not normally a long-lived server to be manually repaired. Pods are replaceable instances. Persistent identity, stable endpoints, and durable data require appropriate higher-level resources rather than assumptions about a Pod's survival.

Kubernetes Resources as an Operating Model

Kubernetes manifests are commonly written in YAML and describe resources such as Deployments, Services, ConfigMaps, Secrets, Ingresses, and autoscalers. A manifest is not simply a script; it is an executable representation of policy and desired state. It should therefore be version-controlled, reviewed, tested, and promoted through environments.

A Deployment specifies a Pod template and a replica count. Its rolling update strategy can limit unavailable and additional Pods, allowing controlled replacement. Readiness probes determine whether a Pod should receive traffic; liveness probes identify a process that requires restart; startup probes protect slow-starting applications from premature liveness failure. Probes must test meaningful health conditions. A liveness probe that checks only whether a process exists may fail to detect dependency problems, while an overly aggressive probe can cause restart loops.

Resource requests tell the scheduler the amount of CPU and memory a container requires. Limits constrain consumption, though their precise effects depend on the resource and cluster configuration. Requests influence placement and capacity planning. Missing requests can lead to inefficient scheduling and unreliable autoscaling. Quality-of-service classifications, node affinity, taints, tolerations, topology spread constraints, and disruption budgets help express availability and placement requirements.

Configuration should be separated from the image. ConfigMaps are appropriate for non-sensitive settings; Secrets are intended for sensitive values, but their presence as a Secret object does not by itself guarantee adequate protection. Encryption at rest, strict access control, external secret managers, rotation, audit logging, and avoidance of secret exposure in logs are essential. Configuration should be environment-specific without requiring a different application image for every environment.

Namespaces provide logical boundaries for teams, applications, or environments. They support naming isolation and can be combined with Role-Based Access Control, quotas, network policies, and policy admission. They are not a complete security boundary in every threat model. Strong isolation may require separate clusters, accounts, or subscriptions.

Networking and Service Discovery

Kubernetes networking aims to give Pods routable IP addresses and allow communication without requiring manual port mappings. A Pod IP is ephemeral, however, so clients should not directly depend on it. A Service provides a stable virtual address and selects Pods through labels. ClusterIP is used for internal access, while NodePort and LoadBalancer expose services in other ways. Ingress or Gateway resources provide HTTP routing, host-based rules, TLS termination, and traffic policies, normally through an installed controller or gateway implementation.

Service discovery commonly uses cluster DNS. A service name resolves to a stable DNS record, allowing applications to communicate by logical identity rather than changing addresses. This reduces coupling between deployment instances and clients. Headless Services are useful when clients need to discover individual Pod addresses, as in some stateful systems.

NetworkPolicy expresses permitted traffic between Pods and, depending on the networking implementation, external endpoints. A default-deny posture followed by explicit allow rules reduces lateral movement. Policies should be designed alongside application flows, including DNS, observability, identity services, databases, and external APIs. A policy that is technically secure but operationally incomplete can cause mysterious failures.

TLS, mutual TLS, API gateways, and service meshes may provide encryption, identity, retries, traffic splitting, and telemetry. A service mesh can be valuable in a large microservice estate, but it adds control-plane, data-plane, debugging, and resource overhead. It should solve a demonstrated problem rather than be adopted as a default badge of maturity.

Storage and Stateful Workloads

Containers are ephemeral by design. Data written to a container filesystem may disappear when the Pod is replaced. Kubernetes separates storage claims from application specifications through PersistentVolumes and PersistentVolumeClaims. A StorageClass describes dynamic provisioning and characteristics such as performance or replication. Access modes indicate how a volume may be mounted, but their exact availability depends on the storage provider.

Stateless services should normally externalize durable data to managed databases, object storage, queues, or other appropriate platforms. Stateful workloads can run in Kubernetes, but they require careful treatment of backup, recovery, replication, consistency, upgrades, failure domains, and operational expertise. A persistent volume is not automatically a backup. Backup schedules, restoration tests, retention, encryption, and recovery objectives must be explicit.

Two useful measures are the recovery point objective (RPO), which describes acceptable data loss, and the recovery time objective (RTO), which describes acceptable restoration time. Storage architecture should be evaluated against both, as well as latency, throughput, cost, compliance, and regional resilience.

Scaling, Reliability, and Release Strategies

Horizontal Pod Autoscaling changes replica count using metrics such as CPU, memory, or application-specific indicators. It is effective only when requests, limits, metrics, and workload behaviour are understood. CPU utilization alone may be a poor signal for queue-driven or latency-sensitive systems. Event-driven autoscaling can respond to queue depth or external events. Vertical scaling changes resource allocations, while cluster autoscaling adds or removes nodes. These mechanisms operate at different levels and can interact unpredictably if poorly tuned.

Autoscaling is not a substitute for capacity planning. A system must have sufficient quota, node capacity, database capacity, network throughput, and downstream service tolerance. Cooldowns, stabilization windows, maximum replicas, and rate limits help prevent oscillation or overload.

Reliability also depends on graceful shutdown, connection draining, retries with bounded backoff, timeouts, idempotency, circuit breaking, and disruption budgets. Retries without limits can amplify an outage. Readiness should be removed before termination so that new requests stop arriving while in-flight requests complete.

Rolling updates are generally safer than replacing all instances simultaneously. Blue-green deployment maintains two environments and shifts traffic between them. Canary release sends a controlled fraction of traffic to a new version and evaluates technical and business signals. The selected strategy should reflect blast radius, rollback speed, database compatibility, observability quality, and regulatory requirements.

Helm and Configuration Reuse

Helm packages Kubernetes resources as charts. A chart contains templates, default values, metadata, and sometimes dependencies. Values can be overridden for development, testing, staging, and production. Helm improves reuse and release management, but templates can become difficult to read and test when they contain excessive conditional logic.

A robust Helm practice keeps common templates generic while placing environment-specific values in separately reviewed files. Secrets should not be casually committed as plaintext values. Chart versions should be pinned, dependencies verified, rendered manifests tested, and upgrades examined for destructive changes. Helm is a packaging and release tool; it does not by itself provide complete promotion governance, drift control, or application lifecycle management.

Alternatives include Kustomize overlays, Jsonnet, purpose-built operators, or plain manifests. The choice should be based on team capability, reuse requirements, policy needs, and the complexity of environment differences. Configuration variation should remain small and intentional. If environments differ so greatly that each is effectively a separate product, the organization should question the architecture and test strategy.

GitOps and Multi-Environment Operations

GitOps treats Git as the authoritative source for desired deployment state. A reconciler running in the cluster observes the repository and applies approved changes. This creates an auditable path: a change is proposed, reviewed, merged, reconciled, and measured. Pull-based reconciliation can reduce direct cluster credentials in CI systems and continuously detect drift.

A practical repository model separates application source from environment configuration while maintaining clear references between them. Promotion may occur by updating an image digest or chart version in a development branch, then promoting the same tested artefact to staging and production. This is stronger than rebuilding separately for each environment. Automated tests, policy checks, vulnerability scanning, manifest validation, and progressive delivery controls should be integrated into the path.

GitOps does not eliminate operational responsibility. Emergency changes, cluster upgrades, credential rotation, policy exceptions, and restoration from repository or cluster failure require documented procedures. The organization should define who may approve changes, how break-glass access is audited, and how drift is reconciled after an incident.

Observability, Security, and Governance

Cloud-native operations require signals at several levels. Metrics reveal resource use, saturation, error rates, latency, and business indicators. Logs provide event detail, while traces connect a request across services. Kubernetes events, audit logs, control-plane health, node conditions, and application telemetry should be correlated. Useful dashboards are designed around service-level objectives rather than an indiscriminate collection of charts.

Security is a lifecycle concern. Images should be built from maintained bases, scanned for vulnerabilities, signed where appropriate, and run with least privilege. Containers should avoid privileged mode, unnecessary Linux capabilities, host filesystem mounts, and host networking. RBAC should grant only required verbs and resources. Network policies limit communication. Admission policies can reject images without approved registries, missing resource requests, or unsafe security contexts.

Governance must include ownership, naming, tagging, cost allocation, data classification, retention, incident response, and change accountability. Managed Kubernetes reduces infrastructure administration but does not transfer responsibility for application security, identity, data protection, or workload configuration. Regulatory requirements may affect data location, encryption, audit retention, access review, and separation of duties.

Cost management is particularly important because Kubernetes can hide waste behind abstractions. Oversized requests, idle clusters, excessive replicas, high-cardinality telemetry, unused load balancers, and inefficient storage all create cost. Chargeback or showback, rightsizing reviews, scheduling policies, and capacity dashboards connect technical decisions to financial outcomes.

Illustrative Case: A Retail Platform Moving to Kubernetes

Background

Northstar Retail, a fictional regional retailer, operates an online ordering platform. Its release process uses manually configured virtual machines. During seasonal campaigns, teams overprovision servers, yet checkout occasionally fails because a recommendation service consumes excessive memory. Releases are infrequent, rollback is slow, and production configuration differs from testing.

Situation and Analysis

The platform team proposes moving every component to Kubernetes immediately. Analysis shows that the ordering API and catalogue service are good initial candidates because they are stateless and already have automated tests. The payment database, however, has strict recovery requirements and is operated by a specialist database team. Moving it into the first Kubernetes phase would introduce unnecessary risk. The recommendation workload is asynchronous and driven by a queue, making queue depth a more suitable scaling signal than CPU alone.

The principal risks are not container construction but operational readiness: insufficient observability, unclear ownership, untested recovery, and insecure configuration. The team therefore defines service-level objectives, dependency maps, RPO and RTO targets, and a production readiness review.

Approach and Solution

The team builds minimal, non-root images through multi-stage pipelines. Each image receives an immutable digest and a software bill of materials. A private registry and admission policy prevent unapproved images from running. Deployments include resource requests, readiness and startup probes, disruption budgets, and graceful termination handling.

Internal communication uses Services and DNS. An Ingress exposes the public API with managed TLS. Network policies permit only required paths: the API may reach the catalogue and payment gateway, while the recommendation worker may consume the recommendation queue. Configuration is stored separately from images, and sensitive values are retrieved from an external secret manager.

The API uses horizontal scaling based on CPU and request latency, while recommendation workers use event-driven scaling based on queue depth. The database remains on the managed platform, with tested backups and a separate recovery procedure. Helm charts package common application resources, and environment values are maintained through a GitOps repository. Development, staging, and production promote the same image digest rather than rebuilding it.

Observability combines application metrics, structured logs, distributed traces, Kubernetes events, and alerts tied to service-level objectives. A canary release first sends a small percentage of traffic to each new API version. Automated rollback occurs when error rate or latency exceeds the agreed threshold.

Outcome

The new operating model improves release repeatability and reduces manual configuration differences. More importantly, the organization gains explicit ownership, measurable reliability targets, and a tested rollback path. The case does not claim that Kubernetes alone created these improvements; they resulted from combining platform capabilities with engineering controls and operating discipline.

Learning Points

The case demonstrates five principles. First, migration should be guided by workload characteristics rather than fashion. Second, stateful systems require explicit recovery design. Third, autoscaling signals must represent demand. Fourth, GitOps and immutable artefacts improve traceability. Finally, observability and governance are prerequisites for safe automation, not optional additions after deployment.

Challenges and Common Mistakes

Kubernetes introduces a substantial learning curve. Teams may expose every service publicly, use privileged containers for convenience, omit resource requests, store credentials in repositories, or treat namespaces as complete isolation. Another common mistake is to run a single large cluster without considering failure domains, upgrade procedures, or ownership boundaries.

Overengineering is also a risk. A small application may not require a service mesh, multiple clusters, elaborate templating, or dozens of operators. Conversely, underengineering a critical platform by relying on manual changes creates hidden fragility. Decisions should be evaluated against reliability objectives, team skills, operational burden, and total cost.

Cluster upgrades, certificate expiry, registry outages, DNS failure, exhausted quotas, node pressure, and API-server unavailability must be included in resilience exercises. Disaster recovery should address not only application data but also cluster configuration, repositories, secrets, external dependencies, and identity systems.

Ethically, engineers should consider the effects of outages on customers, employees, and dependent communities. Automated decisions must be explainable where they affect access or service quality. Logs should avoid unnecessary personal data. Data retention, regional processing, and access rights must comply with applicable law and organizational policy. Convenience is not a justification for excessive collection or weak access controls.

Current Developments and Professional Practice

Cloud-native platforms increasingly use policy as code, software supply-chain attestations, workload identity, confidential computing, and platform engineering portals. OpenTelemetry is encouraging common approaches to telemetry collection. Progressive delivery controllers can automate canary analysis. WebAssembly and lightweight sandboxing may complement containers for selected workloads, although they do not eliminate the need for operational governance.

Artificial intelligence is being applied to anomaly detection, alert correlation, capacity forecasting, and natural-language assistance for incident investigation. These applications are useful only when telemetry is reliable and human accountability remains clear. Automated remediation should begin with bounded, reversible actions and strong audit trails. AI-generated configuration must be reviewed because a syntactically valid manifest may still create security, availability, or cost problems.

Professional practice therefore combines platform automation with human judgement. The mature question is not “Can this run in Kubernetes?” but “What operational model provides the required reliability, security, portability, speed, and cost for this workload?”

Best-Practice Guidance

A practical cloud-native operating baseline includes the following:

• Build small, reproducible, scanned images and deploy immutable digests.
• Use declarative manifests under version control with peer review and automated validation.
• Define ownership, service-level objectives, resource requests, and recovery targets.
• Keep services private by default and expose only deliberate entry points.
• Apply least privilege to identities, containers, namespaces, and network paths.
• Separate configuration and secrets from images, and rotate sensitive credentials.
• Treat storage, backup, restoration, and disaster recovery as one design problem.
• Scale from meaningful workload signals and test scaling under realistic load.
• Use Helm or another packaging method consistently; avoid uncontrolled template complexity.
• Promote the same tested artefact through environments using GitOps or an equivalently auditable process.
• Instrument applications and platforms before relying on automated remediation.
• Review reliability, security, cost, and compliance continuously rather than only at launch.

Chapter Summary

Containers provide portable application packaging through image-based, process-level isolation. Kubernetes supplies a declarative, reconciled operating model in which controllers manage Pods, Deployments, Services, storage, configuration, and scaling. Services and DNS support stable discovery, while Ingress or Gateway resources control external access. Persistent data requires deliberate volume, backup, RPO, and RTO design. Autoscaling depends on meaningful metrics, accurate resource requests, and sufficient underlying capacity.

Helm packages reusable Kubernetes resources, while GitOps provides an auditable, continuously reconciled path from approved configuration to cluster state. Effective cloud-native operations also require observability, least-privilege security, policy, cost management, ownership, and tested recovery. The central professional lesson is that Kubernetes is an enabling platform, not a substitute for architecture, governance, or operational discipline.

Key Terms

Container: An isolated process environment that packages an application and its user-space dependencies while sharing the host kernel.

Container image: An immutable, versioned filesystem and metadata package used to create containers.

Registry: A service that stores, distributes, and often scans container images.

Kubernetes: An open-source platform for declarative orchestration and reconciliation of containerized workloads.

Pod: Kubernetes’ smallest deployable unit, containing one or more co-located containers.

Deployment: A resource that manages replicated, updateable stateless Pods through ReplicaSets.

Service: A stable virtual endpoint that selects and routes traffic to Pods.

Ingress: A resource describing external HTTP or HTTPS routing into cluster services.

ConfigMap: A Kubernetes object for non-sensitive configuration data.

Secret: An object intended for sensitive configuration, requiring additional protection and access control.

Namespace: A logical Kubernetes partition used for naming, access, policy, and resource organization.

PersistentVolumeClaim: A request by a workload for provisioned persistent storage.

NetworkPolicy: A rule set controlling permitted network traffic to or from selected Pods.

Horizontal Pod Autoscaler: A controller that adjusts Pod replicas according to selected metrics.

Helm: A packaging and release tool that uses charts and configurable templates for Kubernetes resources.

GitOps: An operating approach in which version-controlled desired state is continuously reconciled with a running environment.

Reconciliation: The controller process of comparing desired state with actual state and correcting differences.

RPO: The maximum acceptable period of data loss measured backward from a disruption.

RTO: The target time within which a service or system should be restored after disruption.

Service-level objective: A measurable reliability target for a service, such as availability or latency.

Review Questions

Short Answer Questions

1. Why are immutable image digests preferable to the `latest` tag in production?

2. Distinguish a Pod from a container.

3. What problem does a Kubernetes Service solve?

4. Why is a PersistentVolume not itself a backup?

5. State the purpose of a readiness probe.

Long Answer Questions

1. Explain Kubernetes’ declarative operating model and the role of reconciliation.

2. Discuss the design considerations for networking and service discovery in a multi-service application.

3. Explain how resource requests, autoscaling, and capacity planning interact.

4. Compare Helm-based packaging with GitOps-based deployment governance.

5. Discuss the security and governance controls required for a production Kubernetes platform.

Analytical / Application Questions

1. A queue-processing service experiences increasing backlog while CPU utilization remains low. Which scaling approach would you recommend, and why?

2. A team wants to move a payment database into Kubernetes during its first migration phase. Analyse the risks and propose decision criteria.

3. A production deployment has correct replicas but intermittent user failures after release. Design an investigation using probes, Services, metrics, logs, traces, and rollout controls.

22. Multiple Choice Questions with Answers

Question 1

Which statement best explains why a Kubernetes Service is used for an application?

A. It permanently stores the application container filesystem

B. It provides a stable endpoint for a changing set of Pods

C. It compiles the application into a container image

D. It automatically encrypts every database volume

Correct Answer: B. It provides a stable endpoint for a changing set of Pods

Question 2

A queue worker has low CPU utilization but an increasingly large queue. Which autoscaling signal is most appropriate?

A. Node hostname

B. Container image size

C. Queue depth

D. Number of namespaces

Correct Answer: C. Queue depth

Question 3

What is the primary purpose of a readiness probe?

A. To decide whether a Pod should receive traffic

B. To build a new container image

C. To assign a persistent disk to every node

D. To authenticate a user to the cluster

Correct Answer: A. To decide whether a Pod should receive traffic

Question 4

Which practice provides the strongest release traceability?

A. Rebuilding an unpinned image separately in every environment

B. Using the mutable latest tag for all deployments

C. Promoting the same tested image digest across environments

D. Editing production manifests directly on a worker node

Correct Answer: C. Promoting the same tested image digest across environments

Question 5

Which statement about PersistentVolumes is correct?

A. They automatically provide tested disaster recovery

B. They can provide durable storage, but backup and restoration must still be designed

C. They eliminate the need to define RPO and RTO

D. They make all stateful applications equivalent to stateless services

Correct Answer: B. They can provide durable storage, but backup and restoration must still be designed

Question 6

In a GitOps model, what is the normal role of the reconciler?

A. It continuously compares cluster state with approved desired state and corrects differences

B. It replaces all application testing with manual approval

C. It stores passwords in application source code

D. It changes source code without an audit trail

Correct Answer: A. It continuously compares cluster state with approved desired state and corrects differences

Question 7

Which control most directly reduces lateral movement between workloads?

A. Increasing the number of replicas

B. Applying appropriate NetworkPolicy rules

C. Using a larger container base image

D. Disabling application logging

Correct Answer: B. Applying appropriate NetworkPolicy rules

Question 8

A team places database credentials in a ConfigMap because it is convenient. What is the principal concern?

A. ConfigMaps cannot contain text

B. Non-sensitive configuration objects are inappropriate for protecting secrets

C. ConfigMaps automatically delete databases

D. Kubernetes cannot mount ConfigMaps into Pods

Correct Answer: B. Non-sensitive configuration objects are inappropriate for protecting secrets

Question 9

Which deployment approach sends a controlled fraction of traffic to a new version before wider release?

A. Canary deployment

B. Cold backup

C. Node tainting

D. Static provisioning

Correct Answer: A. Canary deployment

Question 10

Why might adopting a service mesh be an inappropriate first response for a small application?

A. A service mesh cannot support networking

B. It adds operational complexity and overhead that may not address a demonstrated need

C. It prevents all telemetry collection

D. It removes the need for application-level timeouts

Correct Answer: B. It adds operational complexity and overhead that may not address a demonstrated need