6. Orchestration Theory and Kubernetes Architecture

Introduction and Context

Containerization solves an important packaging and portability problem: an application and its dependencies can be distributed as a reproducible image and executed in an isolated environment. In production, however, running one container is rarely the principal difficulty. Organizations must operate many containers across multiple machines, replace failed instances, distribute traffic, manage configuration, perform controlled releases, and use computing resources efficiently. These requirements create the need for orchestration.

Orchestration is the automated coordination of workloads and infrastructure according to operational objectives. Kubernetes is the most influential open-source platform for this purpose. It does not merely start containers; it maintains an intended relationship between applications, infrastructure, networking, storage, and policy. Its design reflects a broader systems principle: administrators describe a desired state, while software controllers continuously work to make the observed state converge toward that description.

This chapter examines that principle and relates it to Kubernetes architecture. It explains declarative management, reconciliation, control-plane and worker-node components, Kubernetes API objects, etcd, container runtimes, cluster topologies, and deployment choices. The discussion is intended to support both technical implementation and architectural decision-making.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain orchestration as a desired-state control problem.

2. Distinguish imperative commands from declarative management.

3. Describe reconciliation loops and their operational significance.

4. Evaluate the roles of the Kubernetes API server, scheduler, controllers, and etcd.

5. Explain the principal components of a worker node and the container runtime interface.

6. Interpret common Kubernetes API objects and their relationships.

7. Compare single-control-plane, highly available, managed, and hybrid cluster topologies.

8. Assess deployment options, operational risks, governance requirements, and professional best practices.

From Container Execution to Desired-State Orchestration

A container engine can create, start, stop, and remove containers. These operations are generally imperative: an operator issues a command describing what should happen immediately. Such commands are useful for experimentation, but they become difficult to manage when a system contains hundreds of workloads and several operators.

Orchestration introduces a higher-level contract. Instead of repeatedly issuing commands, an administrator records a desired state. The desired state may specify that three replicas of a web service should run, that they should use a particular image, that they should receive traffic through a stable service address, and that they should be scheduled only on suitable nodes. The orchestration platform observes the cluster and takes actions that move it toward this condition.

The distinction can be expressed simply:

• Imperative management specifies an action: “start this container.”
• Declarative management specifies an outcome: “this service must have three healthy replicas.”

Declarative management does not eliminate procedures. Kubernetes still performs concrete actions such as placing Pods, creating network endpoints, and restarting containers. The difference is that these actions are implementation details of controllers rather than a complete operational script written by the user.

Desired State, Observed State, and Reconciliation

A Kubernetes object normally contains a specification, or `spec`, describing the desired state, and a status section describing the system’s observed or current state. A Deployment may request five replicas. The controller compares that request with the number and condition of existing Pods. If only four suitable Pods exist, it creates another; if six exist, it removes one according to its management logic.

This repeated comparison is called reconciliation. A controller can be understood as a feedback mechanism:

`desired state – observed state → corrective action`

The controller does not assume that a single action permanently solves the problem. A node may fail after a Pod is scheduled, a process may terminate, or a network condition may change. Reconciliation therefore runs continuously. This property gives Kubernetes resilience: recovery is based on the recorded objective rather than on the memory of a particular operator or command sequence.

Reconciliation is usually level-triggered rather than purely event-triggered. Events can prompt a controller to examine an object, but the controller evaluates the current state and determines what is required. Consequently, missed events do not necessarily prevent recovery if the state remains inconsistent. This design also makes controllers comparatively robust in the presence of retries and temporary failures.

Idempotence and Convergence

A well-designed controller should be idempotent. Applying the same desired configuration repeatedly should not create uncontrolled side effects. If the desired number of replicas is already present, another reconciliation should make no unnecessary change. Idempotence is particularly important because distributed systems experience duplicate messages, retries, delayed observations, and partial failures.

Convergence does not imply instantaneous consistency. Kubernetes is generally an eventually consistent control system. A configuration change passes through the API server, is stored, observed by controllers, and translated into actions on nodes. During this interval, the actual cluster may differ from the requested state. Production designs must therefore account for propagation delay, readiness conditions, rollout strategy, and failure domains.

Kubernetes as a Distributed Control System

Kubernetes separates the control plane, which makes cluster-wide decisions, from worker nodes, which execute workloads. The separation is logical rather than necessarily physical: in small clusters, control-plane components may run on the same machines as workloads, while production clusters commonly isolate them.

The Kubernetes API and Object Model

The Kubernetes API is the principal interface to the cluster. Users, command-line tools, controllers, operators, admission components, and external automation interact through it. An API object is a persistent, structured representation of a resource. Typical object fields include metadata, desired specification, and status.

Important object categories include:

• Pod: The smallest deployable unit in Kubernetes. A Pod contains one or more tightly coupled containers sharing networking and, where configured, storage. Containers in the same Pod are scheduled together.
• Deployment: A controller that manages replicated, usually stateless, application Pods through ReplicaSets and supports rolling updates and rollback.
• ReplicaSet: Maintains a specified number of matching Pod replicas. It is commonly managed indirectly by a Deployment.
• Service: Provides a stable virtual endpoint and selects a set of Pods, allowing clients to communicate without depending on changing Pod addresses.
• Namespace: Provides a logical boundary for names, access control, quotas, and organizational separation within a cluster.
• ConfigMap: Stores non-confidential configuration data.
• Secret: Represents sensitive data, although effective protection also depends on encryption, access control, and the security of the cluster itself.
• StatefulSet: Manages workloads requiring stable identities, ordered behavior, or persistent storage relationships.
• DaemonSet: Ensures that a Pod is present on selected nodes, often for logging, monitoring, or networking agents.
• Job and CronJob: Represent finite tasks and scheduled tasks respectively.
• Ingress: Describes HTTP or HTTPS routing rules to services; actual behavior depends on an installed ingress controller.
• PersistentVolume and PersistentVolumeClaim: Separate available storage resources from an application’s request for storage.

Objects are related through labels, selectors, owner references, and namespaces. For example, a Deployment creates a ReplicaSet, which creates Pods. A Service selects Pods by labels rather than by manually maintained addresses. This loose coupling is a major reason Kubernetes can replace individual instances without requiring application clients to know their identities.

The API Server

The `kube-apiserver` is the front door to the cluster. It authenticates requests, authorizes actions, applies admission controls, validates API objects, and exposes resources through the Kubernetes API. It also mediates access to the cluster’s persistent state. Because nearly every administrative and automated operation passes through it, the API server is a critical security and availability boundary.

Authentication establishes who or what is making a request. Authorization determines whether that identity may perform the requested action, commonly through role-based access control. Admission mechanisms can validate or mutate objects after authentication and authorization. Organizations may use them to enforce image policies, required labels, security contexts, resource limits, or approved registries.

The API server should not be treated as a general-purpose database interface. Clients are expected to use the supported API semantics, resource versions, watches, and conflict-handling mechanisms. Direct modification of the underlying data store bypasses important validation and can corrupt cluster behavior.

etcd: Persistent Cluster State

`etcd` is a distributed key-value store used by Kubernetes to persist critical cluster state, including API objects and configuration information. It is not a workload database and should not be used by applications merely because it exists in the cluster architecture.

The reliability of etcd is central to control-plane reliability. Its data must be protected through access restrictions, encryption where appropriate, carefully managed backups, and tested restoration procedures. In a highly available arrangement, etcd members use consensus to maintain a consistent authoritative state. A majority, or quorum, is needed to make progress; therefore, adding members without considering failure domains and quorum behavior can reduce rather than improve operational simplicity.

Performance also matters. Excessive object churn, oversized objects, unbounded event retention, or poorly designed automation can increase storage and API pressure. Monitoring database latency, disk performance, space consumption, and leader health is part of responsible cluster administration.

Controllers and the Controller Manager

Controllers implement desired-state logic. The controller manager hosts or coordinates several control loops, including those concerned with node health, replica management, endpoints, jobs, and namespaces. A controller observes relevant objects, calculates the difference between desired and observed state, and submits changes through the API.

Controllers are intentionally decoupled. A Deployment controller need not know how a particular application processes requests; it manages replicas and rollout state. A node controller can react to node failure without understanding the application’s business logic. This modularity supports extensibility through custom resources and custom controllers, often packaged as operators.

The Scheduler

The scheduler assigns unscheduled Pods to suitable nodes. It considers resource requests, available capacity, node labels, taints and tolerations, affinity and anti-affinity rules, topology constraints, and other policies. Scheduling is not merely a search for free CPU. It is a constraint and preference problem involving availability, performance, compliance, cost, and locality.

A Pod’s resource requests influence placement, while limits constrain consumption at runtime. Incorrect requests can produce poor decisions: underestimation may cause contention, whereas overestimation may leave capacity unusable. Advanced scheduling may distribute replicas across zones, reserve specialized nodes for accelerators, or keep cooperating services close to one another.

The scheduler’s decision is distinct from execution. It records a binding between a Pod and a node; the node agent then attempts to run the assigned workload. If the node cannot satisfy the assignment, further status reporting and reconciliation are required.

Worker-Node Architecture

A worker node provides the environment in which Pods execute. Its major Kubernetes components are the kubelet, a container runtime, and usually kube-proxy or an equivalent networking implementation.

The kubelet is the node agent. It receives Pod specifications through the control plane, ensures that the containers described in those specifications are running and healthy, mounts volumes, reports status, and performs local probes and lifecycle operations. The kubelet does not normally schedule arbitrary Pods on its own; it acts on assignments and node-local configuration.

The container runtime downloads images, creates containers, manages their lifecycle, and implements low-level execution. Kubernetes communicates with runtimes through the Container Runtime Interface (CRI). Modern runtimes commonly use `containerd` or CRI-O, while the older assumption that the Docker Engine itself is required for every Kubernetes node is no longer correct. Docker remains important for image construction and developer workflows, but Kubernetes nodes need a CRI-compatible runtime.

The runtime relies on lower-level operating-system facilities, including namespaces, control groups, filesystem layers, and security mechanisms. Runtime choice can affect image handling, observability, startup behavior, and operational tooling, but Kubernetes aims to abstract common lifecycle operations through the CRI.

kube-proxy, where used, helps implement Service networking by programming node-level packet forwarding rules. Some clusters replace or supplement it with an eBPF-based networking implementation. The precise data path depends on the selected Container Network Interface (CNI) plugin and cluster design. Kubernetes itself defines networking expectations, while CNI implementations provide much of the concrete connectivity, routing, and network-policy behavior.

A Typical Object Lifecycle

Consider a user applying a Deployment that requests three replicas. The API server authenticates and validates the request, applies admission policy, and stores the object through etcd. The Deployment controller observes the new object and creates a ReplicaSet. The ReplicaSet controller creates three unscheduled Pods. The scheduler selects nodes according to resources and constraints and records bindings. Kubelets on those nodes obtain the Pod specifications, ask their runtimes to pull images and start containers, configure volumes and networking, and report status. A Service controller and endpoint mechanisms may then make healthy Pods available to clients.

If one Pod terminates, the ReplicaSet controller observes that the actual count is below the desired count and creates a replacement. If a node fails, node and workload controllers cooperate to mark the node unhealthy and recreate eligible Pods elsewhere, subject to storage, disruption, and scheduling constraints. The example illustrates that no single component “runs Kubernetes.” Cluster behavior emerges from cooperating control loops and agents.

Cluster Topologies and Deployment Choices

A development cluster may use a single control-plane node, local storage, and limited isolation. This arrangement is inexpensive and easy to understand, but it has obvious availability limitations. Loss of the control-plane node can make the cluster unmanageable even if already running workloads continue temporarily.

A production control plane commonly uses multiple API servers behind a stable endpoint, replicated controller and scheduler processes, and a resilient etcd arrangement. High availability requires more than installing extra machines. Control-plane nodes should be distributed across suitable failure domains, certificates and endpoints must be managed, and quorum-sensitive components require disciplined maintenance. Worker nodes should also be spread across zones or racks when application availability depends on such distribution.

Kubernetes can be deployed in several broad ways:

1. Self-managed clusters: The organization installs and operates the control plane, nodes, networking, upgrades, backups, and security. This offers extensive control but requires substantial expertise and operational capacity.

2. Managed Kubernetes services: A cloud or platform provider operates much of the control plane and often integrates identity, networking, load balancing, and storage. The customer still manages workloads, policies, costs, and usually worker-node configuration.

3. On-premises or private-cloud clusters: These support data locality, regulatory, or infrastructure-control requirements but make hardware, capacity planning, and platform reliability the customer’s responsibility.

4. Hybrid or multi-cluster arrangements: Workloads are distributed across environments for resilience, sovereignty, latency, or organizational reasons. They introduce federation, identity, networking, observability, and policy-consistency challenges.

5. Edge and lightweight clusters: These target constrained or intermittently connected environments. Reduced resource consumption must be balanced against limited redundancy and more difficult remote operations.

The correct choice depends on workload criticality, staff capability, regulatory requirements, latency, data gravity, portability objectives, and total cost of ownership. A multi-cluster strategy is not automatically more resilient: it can multiply upgrade, security, and observability complexity.

Applied Case: Modernizing a University Examination Platform

Background

A university operates an online examination platform used heavily during assessment periods. The original deployment consists of manually configured virtual machines and several independently started containers. The platform works under normal load, but operational staff struggle with traffic spikes, failed processes, inconsistent configuration, and emergency releases.

Situation and Problem

During a previous examination, a virtual machine failure reduced capacity. Operators restored services by manually starting containers, but the process was slow and configuration differed between machines. The university requires improved availability while retaining control over examination data and enforcing separation between development, testing, and production.

Analysis

The central problem is not simply container startup. It is the absence of a continuously enforced operational model. The platform needs replicated application Pods, stable service discovery, controlled rollouts, resource reservations, health checks, and a reliable method for restoring cluster state. Examination services also include stateful components, so not every workload should be treated as a disposable stateless container.

Approach and Solution

The university designs a production Kubernetes cluster with multiple control-plane nodes distributed across separate infrastructure failure domains. Worker nodes are labelled according to workload class. The web and API tiers are deployed through Deployments with readiness and liveness probes, resource requests, and replica distribution constraints. Services provide stable internal access, while an ingress controller exposes approved external routes.

Configuration that is not confidential is stored in ConfigMaps; credentials are placed in Secrets and protected through role-based access control and encryption measures. The database is either operated through a carefully evaluated StatefulSet and persistent storage design or provided as a managed database service. A separate namespace and access policy are used for testing. Network policies restrict communication so that the web tier cannot directly access unrelated services.

The team stores declarative manifests in version control and applies them through a controlled delivery pipeline. Admission policies reject images from unapproved registries and require resource specifications. Monitoring tracks API latency, node capacity, Pod restarts, rollout status, and application-level examination metrics. etcd backups and restoration drills are scheduled before the examination period.

Outcome

The resulting platform can replace failed stateless Pods automatically, distribute replicas across suitable nodes, and release new versions through controlled rolling updates. Operators no longer need to remember a sequence of manual container commands. The design does not guarantee that every failure is invisible: database recovery, network provider failure, and incorrect application behavior remain risks. Nevertheless, responsibility becomes explicit and recovery procedures become testable.

Learning Points

The case demonstrates that orchestration is a governance and reliability model as much as a deployment tool. Declarative definitions improve repeatability, but they must be accompanied by appropriate storage design, security controls, capacity planning, observability, and tested recovery. Kubernetes can automate an incorrect architecture just as efficiently as a sound one.

Risks, Limitations, and Common Errors

Kubernetes introduces a substantial operational surface. Misconfigured permissions can expose sensitive workloads; insecure dashboards or overly broad service accounts can provide pathways for compromise. Images may contain vulnerabilities, and a Secret object does not by itself guarantee confidentiality. Supply-chain controls, signed or verified images, scanning, minimal privileges, and runtime restrictions are therefore important.

Another common error is treating Pods as virtual machines. Pods are replaceable scheduling units, not necessarily durable identities. Applications requiring stable identity or persistent data need StatefulSets, appropriate storage, backup, and recovery design. Similarly, a ReplicaSet does not make a database highly available by itself.

Poor resource requests cause scheduling inefficiency and contention. Missing readiness probes can route traffic to an application that has started a process but is not ready to serve requests. Excessive use of privileged containers, host networking, host filesystem mounts, or cluster-admin permissions weakens isolation. Unbounded logging and event production can burden the control plane.

Teams also underestimate upgrades. Kubernetes versions, add-ons, runtimes, networking plugins, storage drivers, and admission policies interact. Upgrade plans should specify compatibility, disruption budgets, rollback limits, and validation steps. Finally, abstractions can hide cost: overprovisioned nodes, unnecessary replicas, idle namespaces, and cross-zone traffic may create significant expenditure.

Governance, Ethics, and Professional Responsibility

Cluster administration involves decisions about access, data, availability, and accountability. Role-based access should follow least privilege, with separate identities for people, pipelines, and workloads. Audit logs should support investigation without exposing more sensitive information than necessary. Namespaces and quotas can support organizational fairness by preventing one team from consuming all shared capacity.

Regulatory obligations may require data locality, retention controls, encryption, documented access, or separation of duties. These requirements should be translated into enforceable policies rather than left as informal expectations. A platform team also has a professional responsibility to communicate the limits of automation. Automatic rescheduling cannot repair corrupted data, a faulty application algorithm, or an unavailable external dependency.

Current Developments and Best Practices

Kubernetes is increasingly used as a platform for policy-driven delivery, internal developer platforms, event processing, machine learning workloads, and edge computing. Custom Resource Definitions and operators extend the API model to domain-specific systems. GitOps practices treat version-controlled declarations as the authoritative source for delivery, while policy engines validate configuration before or during admission. eBPF-based networking and observability can improve visibility and performance, although they add implementation complexity.

Effective practice generally includes:

• define workloads declaratively and keep definitions under version control;
• use namespaces, least-privilege RBAC, network policies, and admission controls;
• specify realistic resource requests and limits;
• implement readiness, liveness, and startup behavior appropriate to the application;
• distribute replicas across meaningful failure domains;
• monitor both application outcomes and control-plane health;
• back up etcd and persistent application data, then test restoration;
• design upgrades, disruption budgets, and rollback procedures before production use;
• choose managed, self-managed, or hybrid deployment according to capability and risk rather than fashion;
• document ownership, escalation paths, and service-level objectives.

Chapter Summary

Orchestration coordinates distributed workloads by maintaining a desired state rather than relying on repeated manual commands. Kubernetes implements this model through an API-driven architecture in which objects describe intent, controllers reconcile observed state, the scheduler assigns Pods to nodes, and kubelets with container runtimes execute those assignments. The API server is the principal control boundary, while etcd persistently stores cluster state and therefore requires strong protection and tested recovery.

Pods, Deployments, Services, StatefulSets, Jobs, configuration objects, and storage objects provide composable abstractions. Worker nodes depend on kubelet, a CRI-compatible runtime, and networking components. Cluster topology and deployment method—single-node, highly available, managed, self-managed, hybrid, or edge—must be selected through analysis of availability, cost, compliance, skills, and operational responsibility. Kubernetes improves repeatability and resilience, but it does not replace sound application design, security governance, observability, capacity planning, or recovery engineering.

Key Terms

Desired state: The condition an operator declares that the system should maintain.

Observed state: The condition currently detected by the platform.

Reconciliation: Continuous comparison of desired and observed state followed by corrective action.

Declarative management: Defining intended outcomes rather than prescribing every execution step.

Imperative management: Directly instructing a system to perform particular actions.

Control plane: Components that store cluster state, make decisions, and coordinate workloads.

Worker node: A machine that executes assigned Pods.

API server: The authenticated and authorized front end to the Kubernetes API.

etcd: Distributed key-value storage for authoritative Kubernetes cluster state.

Controller: A control loop that works to make observed state match desired state.

Scheduler: The component that selects suitable nodes for unscheduled Pods.

Pod: Kubernetes’ smallest deployable unit, containing one or more co-located containers.

Deployment: A controller for replicated, commonly stateless, application Pods and rolling updates.

Service: A stable virtual endpoint that exposes a selected group of Pods.

Kubelet: The node agent that ensures assigned Pods are running and reports their status.

Container Runtime Interface: The interface through which Kubernetes communicates with container runtimes.

Namespace: A logical boundary for names, policy, access, and resource organization.

Quorum: The majority of members required for a consensus-based distributed system to make progress.

Custom Resource Definition: An extension mechanism that adds domain-specific resource types to the Kubernetes API.

GitOps: An operational approach in which version-controlled declarations drive deployment and reconciliation.

Review Questions

Short Answer Questions

1. What is the difference between desired state and observed state in Kubernetes?

2. Why is idempotence important in a reconciliation loop?

3. What is the role of etcd in a Kubernetes cluster?

4. Distinguish a Pod from a Deployment.

5. What responsibilities does the kubelet perform on a worker node?

Long Answer Questions

1. Explain how a Deployment request travels through the Kubernetes control plane to become running containers on worker nodes.

2. Discuss the responsibilities of the API server, scheduler, controllers, kubelet, and container runtime.

3. Explain why etcd availability, quorum, backup, and restoration are central to control-plane reliability.

4. Compare self-managed Kubernetes with managed Kubernetes, including operational and strategic trade-offs.

5. Discuss how declarative management and reconciliation improve reliability while still leaving important limitations.

Analytical / Application Questions

1. A three-replica web application loses one Pod whenever a node fails, but no replacement appears on another node. Identify the control-plane and node-level factors that should be investigated.

2. An organization wants to deploy a database using a Deployment because its web service uses one. Evaluate this decision and recommend a more appropriate architecture.

3. Design a high-level Kubernetes approach for a regulated application requiring workload isolation, persistent data, controlled releases, auditability, and recovery testing. Justify the relevant objects, policies, and topology choices.

22. Multiple Choice Questions with Answers

Question 1

A Kubernetes controller finds that a Deployment requests five replicas but only four healthy Pods exist. What is the most appropriate controller behavior?

A. Modify etcd directly to report five replicas

B. Create or arrange for a replacement Pod through the Kubernetes API

C. Ask the scheduler to delete the remaining four Pods

D. Stop reconciling until an administrator issues a command

Correct Answer: B. Create or arrange for a replacement Pod through the Kubernetes API

Question 2

Which statement best explains declarative management in Kubernetes?

A. The administrator specifies every container command in execution order

B. The administrator specifies the desired outcome and controllers determine corrective actions

C. The scheduler manually configures each application process

D. The runtime stores application requirements outside the API

Correct Answer: B. The administrator specifies the desired outcome and controllers determine corrective actions

Question 3

What is the primary purpose of etcd?

A. To build container images

B. To route all application traffic between Pods

C. To persist authoritative Kubernetes cluster state

D. To schedule Pods onto worker nodes

Correct Answer: C. To persist authoritative Kubernetes cluster state

Question 4

A Pod has been assigned to a node, but its containers are not running. Which component primarily attempts to realize that assignment locally?

A. Kubelet working with the container runtime

B. The Deployment controller alone

C. The etcd leader

D. The namespace controller

Correct Answer: A. Kubelet working with the container runtime

Question 5

Why are resource requests important to Kubernetes scheduling?

A. They determine the container image format

B. They help the scheduler evaluate whether a node has suitable capacity

C. They replace the need for readiness probes

D. They guarantee that an application will never exceed its limit

Correct Answer: B. They help the scheduler evaluate whether a node has suitable capacity

Question 6

A team wants clients to continue reaching an application even when its Pods are replaced. Which object provides the stable virtual endpoint?

A. Secret

B. ReplicaSet

C. Service

D. ConfigMap

Correct Answer: C. Service

Question 7

Which design most directly supports etcd’s consensus and availability requirements?

A. Adding arbitrary members without considering failure domains or quorum

B. Using a carefully planned replicated arrangement with tested backups and restoration

C. Storing application data in etcd to increase utilization

D. Allowing every user to write directly to the data store

Correct Answer: B. Using a carefully planned replicated arrangement with tested backups and restoration

Question 8

An organization requires a Pod on every selected worker node for logging. Which Kubernetes controller is most appropriate?

A. DaemonSet

B. Job

C. Deployment

D. Ingress

Correct Answer: A. DaemonSet

Question 9

Which statement about Kubernetes and Docker is most accurate?

A. Every Kubernetes node must run the Docker Engine directly

B. Kubernetes requires a CRI-compatible container runtime; Docker remains useful for image workflows

C. Kubernetes cannot run images built with Docker tools

D. Docker replaces the Kubernetes API server

Correct Answer: B. Kubernetes requires a CRI-compatible container runtime; Docker remains useful for image workflows

Question 10

A university wants controlled releases, workload isolation, persistent storage, and tested recovery for an examination platform. Which approach is most appropriate?

A. Use manual container commands and place all workloads in one namespace

B. Use declarative workload definitions, namespaces and policies, suitable storage, controlled rollouts, and backup testing

C. Run every component as a privileged Pod on one node

D. Use a ReplicaSet alone to guarantee database durability

Correct Answer: B. Use declarative workload definitions, namespaces and policies, suitable storage, controlled rollouts, and backup testing