2. Cluster Architecture and Production Engineering

Introduction and Context

A Kubernetes cluster is not merely a collection of containers running on virtual machines. It is a distributed control system that continuously compares the desired state declared by users with the observed state of infrastructure and workloads. Production engineering begins when this system must operate reliably under failure, growth, security constraints, organizational change, and operational uncertainty.

A development cluster can tolerate manual intervention, temporary downtime, weak isolation, and approximate capacity planning. A production cluster cannot. Its architecture must account for control-plane availability, data durability, worker-node resilience, network topology, storage behavior, version compatibility, observability, security, and the boundaries imposed by a cloud provider or enterprise platform. Design decisions made at the cluster level influence application availability, deployment velocity, incident response, cost, and regulatory compliance.

This chapter examines the architecture of production Kubernetes clusters and the engineering reasoning required to design them. It treats the cluster as a socio-technical system: software components matter, but so do failure domains, operating procedures, ownership models, upgrade policies, and governance controls.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain the responsibilities and interactions of Kubernetes control-plane and worker-node components.

2. Describe etcd’s role in cluster state, consistency, backup, and recovery.

3. Analyze the Kubernetes API machinery and explain how requests become running workloads.

4. Design high-availability layouts using failure domains, topology, and quorum principles.

5. Estimate capacity and distinguish resource requests, limits, allocatable capacity, and workload demand.

6. Evaluate add-ons, version compatibility, upgrade strategies, and managed Kubernetes offerings.

7. Identify production risks, governance obligations, and common architectural mistakes.

8. Apply these concepts to a realistic cluster-design scenario.

1. Kubernetes as a Distributed Control System

Kubernetes follows a declarative operating model. A user or automation system specifies a desired state—for example, three replicas of a web service, a particular container image, a network policy, and a storage claim. Controllers then observe the cluster and take actions that move the actual state toward that desired state. This repeated process is known as reconciliation.

The architecture separates the control plane from the data plane. The control plane stores cluster state, accepts and validates API requests, schedules workloads, and runs controllers. The data plane consists primarily of worker nodes that execute Pods and provide networking, storage access, and container runtime services. The separation is logical even when components share physical hosts in small installations.

Kubernetes is therefore resilient in a particular sense: it can recreate replaceable workload instances when nodes or containers fail. It is not automatically resilient to every failure. Loss of the API server, corruption of state, exhausted node capacity, unavailable storage, misconfigured networking, or an incorrectly designed identity system can affect the entire platform. Production engineering means identifying these dependencies rather than assuming that the orchestration layer eliminates them.

2. The Control Plane: Coordination, State, and Policy

2.1 The kube-apiserver

The kube-apiserver is the front door to the cluster. Nearly all administrative and automated interactions pass through it, including commands from kubectl, requests from controllers, scheduler decisions, admission webhooks, and node status updates. It exposes the Kubernetes API over HTTPS and is designed to be stateless with respect to durable cluster data: persistent state is stored in etcd.

An API request normally passes through authentication, authorization, admission control, validation, and persistence. Authentication establishes who or what made the request. Authorization determines whether that identity may perform the requested operation on the target resource. Admission control can mutate or reject a request—for example, by applying a default security context or requiring approved registries. Validation checks whether the object conforms to the API schema and resource rules. If accepted, the object is written to etcd.

Because the API server is central to control-plane operations, production deployments generally run multiple replicas behind a highly available endpoint. The endpoint may be provided by a cloud load balancer, an external reverse proxy, or a virtual IP mechanism. Multiple API servers do not remove the need for healthy etcd or functional authentication; they provide availability for the API-serving layer.

2.2 etcd: The Authoritative Cluster Database

etcd is a distributed key-value store used by Kubernetes to persist cluster configuration and state. Objects such as Deployments, Services, Secrets, leases, node records, and custom resources are represented through the Kubernetes API and stored in etcd. The API server is the supported interface for interacting with this state; direct manipulation of the etcd data model is unsafe.

etcd uses a consensus protocol based on quorum. In a typical odd-sized cluster, a majority of members must be available to commit changes. A three-member cluster can tolerate one member failure; a five-member cluster can tolerate two. Adding members does not automatically improve performance or availability if the members share the same power supply, host, zone, or network path. Physical and logical placement are therefore as important as member count.

etcd is sensitive to disk latency, network latency, clock problems, and resource contention. High write latency can delay API operations and controller reconciliation. For this reason, etcd should use fast and reliable storage, be isolated from noisy workloads, and be monitored for database size, commit latency, leader changes, peer health, and defragmentation needs.

Backups are a production requirement. A useful etcd backup strategy includes scheduled snapshots, encryption of sensitive backup material, access control, retention policies, off-cluster storage, and periodic restoration tests. A snapshot that has never been restored is an assumption rather than evidence of recoverability. Recovery procedures should define how the control plane is rebuilt, how endpoint identity is preserved, and how applications are validated after restoration.

2.3 The kube-scheduler

The scheduler assigns unscheduled Pods to suitable nodes. It considers resource requests, node labels, taints and tolerations, affinity and anti-affinity rules, topology spread constraints, and other scheduling predicates or scoring policies. The scheduler does not start containers itself; it records the selected node, after which the kubelet on that node acts.

Scheduling is a policy decision as well as a technical one. If replicas are allowed to concentrate on one node or one availability zone, a single failure can remove the whole service. Conversely, excessive anti-affinity can make Pods unschedulable and reduce utilization. Production design balances resilience with capacity efficiency.

2.4 Controllers and the Cloud Controller Manager

Controllers implement reconciliation loops. A Deployment controller creates or updates ReplicaSets; a ReplicaSet controller maintains the desired number of Pods; a StatefulSet controller manages ordered or identity-sensitive workloads; and the Node controller responds to node health information. Controllers are generally level-based: they act on the current desired and observed state rather than depending only on a sequence of events.

The cloud controller manager separates cloud-provider integration from core Kubernetes controllers. Depending on the platform, it may manage node addresses, cloud load balancers, routes, and volume attachment. In managed services, much of this integration is operated by the provider, but its behavior still affects cluster design, permissions, failure modes, and cost.

3. The API Machinery and Request Lifecycle

Kubernetes resources are accessed through a versioned API. Group, version, and resource identity allow the platform to evolve without immediately breaking every client. For example, a resource may move from an experimental version to a stable version while conversion logic preserves compatibility between representations.

Custom Resource Definitions extend the API with domain-specific objects. A CRD alone defines an object schema; a controller gives that object operational meaning. This combination supports operators for databases, messaging systems, policy engines, and platform services. However, custom resources increase the control-plane surface area. Poorly designed controllers can generate excessive API traffic, create reconciliation loops, or store unnecessarily large objects in etcd.

The request lifecycle can be summarized as follows:

1. A client sends an authenticated HTTPS request to the API endpoint.

2. The API server authenticates the client and applies authorization rules.

3. Admission plugins and webhooks mutate or reject the object.

4. The API server validates the resource and persists it in etcd.

5. Watches notify controllers, schedulers, and kubelets of relevant changes.

6. Controllers reconcile the desired state and create subordinate resources.

7. The kubelet and container runtime realize the assigned workload on a node.

8. Status information flows back through the API, allowing further reconciliation.

This architecture explains why API availability and API efficiency matter. Controllers use watches and caches, but large-scale or poorly designed clients can still create substantial load. Production platforms should apply request authentication, rate controls where appropriate, API audit logging, resource quotas, and careful design of custom controllers.

4. Worker Nodes and the Data Plane

A worker node supplies compute, memory, networking, and storage for Pods. The kubelet is the node agent. It receives Pod specifications assigned to the node, instructs the container runtime through the Container Runtime Interface, mounts volumes, executes probes, reports status, and initiates local remediation when containers fail.

The container runtime, such as containerd or another CRI-compatible implementation, pulls images and manages container processes. The runtime is not the same as the kubelet: the kubelet coordinates Pod lifecycle, while the runtime performs low-level container operations. The node also requires a networking implementation, commonly through a Container Network Interface plugin, to connect Pods and enforce or support network policy.

Kubernetes networking usually gives each Pod an IP address and enables communication between Pods across nodes. Services provide stable virtual identities for changing Pod sets. An Ingress or Gateway layer exposes selected services to external clients. The exact implementation—overlay, routed networking, eBPF-based forwarding, cloud-native interfaces, or a combination—affects performance, observability, policy enforcement, and troubleshooting.

A node has capacity and allocatable capacity. Capacity describes what the node provides, while allocatable capacity is the portion available to Pods after reserving resources for the operating system, kubelet, runtime, and system daemons. A production cluster must reserve adequate CPU and memory for node stability. If workloads consume all memory, the operating system may invoke the out-of-memory killer, and kubelet eviction behavior may remove Pods before or during severe pressure.

Node pools are useful for expressing different operational or hardware profiles. General-purpose pools may run ordinary stateless services; memory-optimized pools may support caches; GPU pools may support machine-learning workloads; and isolated pools may host regulated or especially sensitive applications. Taints, tolerations, labels, and admission policies should be used together so that specialized nodes do not become accidental general-purpose capacity.

5. Failure Domains and High Availability

High availability is a property of an architecture, not a replica count. A failure domain is a set of components likely to fail together, such as a virtual machine, rack, availability zone, region, power circuit, network segment, or cloud control-plane dependency. Replicas are valuable only when they are distributed across meaningful failure domains.

A production control plane commonly uses multiple API-server instances and an odd number of etcd members. Control-plane hosts should be distributed across zones when the underlying network and storage support the required latency. In self-managed clusters, this requires designing the bootstrap process, load-balancer endpoint, certificate distribution, static Pods or system services, upgrade sequence, and recovery process. In managed clusters, the provider typically operates the control plane, but the customer must still understand the provider’s service-level commitments, regional scope, maintenance behavior, and data-plane dependencies.

Worker availability requires more than multiple nodes. Workloads need sufficient replica counts, disruption budgets, topology spread constraints, and a capacity buffer for maintenance and unexpected failure. A PodDisruptionBudget can limit voluntary disruption, but it cannot guarantee availability during every involuntary event. If a service has two replicas and both are placed in one zone, a zone outage defeats the apparent redundancy.

The following design questions are useful:

• Which failures must the cluster tolerate: node, zone, region, or provider service failure?
• Can the application continue if the control plane is temporarily unavailable?
• Are replicas distributed according to the application’s real failure domains?
• Is there enough spare capacity to reschedule workloads after a node or zone loss?
• Does persistent storage support the same topology and recovery objective as the application?

A regional cluster may offer lower operational complexity than several independent clusters, but it can also create a larger blast radius. Multiple clusters improve isolation and regional independence, yet introduce fleet management, identity federation, policy synchronization, and cross-cluster service-discovery challenges.

6. Capacity Engineering and Topology

Capacity planning begins with demand rather than node count. For each workload, engineers should estimate CPU and memory requests, peak consumption, replica count, scheduling constraints, daemon overhead, storage requirements, network throughput, and growth. Requests influence scheduling and often resource allocation; limits constrain consumption, although behavior varies by resource and workload. A request that is too low can cause contention and poor placement, while an inflated request reduces utilization and increases cost.

A simplified capacity model is:

Available workload capacity = sum of node allocatable resources − resilience reserve − platform overhead.

The resilience reserve represents capacity intentionally left unused so that workloads can be rescheduled after failure or during maintenance. Its size depends on the failure objective. A platform designed to survive loss of one node in a small pool must retain enough capacity to absorb that node’s critical workloads, not merely an arbitrary percentage.

Topology is the placement relationship between resources across domains. Labels such as zone, region, instance type, and hardware capability allow the scheduler to express these relationships. Topology spread constraints can distribute replicas evenly, while node affinity can require or prefer particular locations. Storage topology must also be considered: a volume provisioned in one zone may not be attachable in another, and a StatefulSet may require carefully designed volume replication or application-level recovery.

Overcommitment can improve average utilization, but it introduces contention risk. Quality-of-service classes, eviction thresholds, priority classes, and workload isolation help determine which Pods are protected or removed under pressure. Critical system components should not compete blindly with bursty batch jobs. Capacity engineering is therefore linked to scheduling policy and operational priorities.

7. Add-ons as Part of the Platform Contract

A Kubernetes cluster is operationally incomplete without services that support networking, storage, identity, security, observability, and lifecycle management. Add-ons are not decorative extensions; they become dependencies of applications and platform teams.

Common categories include:

• CNI networking and network-policy enforcement.
• CoreDNS or an equivalent service-discovery component.
• CSI storage drivers and snapshot controllers.
• Ingress controllers or Gateway implementations.
• Metrics, logging, tracing, and alerting agents.
• Cluster autoscalers and node-provisioning controllers.
• Certificate, secrets, policy, and image-signing integrations.
• Backup, disaster-recovery, and vulnerability-scanning tools.

Each add-on should have an owner, support policy, resource budget, upgrade path, security review, and recovery procedure. Installing several overlapping controllers can create conflicting behavior. For example, multiple ingress controllers or autoscaling systems may both react to the same resources unless their scopes are clearly separated.

Observability should cover both workloads and the platform. Useful signals include API latency and error rate, etcd health, scheduler and controller queues, node pressure, Pod restart rates, DNS failures, network drops, volume errors, and admission-webhook latency. Logs without actionable alert thresholds are not an operational strategy. Alerts should represent service impact or a credible precursor to it.

8. Version Compatibility and Lifecycle Engineering

Kubernetes evolves through versioned APIs, changing defaults, component compatibility rules, and provider-specific support windows. Production engineering requires a compatibility matrix covering the control plane, kubelets, kubectl clients, CRDs, admission webhooks, CNI and CSI components, ingress or Gateway implementations, monitoring agents, backup tools, and external cloud integrations.

A safe upgrade is a controlled change to a distributed system. Teams should first identify deprecated APIs and remove them from manifests and automation. They should test workloads and add-ons against the target version, confirm backup and rollback procedures, upgrade a non-production environment, and observe a canary node pool or workload set. During a node upgrade, disruption budgets, topology rules, spare capacity, and storage detach behavior become important.

Rollback is not always symmetrical. A control-plane downgrade may be unsupported or unsafe after data or API schema changes. Consequently, recovery planning should emphasize forward repair, tested backups, and restoration procedures rather than assuming that every upgrade can simply be reversed. Version skew policies should be consulted for the chosen Kubernetes distribution and provider.

9. Self-Managed and Managed Kubernetes

In a self-managed cluster, the organization controls the operating system, control-plane lifecycle, etcd, networking, certificates, upgrades, and much of the incident response. This can provide customization and learning value, but it also creates a substantial reliability and security responsibility. The organization must possess the skills and staffing to operate consensus storage, load balancing, node images, patching, and recovery.

Managed Kubernetes services typically abstract or operate the control plane and may provide integrated identity, load balancing, storage, upgrades, and node provisioning. They reduce undifferentiated operational work, but they do not make applications automatically portable or resilient. Provider-specific APIs, IAM models, network constructs, storage classes, maintenance windows, quotas, and regional limitations can create dependency and migration costs.

The choice should be based on operating objectives, not fashion. Relevant criteria include required control, compliance boundaries, available engineering expertise, workload portability, geographic scope, integration requirements, service-level objectives, cost predictability, and exit strategy. A managed control plane still requires customer responsibility for workload security, namespace governance, resource requests, application backups, network policy, and often worker-node configuration.

10. Security, Governance, and Professional Responsibility

Cluster architecture is inseparable from governance. The API is a powerful administrative boundary, and excessive permissions can turn a compromised workload into a platform-wide incident. Role-based access control should follow least privilege, separate human and workload identities, and use short-lived or federated credentials where practical.

Secrets require particular care. Kubernetes Secret objects are not, by themselves, a complete secrets-management solution. Organizations should evaluate encryption at rest, external secret managers, key rotation, access logging, and the risk of exposing secret material through logs, manifests, backups, or improperly configured APIs.

Admission policies can enforce organizational standards such as approved registries, non-root execution, restricted host access, required labels, resource requests, and image provenance. Policy should be designed to reduce risk without blocking emergency operations or legitimate platform functions. Exceptions require ownership, expiry, and auditability.

Audit logs support accountability and investigation. They should be protected from tampering, retained according to legal and organizational requirements, and correlated with identity-provider, cloud, node, and application records. Regulatory obligations may also affect data locality, backup retention, encryption, segregation of duties, and evidence of change approval.

Ethically, platform engineers must recognize that reliability and access decisions affect users and organizations. Concealing known capacity limitations, disabling controls without documentation, or treating production data as a convenient debugging resource violates professional responsibility. Governance should enable safe delivery rather than become an unexamined accumulation of restrictions.

11. Illustrative Case: Designing a Regional Payments Platform

Background

A fictional financial-technology company, Northstar Pay, is moving a payment authorization service from virtual machines to Kubernetes. The service operates in one cloud region with three availability zones. It requires high availability, controlled access to sensitive data, predictable latency, and routine upgrades without prolonged interruption.

Situation and Problem

The initial proposal uses three worker nodes, one API server, a single-zone database volume, and two application replicas. All replicas are permitted to schedule anywhere. The proposal is inexpensive but exposes several risks: loss of one node may remove half the application capacity; a zone failure may eliminate the storage volume; the control plane has a single API endpoint; and no capacity is reserved for maintenance.

Analysis

The service’s real availability objective is regional rather than merely node-level. Therefore, application replicas must be distributed across zones, and storage must provide a recovery design consistent with the payment system’s data requirements. The Kubernetes cluster should not be treated as the system of record for payment transactions; durable transaction storage requires an appropriately designed database service and tested backup process.

The control plane should use a highly available provider-managed configuration or, if self-managed, multiple API-server instances and a quorum-based etcd deployment distributed across zones. The API endpoint needs a stable, monitored load-balancing path. Worker capacity must tolerate the loss of at least one node and support rolling upgrades without evicting all critical replicas.

Approach and Solution

Northstar chooses a managed regional control plane and three worker pools distributed across the zones. The general-purpose pool runs the authorization service, while a smaller system pool is reserved for platform agents. Application Deployments use at least three replicas, topology spread constraints, and a disruption budget that preserves service during voluntary maintenance. Requests are based on measured load, with a reserve for node failure and deployment overlap.

Network policies restrict payment-service communication to explicitly required services. An admission policy requires signed images, non-root execution, resource requests, and approved registries. Secrets are obtained through an external secrets integration, and audit records are exported to a protected logging system. The storage design uses a database service with multi-zone resilience rather than relying on a single-zone application volume.

Before production, the team tests node drain, zone-level workload relocation, API unavailability, failed image pulls, DNS disruption, storage recovery, and restoration from backup. A staged upgrade is performed first on a non-production cluster and then on a canary node pool. Runbooks identify owners, escalation paths, and customer communication procedures.

Outcome

The revised design costs more than the initial proposal but aligns capacity, topology, security, and recovery with the business objective. A single node failure can be absorbed, routine maintenance does not remove all replicas, and the data layer has a recovery model independent of Kubernetes object recovery.

Learning Points

The case illustrates that high availability is end-to-end. Replicas, nodes, the control plane, storage, identity, networking, and operational procedures must support the same objective. It also demonstrates that a managed control plane reduces selected responsibilities but does not replace workload architecture, capacity planning, testing, or governance.

12. Common Failure Patterns and Design Mistakes

Several recurring mistakes appear in production clusters. Treating replica count as availability ignores placement and dependencies. Running all components in one zone creates a hidden shared failure domain. Omitting resource requests produces unreliable scheduling and weak capacity forecasts. Setting limits without understanding application behavior can cause throttling or memory termination. Installing add-ons without ownership creates unsupported critical paths.

Other mistakes include relying on default credentials, granting cluster-admin permissions for convenience, storing unprotected backups, failing to test restoration, upgrading without checking CRD and webhook compatibility, and assuming that a managed service handles every responsibility. Excessive custom controllers can overload the API server and etcd. Conversely, insufficient observability leaves teams unable to distinguish an application failure from a platform failure.

A mature engineering review should examine failure scenarios explicitly. For each critical dependency, ask whether it is replaceable, replicated, recoverable, observable, and governed. If the answer is unknown, the design is not yet production-ready.

13. Emerging Directions

Kubernetes operations are increasingly influenced by automated node provisioning, policy-as-code, software supply-chain security, eBPF-based networking and observability, and fleet management across multiple clusters. Autoscaling systems can provision nodes in response to pending Pods, but they must be constrained by quotas, disruption policies, budget, and availability-zone capacity.

Platform engineering is also moving toward internal platforms that provide approved templates, paved deployment paths, identity integration, policy defaults, and self-service environments. This can reduce cognitive load for application teams while centralizing reliability and security practices. However, abstraction must remain transparent enough that developers understand resource behavior, failure modes, and operational ownership.

AI-assisted operations may help correlate alerts, identify anomalous resource behavior, summarize incidents, or recommend configuration changes. Such tools should support—not replace—human approval, change control, privacy protection, and evidence-based diagnosis. Automated remediation is appropriate only when its scope, safety conditions, and rollback behavior are well understood.

Best-Practice Design Heuristics

A production cluster should have a documented failure model, clear ownership, and tested recovery procedures. Control-plane state should be protected with encrypted, off-cluster backups and restoration drills. Worker nodes should be distributed across relevant failure domains, with explicit reserves for disruption. Workloads should use measured requests, appropriate limits, topology rules, and disruption budgets.

The platform should maintain a compatibility matrix for Kubernetes and add-ons, use staged upgrades, and remove deprecated APIs before they become emergencies. Access should be least-privilege and auditable. Add-ons should be treated as products with lifecycle ownership. Finally, design reviews should evaluate not only whether the cluster works on a successful day, but also how it behaves during loss of nodes, zones, credentials, storage, network paths, and human attention.

Chapter Summary

Kubernetes production engineering is the disciplined design of a distributed control and execution platform. The API server coordinates access, etcd stores authoritative state, the scheduler places workloads, controllers reconcile desired state, and worker nodes execute Pods through the kubelet and container runtime. Reliable operation depends on more than component replication: etcd quorum, API availability, node capacity, topology, storage behavior, networking, and recovery procedures must align with the required failure model.

Capacity planning uses allocatable resources, workload requests, platform overhead, and resilience reserves. Add-ons such as networking, storage, DNS, observability, policy, and backup systems form part of the platform contract. Version compatibility and staged lifecycle management reduce upgrade risk. Managed services can reduce control-plane operations but do not remove customer responsibility for workload architecture, security, governance, and disaster recovery. The central professional lesson is that availability, security, and recoverability are end-to-end properties established through architecture and verified through testing.

Key Terms

Control plane: Kubernetes components that manage cluster state, scheduling, API access, and reconciliation.

Data plane: The worker-node environment in which application and system Pods execute.

kube-apiserver: The authenticated and authorized HTTPS interface to Kubernetes resources and operations.

etcd: A distributed, quorum-based key-value store that persists Kubernetes cluster state.

Quorum: The majority of members required by a consensus system to commit state changes.

Reconciliation: The repeated process of moving observed system state toward declared desired state.

Kubelet: The node agent that manages Pods and reports node and workload status.

Container runtime: Software that pulls images and manages container processes through the CRI.

Scheduler: The control-plane component that selects suitable nodes for unscheduled Pods.

Admission control: Processing that validates, mutates, or rejects API requests before persistence.

Failure domain: A set of resources likely to fail together, such as a node, zone, or region.

Allocatable capacity: The node resources available to Pods after system and platform reservations.

Topology spread constraint: A scheduling rule that distributes Pods across failure or infrastructure domains.

PodDisruptionBudget: A policy limiting voluntary disruption to selected application Pods.

Custom Resource Definition: An extension that adds a new resource type to the Kubernetes API.

Add-on: A supporting platform component providing capabilities such as networking, storage, or observability.

Version skew: The permitted difference in versions among Kubernetes components and clients.

Managed Kubernetes: A service in which a provider operates some or all control-plane responsibilities.

Admission webhook: An external service invoked during API admission to validate or mutate requests.

Resilience reserve: Deliberately unused capacity retained for failures, maintenance, or rescheduling.

Review Questions

Short Answer Questions

1. What is the relationship between the kube-apiserver and etcd?

2. Why does an odd number of etcd members commonly improve fault tolerance?

3. Distinguish node capacity from allocatable capacity.

4. What responsibilities does the kubelet perform?

5. Why should add-ons be assigned explicit owners and upgrade policies?

Long Answer Questions

1. Explain the lifecycle of a Kubernetes API request from client submission to workload realization.

2. Analyze how control-plane and worker-node high availability should be designed across availability zones.

3. Discuss the role of capacity planning, resource requests, limits, and resilience reserves in production clusters.

4. Compare self-managed and managed Kubernetes from the perspectives of control, responsibility, risk, and cost.

5. Explain why backup and restoration testing are essential for etcd and application data recovery.

Analytical / Application Questions

1. A three-zone cluster has six worker nodes, but all replicas of a critical service are scheduled in one zone. Identify the architectural risk and propose scheduling and capacity changes.

2. During an upgrade, admission webhooks begin timing out and new Deployments fail, although existing Pods continue to run. Analyze the likely compatibility and operational issues and formulate a safe response.

3. Design a high-level production architecture for a regulated workload that must tolerate one node failure, preserve auditability, protect secrets, and undergo routine maintenance without total service interruption.

22. Multiple Choice Questions with Answers

Question 1

Which component provides the durable authoritative store for Kubernetes cluster objects?

A. kube-scheduler

B. etcd

C. kubelet

D. Container runtime

Correct Answer: B. etcd

Question 2

A three-member etcd cluster generally tolerates which failure while retaining quorum?

A. Failure of one member

B. Failure of two members

C. Failure of all members in one zone regardless of placement

D. Any number of network partitions

Correct Answer: A. Failure of one member

Question 3

What is the primary purpose of the kube-apiserver?

A. To execute container processes directly

B. To assign IP addresses to every application

C. To provide the authenticated and authorized Kubernetes API

D. To replace application databases

Correct Answer: C. To provide the authenticated and authorized Kubernetes API

Question 4

A team wants replicas to survive a complete availability-zone outage. Which design most directly supports this objective?

A. Increase replicas but permit unrestricted placement

B. Use topology spread constraints across zones and retain failure capacity

C. Place all replicas on the largest node

D. Use a PodDisruptionBudget without changing placement

Correct Answer: B. Use topology spread constraints across zones and retain failure capacity

Question 5

Which statement best describes allocatable node capacity?

A. The node’s total physical capacity before any reservations

B. The capacity available to Pods after reserving resources for the system and platform

C. The combined limits of all containers on the node

D. The capacity of the entire cluster divided by the number of namespaces

Correct Answer: B. The capacity available to Pods after reserving resources for the system and platform

Question 6

Why is restoring an etcd snapshot periodically important?

A. It proves that the scheduler can place Pods faster

B. It verifies that the backup is usable for actual recovery

C. It eliminates the need for API authentication

D. It automatically creates a multi-region application database

Correct Answer: B. It verifies that the backup is usable for actual recovery

Question 7

Which component selects a suitable node for an unscheduled Pod?

A. kube-scheduler

B. CoreDNS

C. The container runtime

D. The cloud load balancer

Correct Answer: A. kube-scheduler

Question 8

A managed Kubernetes service most directly reduces which responsibility?

A. Designing application replica placement

B. Managing every application secret

C. Operating selected control-plane infrastructure

D. Testing application disaster recovery

Correct Answer: C. Operating selected control-plane infrastructure

Question 9

Which practice best reduces upgrade-related compatibility risk?

A. Upgrade production first and inspect failures afterward

B. Ignore deprecated APIs if current workloads still run

C. Maintain a compatibility matrix and test add-ons and webhooks before staged rollout

D. Disable admission controls permanently during every upgrade

Correct Answer: C. Maintain a compatibility matrix and test add-ons and webhooks before staged rollout

Question 10

Why can a PodDisruptionBudget not guarantee availability during every outage?

A. It applies only to voluntary disruptions and cannot prevent all involuntary failures

B. It controls etcd quorum rather than application Pods

C. It forces all replicas into one zone

D. It replaces topology constraints

Correct Answer: A. It applies only to voluntary disruptions and cannot prevent all involuntary failures