10. Resource Management, Scaling and Resilience

Introduction and Context

A containerized application may be packaged consistently, but packaging alone does not make it reliable in production. A Kubernetes cluster must decide where workloads run, how much compute they may consume, when additional replicas are required, how failures are detected, and how services recover during disruption. These decisions form the operational foundation of resource management, scaling, and resilience.

Resource management concerns the allocation and control of CPU, memory, storage, and related cluster capabilities. Scaling concerns the deliberate adjustment of capacity in response to workload demand. Resilience concerns the ability of an application to continue providing an acceptable service despite failures, overload, maintenance, or infrastructure loss. These subjects are closely connected. Poor resource definitions can cause unstable scheduling; inadequate scaling can create overload; weak failure-handling policies can turn a minor fault into a prolonged outage.

In Kubernetes, reliability is not produced by a single feature. It emerges from the interaction of containers, Pods, Deployments, Services, probes, controllers, schedulers, autoscalers, disruption policies, storage systems, observability, and operational procedures. This chapter examines these mechanisms as an integrated design problem.

Learning Objectives

After completing this chapter, the learner should be able to:

1. Distinguish resource requests, limits, quotas, and priorities.

2. Configure scheduling constraints and select suitable placement strategies.

3. Explain horizontal, vertical, and cluster autoscaling.

4. Design workloads for high availability and self-healing.

5. Apply graceful termination, readiness, liveness, and startup controls.

6. Evaluate fault-tolerance, backup, disaster-recovery, and capacity-planning strategies.

7. Identify common operational mistakes and governance risks.

8. Analyze a production scenario involving performance, failure, and recovery.

Resource Management as a Contract

Kubernetes resource management is best understood as a contract between a workload and the cluster. A workload declares what it requires and what it is allowed to consume; the scheduler and kubelet use those declarations to make placement and enforcement decisions.

A container resource request expresses the amount of a resource needed for scheduling. For CPU, it is commonly expressed in millicores, where 1000m represents one CPU core. Memory is expressed in bytes or units such as Mi and Gi. If a Pod requests 500m CPU and 512Mi memory, the scheduler attempts to place it on a node with sufficient allocatable capacity. Requests are not merely performance hints: they influence placement and, for CPU and memory, quality-of-service classification.

A limit defines an upper boundary for consumption. CPU exceeding its limit is generally throttled, whereas a container exceeding its memory limit may be terminated by the kernel through an out-of-memory event. Limits therefore require careful testing. A limit that is too low can create artificial failures, while an unlimited or excessively high limit can permit one workload to harm other workloads.

For a Pod, effective resource requirements are calculated from its containers. For ordinary application containers, requests are additive across containers. Init containers are treated differently because they run sequentially; scheduling must account for the largest applicable init-container requirement and the sum of regular containers. This distinction matters when a short-lived initialization process requires substantial memory.

A namespace can be governed by a ResourceQuota. A quota limits aggregate consumption or object counts, such as total requested CPU, total memory limits, or the number of Pods. Quotas support multi-tenancy by preventing one team from consuming all cluster capacity. A LimitRange can impose default requests and limits or minimum and maximum values for individual containers. Together, quotas and limit ranges create enforceable organizational boundaries.

Kubernetes assigns Pods a quality-of-service class. A Guaranteed Pod has matching CPU and memory requests and limits for every container. A Burstable Pod has at least one request or limit but does not satisfy the Guaranteed conditions. A BestEffort Pod has no requests or limits. Under node memory pressure, BestEffort workloads are generally more vulnerable to eviction, followed by Burstable workloads according to usage relative to requests, while Guaranteed workloads receive stronger protection. QoS is not a substitute for capacity planning, but it provides an important eviction and prioritization signal.

Resource management must include non-compute resources. Ephemeral storage is consumed by writable container layers, logs, and temporary files. Persistent volumes provide durable storage, but they introduce capacity, performance, availability, and backup requirements. Extended resources, such as GPUs, can be advertised by nodes and requested by specialized workloads. Device plugins make such resources schedulable, but they do not remove the need for workload isolation and monitoring.

Scheduling: Placing Workloads Intentionally

The scheduler first filters nodes that cannot satisfy a Pod's requirements and then scores feasible nodes according to scheduling preferences. A production design should not rely on accidental placement. It should express relevant operational intent explicitly.

Node selectors and node affinity constrain placement according to labels. Required node affinity is a hard rule; preferred affinity expresses a soft preference. Taints and tolerations provide the inverse mechanism: a tainted node repels Pods unless those Pods carry a matching toleration. These features are useful for separating system workloads, high-memory services, regulated workloads, or accelerator-dependent applications.

Pod affinity and anti-affinity express relationships between Pods. Anti-affinity can prevent replicas of the same service from sharing a node, reducing the impact of node failure. Topology spread constraints provide a more systematic way to distribute replicas across zones, nodes, or other topology domains. A resilient service commonly combines multiple replicas with spreading across failure domains rather than merely across host names.

PriorityClass allows Pods to be ranked when resources are scarce. Preemption may remove lower-priority Pods to make room for higher-priority work. This is powerful but potentially disruptive. Priority should reflect business criticality and recovery requirements, not the personal preference of a development team. Poorly designed priorities can cause noncritical workloads to evict useful services or create starvation.

Scheduling decisions should account for requests rather than average consumption alone. A cluster may appear lightly utilized in monitoring dashboards while remaining unschedulable because declared requests exhaust allocatable capacity. Conversely, inflated requests waste capacity and increase cost. The objective is not to minimize requests indiscriminately but to make them credible representations of operational need.

Scaling Models and Feedback Control

Scaling is a feedback-control problem. A controller observes signals, compares them with a target, and changes the number or size of resources. Because observation and actuation take time, scaling policies must account for delay, noise, startup time, and oscillation.

Horizontal Pod Autoscaling (HPA) changes the number of replicas in a workload such as a Deployment or StatefulSet. It can use CPU or memory utilization relative to requests, or custom and external metrics. CPU utilization is meaningful only when CPU requests are realistic. A service with a request far below its normal consumption may scale prematurely; a request far above normal usage may scale too late.

Custom metrics often provide better business alignment. Queue length, requests per second, concurrent sessions, or latency-based signals may represent demand more directly than CPU. However, metrics must be stable, available, and carefully interpreted. Latency can rise because of a downstream failure, in which case indiscriminate scaling may amplify pressure rather than solve the root problem.

HPA behavior can be tuned through stabilization windows, scaling policies, and separate rules for scale-up and scale-down. Rapid scale-up and deliberately slower scale-down often reduce oscillation. Autoscaling cannot compensate for an application that has a fixed connection bottleneck, insufficient database capacity, or long initialization time.

Vertical Pod Autoscaling (VPA) recommends or adjusts resource requests and limits. It is valuable for workloads whose resource profile is difficult to estimate, but applying changes may require Pod restarts. VPA and HPA can conflict when both act on the same resource signal. A common design is to use HPA for replica count and VPA for resource recommendation, while avoiding uncontrolled simultaneous changes.

The Cluster Autoscaler changes the number of nodes when Pods cannot be scheduled or when nodes are substantially underutilized and their workloads can be moved. It depends on a cloud or infrastructure provider that can create and remove nodes. Node scale-up is not instantaneous; therefore, critical systems may require a baseline of spare capacity. Autoscaling also needs disruption controls so that node removal does not violate availability objectives.

A useful capacity model distinguishes steady-state demand, expected peaks, failure reserve, and operational headroom. If normal demand is 60 percent of capacity and a zone failure removes one-third of nodes, the remaining cluster must still accommodate critical workloads. Capacity planning should use workload growth forecasts, load tests, historical percentiles, deployment surge requirements, and recovery-time objectives rather than a single average utilization figure.

Designing for High Availability and Self-Healing

A highly available service minimizes common failure domains. A Deployment maintains a desired number of replicas and replaces Pods that terminate or fail scheduling. A ReplicaSet ensures the replica count, while a Service provides stable discovery and load distribution. These controllers provide self-healing, but only within the boundaries of their configuration. If all replicas are placed on one node, a node failure can still cause total service loss.

Readiness, liveness, and startup probes have distinct purposes. A readiness probe answers whether a Pod should receive traffic. A liveness probe determines whether a container is stuck and should be restarted. A startup probe protects slow-starting applications by delaying liveness evaluation until initialization has completed. Confusing these probes is a frequent cause of outages. A readiness failure should normally remove a Pod from service; it should not necessarily restart the process.

Probes should test meaningful conditions without depending excessively on fragile downstream systems. A liveness probe that fails whenever a database is temporarily unavailable can restart every replica simultaneously, increasing the outage. Readiness may include dependency status when serving without that dependency is unsafe, but liveness should usually establish whether the process can make progress.

PodDisruptionBudget (PDB) limits voluntary disruptions, such as node draining or maintenance, by requiring a minimum number of available replicas or limiting simultaneous disruption. A PDB does not protect against involuntary failures such as a sudden node crash. It also cannot create availability when the deployment has too few replicas or insufficient capacity.

Rolling updates should be configured with suitable maxUnavailable and maxSurge values. A zero-unavailability strategy may require additional capacity. A small surge can reduce risk but increases temporary resource demand. Readiness gates, application-level compatibility, and database migration design are equally important. During a rolling update, old and new versions may coexist, so APIs and data schemas should support controlled compatibility.

Graceful Termination and Fault Tolerance

Kubernetes termination is a coordinated process rather than an immediate kill. When a Pod is deleted, it is marked for termination, removed from normal service endpoints, and sent a termination signal such as SIGTERM. The application should stop accepting new work, finish or safely abandon in-flight work, close connections, and exit before the termination grace period expires. If it does not exit, the container may receive SIGKILL.

Application design determines whether graceful termination works. Web servers should drain connections; consumers should stop receiving new messages and acknowledge only completed work; batch jobs should checkpoint progress. PreStop hooks may support coordination, but they consume the termination period and should not replace proper signal handling. Load balancers and endpoint propagation introduce delay, so applications often need a brief draining interval.

Fault tolerance requires identifying failure modes and choosing controls proportionate to their effects. Replication addresses some instance failures. Timeouts prevent indefinite resource occupation. Retries can overcome transient faults but must use bounded exponential backoff and jitter. Circuit breakers prevent repeated calls to an failing dependency. Bulkheads isolate pools of resources so that one dependency cannot consume all threads or connections. Idempotency makes retry and recovery safer, especially for message processing and payment-like operations.

Stateful services require additional care. Replicas are not automatically equivalent when data consistency, leader election, or persistent volumes are involved. StatefulSets provide stable identities and ordered operations, but they do not themselves implement database replication or backup. Storage availability zones, volume attachment behavior, replication, and recovery procedures must be evaluated together.

Backup, Disaster Recovery, and Continuity

High availability and disaster recovery address different failure scales. High availability reduces interruption during ordinary component failures. Disaster recovery restores service after larger events, such as regional loss, corruption, ransomware, operator error, or destructive configuration changes.

A backup strategy should define what is backed up, how often, where copies are stored, how they are encrypted, how long they are retained, and how restoration is verified. Kubernetes object manifests, Helm values, policy definitions, secrets, persistent application data, and external dependencies may all require separate treatment. A backup that captures only cluster objects but not database contents is incomplete for a stateful system.

Recovery Point Objective (RPO) specifies the maximum acceptable data loss measured in time. Recovery Time Objective (RTO) specifies the maximum acceptable restoration time. These objectives influence replication, backup frequency, architecture, staffing, and cost. A low RPO may require synchronous or frequent asynchronous replication; a low RTO may require a warm secondary environment rather than an archive that must be rebuilt manually.

Restoration must be tested. A backup is evidence of recoverability only after it has been restored successfully in an isolated environment. Tests should validate application consistency, identity and access configuration, DNS, certificates, storage, dependencies, and operational runbooks. Disaster recovery also requires protection against restoring compromised or corrupted data; immutable and versioned backup copies are valuable safeguards.

Illustrative Case Study: Scaling a Multi-Zone Order Service

Background

Northstar Retail operates an online ordering platform on Kubernetes. The platform consists of a stateless API, a message-consuming fulfillment service, a relational database, and a cache. Traffic increases sharply during promotional events. The original deployment used three API replicas with no explicit resource requests, a single-zone node pool, and a CPU-based HPA.

Situation and Problem

During a promotion, API latency increased and several Pods were repeatedly restarted. The HPA added replicas, but new Pods remained Pending because node capacity was insufficient. At the same time, a memory-intensive reporting process consumed resources on the same nodes. A later maintenance operation drained the only active zone, causing a complete service interruption. Database backups existed, but restoration had never been tested.

Analysis

The absence of requests made scheduling and HPA behavior unreliable. CPU was not the primary bottleneck; database connections and memory pressure were more significant. Replicas were not distributed across failure domains. The reporting workload had no quota or priority boundary. Finally, the organization had treated node redundancy as equivalent to application availability and had not defined RPO or RTO.

Approach

The engineering team first measured CPU, memory, request rate, latency, queue depth, startup time, and database connection utilization under representative load. They then established realistic requests and limits, applied a namespace quota, and isolated reporting workloads with labels, taints, and a separate node pool. The API was deployed with six replicas distributed across three zones using topology spread constraints.

The HPA was changed to use request rate and queue-related custom metrics, with rapid scale-up and slower scale-down. A minimum baseline of nodes was retained so that normal peak demand did not depend entirely on cloud provisioning time. Readiness and startup probes were redesigned, and the application implemented graceful connection draining. A PDB protected the API during voluntary maintenance.

The database team introduced tested backups, encrypted off-site copies, and a documented restoration process. RPO and RTO were agreed with business stakeholders. A simulated zone failure and a restore exercise were performed before the next promotion.

Outcome

The revised design prevented reporting activity from exhausting application capacity, distributed API replicas across zones, and allowed the platform to scale according to actual demand. The organization also discovered during the restore test that a required secret and an external payment configuration were missing from the initial recovery procedure. Correcting these omissions made the recovery plan materially more credible.

Learning Points

The case demonstrates that autoscaling cannot repair inaccurate resource models, insufficient failure-domain diversity, or an under-capacity cluster. High availability depends on placement, application behavior, storage, dependencies, and tested operational procedures. Recovery planning must include configuration and external services, not only Kubernetes manifests.

Risks, Limitations, and Common Mistakes

Several mistakes recur in production environments:

• Setting identical, arbitrary requests and limits for every service.
• Treating memory limits like CPU limits and overlooking out-of-memory termination.
• Using CPU-only autoscaling for I/O-bound or queue-driven workloads.
• Creating a PDB with a single replica and assuming it provides meaningful protection.
• Making liveness probes depend on every external dependency.
• Deploying multiple replicas without topology distribution.
• Allowing unlimited retries, which can create retry storms.
• Scaling application replicas without scaling databases, queues, or connection pools.
• Removing nodes without considering local storage, PDBs, or lengthy eviction.
• Assuming a successful backup job proves that restoration will work.
• Using priority and preemption without analyzing starvation and fairness.
• Ignoring ephemeral storage, log growth, and image-pull capacity.

Autoscaling also has economic and environmental implications. Excessive replicas, oversized requests, and unnecessary high-performance nodes increase cost and energy consumption. Conversely, aggressive consolidation may reduce resilience. The correct objective is economically sustainable service reliability, not maximum utilization in isolation.

Governance, Security, and Professional Practice

Resource quotas and priority policies are governance mechanisms as well as technical settings. Organizations should define ownership, service classes, escalation procedures, and approval requirements for high-priority workloads. Namespaces and service accounts should follow least-privilege principles. Secrets must not be embedded in images or casually stored in source repositories; backup copies must receive equivalent protection.

Capacity and resilience decisions should be auditable. Teams should document assumptions, service-level objectives, dependency limits, RPO, RTO, and evidence from load and recovery tests. Regulatory environments may impose requirements for data location, retention, access logging, and recoverability. A technically available service may still be unacceptable if it violates privacy, segregation, or data-residency obligations.

Emerging Directions

Modern platforms increasingly combine Kubernetes metrics with distributed tracing, event-driven autoscaling, and predictive capacity analysis. OpenTelemetry can correlate resource pressure with request paths and dependency behavior. Event-driven autoscalers can respond to queue depth or cloud events, while policy engines can enforce limits, security rules, and topology requirements before deployment.

AI-assisted operations may help identify abnormal utilization, forecast demand, or recommend resource settings. Such recommendations require human validation because historical data may encode outages, inefficient configurations, or temporary conditions. Automated remediation should be bounded by clear safety policies, approval levels, and rollback mechanisms. The central principle remains unchanged: automation improves resilience only when its signals and actions are trustworthy.

Best-Practice Design Sequence

A disciplined implementation can proceed as follows:

1. Define service-level objectives, dependencies, RPO, and RTO.

2. Measure workload behavior under normal, peak, and failure conditions.

3. Set evidence-based requests, limits, quotas, and priorities.

4. Design placement across nodes and failure domains.

5. Configure probes, rolling updates, disruption budgets, and termination behavior.

6. Select scaling signals and test controller response times.

7. Validate downstream capacity, storage, connection pools, and external services.

8. Automate backups and regularly perform restoration exercises.

9. Test node, zone, dependency, overload, and deployment failures.

10. Review cost, security, governance, and operational evidence continuously.

Chapter Summary

Resource management establishes the constraints within which Kubernetes schedules and runs workloads. Requests influence scheduling and autoscaling, limits constrain consumption, quotas govern aggregate use, and priorities determine behavior during scarcity. Effective scheduling distributes workloads across suitable nodes and failure domains through affinity, taints, tolerations, and topology constraints.

Scaling may be horizontal, vertical, or cluster-wide, and each approach depends on reliable metrics and realistic timing assumptions. High availability requires replication, appropriate placement, healthy probes, controlled updates, and disruption policies. Self-healing is strengthened by graceful termination, idempotent processing, bounded retries, timeouts, and dependency isolation. Disaster recovery extends beyond cluster objects to include data, secrets, external systems, tested restoration, RPO, and RTO. The mature practitioner treats resilience as an engineered and continuously tested property rather than a default feature of Kubernetes.

Key Terms

Request: The resource amount reserved for scheduling a container.

Limit: The maximum resource consumption permitted for a container.

ResourceQuota: A namespace-level aggregate limit on resources or object counts.

LimitRange: A policy defining defaults and boundaries for individual containers.

Quality of Service: A Kubernetes classification influencing resource and eviction behavior.

Allocatable Capacity: The node capacity available for scheduled workloads after system reservations.

Node Affinity: A rule or preference controlling placement according to node labels.

Taint: A node attribute that repels Pods without matching tolerations.

Topology Spread Constraint: A rule distributing Pods across failure or infrastructure domains.

PriorityClass: A ranking assigned to Pods for scheduling and possible preemption.

Horizontal Pod Autoscaler: A controller that adjusts replica count using observed metrics.

Vertical Pod Autoscaler: A mechanism that recommends or changes resource settings.

Cluster Autoscaler: A controller that adjusts the number of cluster nodes.

Readiness Probe: A check determining whether a Pod should receive traffic.

Liveness Probe: A check used to determine whether a container should be restarted.

Startup Probe: A check that protects slow-starting containers from premature liveness failure.

PodDisruptionBudget: A policy limiting voluntary disruption of replicated workloads.

RPO: The maximum acceptable period of data loss after a disruption.

RTO: The maximum acceptable time required to restore service.

Graceful Termination: Controlled shutdown that drains work and releases resources safely.

Review Questions

Short Answer Questions

1. How do resource requests differ from resource limits?

2. What purpose does a ResourceQuota serve in a multi-team cluster?

3. Distinguish readiness, liveness, and startup probes.

4. Why are topology spread constraints useful for highly available services?

5. What do RPO and RTO measure?

Long Answer Questions

1. Explain how requests, limits, QoS classes, quotas, and limit ranges interact in Kubernetes resource management.

2. Compare HPA, VPA, and Cluster Autoscaler, including appropriate use cases and limitations.

3. Discuss the design requirements for a highly available Deployment operating across multiple zones.

4. Explain how graceful termination, bounded retries, timeouts, circuit breakers, and idempotency contribute to fault tolerance.

5. Design a disaster-recovery strategy for a stateful Kubernetes application, including backup, restoration, RPO, and RTO.

Analytical / Application Questions

1. A service has six replicas but all are scheduled on one node. Which Kubernetes mechanisms would you apply to reduce the impact of node failure, and why?

2. An HPA scales rapidly during a database outage, increasing connection failures. Analyze the feedback problem and propose safer scaling and dependency-protection measures.

3. A cluster reports low average CPU utilization, yet new Pods remain Pending during deployments. Identify likely causes and explain how you would investigate and correct the capacity model.

22. Multiple Choice Questions with Answers

Question 1

Which statement best describes a Kubernetes resource request?

A. It guarantees unlimited access to the resource

B. It is used primarily by the scheduler to determine placement

C. It causes the container to restart when exceeded

D. It limits the total resource use of a namespace

Correct Answer: B. It is used primarily by the scheduler to determine placement

Question 2

A container exceeds its memory limit. What is the most likely result?

A. The container is throttled indefinitely

B. The Pod is automatically moved to another zone

C. The container may be terminated by an out-of-memory event

D. The namespace quota is increased

Correct Answer: C. The container may be terminated by an out-of-memory event

Question 3

Which mechanism is most directly intended to protect a workload during voluntary node maintenance?

A. PodDisruptionBudget

B. Startup probe

C. LimitRange

D. Container image policy

Correct Answer: A. PodDisruptionBudget

Question 4

Why can CPU-based HPA be unsuitable for a queue-consuming service?

A. CPU metrics are never available in Kubernetes

B. Queue depth may represent demand more directly than CPU utilization

C. HPA can scale only StatefulSets

D. CPU requests prevent all horizontal scaling

Correct Answer: B. Queue depth may represent demand more directly than CPU utilization

Question 5

What is the principal purpose of a readiness probe?

A. To decide whether traffic should be sent to the Pod

B. To determine the node's total capacity

C. To assign a PriorityClass

D. To create a persistent volume

Correct Answer: A. To decide whether traffic should be sent to the Pod

Question 6

Which design most effectively reduces the risk that one node failure removes all replicas?

A. Set all replicas to BestEffort QoS

B. Use topology distribution and multiple replicas

C. Remove resource requests

D. Use only a liveness probe

Correct Answer: B. Use topology distribution and multiple replicas

Question 7

What is a major risk of an unlimited retry policy?

A. It prevents all service discovery

B. It can create retry storms and increase dependency overload

C. It disables graceful termination

D. It guarantees duplicate-free processing

Correct Answer: B. It can create retry storms and increase dependency overload

Question 8

What does an RPO define?

A. The maximum acceptable time to restore service

B. The maximum acceptable amount of data loss measured in time

C. The maximum number of replicas in a Deployment

D. The percentage of CPU reserved for system Pods

Correct Answer: B. The maximum acceptable amount of data loss measured in time

Question 9

Why should a backup be restored periodically in a test environment?

A. To increase the cluster's CPU limit

B. To prove that stored data and required configuration are actually recoverable

C. To prevent all node evictions

D. To replace the need for replication

Correct Answer: B. To prove that stored data and required configuration are actually recoverable

Question 10

A cluster has low observed utilization but cannot schedule new Pods. Which explanation is most plausible?

A. Declared resource requests may have consumed allocatable capacity

B. Readiness probes always reserve all node memory

C. Services cannot run more than one Pod

D. PDBs convert CPU into memory

Correct Answer: A. Declared resource requests may have consumed allocatable capacity