3. Workload, Scheduling and Resource Management

Introduction and Context

Kubernetes is often introduced as a platform for deploying containers, but its deeper value lies in the way it continuously reconciles desired application states with the resources available in a distributed cluster. Workload management is therefore not limited to creating Pods. It involves selecting suitable workload controllers, expressing resource requirements, placing workloads intelligently, preserving availability during failures and maintenance, and adapting capacity as demand changes.

A poorly designed workload may be operationally correct yet economically inefficient, difficult to scale, or vulnerable to disruption. Conversely, a well-engineered workload uses requests and limits deliberately, cooperates with the scheduler, tolerates infrastructure failure, and exposes sufficient signals for autoscaling. These concerns are closely related: resource requests influence scheduling; scheduling decisions influence availability and cost; autoscaling changes the number or size of workloads; and disruption policies determine whether maintenance can proceed safely.

This chapter examines Kubernetes Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, resource management, scheduling constraints, priorities, autoscaling, and disruption control. The emphasis is on engineering decisions rather than command memorization.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Select and configure appropriate Kubernetes workload controllers.

2. Distinguish between requests, limits, quality of service classes, quotas, and limit ranges.

3. Explain how the scheduler filters and scores candidate nodes.

4. Apply affinity, anti-affinity, taints, tolerations, and topology spread constraints.

5. Design horizontal, vertical, and cluster-level autoscaling strategies.

6. Use priorities and PodDisruptionBudgets to manage contention and voluntary disruption.

7. Analyse scheduling failures, resource waste, and availability risks.

8. Evaluate workload designs from technical, operational, financial, and governance perspectives.

Workload Controllers as Operational Contracts

A Pod is the smallest deployable unit in Kubernetes, but it is rarely the correct object for directly managing a production application. Controllers encode operational intent: how many instances should exist, whether identity must persist, whether work should run once, or whether every node requires a local agent.

Deployments and ReplicaSets

A Deployment manages a stateless application through ReplicaSets. It declares a desired replica count and a Pod template. Kubernetes creates or adjusts ReplicaSets so that the requested number of Pods is available. A change to the Pod template creates a new ReplicaSet, enabling controlled rollout and rollback.

Rolling updates are governed mainly by `maxUnavailable` and `maxSurge`. A high surge permits faster replacement but temporarily consumes additional capacity. A low unavailability value protects service continuity but may make rollout slower or impossible in a tightly packed cluster. A Deployment should normally include readiness probes so that a newly created Pod is counted as available only when it can serve traffic.

Deployments are appropriate for stateless APIs, web servers, workers whose state is externalized, and services where any replica can handle any request. They are not a substitute for a data-replication system. If application state is stored only inside a container filesystem, replacing a Pod can cause data loss.

StatefulSets

A StatefulSet manages applications requiring stable identity, ordered operations, or persistent storage. Each replica receives a predictable name such as `database-0` and may receive a stable network identity through a headless Service. PersistentVolumeClaims can be generated per replica, allowing storage to follow the logical member rather than the transient Pod.

StatefulSets are useful for databases, distributed queues, and clustered systems, but they do not automatically make an application highly available. The application itself must implement replication, quorum, failover, consistency, and recovery. Operators must also understand update ordering and the consequences of terminating a particular ordinal. A StatefulSet should be paired with tested backup and restoration procedures.

DaemonSets

A DaemonSet ensures that a Pod runs on every eligible node, or on every node satisfying a selector. Typical examples include log collectors, node monitoring agents, networking components, and storage plugins. DaemonSets respond automatically when nodes are added.

DaemonSet Pods consume capacity on every targeted node. Their requests must therefore be included in node sizing calculations. Taints may exclude ordinary workloads from infrastructure nodes, while tolerations allow required system agents to run there. A DaemonSet should not be used merely because an application has many replicas; it is intended for node-local or node-wide functions.

Jobs and CronJobs

A Job represents finite work. It may run one Pod to completion, run several completions, or execute parallel workers. Important controls include `backoffLimit`, completion count, parallelism, and termination grace periods. A Job must be designed for retry behaviour: if a Pod fails after partially completing work, a retry can duplicate effects unless the operation is idempotent or uses a transaction and checkpointing strategy.

A CronJob creates Jobs according to a schedule. It should include a history limit, a concurrency policy, and a defined response to missed schedules. For example, `Forbid` prevents overlapping executions, whereas `Replace` terminates an earlier Job before starting a new one. Schedules must account for time zones, long-running tasks, and external dependencies.

Resource Semantics: Requests, Limits, and Quality of Service

Kubernetes schedules a Pod primarily using resource requests. A CPU request of 500 millicores means that the scheduler accounts for half a logical CPU when deciding placement; it does not guarantee that the container always receives exactly that amount. A memory request similarly represents the amount considered necessary for placement.

Limits define an upper boundary. CPU limits can cause throttling when usage exceeds the limit. Memory limits are enforced differently: exceeding the limit can result in an out-of-memory kill. An excessively low limit can create instability, while an excessively high limit can hide inefficient code and reduce cluster packing efficiency.

Requests should be based on measured usage, startup behaviour, traffic variability, and the consequences of degradation. They should not simply equal the developer's guess or the largest observed spike. A useful operational process gathers historical metrics, establishes service-level objectives, and periodically reviews actual consumption.

Kubernetes assigns a Pod a quality of service class. A Guaranteed Pod has CPU and memory requests equal to limits for all containers. A Burstable Pod has requests or limits that do not satisfy that strict equality. A BestEffort Pod has neither CPU nor memory requests nor limits. Under memory pressure, lower-quality Pods are generally more vulnerable to eviction, although actual eviction decisions also depend on usage relative to requests and node conditions.

Resource management has two levels. `LimitRange` supplies namespace-level defaults and minimum or maximum constraints for individual containers or Pods. `ResourceQuota` limits aggregate consumption in a namespace, including resources such as CPU, memory, object counts, and sometimes storage. Quotas are governance mechanisms, not merely technical restrictions. They help allocate capacity among teams and prevent one namespace from consuming the entire cluster.

A common failure occurs when a namespace has a quota but no LimitRange. A Deployment without explicit requests may be rejected or may receive unsuitable defaults depending on policy. Conversely, a very restrictive quota can prevent a rollout because a Deployment needs temporary surge capacity. Quota design must therefore consider steady state, rollout overhead, batch bursts, and autoscaling headroom.

How the Scheduler Makes Placement Decisions

The Kubernetes scheduler first identifies feasible nodes and then ranks them. Filtering considers resource requests, node selectors, affinity rules, taints, topology constraints, volumes, and other predicates. Scoring ranks feasible nodes according to preferences such as balanced resource usage or preferred affinity.

A scheduling failure should be interpreted as a set of constraints with no common solution. For example, a Pod may request two CPUs, require a particular zone, reject a tainted node, and demand anti-affinity from existing replicas. Each rule may be reasonable independently, but their combination may make placement impossible. Diagnosis should therefore begin with events and node conditions, followed by examination of requests, labels, taints, affinities, quotas, and available capacity.

Node Selection, Affinity, and Anti-Affinity

A `nodeSelector` expresses a simple requirement: the node must possess specified labels. Node affinity provides richer required and preferred expressions. Required affinity is a hard constraint; preferred affinity influences scoring but does not make scheduling impossible if no preferred node exists.

Pod affinity places a Pod near other Pods matching a label rule, which can reduce network latency or co-locate cooperating services. Pod anti-affinity separates replicas, improving failure tolerance. Anti-affinity based only on hostname can distribute replicas across nodes, but it does not necessarily distribute them across zones. The topology key must match the desired failure domain.

Taints and Tolerations

A taint marks a node so that Pods without a matching toleration are not placed there. Effects include `NoSchedule`, `PreferNoSchedule`, and `NoExecute`. The last can also evict existing Pods that do not tolerate the taint. Taints are useful for dedicated hardware, control-plane nodes, GPUs, maintenance states, and isolation of infrastructure workloads.

A toleration permits scheduling; it does not require scheduling. This distinction is important. A GPU workload may tolerate a GPU-node taint, but it should also use node affinity or a device request if it must actually run on a GPU node. Broad tolerations can unintentionally allow application Pods onto nodes reserved for critical infrastructure.

Topology Spread Constraints

Topology spread constraints express a desired distribution of Pods across domains such as zones, regions, or nodes. `maxSkew` controls how uneven the distribution may be, while `whenUnsatisfiable` determines whether the rule is hard or merely advisory. Spread constraints often provide a clearer availability policy than manually combining multiple anti-affinity rules.

They must be evaluated alongside replica count. Three replicas cannot be evenly distributed across four zones. A strict constraint may leave a Pod Pending rather than accept an uneven but operationally acceptable placement. Engineering decisions should reflect the application's failure model and capacity budget.

Priority, Preemption, and Disruption

PriorityClasses assign relative importance to Pods. When capacity is scarce, a higher-priority Pod may preempt lower-priority Pods, subject to scheduling constraints and termination behaviour. Priority is not a replacement for capacity planning. Excessive high-priority classification can cause instability and makes the ordering meaningless.

Preemption should be used carefully for workloads whose absence has materially greater consequences. Batch jobs should generally have lower priority than critical serving workloads, but a low-priority job can still be essential to a business process. Priority policy should therefore be documented and governed rather than assigned informally by individual teams.

A PodDisruptionBudget (PDB) limits voluntary disruption, such as node draining or cluster maintenance. It can specify `minAvailable` or `maxUnavailable`. A PDB does not protect against involuntary failures such as hardware loss, kernel panic, or out-of-memory termination. Nor does it create additional capacity. An overly strict PDB can block maintenance indefinitely, especially when a workload has too few replicas.

Availability requires coordination among replica count, topology distribution, readiness, PDB settings, and spare capacity. A three-replica Deployment with all replicas on one node is not resilient merely because its PDB requires two available replicas.

Autoscaling as a Feedback System

Autoscaling changes workload or infrastructure capacity in response to observed demand. The Horizontal Pod Autoscaler (HPA) changes replica count based on metrics such as CPU, memory, or application-specific indicators. CPU utilization is usually calculated relative to requests, so inaccurate requests can produce misleading scaling decisions. A small request can make a modest absolute CPU use appear highly utilized; a large request can suppress scaling.

The Vertical Pod Autoscaler (VPA) recommends or applies changes to resource requests and limits. It is valuable for workloads whose resource needs are difficult to estimate, but applying recommendations may restart Pods. VPA and HPA should not independently control the same resource dimension without a clear design, because they can produce conflicting feedback.

The Cluster Autoscaler adjusts the number of nodes when Pods cannot be scheduled or when nodes are underutilized and can be safely removed. It cannot solve a Pod that is unschedulable because of an impossible affinity rule, an excessive request, or a restrictive quota. Node provisioning time must also be considered: scaling infrastructure is slower than scaling replicas.

Autoscaling is a control loop with delay, measurement error, and possible oscillation. Stabilization windows, scale-up and scale-down policies, cooldown periods, queue metrics, and minimum and maximum bounds help prevent thrashing. For queue workers, queue depth or oldest-message age may be better signals than CPU. For latency-sensitive APIs, request rate and tail latency can be more meaningful, provided the metric is carefully defined.

A Resource and Scheduling Engineering Method

A practical design method begins with workload classification. Determine whether the workload is stateless, stateful, node-local, finite, or scheduled. Next, identify failure domains and recovery expectations. Then establish resource requests from measurements and define limits only where their operational effect is understood.

The next step is to model placement. Identify required labels, acceptable zones, dedicated nodes, storage topology, and distribution goals. Use hard constraints only for genuine requirements; use preferred constraints for desirable but negotiable outcomes. Test the combined constraints in a cluster that resembles production.

Finally, define scaling and disruption behaviour. Select metrics, bounds, stabilization policies, priorities, and PDBs together. Observe rollout capacity, maintenance procedures, and failure recovery. A design is incomplete until it has been tested under node loss, resource pressure, rolling updates, quota exhaustion, and metric unavailability.

Illustrative Case Study: Multi-Zone Payment Processing Platform

Background

A fictional financial-technology company operates a payment API in a three-zone Kubernetes cluster. The API is stateless, while reconciliation jobs process transactions from a durable queue. The company also runs log collection and node monitoring on every worker node.

Situation and Problem

The original API Deployment used six replicas but had no resource requests, no topology rules, and no PDB. During traffic peaks, several replicas were placed on one node. A node upgrade caused simultaneous termination of those replicas, increasing latency. Reconciliation Jobs then competed with the API for resources, and an HPA based on CPU reacted slowly because requests were absent or inconsistent.

Analysis

The workload consisted of three distinct classes: a latency-sensitive stateless service, finite asynchronous workers, and node-local agents. Treating them alike caused scheduling and priority conflicts. The absence of requests prevented meaningful capacity planning. The lack of spread constraints allowed concentration in a single failure domain. Finally, the maintenance process had no disruption budget to communicate the minimum acceptable service level.

Approach and Solution

The API was placed in a Deployment with measured CPU and memory requests, conservative memory limits, readiness probes, and a minimum of nine replicas. Required topology spread constraints distributed replicas across zones, while preferred node-level anti-affinity reduced co-location. An HPA used request rate and CPU with bounded scale-up and scale-down behaviour. A PDB permitted at most one voluntary unavailable replica, and node pools were sized with rollout and maintenance headroom.

Reconciliation was implemented as a lower-priority worker Deployment consuming queue-depth metrics. Workers were made idempotent so that retries would not create duplicate payments. A namespace quota reserved capacity for the API, while a separate batch namespace limited reconciliation bursts. Log and monitoring agents were managed through DaemonSets and tolerated the taints on infrastructure nodes.

Outcome

The revised design separated failure tolerance from workload identity, made capacity consumption visible, and prevented routine batch activity from displacing the API. Maintenance could proceed without violating the service's availability objective. The organization also gained a repeatable review process for requests, scaling signals, and disruption policies.

Learning Points

The central lesson is that availability is a system property. Replicas alone are insufficient; they must be distributed, ready, protected during voluntary disruption, and supported by adequate capacity. A second lesson is that resource values are operational hypotheses requiring measurement and revision. Finally, workload controllers should reflect application semantics rather than deployment convenience.

Challenges, Risks, and Common Mistakes

One frequent mistake is setting identical high requests for every container. This improves predictability but can create artificial fragmentation and unnecessary node costs. The opposite mistake—omitting requests—causes poor scheduling, unreliable autoscaling, and weak isolation.

Another error is using `latest` images and treating rollback as automatic. A controller can restore a prior template, but only if image versions and configuration are reproducible. Stateful workloads are often deployed without testing restore procedures, while Jobs are made non-idempotent and then configured for aggressive retries.

Overconstrained scheduling is also common. Hard affinity, strict anti-affinity, taints, storage restrictions, and topology rules may collectively eliminate all feasible nodes. Constraints should be reviewed as a system. Similarly, a PDB with `minAvailable` equal to the replica count may prevent voluntary maintenance, while a permissive PDB may provide little protection.

Security and governance must be considered. Resource quotas support fair allocation, but namespaces are not complete security boundaries. Workload identity, RBAC, admission policies, image provenance, and network controls remain necessary. Priority classes should be centrally governed, and autoscaling limits should prevent uncontrolled expenditure.

Current Developments and Professional Practice

Modern clusters increasingly use custom metrics, event-driven autoscaling, and workload-aware node provisioning. Autoscaling systems can react to queue events or external demand rather than waiting for CPU saturation. Scheduling extensions and admission controllers can enforce organizational policies before Pods reach the scheduler.

The growing use of heterogeneous hardware—including GPUs, high-memory nodes, and specialized accelerators—makes resource discovery and node labeling more important. Cost-aware scheduling is also becoming significant: teams may combine spot or preemptible capacity for fault-tolerant batch workloads with reserved capacity for critical services.

AI-assisted operational tools can help identify anomalous resource usage, recommend requests, or explain unschedulable Pods. Such recommendations must remain subject to human review because historical usage may reflect incidents, poor application behaviour, or temporary traffic patterns. Automation should improve evidence-based decision-making, not replace accountability.

Best-Practice Guidance

A mature workload platform generally follows these principles:

• use the controller whose semantics match the workload;
• define and periodically recalibrate resource requests;
• apply limits with awareness of throttling and eviction effects;
• use quotas and limit ranges as explicit governance mechanisms;
• prefer simple, necessary scheduling constraints over complex rules;
• distribute critical replicas across meaningful failure domains;
• make retryable work idempotent;
• design HPA, VPA, and cluster scaling as one feedback system;
• use priorities and disruption budgets consistently across teams;
• test rollouts, drains, node failures, quota exhaustion, and restoration;
• monitor pending Pods, evictions, throttling, OOM kills, utilization, and scaling latency.

Chapter Summary

Kubernetes workload engineering connects application semantics with cluster capacity and failure behaviour. Deployments manage replaceable stateless replicas; StatefulSets preserve identity and storage relationships; DaemonSets provide node-local services; Jobs and CronJobs execute finite or scheduled work. Requests influence scheduling and autoscaling, while limits affect enforcement and stability. Quotas and limit ranges provide namespace-level governance.

The scheduler combines filtering and scoring, using node and Pod affinity, taints and tolerations, topology spread, volumes, and resource availability. Priorities can resolve contention, but preemption requires governance. PodDisruptionBudgets protect against excessive voluntary disruption but do not protect against unexpected failures. HPA, VPA, and cluster autoscaling form interacting feedback loops that require suitable metrics, bounds, and stabilization. Effective designs are measured, failure-aware, and tested under realistic operational conditions.

Key Terms

Pod: The smallest deployable Kubernetes unit, consisting of one or more containers and shared execution context.

Deployment: A controller for managing versioned, usually stateless, replicated applications.

StatefulSet: A controller providing stable identity, ordering, and persistent storage associations.

DaemonSet: A controller that places a Pod on every eligible node.

Job: A controller for finite work that runs until specified completion conditions are met.

CronJob: A controller that creates Jobs according to a schedule.

Resource request: The amount of a resource used for scheduling and capacity accounting.

Resource limit: The maximum resource usage permitted for a container.

Quality of Service class: A Kubernetes classification affecting resource management and eviction behaviour.

ResourceQuota: A namespace-level limit on aggregate resource or object consumption.

LimitRange: A namespace policy defining defaults or bounds for individual resources.

Node affinity: A rule expressing required or preferred node-label placement.

Pod anti-affinity: A rule that discourages or prevents placement near selected Pods.

Taint: A node marker that repels Pods without matching tolerations.

Toleration: A Pod declaration permitting it to be scheduled onto a tainted node.

Topology spread constraint: A rule controlling distribution across failure or infrastructure domains.

PriorityClass: A relative importance value used during scheduling and possible preemption.

PodDisruptionBudget: A policy limiting voluntary disruption of a workload.

Horizontal Pod Autoscaler: A controller that changes replica count using observed metrics.

Cluster Autoscaler: A system that adjusts the number of cluster nodes according to scheduling demand and utilization.

Review Questions

Short Answer Questions

1. Why are resource requests more important than limits during initial scheduling?

2. What distinguishes a StatefulSet from a Deployment?

3. Explain the difference between a taint and a toleration.

4. What type of disruption does a PodDisruptionBudget control?

5. Why can inaccurate CPU requests distort HPA behaviour?

Long Answer Questions

1. Explain the operational differences among Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs, including suitable use cases.

2. Describe how requests, limits, Quality of Service classes, LimitRanges, and ResourceQuotas interact.

3. Explain the scheduler's filtering and scoring process and analyse how affinity, anti-affinity, taints, and topology spread affect it.

4. Compare HPA, VPA, and Cluster Autoscaler, including the risks of poorly coordinated feedback loops.

5. Develop a resilient workload strategy for a multi-zone production service, including scaling, priority, disruption, and capacity considerations.

Analytical / Application Questions

1. A three-replica service is deployed across a cluster with two zones, a strict zone spread rule, and a PDB requiring all replicas to remain available. Analyse the risks during node maintenance and propose changes.

2. A batch Job repeatedly fails after partially writing records, and its `backoffLimit` causes multiple retries. Explain the likely business risk and design an idempotent processing strategy.

3. A Pod remains Pending even though several nodes appear to have free CPU. Analyse at least five possible causes involving requests, labels, taints, affinity, topology, storage, or quotas, and propose a diagnostic sequence.

22. Multiple Choice Questions with Answers

Question 1

Which Kubernetes controller is most appropriate for a stateless HTTP service requiring versioned rolling updates?

A. Deployment

B. StatefulSet

C. DaemonSet

D. CronJob

Correct Answer: A. Deployment

Question 2

What is the primary scheduling role of a container's resource request?

A. It guarantees unlimited access to the resource

B. It determines the maximum memory a container may consume

C. It contributes to node feasibility and capacity accounting

D. It forces the container to run on a GPU node

Correct Answer: C. It contributes to node feasibility and capacity accounting

Question 3

A Pod tolerates a GPU-node taint but has no GPU resource request or node affinity. What does the toleration alone establish?

A. The Pod must run on a GPU node

B. The Pod may be scheduled onto the tainted node if otherwise feasible

C. The Pod automatically receives a GPU

D. The Pod cannot run on ordinary nodes

Correct Answer: B. The Pod may be scheduled onto the tainted node if otherwise feasible

Question 4

Which workload is the strongest candidate for a DaemonSet?

A. A replicated payment API

B. A database cluster with persistent identity

C. A node-local log collection agent

D. A monthly financial report

Correct Answer: C. A node-local log collection agent

Question 5

Why can a CPU-based HPA behave poorly when resource requests are inaccurate?

A. HPA ignores all CPU metrics

B. CPU utilization is commonly interpreted relative to requests

C. Requests affect only storage scheduling

D. Inaccurate requests disable readiness probes

Correct Answer: B. CPU utilization is commonly interpreted relative to requests

Question 6

Which statement correctly describes a PodDisruptionBudget?

A. It prevents hardware failures

B. It guarantees additional cluster capacity

C. It limits selected voluntary disruptions

D. It controls container CPU throttling

Correct Answer: C. It limits selected voluntary disruptions

Question 7

A Pod is unschedulable because it requires zone A, rejects a node taint present in zone A, and demands two CPUs unavailable on the remaining nodes. What is the best interpretation?

A. The scheduler is malfunctioning because some nodes have free disk

B. The combined constraints have eliminated all feasible nodes

C. The Pod should automatically ignore required affinity

D. The HPA must immediately reduce replicas

Correct Answer: B. The combined constraints have eliminated all feasible nodes

Question 8

What is a key reason for making Job processing idempotent?

A. Jobs never retry failed Pods

B. Retries may repeat effects after partial completion

C. Idempotency increases node taints

D. It guarantees zero execution time

Correct Answer: B. Retries may repeat effects after partial completion

Question 9

Which design best distributes critical replicas across independent failure domains?

A. Required topology spread across zones

B. A single node selector selecting one node

C. A broad toleration without topology rules

D. A memory limit equal to zero

Correct Answer: A. Required topology spread across zones

Question 10

What is a major risk of assigning a very strict PDB to a small workload?

A. It can block voluntary maintenance

B. It automatically increases replica count

C. It disables all readiness probes

D. It converts the workload into a StatefulSet

Correct Answer: A. It can block voluntary maintenance