7. Kubernetes Workloads, Scheduling and Application Lifecycle

Introduction and Context

Kubernetes does not deploy an application merely by starting a container. It provides a control system that continuously attempts to make the running cluster match a declared desired state. Workloads are the principal abstractions through which that desired state is expressed. A workload object specifies what should run, how many instances are required, how instances are updated, and how Kubernetes should respond to failure or change.

This distinction is important in production environments. A manually created Pod may work during a demonstration, but it does not by itself provide replication, controlled replacement, progressive delivery, stable identity, scheduled execution, or protection against planned disruption. Kubernetes workload controllers address these operational requirements. Scheduling then determines where Pods run, while probes and lifecycle policies determine whether an application is ready to receive traffic, whether it remains healthy, and how it is terminated.

The chapter develops a connected view of Pods, ReplicaSets, Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs. It also examines labels and selectors, namespaces, scheduling constraints, affinity, taints, tolerations, rollout strategies, and disruption controls. The central professional skill is not memorizing resource definitions, but selecting and combining them to express an application’s operational behaviour accurately.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain the role of Pods and workload controllers in Kubernetes.

2. Distinguish among Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, and ReplicaSets.

3. Use labels, selectors, and namespaces to organize and target resources safely.

4. Configure startup, readiness, and liveness probes appropriately.

5. Describe Kubernetes scheduling, affinity, taints, tolerations, and topology constraints.

6. Design rolling updates, rollbacks, and disruption policies.

7. Diagnose common workload and scheduling failures.

8. Evaluate workload designs from technical, organizational, and governance perspectives.

The Workload Model: From Pod to Controller

A Pod is the smallest deployable unit in Kubernetes. It contains one or more containers that share a network namespace, IP address, and attached storage volumes. Containers in the same Pod are consequently suitable for tightly coupled functions, such as an application container and a logging or proxy sidecar. A Pod is not normally an appropriate unit for independent scaling; all containers in the Pod are scheduled together and share its lifecycle.

Pods are intentionally ephemeral. A Pod can be deleted, evicted, or recreated on another node. Its IP address should therefore not be treated as a permanent identity. Services provide stable network access, while controllers provide replacement and scaling behaviour. A direct Pod manifest can be useful for debugging or for a very limited administrative task, but most long-running applications should be managed by a controller.

A controller implements a reconciliation loop. It observes the current state, compares it with the desired state expressed in an API object, and takes corrective action. For example, if a Deployment specifies three replicas and one Pod disappears, its ReplicaSet creates a replacement. This model is declarative: the administrator states the desired result rather than scripting every recovery step.

Labels, Selectors, and Namespaces

Labels are key-value attributes attached to Kubernetes objects. They may identify an application, version, environment, team, or operational role. Selectors use labels to find related objects. A Deployment’s selector identifies the Pods managed by its ReplicaSet; a Service selector identifies the Pods to which it sends traffic. Because selectors create control relationships, careless label design can produce serious failures, including a controller adopting the wrong Pods or a Service routing to an unintended workload.

A robust label scheme separates stable identity from release identity. For example, `app: payments` may remain constant while `version: v2` changes during a release. Labels should be documented, consistently named, and protected from accidental mutation where they affect ownership or traffic routing. Annotations, by contrast, store non-identifying metadata such as tool configuration, documentation links, or release information.

Namespaces provide logical separation within a cluster. They are useful for environments, teams, or applications and form a boundary for many names and policies. Namespaces do not constitute a complete security boundary: access control, network policies, resource quotas, and admission policies must also be configured. A production namespace should normally have controlled permissions, quotas, default resource policies, and auditing.

Choosing the Appropriate Workload Controller

ReplicaSets

A ReplicaSet maintains a specified number of identical Pods. It is useful for understanding replication, but it is rarely created directly in a modern application platform. Deployments manage ReplicaSets and add revision history, rolling updates, rollback, and declarative release management. A directly managed ReplicaSet may be appropriate for a narrow use case in which update orchestration is unnecessary, but it should not normally coexist with a Deployment using overlapping selectors.

Deployments

A Deployment is the standard controller for stateless, replaceable applications such as HTTP APIs, front-end servers, and worker processes without stable local identity. It manages ReplicaSets and supports scaling, rolling updates, pause and resume operations, and rollback.

A simplified Deployment illustrates the relationship between the object, its selector, and the Pod template:

```yaml

apiVersion: apps/v1

kind: Deployment

metadata:

name: web

spec:

replicas: 3

strategy:

type: RollingUpdate

rollingUpdate:

maxUnavailable: 1

maxSurge: 1

selector:

matchLabels:

app: web

template:

metadata:

labels:

app: web

spec:

containers:

• name: web

image: example/web:2.1

ports:

• containerPort: 8080

readinessProbe:

httpGet:

path: /ready

port: 8080

periodSeconds: 5

```

The selector must match the labels in the Pod template. The `maxUnavailable` and `maxSurge` values govern the balance between capacity and speed during an update. A large surge can accelerate delivery but consume cluster resources; a high unavailability allowance can reduce capacity during the release.

StatefulSets

StatefulSets are designed for applications requiring stable network identity, ordered operations, or persistent storage association. Each replica receives a predictable ordinal name, such as `database-0` and `database-1`. PersistentVolumeClaims can be created per replica so that storage remains associated with the logical instance when its Pod is recreated.

StatefulSets do not make an application stateful by themselves. The application must still implement replication, leader election, backup, recovery, and data consistency. A database operator may be preferable when the system requires sophisticated failover and upgrade procedures. StatefulSet updates can be deliberately conservative because many stateful systems cannot tolerate the same replacement pattern as stateless web servers.

DaemonSets

A DaemonSet ensures that a Pod runs on every eligible node, or on every node matching a selector. Common examples include log collectors, monitoring agents, storage plugins, and network components. A DaemonSet is node-oriented rather than replica-oriented. As nodes are added, matching Pods are created; when nodes are removed, their Pods disappear.

DaemonSets require careful resource planning. A seemingly small agent multiplied across hundreds of nodes can become a significant consumer of CPU and memory. They may also require tolerations so that they can run on control-plane or specially tainted nodes. A DaemonSet should not be used simply because an application has many replicas; it is appropriate only when placement on nodes is itself the requirement.

Jobs and CronJobs

A Job represents finite work that should complete successfully. It creates Pods and tracks completion. Jobs can run one task, several tasks in parallel, or a required number of successful completions. `backoffLimit`, completion count, parallelism, and restart policy influence failure handling. Idempotent task design is essential because retries can repeat partially completed operations.

A CronJob creates Jobs according to a schedule. It is suitable for backups, reports, reconciliation tasks, and maintenance operations. Production CronJobs should specify concurrency behaviour, history limits, deadlines, and an explicit time-zone policy where supported by the Kubernetes version. A schedule does not guarantee that a task will run exactly once: controller restarts, missed schedules, retries, and overlapping executions must be considered. The task should therefore tolerate duplicate execution or implement an external idempotency mechanism.

Application Lifecycle and Health

Kubernetes distinguishes process existence from application usability. A container can be running while an application is unable to serve requests. Probes communicate these different states to the kubelet and controllers.

A startup probe is used for applications that require substantial initialization time. Until it succeeds, liveness and readiness checks are normally held back. This prevents a slow-starting application from being repeatedly killed by an overly aggressive liveness test.

A liveness probe determines whether a container should be restarted. It should detect conditions from which the process cannot recover without a restart, such as a permanently deadlock state. It should not be used to test temporary dependency failure; otherwise an overloaded database or network interruption may cause unnecessary restart storms.

A readiness probe determines whether a Pod should receive Service traffic. It is appropriate for checking that the application has completed initialization and can serve requests. A Pod failing readiness is not necessarily broken and is not automatically restarted. Readiness is therefore central to safe rolling updates and graceful degradation.

Probes may use HTTP, TCP, or an executed command. Timeouts, initial delays, periods, and failure thresholds must reflect actual application behaviour. A common error is to probe an endpoint that depends on every external service. This may remove all instances from service during a partial dependency outage. Readiness should represent the application’s ability to provide useful service, while liveness should remain comparatively narrow.

Termination also forms part of the lifecycle. When a Pod is deleted, Kubernetes marks it for termination, runs lifecycle hooks where configured, and sends termination signals to containers. Applications should handle `SIGTERM`, stop accepting new work, complete or safely abandon in-flight requests, and exit within the termination grace period. A `preStop` hook can assist coordination but should not replace correct signal handling.

Scheduling as a Policy Decision

The scheduler selects a suitable node for each unscheduled Pod. It considers resource requests, node availability, taints and tolerations, affinity rules, topology constraints, and other predicates and priorities. Scheduling is therefore not simply a search for an empty machine; it is a policy decision constrained by the declared workload requirements.

Resource requests represent the amount of CPU and memory needed for scheduling. Limits constrain consumption at runtime, though their exact behaviour differs by resource. Omitting requests makes placement less predictable and can cause important workloads to compete unfairly. A production platform should derive requests from measurement, load testing, and review rather than copying arbitrary values.

Node selectors and node affinity express placement preferences or requirements. Required affinity prevents scheduling unless the rule is satisfied. Preferred affinity influences scoring but allows placement elsewhere. For example, a latency-sensitive service might prefer nodes in a particular zone while still remaining schedulable in another zone during capacity loss.

Pod affinity places Pods near other Pods, whereas pod anti-affinity discourages or prevents co-location. Anti-affinity can improve failure isolation by distributing replicas across nodes or zones, but strict rules can make a workload unschedulable in a small cluster. Topology spread constraints provide a more explicit way to distribute replicas across domains such as zones or hostnames.

Taints are applied to nodes and repel Pods. A Pod must have a matching toleration to be considered for a tainted node. A toleration does not force placement; it merely permits it. This distinction is frequently misunderstood. For example, a GPU node may carry a taint so that only workloads with GPU requirements and the corresponding toleration are admitted. Taints with effects such as `NoSchedule`, `PreferNoSchedule`, and `NoExecute` have different implications; `NoExecute` can also evict existing Pods that do not tolerate the taint.

Scheduling decisions should be evaluated against failure domains. Placing all replicas on one node may satisfy a replica count while providing no meaningful availability. Zone-aware distribution, appropriate anti-affinity, and capacity reserves are often more valuable than maximizing immediate utilization.

Updates, Rollouts, and Disruption Controls

A Deployment’s rolling update replaces old Pods gradually. The update must be considered together with readiness probes, capacity, database compatibility, and client behaviour. If a new version becomes ready before it is actually safe, Kubernetes may route traffic prematurely. If it never becomes ready, the rollout may stall rather than automatically solve the application defect.

Useful operational commands include:

```bash

kubectl rollout status deployment/web -n production

kubectl rollout history deployment/web -n production

kubectl rollout undo deployment/web -n production

kubectl scale deployment/web --replicas=5 -n production

```

A rollback restores a previous Deployment revision, but it cannot automatically reverse incompatible data migrations or external side effects. Release engineering must therefore separate backward-compatible schema changes from irreversible operations and maintain a tested recovery procedure.

A PodDisruptionBudget (PDB) limits voluntary disruption, such as node draining or cluster maintenance. It can specify a minimum available count or a maximum unavailable count. A PDB does not protect against every failure: node crashes, out-of-memory events, and application defects are not necessarily voluntary disruptions. Nor does it create capacity. An excessively strict PDB can prevent legitimate maintenance by making eviction impossible.

PDBs should be designed alongside replica counts and topology. A single-replica workload with a requirement for one available Pod cannot be voluntarily disrupted without violating its own policy. This is a design constraint, not a PDB malfunction. Availability objectives, maintenance requirements, and budget must be negotiated at the service-owner level.

An Applied Case: Modernizing an Order Platform

Background

Consider an illustrative case involving Northstar Retail, an organization operating an online order platform. The platform includes a stateless order API, a PostgreSQL database, a node-level log collector, and a nightly reconciliation task. The organization is moving from manually managed virtual machines to Kubernetes.

Situation and Problem

The first migration placed the API in one manually created Pod. During node maintenance the API became unavailable. A later attempt created three replicas, but all were scheduled on one node. A new version was then deployed without a readiness probe, causing traffic to reach Pods before their caches and connection pools were initialized. Finally, the reconciliation script ran twice during a controller interruption and duplicated records.

Analysis

The failures resulted from treating Kubernetes as a container launcher rather than a declarative control system. The API required a Deployment, a Service, resource requests, readiness and liveness probes, and distribution rules. The database required stable identity and persistent storage, but a StatefulSet alone would not provide complete database failover. The reconciliation task needed a Job created by a CronJob with idempotent processing and an explicit concurrency policy.

Strategy and Solution

Northstar adopted the following design:

• The API was placed in a Deployment with three initial replicas, a rolling-update policy, and a readiness endpoint that returned success only after initialization.
• Requests were established through load testing. Topology spread constraints distributed replicas across zones where capacity permitted.
• A PDB allowed one API Pod to be voluntarily unavailable during maintenance.
• The database used a managed service rather than an in-cluster database during the first migration. This reduced operational risk while the team developed database expertise.
• A DaemonSet collected node and container logs, with resource limits and tolerations appropriate to the node fleet.
• The reconciliation process used a CronJob with non-overlapping execution, a deadline, limited history, and an idempotency key for each order period.
• Namespaces, role-based access control, resource quotas, and admission checks were introduced for production workloads.
• Rollouts were first tested in a staging namespace and then released using a controlled traffic strategy.

Outcome

Node maintenance no longer caused complete API loss because replicas were distributed and voluntary disruption was constrained. Premature traffic shifted to new Pods was reduced by readiness checks. The reconciliation process could safely retry without duplicating records. The platform team also gained clearer ownership boundaries: application teams managed Deployments and release health, while the platform team managed node agents, quotas, and cluster policy.

Learning Points

The case demonstrates that reliability is produced by a combination of workload choice, application behaviour, scheduling policy, and governance. No single Kubernetes object can compensate for a poor application contract. It also shows why “running” is not equivalent to “available,” why replicas without topology rules may not provide resilience, and why idempotency is essential for scheduled and retried work.

Common Errors, Risks, and Governance Considerations

A frequent error is using a Deployment for a workload that requires stable identity or ordered storage, or using a StatefulSet for an application that needs only interchangeable replicas. Another is modifying a controller’s selector after creation. Selectors are fundamental ownership relationships and should be treated as effectively immutable in normal operations.

Overly broad selectors can cause accidental adoption or traffic routing. Missing resource requests lead to poor scheduling and noisy-neighbour effects. Excessive limits may cause throttling or memory termination. Probe misconfiguration creates either false failures or silent routing to unhealthy applications. Strict affinity, taints, and PDB combinations can make a workload impossible to schedule or maintain.

Security and governance are inseparable from workload management. Images should be scanned and pinned using an appropriate versioning policy; containers should run with least privilege and non-root identities where feasible. Role-Based Access Control should limit who can modify Deployments, secrets, or admission-sensitive labels. Resource quotas and limit ranges protect shared clusters. Network policies restrict unintended communication, while audit logs support accountability.

Teams should define ownership for production objects, establish change approval appropriate to risk, and document emergency rollback procedures. Automation must remain observable: events, controller status, probe failures, scheduling reasons, and rollout histories should be integrated into operational monitoring. AI-assisted manifest generation may accelerate configuration, but generated policies require human review because a plausible YAML file can still create unsafe permissions, weak isolation, or an impossible scheduling requirement.

Current Developments and Professional Practice

Modern Kubernetes platforms increasingly combine native controllers with operators, policy engines, GitOps workflows, and progressive delivery tools. Operators encode domain-specific knowledge for systems such as databases and message brokers. GitOps treats version-controlled declarations as the desired source of truth and makes drift detectable. Progressive delivery extends the basic rolling update with canary or blue-green techniques, automated metrics analysis, and controlled traffic shifting.

The emerging direction is toward platform engineering: reusable workload templates, standardized probes, policy-as-code, workload identity, automated right-sizing, and service-level objectives. These capabilities should not obscure first principles. A sophisticated platform remains dependent on accurate selectors, realistic health signals, explicit resource requirements, and an application that behaves safely under replacement and retry.

Best-Practice Design Sequence

A practical design sequence is to first classify the workload: stateless service, stateful service, node agent, finite task, or scheduled task. Next define identity, storage, replication, and failure semantics. Then design labels and selectors, resource requests, health probes, and termination behaviour. Add scheduling requirements only where they express a real availability or hardware need. Finally define update, rollback, disruption, security, and observability policies.

Validate the design under realistic events: Pod deletion, node drain, zone loss, slow startup, dependency failure, failed image pull, missed schedule, and partial rollout. A workload is production-ready not when its YAML is accepted, but when its behaviour under these events is understood and acceptable.

Chapter Summary

Kubernetes workloads express how applications should run and recover. Pods provide the execution unit, while ReplicaSets, Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs provide different reconciliation models. Deployments suit stateless services; StatefulSets provide stable identity and storage association; DaemonSets place agents on nodes; Jobs and CronJobs manage finite and scheduled work.

Labels, selectors, and namespaces organize resources and establish critical ownership and routing relationships. Probes distinguish startup, readiness, and liveness, enabling safer traffic management and recovery. Scheduling uses requests, affinity, topology rules, taints, and tolerations to place workloads according to capacity and failure-domain requirements. Rolling updates, rollbacks, and PodDisruptionBudgets manage change and voluntary maintenance but cannot replace sound application design.

Professional Kubernetes practice combines declarative configuration with observability, governance, least privilege, idempotent processing, and tested failure scenarios. Workload selection should follow application semantics rather than convenience.

Key Terms

Pod: The smallest deployable Kubernetes unit, containing one or more co-scheduled containers.

Workload controller: A Kubernetes controller that maintains the desired state of application execution.

ReplicaSet: A controller that maintains a specified number of matching Pod replicas.

Deployment: A controller for stateless applications that manages ReplicaSets and rolling releases.

StatefulSet: A controller providing stable Pod identity, ordered operations, and storage association.

DaemonSet: A controller that runs a Pod on each eligible node.

Job: A controller for finite work that should eventually complete.

CronJob: A controller that creates Jobs according to a schedule.

Label: A key-value attribute used to organize and select Kubernetes objects.

Selector: A label-based expression identifying objects managed or targeted by another resource.

Namespace: A logical scope for organizing and governing resources in a cluster.

Readiness probe: A health check determining whether a Pod should receive traffic.

Liveness probe: A health check indicating whether a container should be restarted.

Startup probe: A health check protecting slow-starting applications during initialization.

Affinity: A scheduling rule that attracts or requires placement near selected nodes or Pods.

Taint: A node attribute that repels Pods lacking a matching toleration.

Toleration: A Pod declaration permitting it to be considered for a tainted node.

Rolling update: A gradual replacement of old workload instances with new ones.

PodDisruptionBudget: A policy limiting voluntary disruption of a workload.

Idempotency: The property that repeating an operation produces the same intended result without harmful duplication.

Review Questions

Short Answer Questions

1. Why should long-running applications normally be managed by a controller rather than a bare Pod?

2. Distinguish a readiness probe from a liveness probe.

3. What is the operational purpose of a DaemonSet?

4. Why are resource requests important to scheduling?

5. Explain why a toleration does not force a Pod onto a tainted node.

Long Answer Questions

1. Compare Deployments and StatefulSets in terms of identity, storage, scaling, and update behaviour.

2. Explain how labels, selectors, and namespaces contribute to safe multi-team cluster management.

3. Discuss how startup, readiness, and liveness probes should be designed for a slow-starting API.

4. Describe the factors that should be considered when designing a rolling update and rollback strategy.

5. Explain how affinity, anti-affinity, topology spread constraints, taints, and tolerations interact in scheduling.

Analytical / Application Questions

1. A three-replica API is running on one node, has no readiness probe, and uses a strict PDB. Analyse the risks and propose a redesign.

2. Design a Kubernetes workload model for a log collector, a payment API, a database, and a nightly reconciliation task. Justify each controller choice.

3. A CronJob sometimes overlaps with a previous execution and duplicates records after retries. Analyse the causes and recommend application and Kubernetes-level controls.

22. Multiple Choice Questions with Answers

Question 1

Which Kubernetes workload controller is most appropriate for a stateless HTTP API requiring rolling updates and interchangeable replicas?

A. Deployment

B. StatefulSet

C. DaemonSet

D. CronJob

Correct Answer: A. Deployment

Question 2

A Pod is running, but its application has not completed initialization. Which probe should prevent the Pod from receiving Service traffic?

A. Liveness probe

B. Readiness probe

C. Startup command

D. Node probe

Correct Answer: B. Readiness probe

Question 3

What does a toleration do in Kubernetes scheduling?

A. Forces a Pod onto every matching node

B. Allows a Pod to be considered for a tainted node

C. Removes the taint from a node

D. Creates additional replicas on the node

Correct Answer: B. Allows a Pod to be considered for a tainted node

Question 4

Which workload is the best candidate for a DaemonSet?

A. A replicated web API

B. A database requiring stable ordinal identities

C. A log collector required on each eligible node

D. A monthly billing report

Correct Answer: C. A log collector required on each eligible node

Question 5

Why is a PodDisruptionBudget insufficient to protect a workload from node crashes?

A. It applies only to Jobs

B. It controls voluntary rather than every involuntary disruption

C. It prevents all Pod scheduling

D. It replaces readiness probes

Correct Answer: B. It controls voluntary rather than every involuntary disruption

Question 6

A scheduled reconciliation task may be retried after partial failure. Which application property is most important?

A. High CPU consumption

B. Fixed Pod IP addresses

C. Idempotency

D. Strict pod affinity

Correct Answer: C. Idempotency

Question 7

What is the principal risk of an incorrectly broad Service selector?

A. The Service cannot obtain a namespace

B. The Service may route traffic to unintended Pods

C. The scheduler ignores resource requests

D. The Deployment automatically rolls back

Correct Answer: B. The Service may route traffic to unintended Pods

Question 8

Which controller is designed to provide stable network identity and per-replica storage association?

A. ReplicaSet

B. Deployment

C. StatefulSet

D. DaemonSet

Correct Answer: C. StatefulSet

Question 9

What is the primary purpose of a startup probe?

A. To distribute Pods across zones

B. To protect slow-starting applications from premature liveness failures

C. To replace a PodDisruptionBudget

D. To permit scheduling onto tainted nodes

Correct Answer: B. To protect slow-starting applications from premature liveness failures

Question 10

During a rolling update, what is the likely effect of setting a large maxSurge value?

A. More temporary capacity may be consumed to accelerate replacement

B. All old Pods are immediately deleted

C. The Deployment becomes a StatefulSet

D. Readiness checks are disabled

Correct Answer: A. More temporary capacity may be consumed to accelerate replacement