Configuration Management, DevOps and Reliable Delivery

Introduction

Modern software is not delivered once; it is continuously changed, tested, deployed, observed, and improved. A reliable product therefore depends not only on source code but also on configuration files, build tools, dependencies, databases, cloud resources, deployment procedures, secrets, monitoring rules, and operational knowledge. Configuration management and DevOps provide the practices for controlling these elements as a coherent system.

Configuration management establishes known, reproducible versions of software and its supporting environment. Continuous integration and continuous delivery automate the movement from code change to deployable artifact. Infrastructure as code expresses environments in version-controlled, reviewable definitions. Containers package applications consistently, while deployment strategies reduce release risk. Observability and site reliability engineering connect technical operation with user experience, enabling teams to learn from production rather than treating operations as an afterthought.

The central principle of this chapter is that reliable delivery is an engineered capability. It requires automation, traceability, feedback, suitable risk controls, and explicit responsibility.

Learning Objectives

By the end of this chapter, students should be able to:

• explain configuration management, baselines, versioning, and release control;
• distinguish continuous integration, continuous delivery, and continuous deployment;
• design a CI/CD pipeline with appropriate quality gates;
• describe infrastructure as code, containerization, and environment promotion;
• compare blue-green, rolling, canary, and feature-flag deployment strategies;
• use service-level indicators, objectives, and error budgets to guide reliability work;
• interpret logs, metrics, and traces as elements of observability;
• analyse incident response and post-incident learning;
• evaluate security, governance, and ethical issues in automated delivery.

1. Configuration Management as Control of System State

A configuration item is a component whose identity, version, or settings must be controlled. Examples include source files, libraries, compiler versions, container definitions, database schemas, infrastructure definitions, environment variables, certificates, and operational runbooks. Configuration management answers four questions:

1. What components constitute the system?
2. Which version of each component is approved?
3. How was a particular release built and deployed?
4. Who authorised and changed the system state?

A configuration baseline is an approved, identifiable state used as a reference for further change. A baseline may represent a release candidate, a production release, or an environment specification. Changes after a baseline should be explicit rather than invisible edits.

A practical configuration-management process includes:

• Identification: define configuration items and naming conventions.
• Version control: record changes, authorship, and history.
• Change control: review and approve significant changes.
• Status accounting: report which versions are in development, test, or production.
• Verification and audit: confirm that the implemented state matches the approved state.

Version control systems support branching, merging, tagging, and comparison. A commit should represent a coherent change, and a release tag should identify the exact source revision used to produce an artifact. Long-lived branches can isolate work but also increase merge complexity. Trunk-based development reduces divergence by integrating small changes frequently, usually with feature flags protecting incomplete functionality.

Builds and Reproducibility

A build transforms source and dependencies into an artifact, such as a compiled binary, package, or container image. A reliable build is reproducible: given the same declared inputs, it produces materially equivalent output. Reproducibility requires controlling dependency versions, build tools, operating-system assumptions, generated files, and timestamps where they affect output.

Build artifacts should be immutable. If version 2.4.1 has been tested, its contents should not be silently replaced. Promotion should move the same artifact from testing to production rather than rebuilding it differently for each environment. Artifact repositories should retain metadata such as source revision, dependency information, build identity, test results, and security-scan results.

A software bill of materials (SBOM) records the components contained in an artifact. It supports vulnerability response, licensing analysis, and supply-chain visibility. However, an SBOM does not itself prove that components are safe; it improves the ability to investigate and manage risk.

2. DevOps and the Delivery Value Stream

DevOps is a sociotechnical approach that integrates development, operations, security, and other stakeholders around the delivery and operation of software. It is not merely a toolset. Its foundations include shared ownership, automation, small changes, rapid feedback, measurable outcomes, and a culture that encourages learning rather than blame.

The delivery value stream commonly includes:

Plan → Code → Review → Build → Test → Package → Deploy → Observe → Learn

The stages are connected. If testing is slow, deployment is delayed. If production telemetry is poor, teams cannot evaluate whether a release helped users. If operations cannot influence design, systems may be difficult to run. DevOps therefore seeks to reduce handoffs and make the entire flow visible.

Three terms must be distinguished:

• Continuous integration (CI): developers integrate changes frequently, and automated builds and tests provide rapid feedback.
• Continuous delivery: every change that passes the pipeline is kept in a deployable state; production release may require an explicit business decision.
• Continuous deployment: every change that passes automated controls is automatically released to production.

Continuous deployment is not inherently superior. Regulated systems, safety-critical applications, or products requiring coordinated communication may appropriately retain approval gates. The relevant question is whether controls are proportionate to risk and whether they preserve useful feedback.

Pipeline Design

A robust pipeline usually contains the following controls:

1. source checkout and dependency resolution;
2. formatting, linting, and static analysis;
3. unit and component tests;
4. security and dependency scanning;
5. artifact creation and signing;
6. integration, contract, and end-to-end tests;
7. deployment to an isolated environment;
8. smoke tests and health checks;
9. controlled promotion or rollback.

Fast checks should run early. Expensive tests may run later, but critical defects should be detected as close as possible to the change that introduced them. A failed pipeline should provide actionable evidence rather than simply reporting “failed.”

Pipeline security is part of software security. Credentials should be short-lived and stored in a managed secret system, not in source code or logs. Build agents should be isolated, dependencies should be verified, and pipeline definitions should receive code review. Access should follow least privilege: a test job should not automatically have unrestricted production access.

3. Environments and Infrastructure as Code

Typical environments include local development, test, staging, and production. Their purpose is not to become unrelated systems. Excessive differences create “works on my machine” failures. At the same time, production data and credentials must not be copied carelessly into lower environments.

Environment configuration should be externalised from application code. The same artifact should receive environment-specific settings through controlled configuration. Secrets such as passwords and signing keys require stricter handling than ordinary configuration, including encryption at rest, access logging, rotation, and emergency revocation.

Infrastructure as code (IaC) defines networks, compute resources, databases, permissions, and services in machine-readable files. IaC provides version history, review, repeatability, and the ability to reconstruct an environment. Declarative IaC states the desired outcome; the tool determines the operations needed to reach it. Imperative scripts specify procedural steps and may be useful for specialised actions, but they can become fragile when state changes unexpectedly.

A safe IaC workflow includes plan, review, apply, and verification. State files must be protected because they may contain resource identifiers or sensitive values. Providers and modules should be version-pinned and tested. Destructive changes require explicit safeguards, backups, and approval appropriate to their impact.

Configuration drift occurs when the actual environment diverges from its declared definition. Drift may result from emergency manual changes, provider behaviour, or undocumented administration. Detection can compare observed state with desired state; remediation should be deliberate because automatically overwriting a legitimate emergency change may increase damage.

4. Containers and Image-Based Delivery

A container packages an application with user-space dependencies while sharing the host operating system kernel. It is lighter than a full virtual machine, but it is not a complete security boundary in every context. Containers improve portability and consistency, yet they do not eliminate dependency vulnerabilities, insecure privileges, network exposure, or poor application design.

A container image is built in layers from a base image and application content. Good practice includes using minimal maintained base images, pinning versions, running as a non-root user, excluding unnecessary tools, scanning dependencies, and signing images. Images should be immutable after publication. Runtime configuration, secrets, and persistent data should not be embedded in an image.

Orchestrators can schedule containers, restart failed instances, expose services, manage configuration, and perform rolling updates. They also introduce complexity in networking, storage, identity, and debugging. Teams should adopt orchestration when its operational benefits justify that complexity.

5. Release and Deployment Strategies

A release is a business or product decision to make functionality available; a deployment is the technical act of placing software into an environment. Separating these concepts enables deployment without immediate exposure, particularly through feature flags.

Rollback is effective only if the previous version remains available, data changes are compatible, and the failure can be isolated. Database migrations deserve special attention. A safer pattern is often expand-and-contract: add a backward-compatible schema element, deploy code that can use both forms, migrate data, then remove the obsolete element in a later change.

6. Reliability Engineering and Service Objectives

Reliability is the probability that a system performs its required function for a stated period under stated conditions. In digital services, reliability includes availability, latency, correctness, durability, and recovery capability.

An availability service-level indicator (SLI) can be defined as:

The result is dimensionless and commonly expressed as a percentage. The definition must specify what counts as a valid request and a successful response. For example, excluding overloaded requests could make a service appear more reliable than users experience.

A service-level objective (SLO) is a target for an SLI over a stated window. If the availability objective is 99.9% for a 30-day month, the approximate error budget is:

Using a 30-day window, total time is 30 × 24 × 60 = 43,200 minutes. Thus:

This calculation assumes a time-based availability SLO and treats all unavailable time equally. A request-based SLO would instead use failed-request allowance, and a real policy must define exclusions, measurement boundaries, and rounding.

The error budget is not permission to fail. It is a decision tool. When the budget is being consumed rapidly, teams may pause risky releases and prioritise reliability. When substantial budget remains, the organisation may accept measured change risk. SLOs should reflect user needs rather than arbitrary infrastructure targets.

The Four Golden Signals

A widely useful monitoring model focuses on:

• Latency: time required to serve requests;
• Traffic: demand placed on the system;
• Errors: failed or incorrect requests;
• Saturation: how fully a resource or service is utilised.

Monitoring only CPU utilisation is inadequate. A service may have low CPU usage while suffering from database locks, queue delays, dependency failures, or incorrect responses.

7. Observability and Operational Feedback

Monitoring reports known conditions; observability concerns how well system outputs permit engineers to infer internal state. The principal signals are:

• Logs: timestamped event records, preferably structured and correlated with request or trace identifiers;
• Metrics: numerical time-series measurements used for trends, thresholds, and aggregation;
• Traces: records of a request’s path across distributed components.

Good telemetry is purposeful. Logs should avoid secrets and personal data, use consistent severity levels, and provide context. Metrics require carefully chosen labels; unbounded labels such as arbitrary user IDs can create high-cardinality cost and performance problems. Traces are powerful for distributed latency analysis but require sampling and privacy consideration.

Alerts should be actionable and tied to user impact or imminent risk. Alert fatigue occurs when excessive low-value alerts train responders to ignore signals. Runbooks should state diagnosis steps, escalation paths, mitigation actions, and rollback procedures.

8. Incident Response and Learning

An incident is an event that materially threatens service quality, security, safety, or business operation. Incident response normally includes detection, triage, assignment of an incident commander, communication, mitigation, recovery, and follow-up.

During an incident, restoring an acceptable service level is usually more urgent than finding the complete root cause. Teams may reduce traffic, disable a feature, roll back a release, fail over, or apply a temporary capacity change. Actions and decisions should be recorded in a timeline.

A blameless post-incident review does not mean that accountability disappears. It means analysis focuses on system conditions, decisions, incentives, and safeguards rather than treating one person as the entire cause. A useful review identifies contributing factors, what went well, what failed, and concrete actions with owners and deadlines.

Common reliability measures include mean time to detect (MTTD), mean time to restore (MTTR), change failure rate, deployment frequency, and lead time for changes. These metrics should be interpreted together. Optimising deployment frequency alone can encourage unsafe releases; reducing MTTR may be valuable even when incidents cannot be eliminated.

9. Illustrative Case: Online Learning Platform Release

Background

An illustrative university learning platform serves video lessons, assessments, and discussion forums. Its team historically deployed manually every six weeks. Releases frequently caused configuration errors, and the operations team discovered failures through student complaints.

Problem

The platform had inconsistent staging and production settings, no reliable rollback procedure, and dashboards focused on server CPU rather than failed submissions or latency. A database schema change caused assessment submissions to fail after a release.

Analysis

The team mapped the delivery value stream and found that the same source was rebuilt separately for staging and production. Secrets were stored in deployment scripts, and manual changes to cloud resources were not recorded. There was no SLO for assessment submission success.

Approach

The team introduced a version-controlled pipeline, immutable container images, and IaC for the application network and database permissions. The artifact promoted to production was exactly the artifact tested in staging. Secrets moved to a managed secret store. Database changes used an expand-and-contract migration. A canary deployment exposed the release to one faculty before broader rollout.

They defined an SLI as successful assessment submissions divided by valid submission attempts, with an SLO of 99.95% over a rolling month. Dashboards combined submission errors, latency, traffic, and database saturation. Alerts were linked to user impact, and a runbook documented rollback and communication.

Outcome

The next release revealed a compatibility defect during canary testing and was halted before university-wide exposure. The team corrected the defect, repeated the pipeline, and deployed safely. Manual release effort fell, configuration drift became visible, and post-release reviews produced improvements rather than blame.

Learning Points

The case demonstrates that reliability is created by the interaction of version control, reproducible artifacts, IaC, compatible data changes, progressive delivery, meaningful SLOs, and operational learning. No single tool would have solved the original problem.

10. Governance, Security, and Ethical Considerations

Automated delivery changes who can affect production and how quickly. Governance should therefore define approval authority, segregation of duties where justified, audit records, emergency-change procedures, retention policies, and recovery objectives. Excessive approval bureaucracy can encourage bypassing controls; inadequate control can permit unauthorised or unsafe change.

Security must be integrated into the pipeline. Software supply-chain threats include compromised dependencies, malicious build steps, stolen credentials, poisoned images, and unauthorised artefacts. Defences include dependency provenance, signed commits or artefacts where appropriate, isolated builds, secret scanning, vulnerability management, and restricted permissions.

Observability also raises privacy issues. Logs and traces may contain identifiers, search terms, locations, or educational and health information. Data minimisation, access control, retention limits, masking, and legally appropriate processing are essential. AI-assisted coding and operations can improve productivity, but generated code and automated remediation require human review, testing, provenance awareness, and controls against exposing confidential data to external services.

11. Best Practices and Limitations

Effective practice includes:

• keep changes small, reviewable, and reversible;
• build once and promote the same immutable artifact;
• automate repeatable verification, but retain risk-appropriate human judgement;
• define SLOs from user outcomes;
• test rollback, backup restoration, and disaster recovery rather than merely documenting them;
• treat infrastructure and pipeline code as production software;
• design database changes for compatibility;
• use progressive delivery for high-risk changes;
• reduce alert noise and maintain operational runbooks;
• conduct blameless, action-oriented reviews.

Automation has limits. A green pipeline cannot prove absence of all defects. Staging cannot perfectly represent production. Containers do not automatically provide security. IaC can reproduce an incorrect design at scale. Metrics can be gamed or misinterpreted. SLOs can obscure harm if their measurement boundary excludes affected users. Reliable delivery therefore requires technical controls combined with critical judgement and continuous validation.

Chapter Summary

Configuration management controls the identity, version, and approved state of software and its environment. Baselines, version control, immutable artefacts, traceability, and verification make change understandable and reversible. DevOps integrates development and operations through shared ownership, automation, small changes, and feedback. CI validates frequent integration; continuous delivery maintains deployability; continuous deployment automates production release when appropriate.

Infrastructure as code makes environments reviewable and repeatable, while containers improve packaging consistency but do not remove operational or security responsibilities. Blue-green, rolling, canary, and feature-flag strategies manage release risk in different ways. Observability combines logs, metrics, and traces to explain system behaviour. SRE uses SLIs, SLOs, and error budgets to connect reliability with engineering decisions. Incident response prioritises mitigation and communication, while blameless reviews convert failure into learning. Secure pipelines, privacy-aware telemetry, supply-chain controls, and human oversight are essential to responsible reliable delivery.

Key Terms

Configuration item: A component whose version or state is controlled.

Baseline: An approved, identifiable system state.

Configuration drift: Divergence between declared and actual system state.

Artifact: A produced, deployable software package.

Continuous integration: Frequent integration supported by automated build and test feedback.

Continuous delivery: Keeping every accepted change deployable.

Continuous deployment: Automatically releasing accepted changes to production.

Infrastructure as code: Machine-readable management of infrastructure definitions.

Container: An isolated process environment packaging application user-space dependencies.

Immutable artifact: An artifact whose contents are not changed after publication.

Canary deployment: Progressive release to a small representative population.

Feature flag: A runtime control that changes feature exposure.

Service-level indicator: A measured representation of service performance.

Service-level objective: A target value for an SLI over a defined period.

Error budget: The permitted deviation from an SLO during its measurement window.

Observability: The ability to infer internal system state from outputs.

MTTR: Mean time to restore service after failure.

Blameless post-incident review: A learning-focused analysis of contributing system conditions.

Software bill of materials: An inventory of components in a software artifact.

Configuration drift: Unplanned difference between intended and implemented configuration.

Review Questions

Short Answer Questions

1. What is a configuration baseline, and why is it useful?
2. Distinguish continuous delivery from continuous deployment.
3. State two advantages and one limitation of infrastructure as code.
4. What are the four golden signals of monitoring?
5. Why should database migrations be designed for compatibility during progressive delivery?

Long Answer Questions

1. Explain how version control, immutable artifacts, and CI/CD combine to improve release reliability.
2. Compare rolling, blue-green, and canary deployment strategies, including suitable use cases and risks.
3. Explain how SLOs and error budgets influence the balance between feature development and reliability work.
4. Describe an end-to-end incident-response process, from detection through post-incident learning.
5. Analyse the security and privacy risks of automated pipelines and observability systems, and propose controls.

Analytical / Application Questions

1. A service has a 99.9% availability SLO over a 30-day month. Calculate its approximate time-based error budget and explain what assumptions your calculation uses.
2. Design a delivery pipeline for a payment service. Identify at least five quality or security gates and justify their order.
3. A canary release shows normal CPU usage but a sharp increase in failed transactions and latency. Decide whether to proceed, pause, or roll back, and explain which signals and governance controls support your decision.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a configuration baseline?

A. An automatically generated log file

B. An approved and identifiable system state

C. A temporary developer workspace

D. A list of untested feature ideas

Correct Answer: B. An approved and identifiable system state

Question 2

What is the defining characteristic of continuous integration?

A. Releasing every change directly to customers

B. Manually rebuilding production servers

C. Frequently integrating changes with automated build and test feedback

D. Prohibiting branching in all projects

Correct Answer: C. Frequently integrating changes with automated build and test feedback

Question 3

Why should an artifact be immutable after publication?

A. To ensure tested contents remain traceable and unchanged

B. To prevent all future versions from being created

C. To eliminate the need for testing

D. To make source control unnecessary

Correct Answer: A. To ensure tested contents remain traceable and unchanged

Question 4

Which deployment strategy initially exposes a release to a small representative group?

A. Recreate

B. Canary

C. Big-bang deployment

D. Manual installation

Correct Answer: B. Canary

Question 5

Which formula represents a time-based error budget?

A. SLO × total time

B. (1 − SLO) × total time

C. Total time ÷ SLO

D. SLO + total time

Correct Answer: B. (1 − SLO) × total time

Question 6

Which item is one of the four golden signals?

A. Source-code length

B. Developer seniority

C. Saturation

D. Number of branches

Correct Answer: C. Saturation

Question 7

What is a primary purpose of infrastructure as code?

A. To make infrastructure changes unreviewable

B. To express and reproduce infrastructure through version-controlled definitions

C. To store production passwords in source files

D. To eliminate all infrastructure testing

Correct Answer: B. To express and reproduce infrastructure through version-controlled definitions

Question 8

What is the main purpose of an error budget?

A. To establish a fixed number of software defects allowed in source code

B. To guide the trade-off between reliability and change

C. To calculate developer salaries

D. To replace incident response

Correct Answer: B. To guide the trade-off between reliability and change

Question 9

Which practice best supports a blameless post-incident review?

A. Identifying one person as the complete cause

B. Deleting the incident timeline

C. Examining system conditions and assigning concrete improvement actions

D. Avoiding all discussion of contributing factors

Correct Answer: C. Examining system conditions and assigning concrete improvement actions

Question 10

For a 99.9% SLO over 30 days, approximately how many minutes are in the time-based error budget?

A. 4.32 minutes

B. 43.2 minutes

C. 432 minutes

D. 4,320 minutes

Correct Answer: B. 43.2 minutes