6. Configuration, Templates and Secrets Management

Introduction and Context

DevOps automation depends on the ability to create and operate systems consistently across development, testing, staging, and production environments. The same application may require different database endpoints, network ranges, replica counts, feature settings, credentials, and resource sizes in each environment. If these values are embedded directly in Python scripts, infrastructure definitions, or deployment commands, automation becomes fragile, difficult to audit, and unsafe. Configuration management addresses this problem by separating operational policy and environment-specific values from reusable automation logic.

Templates extend this principle. Instead of creating separate files for every environment, a template describes a general structure and inserts approved values at execution time. Secrets management deals with the most sensitive subset of configuration: passwords, tokens, private keys, certificates, and other data whose disclosure could compromise systems or people. These three concerns are closely related but should not be confused. Configuration describes how a system should operate; templates express how configuration is transformed into executable artifacts; secrets management protects sensitive values throughout their lifecycle.

In a Python-based DevOps programme, these practices are implemented through structured files such as YAML or TOML, environment variables, Python configuration objects, Jinja2 templates, inventory systems, encryption tools, and centralized services such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. The objective is not merely to hide passwords. It is to establish a controlled, reproducible, auditable, and adaptable configuration supply chain.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish configuration data, templates, inventories, feature settings, and secrets.

2. Design a configuration hierarchy that supports reuse, controlled precedence, and environment isolation.

3. Use Python and Jinja2 to generate environment-specific infrastructure or application files.

4. Explain secure storage, retrieval, encryption, access control, rotation, and revocation of secrets.

5. Evaluate centralized secret-management services and select an appropriate approach.

6. Identify common configuration failures, security risks, governance obligations, and operational trade-offs.

7. Apply these principles to a realistic DevOps deployment scenario.

Configuration as a Control Plane for Automation

Configuration is the set of values and policies that determine how software or infrastructure behaves without changing its implementation. Examples include a service port, a database connection pool size, a Kubernetes namespace, an autoscaling threshold, or whether a feature is enabled. Configuration may be static for long periods or may change frequently as operational conditions change.

A useful distinction is between code, configuration, and state. Code defines what a program can do. Configuration selects how it should behave. State records what has actually happened, such as the current contents of a database or the identifier of a provisioned virtual machine. Confusing these categories creates design problems. For example, storing an automatically generated cloud resource identifier as if it were a manually maintained setting may cause drift and failed redeployments.

Configuration can also be classified by scope:

• Global configuration applies across many services, such as an organization-wide region or approved log format.
• Application configuration belongs to one application, such as a cache timeout.
• Environment configuration distinguishes development, staging, and production.
• Instance configuration applies to a particular host, container, or deployment replica.
• Secret configuration contains values whose confidentiality or integrity is especially important.

Good configuration is explicit, validated, versioned where appropriate, and easy to trace to an owner. It should not be scattered across shell history, undocumented manual changes, CI variables, source code, and multiple configuration files without a defined precedence model.

Inventories and Environment Identity

An inventory identifies the systems, services, or targets to which automation applies. In infrastructure scripting, an inventory may contain hostnames, IP addresses, cloud instance identifiers, operating-system groups, regions, or service metadata. An inventory should answer two questions: what is being managed, and which attributes are needed to manage it?

A simple YAML inventory might be represented as follows:

```yaml

environments:

staging:

region: eu-west-1

application_hosts:

• app-stg-01
• app-stg-02

database_endpoint: db-stg.internal

production:

region: eu-west-1

application_hosts:

• app-prd-01
• app-prd-02
• app-prd-03

database_endpoint: db-prd.internal

```

In mature platforms, inventories are often dynamic. A Python program may query a cloud provider for instances tagged `Environment=production` and `Role=web`, then generate an inventory. Dynamic inventory reduces manual maintenance but introduces dependency on provider APIs, permissions, availability, and consistent resource tagging.

Environment identity should be explicit rather than inferred from a hostname or a developer's local directory. A deployment command should be required to name the target environment, and automation should validate that the selected account, subscription, region, and cluster correspond to that environment. This protects against the particularly serious failure in which a script intended for testing modifies production.

Configuration Hierarchies and Precedence

A configuration hierarchy combines values from multiple sources according to a defined order. A typical hierarchy might contain defaults, organization-wide values, application values, environment overrides, deployment-time parameters, and runtime secret references. The hierarchy supports reuse: common values are defined once, while exceptional values are overridden narrowly.

For example, a service might use the following precedence, from lowest to highest:

1. Built-in safe defaults.

2. Organization or platform defaults.

3. Application-level configuration.

4. Environment-specific configuration.

5. Deployment-job parameters.

6. Explicit command-line overrides.

7. Runtime-injected values from a secret or configuration service.

The precise order is a design decision. It must be documented and tested because an unexpected override can be as damaging as a missing value. A configuration system should also reject ambiguous duplicate definitions rather than silently choosing one.

A practical Python representation is a validated configuration object rather than an unstructured dictionary:

```python

from dataclasses import dataclass

import os

@dataclass(frozen=True)

class AppConfig:

environment: str

log_level: str

replicas: int

database_url: str

def load_config() -> AppConfig:

environment = os.environ.get("APP_ENV", "development")

replicas = int(os.environ.get("APP_REPLICAS", "1"))

if replicas < 1:

raise ValueError("APP_REPLICAS must be at least 1")

return AppConfig(

environment=environment,

log_level=os.environ.get("LOG_LEVEL", "INFO"),

replicas=replicas,

database_url=os.environ["DATABASE_URL"],

)

```

The example illustrates several principles: values have defaults only when safe, required values are accessed explicitly, types are converted, and validation occurs before deployment. In production, a library such as Pydantic may provide richer type checking, constraints, and structured error messages. Validation should occur at the boundary where configuration enters the system, not after partially applying infrastructure changes.

Templates: From Intent to Executable Artifacts

A template is a parameterized document containing fixed structure and replaceable expressions. It may generate a systemd unit, Nginx configuration, Kubernetes manifest, cloud-init file, Terraform variable file, or application settings document. Jinja2 is widely used with Python because it supports variable substitution, conditionals, loops, filters, and template inheritance.

Consider a deployment template:

```jinja2

apiVersion: apps/v1

kind: Deployment

metadata:

name: {{ app_name }}

namespace: {{ namespace }}

spec:

replicas: {{ replicas }}

template:

spec:

containers:

• name: {{ app_name }}

image: {{ image_repository }}:{{ image_tag }}

env:

• name: FEATURE_CHECKOUT_V2

value: "{{ feature_checkout_v2 | lower }}"

```

A Python rendering function can supply validated values:

```python

from jinja2 import Environment, FileSystemLoader, StrictUndefined

jinja = Environment(

loader=FileSystemLoader("templates"),

undefined=StrictUndefined,

autoescape=False,

)

template = jinja.get_template("deployment.yaml.j2")

rendered = template.render(

app_name="orders",

namespace="staging",

replicas=2,

image_repository="registry.example/orders",

image_tag="2025.04.2",

feature_checkout_v2=False,

)

```

`StrictUndefined` is important because an omitted variable produces an error instead of silently creating an invalid or incomplete file. Templates should remain focused on presentation and structure. Complicated business logic inside templates is difficult to test and maintain; such logic belongs in Python, where it can be expressed and tested more clearly.

Generated files should be examined before application. A robust pipeline renders templates, parses the output, validates it against a schema or platform command, performs policy checks, and only then applies it. For Kubernetes, this might include YAML parsing, server-side dry runs, and policy evaluation. For an operating-system configuration, it may include syntax checks and a controlled test instance.

Feature Settings and Progressive Delivery

Feature settings determine whether a capability is available, to whom, and under what conditions. A feature flag may be a boolean, but mature systems also use percentage rollouts, user segments, geographic rules, time windows, or dependency checks. Feature configuration is operationally valuable because deployment and release can be separated: code can be deployed while a feature remains disabled.

This flexibility creates governance requirements. Flags need owners, descriptions, creation dates, expiry dates, and a removal plan. A permanently retained flag increases cognitive complexity and may leave inactive code paths vulnerable. A flag that controls payment, authentication, or data migration requires stronger approval and audit controls than a cosmetic user-interface setting.

Secrets: Protection Across the Lifecycle

A secret is sensitive information used to authenticate, authorize, encrypt, or otherwise protect a resource. Common examples are passwords, API tokens, database credentials, signing keys, SSH private keys, OAuth client secrets, and TLS private keys. A secret has a lifecycle: generation, distribution, storage, use, rotation, revocation, archival where required, and destruction.

The central security principle is to minimize both exposure and privilege. A process should receive only the secrets it needs, for only as long as it needs them, through a channel that can be monitored. Secrets should not be placed in source code, container images, ordinary configuration repositories, command-line arguments, build logs, exception messages, or unencrypted artifacts.

Encryption is necessary but not sufficient. Encryption at rest protects stored data if storage is accessed without authorization. Encryption in transit protects data moving between a workload and a secret service. Access control, identity verification, key management, auditing, and rotation determine whether the system remains secure in operation.

Secret Storage Models

Several approaches are used in practice:

• Environment variables are convenient for short-lived process configuration, but may be exposed through debugging, process inspection, crash reports, or careless logging. They are a delivery mechanism, not a complete secret-management strategy.
• Encrypted files, such as those protected with Ansible Vault or a cloud key-management service, can support Git-based workflows. They require careful key distribution and protection of decryption capabilities.
• Platform secret stores, such as Kubernetes Secrets, improve integration but do not automatically provide strong protection unless encryption at rest, access policies, and cluster security are correctly configured.
• Centralized secret-management services provide identity-based access, versioning, audit events, dynamic credentials, leases, and automated rotation. They introduce operational dependency and require reliable availability and access design.

A centralized service generally stores a secret under a path or named object and returns it only after authenticating the workload. Authentication may use cloud workload identity, Kubernetes service-account federation, a machine identity, or a short-lived token. The application should retrieve the value at runtime or during deployment without exposing it to unrelated operators or pipeline stages.

Rotation, Revocation, and Dynamic Credentials

Rotation replaces a secret with a new value according to a schedule, an event, or a security policy. Rotation is effective only when consumers can adopt the new value without unacceptable downtime. A database credential rotation, for example, may require a dual-credential period: the new credential is created, applications are updated, old and new credentials are both accepted temporarily, and the old credential is then revoked.

Revocation is different from rotation. Rotation is planned replacement; revocation is immediate invalidation because a credential is suspected to be compromised or no longer authorized. Incident procedures should identify who can revoke a secret, how dependent services are updated, and how evidence is preserved.

Dynamic credentials are generated for a specific client and limited lifetime. They reduce the value of stolen credentials because the exposure window is short. However, applications must handle expiration, renewal, temporary secret-service failure, and connection re-establishment. Dynamic access is therefore a systems-design issue, not simply a storage feature.

A Secure Configuration Workflow

A reliable workflow begins with classification. Each setting is labelled as public, internal, confidential, or highly sensitive. The classification determines storage, access, retention, and logging rules. Next, common defaults and environment-specific overrides are designed, with schemas defining required fields, data types, allowable ranges, and relationships.

The workflow then separates rendering from application. A pipeline retrieves approved configuration references, obtains secrets through an identity-aware mechanism, renders templates in a controlled workspace, and validates the result. Secret values should be masked in logs and should not be written to artifacts unless the artifact itself is encrypted and access-controlled.

Before applying changes, the pipeline should perform policy checks. Examples include preventing production deployments from using development databases, disallowing public network exposure, requiring encryption for storage, and blocking images with prohibited tags. Approval gates may be appropriate for high-risk changes, but approval should not substitute for automated validation.

After application, the system should record the configuration version, actor or workload identity, target environment, result, and relevant policy decisions. It should not record secret values. Drift detection compares the declared configuration with actual state and raises an alert when manual or unauthorized changes occur.

Illustrative Case Study: Securing a Multi-Environment Order Platform

Background

The fictional company Northstar Retail operates an order-processing platform composed of an API, a worker service, and a PostgreSQL database. The team initially maintains separate YAML files for development, staging, and production. Passwords are stored in a private Git repository, and a Python deployment script substitutes values into Kubernetes manifests.

Situation and Problem

A developer accidentally copied a production database password into a troubleshooting issue. Although the issue was private, several contractors could access it. The team also discovered that staging used a production feature-flag default because one environment file had been copied from another. Deployments were difficult to audit because the script accepted arbitrary command-line overrides and did not record their source.

Analysis

The incident involved more than one misplaced password. Secrets were treated as ordinary configuration, there was no formal classification, and the repository's access boundary was broader than the database's access boundary. The configuration hierarchy was implicit, so copied files caused unintended inheritance. Finally, the deployment script lacked validation and traceability.

Approach and Solution

Northstar introduced a versioned, non-secret configuration repository containing schemas, defaults, feature definitions, and environment overlays. The team defined a precedence order: safe defaults, application configuration, environment overlay, and approved deployment parameters. Arbitrary production overrides were prohibited.

Secrets were moved to a centralized secret-management service. The CI system authenticated using workload identity, while production workloads received narrowly scoped access policies. The database service was changed to support two active credentials during rotation. A scheduled process generated a replacement credential, updated the secret version, restarted or refreshed consumers as required, verified connectivity, and revoked the old credential.

Jinja2 templates were configured with strict undefined-variable handling. The pipeline rendered manifests into a temporary protected workspace, parsed them, ran policy checks, and used a dry-run operation before applying them. Log filters masked known secret patterns, and the deployment record stored the configuration commit, template version, target environment, and identity, but not secret values.

Feature flags received owners and expiration dates. The production checkout feature was initially disabled, enabled for internal users, and then expanded after monitoring error rates and order completion. A Python configuration model validated replica counts, allowed environments, and required references before rendering.

Outcome

The organization reduced the number of people and processes that could access production credentials, established a repeatable rotation procedure, and made environment differences visible in version control. The team also gained a safer release mechanism because feature activation no longer required redeploying application code. The new system did not eliminate all risk: availability of the secret service and correct workload identity configuration became important operational dependencies.

Learning Points

The case demonstrates that secure configuration is an architectural discipline. Encryption alone would not have solved excessive access or accidental logging. A hierarchy is useful only when precedence is explicit. Templates improve consistency only when their output is validated. Finally, rotation must be designed together with application behavior and incident response.

Risks, Limitations, and Common Mistakes

One frequent mistake is committing secrets to Git and assuming that deleting the file removes them. Git history, forks, caches, and CI artifacts may retain the value. The response should include immediate revocation and repository and artifact investigation, not merely deletion.

Another mistake is excessive configuration flexibility. If every value can be overridden at every layer, operators cannot predict the final result. Configuration should have a small number of deliberate extension points. Silent defaults are also dangerous for security-sensitive values; a missing encryption setting should fail closed rather than select an insecure default.

Template injection and unsafe rendering deserve attention. Never evaluate arbitrary Python expressions from untrusted configuration. Validate values before placing them into shell commands, SQL statements, URLs, or structured documents. Quoting and escaping must match the target format. A value safe in YAML may not be safe in a shell command.

Centralized services create dependency, cost, and failure-mode concerns. If a service is unavailable during startup, applications need a carefully designed response: controlled retry, cached short-lived material where policy permits, or a safe failure. Caching secrets indefinitely undermines rotation. Conversely, requiring a network call for every request may create latency and availability problems.

Secret scanning tools are useful but imperfect. They may miss novel credentials, generate false positives, or identify a secret only after it has already been copied. Prevention, least privilege, runtime controls, and response procedures remain necessary.

Governance, Ethics, and Professional Responsibility

Configuration and secrets management involve stewardship of organizational and personal data. Engineers must respect access boundaries, privacy obligations, contractual requirements, and applicable data-protection and sector regulations. A database password may provide access to personal records; therefore, its protection is connected to the organization's broader privacy responsibilities.

Governance should define ownership, approval thresholds, retention, access review, emergency access, audit-log protection, and separation of duties. Operators should not use production secrets for local development when synthetic or restricted data is sufficient. Access should be granted to identities rather than shared human accounts, and privileged access should be reviewed periodically.

Auditability must be balanced with confidentiality. It is appropriate to record that an identity read a secret version at a particular time; it is not appropriate to record the secret itself. Engineers should also avoid using secret values as identifiers in metrics or trace labels, because observability systems often have broad retention and access.

Current Developments and Emerging Practice

Workload identity is increasingly replacing long-lived CI credentials. Instead of storing a cloud access key in a pipeline, a trusted job exchanges an ephemeral identity token for narrowly scoped access. Secretless patterns go further by allowing a proxy, sidecar, or platform integration to handle authentication so that the application does not directly process a long-lived credential.

Policy-as-code is becoming a normal companion to configuration management. Tools can evaluate rendered resources against organizational rules before deployment. Software bills of materials, signed artifacts, and provenance attestations also extend configuration assurance by helping teams verify what is being deployed and how it was produced.

External secret operators and managed identity integrations allow orchestrators to synchronize references from a central store into workloads. These integrations simplify delivery but require careful consideration of synchronization timing, namespace isolation, access policies, and the possibility that a synchronized value becomes persistent in the platform.

AI-assisted configuration generation can accelerate the creation of templates and policy rules, but generated configuration must be reviewed, validated, and tested. An AI system may invent an insecure default, expose a credential in an example, or misunderstand an environment boundary. Human accountability and automated controls remain essential.

Best-Practice Checklist

A professional configuration system should embody the following practices:

• Define configuration ownership, classification, schema, and precedence.
• Keep reusable logic separate from environment-specific data.
• Use explicit environment identity and prevent accidental cross-environment access.
• Validate types, ranges, required fields, and relationships before applying changes.
• Render templates with strict handling of missing variables and correct escaping.
• Keep secrets out of source code, images, logs, command lines, and ordinary artifacts.
• Prefer short-lived, identity-based access and least-privilege policies.
• Design rotation and revocation before the first production deployment.
• Test generated output through parsing, dry runs, policy checks, and integration tests.
• Audit access and changes without recording secret values.
• Assign owners and expiry dates to feature flags.
• Practise incident response for leaked credentials and secret-service unavailability.

Chapter Summary

Configuration management separates operational values from reusable automation and enables the same code to support multiple environments. Inventories identify targets and their attributes, while configuration hierarchies combine defaults and controlled overrides according to documented precedence. Templates such as Jinja2 transform validated data into deployment artifacts; strict undefined-variable handling and output validation reduce failure risk. Feature settings support progressive delivery but require ownership and lifecycle control.

Secrets are a sensitive class of configuration whose lifecycle includes generation, distribution, storage, use, rotation, revocation, and destruction. Encryption must be combined with identity, least privilege, auditing, and safe delivery. Centralized secret-management services, workload identity, dynamic credentials, and policy-as-code strengthen the automation supply chain, while introducing availability and integration considerations. Secure configuration is therefore both a technical and governance responsibility.

Key Terms

Configuration: Operational data that controls software or infrastructure behaviour.

Configuration hierarchy: An ordered system for combining values from multiple configuration sources.

Precedence: The rule determining which value wins when several sources define the same setting.

Inventory: A structured list of systems, services, or targets managed by automation.

Dynamic inventory: An inventory generated from a provider, API, or discovery process.

Template: A parameterized document from which a concrete configuration artifact is generated.

Jinja2: A Python-compatible templating engine supporting expressions, conditions, loops, and filters.

Environment overlay: A configuration layer containing values specific to an environment.

Feature flag: A controllable setting that changes feature availability independently of code deployment.

Secret: Sensitive data used for authentication, authorization, encryption, or protection.

Encryption at rest: Protection of stored data through encryption.

Encryption in transit: Protection of data while it moves between systems.

Workload identity: An identity assigned to an automated workload for authenticated resource access.

Least privilege: Granting only the permissions necessary for an identity or process.

Rotation: Planned replacement of a credential or cryptographic value.

Revocation: Immediate invalidation of a credential or authorization.

Dynamic credential: A short-lived credential generated for a particular consumer or session.

Secret sprawl: Uncontrolled distribution and duplication of sensitive values across systems.

Drift: A difference between declared configuration and actual deployed state.

Policy-as-code: Machine-enforceable rules that evaluate systems or configurations against defined requirements.

Review Questions

Short Answer Questions

1. Distinguish code, configuration, and state in a DevOps system.

2. What is the purpose of a configuration hierarchy?

3. Why should a Jinja2 environment use strict handling for undefined variables?

4. Differentiate secret rotation from secret revocation.

5. State two risks associated with using environment variables for secrets.

Long Answer Questions

1. Explain how inventories, environment overlays, and precedence rules work together in multi-environment automation.

2. Describe a secure end-to-end workflow for rendering and applying infrastructure templates.

3. Discuss the lifecycle of a secret and the controls required at each stage.

4. Compare encrypted files, platform secret stores, and centralized secret-management services.

5. Explain why feature flags require ownership, expiry, and governance rather than being treated as ordinary Boolean settings.

Analytical / Application Questions

1. A deployment succeeds in staging but uses the production database endpoint because a command-line override silently replaced the environment overlay. Design validation and precedence controls that would prevent this incident.

2. A database credential must be rotated without interrupting a high-volume order service. Develop a rotation sequence, including verification, consumer update, rollback considerations, and revocation.

3. An organization wants to store cloud access keys in a CI variable and inject them into every pipeline. Evaluate this design and propose a safer identity and secret-delivery architecture.

22. Multiple Choice Questions with Answers

Question 1

Which configuration source should normally have the highest precedence when an explicit, approved deployment override is permitted?

A. Built-in default

B. Organization-wide default

C. Application-level configuration

D. Approved deployment parameter

Correct Answer: D. Approved deployment parameter

Question 2

What is the principal benefit of using StrictUndefined in a Jinja2 environment?

A. It encrypts rendered templates

B. It fails rendering when a required variable is missing

C. It automatically validates Kubernetes permissions

D. It prevents all template injection attacks

Correct Answer: B. It fails rendering when a required variable is missing

Question 3

Which statement best distinguishes a secret from ordinary configuration?

A. A secret always changes more frequently

B. A secret is always stored outside version control

C. A secret requires stronger confidentiality and access controls

D. A secret cannot be represented in YAML

Correct Answer: C. A secret requires stronger confidentiality and access controls

Question 4

A team deletes a committed password from the current Git file. What is the most appropriate immediate security response?

A. Assume the password is no longer accessible

B. Rename the repository

C. Revoke or rotate the password and investigate retained copies

D. Add the password to a second private repository

Correct Answer: C. Revoke or rotate the password and investigate retained copies

Question 5

Which approach most directly reduces the value of a stolen credential by limiting its lifetime?

A. Dynamic credentials

B. A larger configuration file

C. A permanent environment variable

D. A manually copied inventory

Correct Answer: A. Dynamic credentials

Question 6

Why is validation before infrastructure application important?

A. It guarantees that the target cloud provider is always available

B. It detects invalid types, missing values, and policy violations before changes occur

C. It eliminates the need for access control

D. It makes secrets public for easier debugging

Correct Answer: B. It detects invalid types, missing values, and policy violations before changes occur

Question 7

In the Northstar case, why was supporting two active database credentials useful during rotation?

A. It allowed a transition period while consumers adopted the new credential

B. It removed the need to revoke old credentials

C. It made passwords unnecessary

D. It allowed all users to access the database

Correct Answer: A. It allowed a transition period while consumers adopted the new credential

Question 8

Which practice is most appropriate for audit logs involving secret access?

A. Record the complete secret value

B. Record the secret in a metric label

C. Record the identity, time, and secret object accessed without recording its value

D. Disable logging to preserve confidentiality

Correct Answer: C. Record the identity, time, and secret object accessed without recording its value

Question 9

What is the main danger of allowing unrestricted overrides at every configuration layer?

A. Templates become impossible to encrypt

B. The final effective configuration becomes difficult to predict and govern

C. Inventories become dynamic automatically

D. Feature flags cannot be used

Correct Answer: B. The final effective configuration becomes difficult to predict and govern

Question 10

Which technology trend replaces long-lived CI cloud keys with short-lived access based on the identity of a trusted job?

A. Workload identity

B. Static inventory generation

C. Template inheritance alone

D. Manual password sharing

Correct Answer: A. Workload identity