11. Runtime Security, Observability and Incident Response

Introduction and Context

DevSecOps does not end when an application passes its build, test, and deployment stages. A deployed service is exposed to changing workloads, identities, networks, dependencies, users, and adversaries. A secure pipeline must therefore continue to produce evidence about what is happening at runtime, detect deviations from expected behaviour, protect workloads while they operate, and support an organised response when controls fail.

Runtime security concerns the protection of applications, containers, hosts, platforms, identities, data, and communications during execution. Observability provides the evidence needed to understand system behaviour through logs, metrics, traces, events, and security signals. Incident response transforms those signals into decisions and actions: triage, containment, eradication, recovery, and learning. These capabilities are closely connected. Security without observability is largely unable to explain an attack; observability without response becomes an expensive collection of unacted-upon data.

For DevSecOps teams, the challenge is intensified by delivery velocity. Infrastructure may be created through code, workloads may be short-lived, releases may occur many times per day, and a compromise of the delivery pipeline can affect numerous production systems. Runtime security must therefore be automated where possible, governed through policy, and integrated with the software delivery lifecycle.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the relationship among runtime security, observability, detection, and incident response.

2. Design a telemetry strategy using logs, metrics, traces, events, and security context.

3. Evaluate SIEM, SOAR, anomaly detection, runtime protection, and continuous control monitoring.

4. Conduct incident triage, containment, forensics, recovery, and post-incident learning.

5. Respond to compromise of source repositories, build systems, artefact stores, or deployment platforms.

6. Analyse the technical, managerial, ethical, and governance implications of security monitoring.

Runtime Security as a Continuation of DevSecOps

Runtime security protects a system while it is processing requests, executing code, accessing data, or communicating with other systems. It includes preventive controls, such as identity policies and network segmentation; detective controls, such as audit events and intrusion detection; and responsive controls, such as workload isolation or credential revocation.

A useful distinction is between workload security and platform security. Workload security addresses the application and its immediate execution environment: insecure process behaviour, malicious packages loaded at runtime, unexpected child processes, unauthorised file access, and exploitation of application vulnerabilities. Platform security addresses the orchestration layer, cloud accounts, hosts, control planes, registries, secrets systems, and network services on which workloads depend.

Runtime security should be risk-based rather than merely tool-based. A production payment service, for example, warrants stronger monitoring and shorter response objectives than a disposable development environment. Risk assessment should consider asset value, exposure, identity privileges, data sensitivity, business criticality, and likely attack paths.

The principal runtime security objectives are:

• preserve confidentiality, integrity, availability, and accountability;
• establish that identities and workloads are authorised for their actions;
• detect deviations from approved behaviour;
• limit blast radius when compromise occurs;
• retain sufficient evidence for investigation and legal or regulatory needs; and
• restore trustworthy service without reintroducing the cause of failure.

These objectives require security controls to be represented as code and policy where practical. Examples include admission policies that reject unsigned images, infrastructure rules that prohibit public storage, and identity policies that prevent production workloads from obtaining administrative privileges.

Observability: From Data Collection to Operational Understanding

Observability is the ability to infer the internal state of a system from its externally available outputs. In DevSecOps, it is broader than conventional application monitoring because the relevant state includes security posture, identity use, configuration changes, supply-chain events, and control effectiveness.

The traditional telemetry signals are logs, metrics, and traces. Logs record discrete events, such as an authentication failure or a policy decision. Metrics represent numerical measurements over time, such as request latency, error rate, CPU use, or rejected connections. Traces follow a transaction across distributed services and show where time was spent or where an error occurred. Security observability adds audit records, endpoint events, cloud-control-plane activity, vulnerability findings, identity-provider events, and deployment metadata.

High-quality telemetry has several properties. It is attributable, because an event can be associated with a user, workload, service account, or automation process. It is contextual, because it includes environment, version, location, request, and correlation identifiers. It is time-consistent, normally using synchronised clocks and a clear time standard. It is tamper-resistant, because collection and storage are protected from the workloads being monitored. Finally, it is useful: data is retained and processed according to detection, investigation, compliance, and operational requirements rather than collected indiscriminately.

A telemetry design should begin with questions rather than products. Examples include: Which identity changed a production policy? Did a newly deployed version access a database it had never previously used? Was a secret used from an unusual location? Which requests were affected by a suspected compromise? What evidence would be required to reconstruct the sequence of events? These questions determine the events, fields, retention, and correlation mechanisms needed.

Structured logging is generally preferable to unstructured text. A security-relevant event may contain an event type, timestamp, actor, action, target, result, source address, workload identity, deployment version, trace identifier, and risk classification. Sensitive values such as passwords, access tokens, and unnecessary personal data must not be recorded. Logging a secret for investigation creates a second security incident.

Observability also has a cost dimension. Excessive telemetry creates storage expense, processing load, alert fatigue, and privacy risk. Sampling can reduce trace volume, but security-critical events should not be sampled away. Retention should be tiered: high-value security events may require longer retention than ordinary debug logs, while legal and organisational requirements must be considered explicitly.

Detection Engineering and Security Analytics

Detection is the disciplined conversion of telemetry into a statement that a potentially harmful condition exists. A detection rule should identify a threat scenario, define required data, specify the logic, describe expected false positives, and state the action required when it fires. Detection engineering therefore resembles software engineering: rules need version control, testing, review, deployment, monitoring, and retirement.

A simple rule might identify repeated failed administrative logins followed by a successful login from an unfamiliar location. A more advanced rule could correlate a deployment event, creation of a privileged service account, access to a secrets store, and outbound transfer from a production workload. Correlation across time and systems is essential because individual events may appear legitimate in isolation.

Anomaly detection identifies behaviour that differs from a baseline. Baselines may describe normal request volume, process trees, API calls, network destinations, deployment frequency, or administrative activity. Statistical models, clustering, time-series analysis, and machine-learning methods can assist, but an anomaly is not automatically an attack. Seasonal workload changes, emergency maintenance, and new releases can produce legitimate deviations. Human review and contextual enrichment remain important.

Useful detection measures include precision, recall, false-positive rate, mean time to detect, and the proportion of high-risk assets covered by effective detections. A high alert count is not evidence of good security. An organisation should ask whether analysts can understand alerts, whether alerts lead to timely action, and whether important attack paths remain invisible.

SIEM and SOAR Integration

A security information and event management system, or SIEM, centralises, normalises, stores, searches, and correlates security-relevant data. It can link events from identity providers, cloud platforms, endpoints, networks, source-control systems, CI/CD platforms, and applications. Normalisation is valuable because equivalent events from different technologies can then be queried consistently.

SIEM implementation should include a defined data catalogue, source ownership, parsing validation, time synchronisation, access control, retention policy, and health monitoring. A missing or delayed data source can create false confidence. The SIEM itself is a critical system and must be protected against unauthorised modification, log deletion, and excessive privilege.

Security orchestration, automation, and response, or SOAR, coordinates response actions through playbooks. A playbook for a suspected stolen token might enrich the alert with identity and asset data, check recent use, ask an analyst for approval, revoke the token, disable related sessions, notify stakeholders, and create an investigation record. Automation should be graduated. Low-risk, reversible actions can be fully automated; destructive actions, such as deleting a production workload, normally require authorisation and safeguards.

Integration should preserve evidence and accountability. Every automated action should record who or what initiated it, the reason, the result, and any approval. Playbooks require testing in controlled environments because an incorrect rule can interrupt legitimate operations or spread an outage.

Runtime Protection and Continuous Control Monitoring

Runtime protection can operate at several layers. Application controls include input validation, secure session handling, authorisation checks, and protection against unsafe dependencies. Host and container controls restrict processes, files, system calls, capabilities, and network connections. Cloud and orchestration controls enforce identity, admission, configuration, and segmentation policies. Network controls limit east-west movement and unusual egress. Data controls protect sensitive information in use and in transit.

Preventive runtime mechanisms include least-privilege identities, workload identity federation, immutable or minimal images, signed artefacts, read-only filesystems where feasible, network policies, secret rotation, policy-as-code, and isolation boundaries. Detective mechanisms include endpoint detection, cloud audit monitoring, file-integrity monitoring, runtime behavioural rules, and service-level security metrics.

Continuous control monitoring evaluates whether required controls remain present and effective. It differs from a one-time audit because it repeatedly tests actual state against an approved condition. Examples include checking that production images are signed, public access remains disabled, encryption is enabled, privileged accounts are reviewed, agents are reporting, and deployed versions correspond to approved release records.

A control-monitoring result should distinguish between control absence, control failure, telemetry failure, and legitimate exception. If an endpoint agent stops reporting, the organisation should not interpret the absence of alerts as evidence of safety. Exceptions should have an owner, reason, expiry date, compensating control, and approval. Permanent exceptions usually indicate that the baseline is unrealistic or governance is weak.

Incident Response as a Decision Process

Incident response is not simply the execution of a technical checklist. It is a decision process conducted under uncertainty, time pressure, incomplete evidence, and competing business interests. A mature response capability defines roles, authority, communication channels, severity levels, evidence procedures, and recovery criteria before an incident occurs.

The common lifecycle consists of preparation, detection and analysis, containment, eradication, recovery, and post-incident activity. Preparation includes inventories, contact lists, playbooks, backups, forensic capability, tabletop exercises, and tested access to emergency tools. Detection and analysis establish whether an alert represents a security incident, its scope, affected assets, likely technique, and business impact.

Triage prioritises work. Severity may depend on data sensitivity, privilege level, persistence, spread, service criticality, confidence of compromise, and regulatory consequences. Analysts should record hypotheses and confidence rather than treating early assumptions as facts. A useful incident record includes a timeline, evidence references, decisions, owners, and unresolved questions.

Containment aims to limit damage. Short-term measures may include isolating a workload, blocking an indicator, disabling an account, restricting egress, or pausing a deployment. Long-term containment may involve rebuilding hosts, rotating credentials, applying a policy correction, or moving traffic to a trusted version. Containment must balance safety with availability; indiscriminate shutdown can create a second incident.

Eradication removes the cause and persistence mechanism. It may require deleting unauthorised accounts, removing malicious code, correcting vulnerable configuration, replacing compromised images, and revoking tokens. Recovery restores service from a known-good state and increases monitoring while confidence is rebuilt. Recovery is not complete merely because the application responds; integrity of code, configuration, identities, and data must also be assessed.

Forensics is the systematic preservation and analysis of evidence. Relevant evidence may include cloud audit trails, identity events, container metadata, process information, memory or disk images where available, source-control history, build logs, artefact digests, and network records. Investigators should maintain chain of custody, minimise alteration of evidence, document collection methods, and distinguish observed facts from interpretation. In a cloud-native environment, short-lived resources make rapid collection and centralised evidence retention especially important.

Post-incident learning should examine both technical and organisational causes. A blameless review asks which conditions allowed the event, why controls did not prevent or detect it, which decisions delayed response, and how improvements will be verified. Actions should have owners, deadlines, risk justification, and measurable completion criteria. Merely conducting a meeting without changing controls is not learning.

Responding to Pipeline Compromise

A compromised CI/CD pipeline has unusual potential impact because it can distribute malicious code or configuration to many environments. Warning signs include unexpected workflow changes, altered build definitions, unknown runners, unusual package downloads, access to signing keys, artefacts whose provenance does not match the source revision, and deployments performed outside the normal approval path.

The first priority is to preserve evidence and stop further propagation. Depending on the evidence, actions may include pausing releases, disabling suspicious runners, restricting repository and artefact permissions, revoking exposed credentials, quarantining affected artefacts, and preventing promotion to production. Teams should avoid destroying runners or logs before collecting relevant evidence.

The investigation should establish the initial access vector, affected repositories and workflows, changed files, executed commands, secrets accessed, artefacts produced, environments reached, and whether signing keys or trust metadata were compromised. Rebuilding from a clean, reviewed source and trusted build environment is generally safer than attempting to repair an untrusted runner. Credentials must be rotated from a trustworthy system, and previously produced artefacts should be re-evaluated rather than assumed safe.

Pipeline integrity controls include protected branches, mandatory review, isolated ephemeral runners, restricted network access, short-lived credentials, signed commits or artefacts where appropriate, provenance attestations, separation of build and deployment privileges, reproducible builds, immutable artefact repositories, and independent verification at promotion gates. These controls reduce the probability and blast radius of pipeline compromise but do not eliminate the need for monitoring and response.

Illustrative Case: The Unauthorised Deployment

Background

A fictional online retail organisation operates containerised services in a managed cloud platform. Its delivery process uses source control, automated builds, an artefact repository, and a production deployment service. The organisation has centralised identity and SIEM capabilities, but runtime alerts are still being tuned.

Situation

The SIEM reports that a newly created service account accessed the secrets platform and initiated a production deployment outside the normal release window. Shortly afterward, a payment service begins making outbound connections to an unfamiliar domain. The deployed image digest does not correspond to the approved release record.

Analysis

The response team classifies the event as high severity because it combines unauthorised identity activity, secrets access, an unapproved artefact, and a sensitive service. Analysts preserve cloud audit logs, deployment records, source-control history, build logs, and image metadata. They discover that a build runner was modified through an exposed administrative token. The runner accessed a signing credential and produced an image containing an unauthorised startup process.

The team does not immediately delete the runner. It isolates the runner, suspends promotion, blocks the suspicious egress destination, and revokes the exposed token and signing credential. The affected workload is removed from service and replaced with a previously verified image while transaction processing is monitored for integrity.

Approach and Outcome

Investigators identify all artefacts built by the compromised runner during the exposure period and quarantine them. A clean runner is provisioned from approved code. Credentials are rotated, branch and workflow protections are strengthened, and deployment requires independent verification of image provenance. Runtime rules are updated to alert on unexpected child processes and access to secrets by workloads without an approved relationship.

The organisation finds no evidence of unauthorised payment data modification, but it treats the signing-key exposure as significant and conducts a broader review of artefact trust. The incident review identifies excessive runner privileges, inadequate runner isolation, and the absence of a control verifying that deployment provenance matched the approved build.

Learning Points

The case demonstrates that runtime indicators often reveal a pipeline compromise, that evidence preservation must precede destructive cleanup, and that identity, build, artefact, and deployment telemetry must be correlated. It also shows why continuous control monitoring should test the relationship between source revision, build execution, artefact digest, approval, and production deployment.

Challenges, Governance, and Professional Judgement

Runtime environments generate ambiguity. A legitimate administrator may resemble an attacker, a security test may resemble exploitation, and an automated deployment may appear anomalous during an emergency. Excessive false positives cause alert fatigue, while aggressive thresholds create blind spots. Detection logic should therefore be tested against normal operational scenarios and reviewed when architecture or release practices change.

Privacy is another significant concern. Telemetry may contain user identifiers, location information, request content, or employee activity. Organisations should collect only what is necessary, restrict access, apply retention limits, protect data in transit and at rest, and communicate monitoring purposes. Legal requirements vary by jurisdiction, so security teams should work with privacy, legal, and human-resources specialists.

Incident actions also raise governance questions. Who can disable a production identity? Who can stop a release? When must customers, regulators, partners, or law-enforcement agencies be informed? These decisions should be defined in policy and exercised through simulations. Technical teams should not improvise notification obligations during a crisis.

Common mistakes include collecting logs without an investigation purpose, failing to monitor the monitoring system, retaining secrets in logs, granting SOAR excessive authority, relying on a single indicator, rebuilding before preserving evidence, treating vulnerability scans as runtime protection, and closing incidents without verifying remediation. Another mistake is measuring success by the number of alerts rather than by risk reduction and response quality.

Emerging Directions and Best Practices

Security observability is increasingly linked with software supply-chain provenance, identity-centric detection, eBPF-based runtime telemetry, cloud-native application protection platforms, and OpenTelemetry-based instrumentation. Artificial intelligence can assist with alert grouping, summarisation, query generation, and anomaly analysis, but its outputs require validation. Models may amplify biased baselines, misinterpret novel behaviour, or expose sensitive telemetry through inappropriate prompts. Human accountability remains essential for consequential actions.

Practical best practices include:

• define critical assets, trust boundaries, and high-impact attack paths;
• instrument identity, deployment, control-plane, workload, and application events;
• correlate telemetry with version, image digest, owner, and environment metadata;
• maintain tested detection rules and response playbooks as code;
• automate reversible, low-risk actions and require approval for disruptive actions;
• protect logs, evidence stores, signing keys, and response tooling independently;
• test backups, rebuilds, credential rotation, and clean deployment paths;
• conduct tabletop exercises and controlled failure simulations;
• measure detection coverage, response time, false positives, control health, and remediation completion; and
• treat post-incident actions as engineering work with accountable owners.

Chapter Summary

Runtime security extends DevSecOps beyond deployment by protecting workloads, platforms, identities, data, and communications during execution. Observability combines logs, metrics, traces, audit events, and security context to make system behaviour explainable. SIEM platforms correlate telemetry, while SOAR playbooks coordinate response actions under appropriate human control. Anomaly detection is valuable when supported by reliable baselines and contextual analysis.

Continuous control monitoring tests whether security requirements remain effective in actual environments. Incident response requires preparation, disciplined triage, evidence preservation, containment, eradication, recovery, and post-incident learning. Pipeline compromise deserves special attention because malicious changes can be propagated through trusted automation. Effective programmes combine technical safeguards, measurable controls, governance, privacy protection, and regular exercises.

Key Terms

Runtime security: Protection of systems and workloads while they are executing.

Observability: The ability to infer internal system state from collected outputs and telemetry.

Telemetry: Data generated by systems that describes operation, activity, or security events.

Structured logging: Recording events in a consistent machine-readable format.

SIEM: A platform that collects, correlates, searches, and stores security events.

SOAR: Technology that orchestrates and automates security investigation and response.

Detection engineering: The systematic design, testing, deployment, and maintenance of detections.

Anomaly detection: Identification of behaviour that differs from an established baseline.

Runtime protection: Controls that prevent, detect, or constrain harmful behaviour during execution.

Continuous control monitoring: Repeated assessment of whether required controls remain present and effective.

Triage: Initial assessment and prioritisation of a suspected security incident.

Containment: Actions taken to limit an incident’s spread or impact.

Eradication: Removal of malicious code, persistence, access, or enabling weaknesses.

Recovery: Restoration of trustworthy service and normal operations after an incident.

Forensics: Systematic collection and analysis of evidence concerning an event.

Chain of custody: Documentation showing how evidence was collected, handled, and preserved.

Blast radius: The extent of systems, data, or users affected by a compromise.

Provenance: Verifiable information about the origin and production history of an artefact.

Alert fatigue: Reduced analyst effectiveness caused by excessive or low-value alerts.

Review Questions

Short Answer Questions

1. Distinguish workload security from platform security.

2. Why are correlation identifiers valuable in distributed observability?

3. What is the purpose of continuous control monitoring?

4. Define containment and explain how it differs from eradication.

5. Why is a compromised CI/CD runner a high-risk event?

Long Answer Questions

1. Explain how logs, metrics, traces, audit events, and deployment metadata combine to support security investigations.

2. Describe the incident-response lifecycle and the decisions required at each major stage.

3. Discuss the design principles of an effective SIEM and SOAR integration.

4. Analyse the benefits and limitations of anomaly detection in a continuously deployed environment.

5. Explain the controls that reduce the likelihood and blast radius of pipeline compromise.

Analytical / Application Questions

1. A production workload begins accessing a secrets store after an unplanned deployment. Develop a triage and containment plan, including evidence to preserve and actions requiring approval.

2. An organisation receives thousands of alerts but misses a long-running privileged account compromise. Analyse likely weaknesses in its detection and observability programme and propose measurable improvements.

3. A cloud provider audit log stops arriving while production services continue normally. Determine how continuous control monitoring should classify and escalate the condition, and justify your decision.

22. Multiple Choice Questions with Answers

Question 1

Which telemetry combination is most useful for linking a production deployment to a suspicious workload action?

A. CPU percentage and screen captures

B. Image digest, deployment record, workload identity, and audit event

C. Monthly vulnerability report and user survey

D. Application colour scheme and database size

Correct Answer: B. Image digest, deployment record, workload identity, and audit event

Question 2

What is the principal distinction between SIEM and SOAR?

A. SIEM compiles source code, while SOAR encrypts databases

B. SIEM collects and correlates events, while SOAR coordinates response actions

C. SIEM protects containers, while SOAR replaces backups

D. SIEM performs forensics, while SOAR eliminates the need for analysts

Correct Answer: B. SIEM collects and correlates events, while SOAR coordinates response actions

Question 3

A newly deployed service accesses a database never previously used by that service. What is the most appropriate initial interpretation?

A. It is automatically confirmed malware

B. It is irrelevant because the deployment succeeded

C. It is an anomaly requiring contextual investigation

D. It proves that the database has been deleted

Correct Answer: C. It is an anomaly requiring contextual investigation

Question 4

Which action best preserves forensic value during a suspected compromised build runner?

A. Immediately destroy the runner without collecting records

B. Publish all runner logs publicly

C. Isolate the runner while preserving relevant logs and metadata

D. Ignore the runner and rotate only application passwords

Correct Answer: C. Isolate the runner while preserving relevant logs and metadata

Question 5

Which is an example of continuous control monitoring?

A. Checking once whether an image was signed

B. Repeatedly verifying that production images remain signed and approved

C. Allowing every production image after a manual review

D. Measuring only application response time

Correct Answer: B. Repeatedly verifying that production images remain signed and approved

Question 6

Why should destructive SOAR actions commonly require approval?

A. Automation cannot record actions

B. Disruptive actions may interrupt legitimate services or worsen an incident

C. Approval makes logs unnecessary

D. Destructive actions always fail when automated

Correct Answer: B. Disruptive actions may interrupt legitimate services or worsen an incident

Question 7

During recovery from a pipeline compromise, which approach provides the strongest basis for trust?

A. Repairing the suspected runner in place without investigation

B. Reusing all artefacts produced during the exposure period

C. Rebuilding from reviewed source in a clean, trusted environment

D. Disabling monitoring to accelerate deployment

Correct Answer: C. Rebuilding from reviewed source in a clean, trusted environment

Question 8

Which metric best indicates detection quality rather than merely alert volume?

A. Number of alerts generated

B. Amount of log storage purchased

C. False-positive rate and time to detect high-risk scenarios

D. Number of dashboards created

Correct Answer: C. False-positive rate and time to detect high-risk scenarios

Question 9

What is the primary purpose of containment?

A. To identify every historical vulnerability

B. To limit the spread or impact of an active incident

C. To complete the post-incident review

D. To permanently redesign the organisation

Correct Answer: B. To limit the spread or impact of an active incident

Question 10

Which practice most directly reduces the blast radius of a compromised CI/CD runner?

A. Granting runners permanent production administrator access

B. Using isolated ephemeral runners and short-lived credentials

C. Storing signing keys in build logs

D. Removing independent deployment verification

Correct Answer: B. Using isolated ephemeral runners and short-lived credentials