Cloud security depends on continuous visibility. Preventive controls such as identity policies, network segmentation, encryption and secure configuration reduce exposure, but they cannot guarantee that every malicious action will be blocked. Misconfigurations, stolen credentials, vulnerable workloads, insider misuse and supply-chain compromise may bypass preventive controls. Security monitoring provides the capability to observe activity, identify abnormal or malicious behaviour, investigate evidence and coordinate response.

Cloud monitoring is more complex than traditional data-centre monitoring because responsibility is distributed between the cloud service provider, the customer and sometimes managed-service partners. Resources are created dynamically, identities are often more important than network addresses, and telemetry is distributed across control planes, workloads, SaaS platforms, identity systems and security services. Effective monitoring therefore requires an engineered combination of logging, analytics, detection rules, threat intelligence, automation and governance.

This chapter examines how organisations collect and analyse cloud logs, events and telemetry using security information and event management (SIEM), security orchestration, automation and response (SOAR), cloud security posture management (CSPM), cloud-native application protection platforms (CNAPP) and threat intelligence. It also explains baselines, dashboards, key performance indicators (KPIs), key risk indicators (KRIs) and risk-informed alerting.

Learning Objectives

1. The Monitoring Problem in Cloud Environments

Security monitoring is the systematic collection, processing and interpretation of evidence about activity in an information system. A cloud monitoring programme must answer five questions:

Cloud telemetry has three broad characteristics. First, it is high-volume: routine API calls, network flows, authentication events and application transactions may generate millions of records. Second, it is heterogeneous: different providers and services use different schemas, timestamps, identifiers and retention periods. Third, it is contextual: an event is rarely meaningful in isolation. A successful login from a new country may be benign for a travelling administrator but suspicious when followed by creation of an access key and disabling of logging.

Monitoring must therefore be designed as a detection system rather than a data-collection exercise. More logs do not automatically produce better security. Unnormalised, poorly retained and unactionable data can increase cost and analyst fatigue while failing to detect important attacks.

Shared responsibility and monitoring boundaries

The provider normally secures the underlying facilities, physical infrastructure and managed service platform. The customer remains responsible for appropriate configuration, identities, data, workloads and use of security services, although the precise division depends on the service model. Infrastructure as a service generally leaves the customer with more monitoring responsibilities than software as a service.

2. Cloud Telemetry and Log Architecture

Principal telemetry sources

Source	Typical evidence	Security value
Identity and access	Logins, token use, MFA, role assumption, privilege changes	Detect account compromise and privilege abuse
Cloud control plane	Resource creation, deletion, policy changes, API calls	Detect unauthorised administration
Data plane	Object reads, database queries, secret access	Detect data access and exfiltration
Network	Flow records, DNS, firewall and proxy events	Identify scanning, command-and-control and unusual transfer
Workload	Operating-system, container, Kubernetes and runtime events	Detect malware and exploitation
Application	Authentication, transactions, errors and audit trails	Connect technical activity to business impact
Security services	Vulnerability, posture, malware and threat detections	Enrich investigations and prioritise risk
SaaS and collaboration	Mailbox access, sharing, administrative changes	Detect identity and data abuse outside infrastructure

A useful distinction is between control-plane telemetry and data-plane telemetry. Control-plane events describe management operations, such as changing a firewall rule or creating a virtual machine. Data-plane events describe use of a resource, such as downloading an object or querying a database. Collecting only control-plane data may reveal who created a storage bucket but not who accessed its contents.

The telemetry pipeline

A robust pipeline normally includes collection, transport, normalisation, enrichment, storage, detection, investigation and response. Collection should occur as close as possible to the source and should use provider-native export mechanisms where available. Transport should protect confidentiality and integrity. Normalisation maps provider-specific fields to a common vocabulary, while enrichment adds asset ownership, business criticality, identity role, geolocation, vulnerability status and threat-intelligence context.

Time synchronisation is essential. Events should use a consistent time standard, normally UTC, and preserve the original event time as well as the ingestion time. A useful event record commonly contains:

Logs should be protected from the systems being monitored. A privileged administrator who can erase both the activity and the evidence creates a serious detection gap. Common controls include centralised export, separate logging accounts, immutable or write-once retention, restricted access, cryptographic integrity controls and independent monitoring of the logging service.

Retention and cost

Retention should reflect threat-detection needs, legal obligations, investigation time and storage cost. Hot storage supports rapid searching; warm or cold storage supports economical historical analysis. The correct retention period is not simply the longest possible period. It should be justified by risk, regulation, contractual requirements and the organisation’s ability to investigate historical data.

Sensitive fields should be minimised or protected. Logs may contain usernames, IP addresses, customer identifiers, query text, tokens or personal data. Redaction, tokenisation, field-level access control and encryption can reduce exposure while preserving analytical value.

3. Platforms and Their Roles

Security Information and Event Management

A SIEM aggregates security-relevant events, normalises them, correlates related activity and presents alerts and investigations. Typical SIEM capabilities include search, rules, correlation, case management, dashboards, threat-intelligence integration and long-term analysis.

SIEM is not a substitute for good source logging. If an identity provider does not record MFA results or a storage service does not record data access, the SIEM cannot reconstruct those facts reliably. SIEM rules should be version-controlled, tested and mapped to threats or control objectives.

Security Orchestration, Automation and Response

SOAR connects alerts to procedures and tools. It may automatically enrich an alert with asset ownership and reputation data, create a ticket, request approval, disable a credential or isolate a workload. Automation should be proportional to confidence and impact. Automatically disabling a production identity based on a weak signal may cause a serious outage; automatically adding context to an alert is generally safer.

A sound playbook defines triggers, prerequisites, actions, approvals, rollback steps, evidence requirements and escalation paths. High-impact actions should normally require human approval unless the organisation has explicitly accepted the operational risk.

Cloud Security Posture Management

CSPM evaluates cloud configurations against policy, benchmarks and organisational standards. It can identify publicly exposed storage, excessive permissions, unencrypted resources, missing logging and insecure network rules. CSPM is primarily posture-oriented: it finds conditions that create risk, whether or not an attack is occurring.

CSPM findings require ownership and prioritisation. A public resource containing no sensitive information may be less urgent than a private database holding regulated data but exposed through a compromised identity. Severity should therefore combine technical exposure with asset criticality, exploitability and business impact.

Cloud-Native Application Protection Platforms

CNAPP combines or integrates capabilities such as CSPM, cloud workload protection, cloud infrastructure entitlement management, container and Kubernetes security, application security and attack-path analysis. Its value lies in connecting development-time and runtime context. For example, a vulnerable package in an internet-facing workload with a high-privilege role is more urgent than the same package in an isolated test system.

CNAPP does not eliminate the need for a SIEM. CNAPP focuses on cloud-native posture, workload and application context, whereas the SIEM commonly provides enterprise-wide event correlation, investigation and case management.

Threat intelligence

Threat intelligence converts information about threats into usable knowledge. Strategic intelligence describes trends and adversary objectives; operational intelligence describes campaigns; tactical intelligence supports detection through indicators and behaviours. Indicators such as IP addresses, domains and hashes can become obsolete or be shared by benign infrastructure. Behavioural intelligence, including attack techniques and sequences, often remains useful for longer.

Intelligence should be assessed for source reliability, information credibility, timeliness, relevance and false-positive risk. Importing every available indicator can pollute detection systems. Intelligence should be mapped to assets, sectors, technologies and known adversary behaviours before being operationalised.

4. Detection Engineering

Detection engineering is the disciplined design, testing, deployment and maintenance of methods for identifying suspicious activity. A detection should state its purpose, data sources, logic, severity, owner, response guidance, testing method and known limitations.

Detection types

A high-quality detection is specific enough to be actionable but broad enough to detect meaningful variations. It should include suppression for approved automation, maintenance windows and known service accounts, while avoiding exclusions so broad that they conceal misuse.

Baselines and anomaly analysis

A baseline describes expected behaviour for an entity during a defined period. Entities may include users, roles, workloads, applications, accounts or data stores. Baselines must account for seasonality, deployment cycles, business hours, geographical distribution and legitimate administrative activity.

Here, μ is the mean and xᵢ is an observation. If observations are strongly skewed, the median may represent normal behaviour better than the mean. A common standard-deviation rule for approximately normal data is:

where x is the current observation and σ is the standard deviation. A large absolute z-score may indicate unusual activity, but it is not proof of an attack. The rule is most appropriate when observations are reasonably independent, the baseline is stable and the distribution is not severely skewed. In cloud security, counts and transfer volumes are often skewed, seasonal or correlated; percentile thresholds, robust statistics or peer-group comparisons may therefore be preferable.

A baseline should not be trained blindly on a compromised period. It should be reviewed, bounded and recalculated when roles, architectures or business processes change.

Risk-informed alerting

Alert priority should consider more than event severity. A practical conceptual model is:

This is a prioritisation model, not a universal law. The factors must be defined consistently, and multiplying ordinal scores does not create a precise probability. Likelihood may reflect threat activity and behavioural abnormality; impact may reflect data sensitivity and service criticality; exposure may reflect internet reachability and privilege; confidence may reflect data quality and detection specificity.

For example, an unusual read from a low-value development bucket may receive lower priority than the same behaviour against a regulated production database. A high-confidence alert involving a privileged identity and an exposed asset deserves rapid action even if the event count is small.

5. Measuring Detection Performance

Security teams need measures that support improvement rather than reward the production of large numbers of alerts.

In a classification setting, true positives (TP) are malicious events correctly alerted; false positives (FP) are benign events incorrectly alerted; true negatives (TN) are benign events correctly ignored; and false negatives (FN) are malicious events missed.

Precision answers: “Of the alerts raised, what proportion were genuinely relevant?” Recall answers: “Of the relevant malicious events, what proportion were detected?” These measures require a labelled evaluation set. In real operations, FN is difficult to know because undetected attacks are not directly observable. Precision alone can also be misleading if a rule generates only one alert.

It is useful when both measures matter and neither should dominate, but it assumes that precision and recall are comparable and does not represent business impact.

Operational metrics include mean time to detect (MTTD), mean time to acknowledge, mean time to contain and mean time to recover. A mean is commonly calculated as total elapsed time divided by the number of relevant incidents. Organisations should specify whether measurement begins at the first malicious event, first observable event or alert creation, because different definitions produce different results.

KPIs measure performance against objectives. Examples include the percentage of critical log sources reporting successfully, median investigation time, percentage of high-risk alerts triaged within target and detection-rule coverage for priority techniques. KRIs indicate increasing exposure or risk, such as the number of privileged accounts without phishing-resistant MFA, percentage of critical assets lacking data-plane logging, or unresolved internet-exposed high-severity findings.

A dashboard should support decisions, not merely display attractive charts. It should show trends, thresholds, ownership, confidence, business impact and ageing. Every metric needs a definition, data source, refresh frequency, target and accountable owner.

6. Governance, Ethics and Compliance

Monitoring can affect employees, customers and other individuals. Organisations should establish a lawful purpose, collect only necessary data, restrict access, document retention and communicate acceptable-use expectations. Excessive employee surveillance may damage trust and may violate privacy or employment requirements.

Governance controls include separation of duties, role-based access, approval for sensitive searches, audit trails for analyst activity, documented escalation, evidence preservation and periodic review of detection logic. Automated decisions should be explainable enough for analysts and affected stakeholders to understand why an account was blocked or an alert was prioritised.

Regulatory obligations vary by jurisdiction and sector. Requirements may concern breach notification, financial records, health information, payment data, cross-border transfer, records retention or auditability. Compliance should not be treated as proof of security: a compliant logging configuration may still fail to detect a novel attack.

Artificial intelligence can help summarise cases, cluster related alerts, generate hypotheses and identify unusual patterns. However, AI systems may hallucinate explanations, reproduce bias, expose sensitive logs or be manipulated by adversarial inputs. Human oversight, restricted data access, validation against known cases, provenance and an approval boundary for disruptive actions are essential.

7. Applied Case: Illustrative Retail Cloud Investigation

Background

A fictional online retailer operates customer accounts and payment-related services across two cloud regions. It uses a central SIEM, CSPM and SOAR platform. A production application role can read customer profile data but is not intended to administer identity policies.

Problem

The SIEM receives an alert that the application role invoked an identity-management API. The event is unusual but not automatically malicious because deployment automation occasionally assumes the role. At the same time, CSPM reports that a storage bucket containing exports is publicly readable. A threat-intelligence feed reports recent attacks targeting exposed cloud credentials.

Analysis

The analyst correlates the events by account, identity and time. The identity event occurred from a workload subnet outside the normal deployment window. Shortly afterward, the role created a new access key, listed objects in the export bucket and transferred an unusually large volume to an unfamiliar destination. The workload also contains a critical unpatched dependency.

The evidence is stronger when considered as a sequence rather than as independent alerts:

Approach and solution

The organisation assigns high priority because the identity is associated with production data, the sequence has high detection confidence and exposure is substantial. The SOAR playbook automatically preserves relevant logs, tags the case, enriches the asset and requests approval for containment. The incident team then disables the newly created key, restricts the bucket, isolates the workload, rotates related secrets and deploys a known-good application image. They review access logs to determine the data scope and notify legal and privacy teams according to the incident plan.

The detection team improves the rule by adding a condition for identity-policy calls from application roles, while excluding an explicitly approved deployment service through a narrowly scoped identity and change-controlled exception. CSPM policy is updated to prevent public access to regulated export buckets. The architecture team removes unnecessary identity permissions from the application role.

Outcome

The investigation identifies unauthorised access to a limited export set and prevents further transfer. The organisation records the incident, validates notification obligations, and measures improvements using reduced time to contain, improved critical-log coverage and removal of excessive application permissions.

Learning points

The case demonstrates why posture, identity, workload, data-plane and threat-intelligence signals should be correlated. It also shows that automation should preserve evidence and enrich context before taking disruptive action. Most importantly, a detection programme must address both immediate incidents and structural causes such as excessive privilege, public exposure and vulnerable software.

8. Best Practices and Limitations

Important limitations remain. Logs may be incomplete, delayed, sampled or unavailable in managed services. Encryption and privacy controls may limit content inspection. Attackers can use legitimate credentials and blend into normal activity. Anomaly detection can produce excessive false positives when behaviour changes rapidly. Threat intelligence can be stale. Metrics can be gamed, for example by closing alerts quickly without improving investigations. Monitoring is therefore one layer of a broader risk-management system, not an assurance that compromise is impossible.

Contemporary Developments

Cloud detection is moving toward identity-centric and graph-based analysis. Rather than treating each event as a record, platforms increasingly model relationships among identities, permissions, workloads, vulnerabilities, data stores and network paths. Attack-path analysis can reveal that a seemingly moderate misconfiguration creates a route to a sensitive asset.

Detection-as-code practices apply software engineering to analytics. Rules are stored in repositories, reviewed, tested against sample data, deployed through controlled pipelines and monitored for performance. Open schemas and cross-platform detection languages can reduce dependence on one provider, although field meanings and data availability still differ.

Extended detection and response (XDR) integrates endpoint, identity, email, network and cloud signals. Its effectiveness depends on reliable entity resolution and clear ownership across teams. Privacy-preserving analytics, confidential computing and improved workload provenance may also strengthen monitoring, but each introduces new operational and assurance requirements.

Chapter Summary

Cloud security monitoring transforms logs, events and telemetry into evidence for detection, investigation and risk decisions. Effective programmes distinguish control-plane from data-plane activity, protect log integrity, normalise and enrich events, and define responsibilities under the shared-responsibility model. SIEM supports central correlation and investigation; SOAR coordinates response; CSPM identifies insecure posture; CNAPP connects cloud-native application, workload and entitlement risks; and threat intelligence adds information about adversaries and behaviours.

Detection engineering requires explicit logic, tested data sources, carefully managed exceptions and continuous tuning. Baselines help identify unusual activity, but statistical thresholds must respect skew, seasonality and changing business behaviour. Risk-informed alerting combines confidence with asset impact and exposure rather than treating every alert equally. Precision, recall, F1, MTTD and containment time help evaluate performance, while KPIs track capability and KRIs reveal increasing exposure.

Monitoring must also respect privacy, legal duties, evidence integrity, access governance and human oversight. The illustrative retail case shows that the strongest conclusions often emerge by correlating identity, posture, workload, data-access and threat-intelligence signals. Monitoring is most valuable when it produces timely, explainable and actionable decisions that reduce organisational risk.

Key Terms

Data plane: Operations that use or access the data and services hosted by resources.

SIEM: A platform for collecting, correlating, analysing and investigating security events.

SOAR: Technology and processes for orchestrating and automating security response.

CNAPP: An integrated platform for cloud-native application, workload, posture and entitlement protection.

Threat intelligence: Evaluated information about threats, adversaries, indicators and behaviours.

Detection engineering: The disciplined creation, testing and maintenance of security detections.

False positive: A benign event incorrectly classified as suspicious or malicious.

Detection-as-code: Managing detection logic with software-development practices.

Shared responsibility: Allocation of security duties between provider, customer and other parties.

Review Questions

Short Answer Questions

Long Answer Questions

Analytical / Application Questions

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes control-plane telemetry from data-plane telemetry?

A. Control-plane telemetry records resource administration, while data-plane telemetry records use of hosted resources or data.

B. Control-plane telemetry is collected only from endpoints, while data-plane telemetry is collected only from networks.

D. Control-plane telemetry is generated by users, while data-plane telemetry is generated only by providers.

Correct Answer: A. Control-plane telemetry records resource administration, while data-plane telemetry records use of hosted resources or data.

Question 2

Which platform is primarily responsible for identifying insecure cloud configurations such as public storage and missing encryption?

Question 3

An alert system produces 80 true positives and 120 false positives. What is its precision?

Question 4

Correct Answer: C. Enriching an alert with asset ownership and reputation data

Question 5

Correct Answer: B. The proportion of relevant malicious events that are detected

Question 6

Correct Answer: C. It may show resource administration but not actual access to hosted data

Question 7

Correct Answer: C. Number of privileged accounts without phishing-resistant MFA

Question 8

A detection has precision 0.80 and recall 0.50. What is its F1 score, approximately?

Question 9

B. Storing rules in version control and testing them against representative events

Correct Answer: B. Storing rules in version control and testing them against representative events

Question 10

B. Indicators may be stale, shared by benign systems or irrelevant to the organisation

Correct Answer: B. Indicators may be stale, shared by benign systems or irrelevant to the organisation

Chapter 10 Security Monitoring, Analytics and Cloud Detection

Security Monitoring, Analytics and Cloud Detection

Introduction