Logging, Monitoring and Threat Detection

Introduction

Cloud security depends on evidence. A control that cannot be observed cannot be reliably tested, and an incident that cannot be reconstructed cannot be effectively contained or learned from. Logging records events; monitoring interprets events over time; threat detection identifies activity that may indicate compromise, misuse, or policy violation. Together, these capabilities support prevention, investigation, compliance, service reliability, and continuous improvement.

Cloud environments make this work both more important and more difficult. Resources are created and destroyed rapidly, identities may be temporary, workloads are distributed across regions and providers, and responsibility is shared between the cloud provider and the customer. A useful security programme therefore collects evidence from several layers: identity, control-plane activity, networks, operating systems, applications, databases, containers, serverless services, and security tools.

The purpose of logging is not to collect everything indiscriminately. Excessive collection increases cost, noise, privacy risk, and the probability that important signals will be missed. The objective is to collect the right evidence, protect it from alteration, analyse it responsibly, and retain it for as long as business, legal, and investigative requirements demand.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the distinction between logging, monitoring, alerting, and threat detection.
2. Identify important cloud log sources and design a practical collection strategy.
3. Describe how logs are protected, normalised, retained, and made available for investigation.
4. Define indicators, detection rules, and useful alerts while controlling false positives.
5. Explain the roles of SIEM platforms, behavioural analytics, and threat intelligence.
6. Apply responsible practices for privacy, governance, and AI-assisted detection.
7. Develop a basic response workflow from a suspicious event to a documented decision.

1. The Evidence Model of Cloud Security

A security event is an observable occurrence, such as a successful login, an access-policy change, a rejected network connection, or an application exception. A log is a recorded representation of that event. Monitoring is the continuing observation and interpretation of system behaviour. An alert is a notification generated when a defined condition is met. Threat detection is the process of determining whether observed activity is potentially malicious or otherwise unacceptable.

These concepts are related but not interchangeable. A system may log an event without alerting on it. A monitoring dashboard may display a metric without identifying an attack. An alert may be technically correct but operationally unimportant. Effective detection links evidence to a decision or action.

A helpful chain is:

Event → Log record → Normalisation → Correlation or analysis → Alert → Triage → Response → Learning

The chain has weaknesses if any stage is absent. Missing logs produce blind spots; inaccurate timestamps prevent correlation; poorly designed rules produce alert fatigue; weak triage delays response; and failure to learn allows recurring incidents.

Security properties of logs

Security logs should support four primary properties:

• Authenticity: the record genuinely originated from the stated source.
• Integrity: the record has not been changed without detection.
• Availability: authorised investigators can access it when needed.
• Confidentiality: sensitive contents are protected from unauthorised disclosure.

Logs are often evidence in an investigation. Consequently, access should be restricted, administrative actions should themselves be logged, and retention and deletion should be governed rather than improvised.

2. What Should Be Logged?

A cloud logging strategy begins with risk and use cases, not with a product catalogue. The organisation should ask: Which actions must be attributable? Which events could indicate compromise? Which records are needed to demonstrate compliance? Which evidence is required to restore service or investigate fraud?

2.1 Identity and access logs

Identity logs record authentication and authorisation activity. Useful fields include subject or principal, authentication method, source address, device or workload identity, target resource, action, result, timestamp, and reason for denial. Examples include:

• successful and failed sign-ins;
• multi-factor authentication challenges and changes;
• creation, deletion, or modification of users, roles, groups, and service accounts;
• privilege elevation and role assumption;
• changes to access policies;
• use of emergency or break-glass accounts; and
• token issuance, revocation, and unusual geographical activity.

Identity events are especially valuable because attackers frequently seek valid credentials rather than exploiting a technical vulnerability directly.

2.2 Control-plane and audit logs

Control-plane logs record administrative actions against cloud resources: creating a storage bucket, changing a firewall rule, disabling encryption, modifying a key policy, or deleting a database. They commonly provide the strongest account of who changed the environment and when.

A practical baseline is to capture both successful and failed administrative actions. Repeated failures may indicate reconnaissance or attempted abuse, while a successful change may be dangerous even if it is later reversed.

2.3 Network logs

Network evidence can include flow records, firewall decisions, load-balancer logs, DNS queries, web application firewall events, VPN activity, and private endpoint connections. Flow records typically show communicating addresses, ports, protocols, direction, volume, and decision, but not the full content of packets. This distinction matters: flow data supports behavioural analysis, whereas packet or application logging may be needed to understand content.

Network logs can reveal scanning, unexpected outbound communication, lateral movement, data exfiltration, and connections to known malicious infrastructure. They should be interpreted in context because cloud services may legitimately communicate across many regions and addresses.

2.4 Workload and application logs

Operating-system logs include process creation, service changes, authentication, kernel events, and endpoint-security findings. Container and orchestration logs include image pulls, workload creation, privilege settings, admission decisions, and control-plane activity. Serverless logs may include invocation results, execution duration, errors, and identity context.

Application logs should record security-relevant business events such as password resets, changes to payment details, administrator actions, data exports, and repeated authorisation failures. They should not record passwords, private keys, authentication tokens, or unnecessary personal data.

2.5 Data and database logs

Database audit records may identify queries, schema changes, administrative access, failed connections, and exports. Storage access logs can show reads, writes, object deletion, and changes in sharing permissions. Logging every data value is rarely necessary and can create serious privacy risk. Metadata about access is often sufficient for detection.

3. Designing a Collection and Protection Architecture

A common pattern is to collect logs close to their sources, forward them to a central aggregation layer, transform them into a consistent schema, store an immutable or tamper-evident copy, and send suitable data to an analysis platform.

Centralisation improves search and correlation. For example, a suspicious sign-in becomes more meaningful when correlated with a new access-policy change and an unusual outbound connection. However, centralisation also creates an attractive target. The logging account, storage location, and ingestion pipeline must be protected as critical security infrastructure.

3.1 Collection decisions

For each log source, document:

Collection should cover production and, where risk warrants, development and test environments. Attackers may use less-protected environments as a route to production.

3.2 Time and identity consistency

Correlation depends on reliable time. Systems should use an approved time-synchronisation service, record timestamps consistently, and preserve the original timestamp as well as the ingestion timestamp. Clock drift, delayed delivery, and daylight-saving conversions can otherwise produce misleading event sequences.

Resource and identity naming should also be stable. A temporary virtual machine identifier may be less useful than a persistent workload, account, project, or deployment identifier. A log schema should distinguish the actor, the affected resource, the action, the result, and the source location.

3.3 Integrity and access controls

Important controls include:

• write permissions separated from read and delete permissions;
• restricted access through dedicated security roles;
• encryption in transit and at rest;
• immutable or append-only storage for selected evidence;
• retention locks or equivalent deletion safeguards;
• cryptographic hashes or signatures where tamper evidence is required;
• independent monitoring of logging configuration; and
• alerts when sources stop reporting or logging is disabled.

Encryption protects confidentiality but does not, by itself, prove that a record has not been altered by an authorised administrator. Access separation, immutable storage, and audit trails address that risk more directly.

3.4 Retention and cost

Retention should be based on use, risk, contractual commitments, and applicable law. A single period is not always appropriate. Recent high-value logs may remain readily searchable, while older logs may be compressed or archived. Deletion should be deliberate and documented.

Storage cost is only one part of total cost. Processing, indexing, network transfer, analyst time, and alert investigation also matter. Sampling may be acceptable for high-volume performance telemetry, but security-critical events should not be sampled without a documented risk decision.

4. Indicators, Detection Rules, and Alerts

An indicator is an observable sign associated with a condition of interest. Indicators may be technical, behavioural, or contextual. Examples include a known malicious domain, a disabled audit trail, impossible travel between sign-ins, an unusual privilege grant, or a sudden increase in data transfer.

Indicators are not proof of compromise. A malicious IP address may be shared by legitimate services; an unusual login may be a travelling employee; and a large export may be a scheduled business process. Detection should therefore combine evidence and context.

Rule types

Signature or indicator-based rules match known values or patterns, such as a malware hash or a connection to a threat-intelligence domain. They are precise when indicators are reliable but perform poorly against new or modified attacks.

Threshold rules detect counts or rates exceeding a boundary, such as many failed sign-ins in a short interval. The threshold must reflect normal behaviour and account for legitimate batch activity.

Correlation rules combine events, often across sources. For example, a new privileged role assignment followed by access to sensitive storage from a previously unseen location is more significant than either event alone.

Sequence rules identify an ordered progression, such as reconnaissance, credential use, privilege escalation, and data access. They require accurate time and entity correlation.

Behavioural rules compare current activity with a baseline for a user, service, device, or workload. They can detect novel activity but may be difficult to explain and can be sensitive to changes in business practice.

Alert quality

A useful alert should state what happened, why it matters, affected entities, supporting evidence, confidence, severity, and recommended next action. Severity should not be confused with confidence. An event can be highly severe but uncertain, or highly certain but low impact.

Alert fatigue occurs when analysts receive too many low-value notifications. It can be reduced by suppressing duplicates, grouping related events, tuning thresholds, adding allow-lists with expiry dates, enriching alerts with ownership and asset criticality, and measuring outcomes. Tuning must not simply hide uncomfortable evidence.

5. SIEM and Behavioural Analytics

A security information and event management (SIEM) platform generally provides central ingestion, parsing, search, correlation, dashboards, alerting, case management, and reporting. It can connect identity, network, workload, and application evidence into a common investigative view.

A SIEM is not automatically a complete detection programme. Its effectiveness depends on source coverage, data quality, rule design, analyst capability, and response integration. A high-priced platform with missing audit logs remains ineffective.

Security orchestration, automation, and response (SOAR) tools may enrich an alert, open a case, disable a token, or isolate a workload. Automated actions should be proportionate and reversible. An incorrect automatic account disablement can interrupt critical services and create operational harm.

User and entity behaviour analytics (UEBA) establishes a model of usual activity for identities and resources. It might consider login times, locations, resource types, command patterns, or data volumes. A simple conceptual deviation score can be written as:

Here, x is an observed value, μ is the baseline mean, and σ is the baseline standard deviation. The score is dimensionless and is useful only when the baseline is reasonably stable and the distribution is not badly misrepresented by a mean and standard deviation. A large absolute score indicates unusualness, not maliciousness. Many operational behaviours are non-normal, seasonal, or multimodal; therefore, a production system may use percentiles, peer groups, time-of-day baselines, or other models instead.

6. Detection Engineering and Performance

Detection engineering treats rules as maintainable software. Each rule should have a purpose, data dependencies, logic, owner, test cases, severity, response guidance, and review date. Rules should be tested against historical benign events and known attack simulations where permitted.

A confusion matrix helps explain outcomes:

The false-positive rate is commonly defined as:

The precision of alerts is:

Here, TP, FP, TN, and FN denote true positives, false positives, true negatives, and false negatives. These measures apply to a defined dataset and classification threshold. They can be misleading when incidents are rare, when labels are incomplete, or when one incident generates many duplicate alerts. Detection programmes should also measure time to acknowledge, time to investigate, time to contain, coverage of important attack paths, and analyst workload.

A worked example illustrates the difference. Suppose a review identifies 20 true threat alerts and 80 false alerts. Precision is 20 ÷ (20 + 80) = 0.20, or 20 percent. This does not mean the detector is useless: if it identifies high-impact attacks that are otherwise difficult to find, it may be justified. However, the investigation cost and the possibility of missed threats should be assessed.

7. Responsible AI-Assisted Detection

Machine learning can support anomaly detection, event prioritisation, entity resolution, and summarisation. Generative AI can help analysts query data, explain a rule, draft an incident timeline, or suggest investigative steps. These systems should assist human judgement rather than silently determine guilt or impose irreversible action.

Important risks include:

• biased baselines that treat certain users or locations as inherently suspicious;
• drift when normal business behaviour changes;
• poor explainability;
• leakage of sensitive logs to an unauthorised model or provider;
• fabricated explanations or inaccurate summaries;
• adversarial manipulation of input data; and
• automation bias, in which analysts accept a system recommendation without scrutiny.

Responsible deployment requires approved data handling, access controls, model and prompt logging where appropriate, validation against representative cases, human review for consequential actions, and a means to challenge or correct decisions. Personal data should be minimised, pseudonymised where practical, and processed under a clear purpose. Retaining raw prompts and model outputs may itself create additional sensitive records.

8. Privacy, Governance, and Legal Considerations

Security monitoring can observe employees, customers, contractors, and members of the public. A legitimate security purpose does not eliminate privacy obligations. Organisations should define purpose, necessity, proportionality, access, retention, and transparency. They should distinguish security investigation from employee performance surveillance.

Governance questions include:

• Is collection limited to what is necessary?
• Are individuals informed where required?
• Are sensitive categories, such as health or biometric data, excluded or specially controlled?
• Are cross-border transfers permitted and protected?
• Who may access investigation records?
• How are legal holds and deletion requests handled?
• Is monitoring tested for discriminatory effects?

Logs may contain identifiers, addresses, message content, or query parameters. Masking and tokenisation can reduce exposure, but investigators may require controlled access to original data. A documented break-glass process should record the reason, approver, scope, and duration of exceptional access.

9. Operational Workflow: From Alert to Learning

A basic workflow is:

1. Validate ingestion. Confirm that relevant sources are reporting and timestamps are credible.
2. Triage. Determine what happened, which entities are involved, and whether the alert is duplicate or benign.
3. Enrich. Add asset criticality, ownership, identity context, threat intelligence, and recent changes.
4. Scope. Search for related activity across accounts, regions, workloads, and time periods.
5. Contain carefully. Revoke tokens, restrict network access, isolate a workload, or disable a rule only with appropriate authority and consideration of service impact.
6. Investigate and preserve. Protect relevant logs and document actions, assumptions, and evidence.
7. Eradicate and recover. Remove persistence, correct the root cause, restore trusted operation, and monitor for recurrence.
8. Review. Improve controls, detection logic, documentation, and training.

The chain of custody for evidence should identify who collected it, when, from where, how it was protected, and who accessed it. This is particularly important when incidents may lead to disciplinary, contractual, or legal proceedings.

Applied Case: Illustrative Cloud Storage Investigation

Background

A fictional online education provider stores course materials and limited learner records in cloud object storage. A central SIEM receives identity, storage-access, network-flow, and control-plane logs. A rule alerts when a privileged identity accesses a sensitive bucket from a new location and transfers an unusually large volume of data.

Problem

At 02:15, the SIEM reports that an administrator account accessed the learner-records bucket from an unfamiliar network address. Shortly afterwards, a storage policy was changed and 18 GB of data was read. The administrator normally works during business hours from a corporate network.

Analysis

The security analyst first verifies that the identity logs are complete and that the timestamp is trustworthy. The administrator confirms no scheduled export was planned. Correlation shows a successful sign-in without the expected second authentication factor, followed by a new access key and policy modification. Network logs show outbound connections to an address not previously associated with the organisation.

No single indicator proves compromise. Together, the unusual access time, missing expected authentication factor, credential change, policy modification, and data volume create a high-confidence incident. The analyst also checks whether the storage policy was reverted and searches for the same access key in other accounts.

Approach and solution

The response team disables the new key, revokes active sessions, temporarily restricts the affected account, and preserves relevant logs in protected storage. They examine identity-provider records and endpoint telemetry to determine whether the administrator’s workstation was compromised. Storage access records identify the objects read, supporting notification and impact assessment. The original policy is restored through a reviewed change process, and stronger conditions are applied to sensitive storage.

The SIEM rule is then improved. It is changed from a single-location threshold to a correlation rule requiring several conditions: privileged access, a new credential or policy change, sensitive data access, and unusual transfer behaviour. An approved service account is added as a documented exception with an expiry date and continued monitoring.

Outcome and learning points

The investigation identifies unauthorised use of a privileged credential and confirms that logging was sufficient to establish scope. No passwords or raw learner content were placed in application logs. The provider improves phishing-resistant authentication, reduces standing privilege, tests alert coverage, and reviews whether the retained data is proportionate.

The case demonstrates that detection is strongest when identity, control-plane, storage, and network evidence are correlated. It also shows why exceptions require ownership and expiry, why alerts need context, and why response actions should preserve evidence before destructive changes are made.

Challenges and Limitations

Cloud logging has unavoidable limitations. Provider-managed services may expose less detail than self-managed systems. Logs may be delayed, incomplete, sampled, or unavailable during an outage. Encryption and privacy controls may restrict content inspection. Attackers may use legitimate credentials and perform actions that resemble normal administration. Detection systems can also be manipulated through log flooding or by disabling telemetry.

False negatives are especially dangerous because they are difficult to observe. Periodic control testing, attack simulations, threat-led reviews, and independent assessment help reveal blind spots. Conversely, indiscriminate collection can reduce security by overwhelming analysts and expanding the privacy attack surface. The correct objective is risk-based observability, not maximum volume.

Best-Practice Checklist

• Define security use cases before selecting log sources.
• Collect identity, control-plane, network, workload, application, and data-access evidence appropriate to risk.
• Synchronise time and preserve original event timestamps.
• Centralise logs while maintaining a protected, tamper-evident copy.
• Separate log-writing, searching, administration, and deletion privileges.
• Monitor the health of the logging pipeline itself.
• Minimise sensitive content and prohibit secrets in logs.
• Give every detection rule an owner, purpose, test method, and review date.
• Enrich alerts with asset and identity context.
• Measure both missed threats and analyst burden.
• Keep automated response actions reversible and subject to governance.
• Validate AI-assisted tools, protect their inputs, and retain human accountability.
• Conduct post-incident reviews and update controls based on evidence.

Chapter Summary

Logging creates records of cloud activity; monitoring interprets activity over time; alerting communicates a condition; and threat detection evaluates whether activity may represent harm. Effective programmes collect evidence from identity, control-plane, network, workload, application, and data layers. Centralisation and normalisation support correlation, while time synchronisation, access separation, encryption, immutable storage, retention governance, and pipeline monitoring protect the evidence.

Indicators may be signature-based, threshold-based, correlated, sequential, or behavioural. SIEM platforms provide collection, search, correlation, and case management, while UEBA and AI can identify unusual behaviour and assist analysts. These tools are not substitutes for sound data, skilled investigation, and accountable response. Precision, false-positive rate, response time, and coverage are useful measures when their assumptions and data quality are understood.

Finally, security monitoring must respect privacy, proportionality, purpose limitation, and legal requirements. A mature capability connects reliable evidence to well-designed decisions, preserves human oversight, and learns from every investigation.

Key Terms

Alert: A notification that a defined condition has been detected.

Audit log: A record of activity maintained to support accountability and review.

Behavioural analytics: Analysis that identifies deviations from expected activity patterns.

Chain of custody: Documentation showing how evidence was collected, protected, accessed, and transferred.

Control-plane log: A record of administrative actions on cloud resources.

Detection rule: Logic used to identify a condition of security interest.

False negative: A threat that is present but not detected.

False positive: An alert generated for activity that is not a threat.

Flow record: Network metadata describing communications without necessarily recording content.

Indicator: An observable sign associated with a condition, threat, or investigation.

Log integrity: Assurance that records have not been altered without detection.

Monitoring: Ongoing observation and interpretation of system activity.

Precision: The proportion of generated alerts that are true positives.

SIEM: Security information and event management platform for collecting and analysing security data.

SOAR: Security orchestration, automation, and response capability.

Threat intelligence: Information about threats, including indicators, behaviours, and context.

Triage: Initial assessment that establishes priority, scope, and likely significance.

UEBA: User and entity behaviour analytics.

Review Questions

Short Answer Questions

1. Distinguish between a log, a monitoring activity, an alert, and threat detection.
2. Why are identity and control-plane logs particularly valuable in cloud investigations?
3. State two controls that protect log integrity.
4. What is alert fatigue, and how can it be reduced?
5. Why should AI-assisted detection retain human oversight?

Long Answer Questions

1. Explain how an organisation could design a risk-based cloud log-collection strategy.
2. Compare signature-based, threshold-based, correlation-based, and behavioural detection rules.
3. Describe the role of a SIEM and explain why purchasing one does not guarantee effective detection.
4. Discuss privacy and governance issues arising from employee and customer monitoring.
5. Explain the complete workflow from receiving a suspicious alert to completing a post-incident review.

Analytical / Application Questions

1. A company receives 500 alerts in one week, of which 50 are confirmed threats. Calculate the precision and explain what the result implies for analyst workload.
2. A privileged account creates a new access key, changes a storage policy, and downloads sensitive data from an unfamiliar location. Design a correlation rule and identify three investigative checks.
3. A proposed AI system classifies employees as high or low risk using login location and time. Evaluate two technical risks, two privacy risks, and three safeguards that should be required before deployment.

Multiple Choice Questions with Answers

Question 1

Which log source most directly records a change to a cloud storage access policy?

A. Control-plane audit log

B. Application performance log

C. CPU utilisation metric

D. Customer support ticket

Correct Answer: A. Control-plane audit log

Question 2

What is the primary purpose of centralising security logs?

A. To eliminate the need for access control

B. To support correlation and consistent investigation

C. To guarantee that every alert is accurate

D. To remove all privacy obligations

Correct Answer: B. To support correlation and consistent investigation

Question 3

Which statement about an indicator is most accurate?

A. It always proves that an attack occurred

B. It is an observable sign requiring contextual interpretation

C. It is only a legal document

D. It can be used only in network monitoring

Correct Answer: B. It is an observable sign requiring contextual interpretation

Question 4

Which control most directly helps prevent an administrator from deleting audit evidence?

A. Disabling time synchronisation

B. Separating deletion privileges and using immutable storage

C. Increasing dashboard colours

D. Sampling all security events

Correct Answer: B. Separating deletion privileges and using immutable storage

Question 5

A detector generates 20 true-positive alerts and 80 false-positive alerts. What is its precision?

A. 5%

B. 20%

C. 80%

D. 100%

Correct Answer: B. 20%

Question 6

Which detection method combines several related events across systems?

A. Correlation rule

B. Password rotation

C. Data compression

D. Static backup

Correct Answer: A. Correlation rule

Question 7

What is a major limitation of a large behavioural deviation score?

A. It automatically proves malicious intent

B. It may indicate unusual activity without identifying its cause

C. It cannot be calculated from any baseline

D. It removes the need for investigation

Correct Answer: B. It may indicate unusual activity without identifying its cause

Question 8

Which practice is most appropriate for responsible AI-assisted detection?

A. Allowing irreversible actions without review

B. Sending unrestricted raw logs to any public model

C. Validating outputs and retaining human accountability

D. Suppressing all explanations to reduce workload

Correct Answer: C. Validating outputs and retaining human accountability

Question 9

Which event is an example of a false negative?

A. A legitimate export produces an alert

B. A real compromise occurs but produces no alert

C. A true threat produces an alert

D. A dashboard displays a normal metric

Correct Answer: B. A real compromise occurs but produces no alert

Question 10

Why should security-critical events generally not be sampled without a documented decision?

A. Sampling may remove evidence needed to detect or investigate incidents

B. Sampling always increases storage cost

C. Sampling prevents encryption

D. Sampling guarantees false positives

Correct Answer: A. Sampling may remove evidence needed to detect or investigate incidents