Security Monitoring, Analytics and AI

Introduction and Context

Security monitoring is the continuous collection, processing and interpretation of evidence about activity in information systems. Its purpose is not merely to record events, but to identify conditions that may threaten confidentiality, integrity, availability, safety or regulatory compliance. Modern organisations generate telemetry from firewalls, identity systems, cloud platforms, applications, databases, endpoints, networks and operational technology. Without disciplined collection and analysis, this evidence becomes fragmented, delayed and difficult to act upon.

A contemporary security operations capability combines four elements:

1. Telemetry collection from relevant systems.
2. Centralised analysis and correlation, commonly through a security information and event management (SIEM) platform.
3. Orchestrated response, often supported by security orchestration, automation and response (SOAR).
4. Human judgement and governance, which determine whether an alert is credible, important and lawful to investigate.

Artificial intelligence and machine learning can improve detection of unusual behaviour, but they do not eliminate the need for sound data engineering, carefully defined use cases or experienced analysts. A model can detect a statistical deviation while failing to understand its business meaning. Conversely, a rule may be simple but highly valuable when it reflects a known attack technique.

Learning Objectives

By the end of this chapter, students should be able to:

• describe the architecture and purposes of SIEM, SOAR, network monitoring and endpoint telemetry;
• distinguish events, alerts, incidents, indicators and evidence;
• design detection use cases and establish meaningful baselines;
• apply correlation, statistical analysis and selected AI-assisted methods;
• evaluate false positives, false negatives, model drift and adversarial manipulation;
• explain privacy, proportionality, auditability and human-oversight requirements; and
• design a defensible monitoring and response workflow for an organisational environment.

1. The Monitoring Information Model

An event is a recorded occurrence, such as a successful login, process creation or DNS request. An alert is an event or combination of events judged sufficiently suspicious to warrant review. An incident is a confirmed or reasonably suspected compromise, policy violation or security-impacting occurrence requiring coordinated response. An indicator is an observable artefact associated with activity, such as an IP address, file hash, domain, user-agent string or behavioural pattern. Telemetry is the broader stream of measurements and records collected from technical systems.

A useful monitoring process converts raw observations into progressively more meaningful representations:

Raw telemetry → Normalised events → Enriched events → Correlated detections
       → Analyst triage → Incident determination → Response and learning

Raw logs are rarely sufficient by themselves. A login record becomes more informative when enriched with identity role, device ownership, geographic context, authentication method, threat-intelligence information and the user's normal working pattern. Enrichment must be controlled: inaccurate geolocation or stale asset ownership can create misleading conclusions.

Log quality

A log source should be evaluated for:

• Coverage: whether it records the events relevant to the threat model;
• Completeness: whether required fields are present;
• Accuracy: whether timestamps, identities and outcomes are trustworthy;
• Timeliness: the delay between occurrence and availability for analysis;
• Integrity: protection against alteration or deletion; and
• Retention: whether records remain available for investigation and legal requirements.

Time synchronisation is fundamental. Systems should use a common time reference, normally through authenticated Network Time Protocol services. Analysts must also understand time zones, daylight-saving changes, clock drift and delayed delivery. A sequence reconstructed from inconsistent timestamps can produce an incorrect attack narrative.

2. SIEM Architecture and Correlation

A SIEM normally performs collection, parsing, normalisation, storage, search, correlation, visualisation, alerting and case management. Sources may include:

• identity and access management systems;
• operating-system and application logs;
• firewalls, proxies, DNS and email gateways;
• cloud control planes and SaaS audit logs;
• vulnerability and asset-management platforms;
• endpoint detection and response (EDR) agents; and
• network detection and response (NDR) sensors.

Normalisation maps different vendor fields into a common schema. For example, src_ip, sourceAddress and client_ip may be represented as source.ip. Normalisation supports portable detections and consistent investigation. It does not make the underlying data equivalent; an authentication event from a cloud identity provider may have different assurance and semantics from a local operating-system login.

Correlation links separate events using shared attributes or temporal relationships. Common patterns include:

• repeated failed logins followed by success;
• a new administrator account followed by privilege use;
• execution of a scripting interpreter followed by an outbound connection;
• a suspicious email attachment followed by a new process and credential access; and
• unusual data access followed by large-volume transfer.

A correlation rule should specify its data sources, logic, time window, thresholds, exclusions, severity, owner and response action. It should also map to an established threat framework, such as a tactic or technique in MITRE ATT&CK, where appropriate. Mapping is useful for coverage analysis, but it does not prove that a detection is effective.

Detection logic and risk

A practical alert priority can combine several dimensions:

• confidence that the activity is malicious;
• impact of the affected asset or account;
• scope and potential spread;
• urgency and attacker dwell time; and
• quality of available evidence.

These dimensions should not be reduced mechanically to a single number without governance. A low-confidence event on a domain controller may deserve faster review than a high-confidence event on an isolated test machine.

3. SOAR and the Response Feedback Loop

SOAR platforms connect alerts to playbooks and external tools. A playbook may enrich an alert with asset ownership, query endpoint telemetry, check reputation services, create a case, request analyst approval and then disable an account or isolate a device.

Automation is safest when actions are:

• reversible;
• bounded by explicit conditions;
• logged with actor, time and reason;
• tested against failure modes; and
• subject to approval for high-impact operations.

For example, automatically blocking a malicious domain at a web gateway may be relatively reversible. Automatically deleting an account or isolating a hospital device may create serious operational harm. SOAR should therefore implement graduated automation: automatic enrichment first, recommendation second, and autonomous containment only where the risk has been formally accepted.

A useful feedback loop is:

Alert → Triage → Decision → Response → Recovery → Lessons learned
  ↑                                                ↓
  └──────────── rule, data and playbook improvement ────────────┘

Analyst decisions should be recorded. Labels such as true positive, benign positive, duplicate, insufficient evidence and false positive support later tuning and supervised learning. Labels must be defined consistently; otherwise historical data teaches a model the preferences of individual analysts rather than reliable security outcomes.

4. Network and Endpoint Telemetry

Network monitoring

Network monitoring observes communications and protocols. Relevant data includes flow records, DNS queries, TLS metadata, proxy logs, packet captures and network-device events. Full packet capture can provide rich evidence but is expensive and may create significant privacy obligations. Flow telemetry is less detailed but often more scalable.

Useful network features include source and destination, port and protocol, byte counts, packet counts, duration, direction, DNS frequency, certificate metadata and connection timing. Encrypted traffic reduces content visibility, so monitoring increasingly relies on metadata, endpoint evidence, identity context and carefully governed decryption where lawful and technically appropriate.

Endpoint telemetry

Endpoint detection and response systems collect process creation, command lines, parent-child relationships, file activity, registry or configuration changes, driver loading, authentication events, persistence mechanisms and network connections. Endpoint telemetry is valuable because it can reveal execution and user context that network data cannot.

However, endpoint data can be incomplete. An agent may be disabled, a process may execute in memory, operating-system logs may be overwritten, or a cloud workload may be short-lived. Detection engineering should therefore avoid dependence on a single sensor for high-value threats.

5. Detection Engineering and Baselines

A detection use case begins with a security question, not with a vendor feature. Examples are:

• Has an account authenticated from an implausible combination of locations and devices?
• Has a privileged identity performed an unusual administrative action?
• Has a host executed a scripting interpreter from an office document?
• Has a workload accessed substantially more sensitive data than its normal role requires?

A use-case specification should contain:

A baseline describes normal behaviour for a defined entity and time period. It may be a typical login hour, normal data-transfer volume, usual process lineage or expected administrative action. Baselines must be segmented. Comparing a database administrator with a call-centre worker, or a production server with a developer laptop, produces misleading anomalies.

For a measured variable x, a simple sample mean is:

Here, n is the number of observations and xᵢ is observation i. The mean is appropriate when observations are reasonably comparable and extreme values do not dominate interpretation. For skewed quantities such as transferred bytes, the median or a percentile may be more representative.

Sample variance is:

The n − 1 denominator estimates population variance from a sample when observations are independently drawn from the same distribution. Security telemetry often violates independence because events cluster during incidents, maintenance or business cycles. Therefore, variance-based thresholds should be treated as operational heuristics, not automatic proof of maliciousness.

A standardised score can be written as:

where μ is the reference mean and σ is the reference standard deviation. The score is dimensionless. Interpreting a value as a probability requires stronger distributional assumptions; a large z-score is not, by itself, evidence of an attack. Robust alternatives use the median and median absolute deviation, particularly when the baseline contains outliers.

Worked example: threshold interpretation

Suppose a service account normally creates 2 to 8 tickets per hour, with a baseline mean of 5 and standard deviation of 2. During one hour it creates 15 tickets. The standardised score is:

This is an unusual observation relative to the stated baseline. It should trigger investigation, but it does not establish compromise. A scheduled batch job, a new deployment or a changed business process could explain it. The analyst must examine identity, source host, ticket targets, timing and related events.

6. Statistical and AI-Assisted Analytics

Security analytics may be descriptive, diagnostic, predictive or prescriptive. Descriptive analytics summarises activity; diagnostic analytics explores causes; predictive analytics estimates future or unseen conditions; prescriptive analytics recommends actions.

Common methods include:

• Rule-based detection: transparent and effective for known patterns;
• Thresholding: flags values above or below an operational boundary;
• Time-series analysis: models recurring patterns and seasonality;
• Clustering: groups similar behaviour without labelled outcomes;
• Classification: predicts categories from labelled examples;
• Sequence analysis: identifies suspicious event orderings; and
• Graph analysis: examines relationships among users, devices, processes and resources.

Machine learning terminology must be used precisely. In supervised learning, a model learns from labelled examples. In unsupervised learning, it identifies structure without target labels. In semi-supervised learning, a small labelled set is combined with a larger unlabelled set. Anomaly detection identifies observations that differ from a reference population; it does not necessarily identify attacks.

An operational classifier can be assessed using a confusion matrix:

Precision and recall are:

Here, TP, FP and FN denote true positives, false positives and false negatives. Precision answers, “Of the alerts raised, how many were malicious?” Recall answers, “Of the malicious cases, how many were found?” Both require reliable labels and a defined evaluation population.

The F1 score is the harmonic mean of precision and recall:

It is useful when a balance is desired, but it can conceal operational priorities. In a safety-critical environment, missing one severe attack may matter more than generating additional false positives. Conversely, an overwhelmed SOC may need to prioritise precision to preserve analyst capacity.

Model limitations

AI-assisted detection is vulnerable to:

• Concept drift: normal behaviour changes over time;
• Data drift: the distribution of inputs changes;
• Label bias: labels reflect incomplete investigations;
• Class imbalance: malicious events are much rarer than benign events;
• Feature leakage: training data accidentally includes information unavailable at deployment;
• Explainability limitations: a score may not provide a defensible reason;
• Feedback loops: analysts investigate only model-selected activity; and
• Adversarial manipulation: attackers alter inputs to evade or poison the model.

A model must be evaluated on temporally separated data where possible. Randomly splitting events from the same incident across training and testing can produce inflated performance. Validation should include changes in operating conditions, new software, identity migrations and known attack simulations.

7. False Positives, False Negatives and Operational Quality

A false positive is an alert classified as malicious when it is benign. A false negative is malicious activity that is not detected. Neither is merely a technical nuisance. False positives consume analyst time, cause alert fatigue and may lead to disabling valuable rules. False negatives create exposure and may allow attacker persistence.

Tuning should proceed through evidence:

1. define the desired security outcome;
2. examine representative alert samples;
3. identify recurring benign causes;
4. add precise context rather than broad exclusions;
5. test against known malicious and benign scenarios;
6. measure alert volume, precision, investigation time and missed cases; and
7. review after environmental or threat changes.

A broad exclusion such as “ignore all activity from administrators” is dangerous. A safer control may restrict the exclusion to a named service, approved change window, managed device and documented ticket. Suppression rules require owners, expiration dates and periodic review.

Useful operational measures include mean time to detect, mean time to acknowledge, mean time to contain, percentage of critical assets covered, alert-to-incident conversion, data-source availability and detection-test pass rate. Metrics should not reward superficial speed at the expense of investigation quality.

8. Privacy, Governance and Human Oversight

Monitoring can involve personal data, communications metadata, employee behaviour and sensitive business information. Organisations should apply purpose limitation, data minimisation, access control, retention limits, transparency and lawful processing. Security necessity does not automatically justify collecting every possible field.

Governance should address:

• who may access raw and enriched telemetry;
• whether monitoring is proportionate to the stated purpose;
• how employee and customer data are separated or masked;
• retention and secure deletion schedules;
• cross-border transfer and third-party processing;
• audit trails for searches and automated actions;
• model documentation, testing and approval; and
• appeal and review mechanisms for high-impact decisions.

Human oversight is essential where an automated conclusion may affect employment, access to essential services, safety or legal rights. Analysts should be able to inspect supporting evidence, challenge model output and record uncertainty. Automation should assist accountable decision-makers rather than create an unreviewable authority.

9. Illustrative Case: Suspicious Cloud Account Activity

Background

A fictional financial-services organisation operates a hybrid environment. Its SIEM receives cloud audit logs, identity events, endpoint telemetry and proxy records. A privileged engineer’s account is used from a managed laptop during business hours.

Problem

The SOC receives an alert indicating that the account authenticated from the usual country but from a previously unseen device. Within 20 minutes, the account creates an access key, changes a storage policy and downloads an unusually large volume of customer-support records.

Analysis

No single event proves compromise. The SIEM correlates the new-device login, creation of a long-lived credential, privilege modification and unusual data access. Endpoint telemetry shows that the engineer’s laptop launched a browser process from an unsigned extension shortly before authentication. The proxy records connections to an unfamiliar domain.

The SOC checks the change-management system and confirms that no approved task explains the activity. Identity logs show that multifactor authentication was completed, but the authentication method was vulnerable to session-token theft. The sequence therefore supports a credential-compromise hypothesis rather than simply an anomalous login.

Approach and solution

The organisation invokes its response playbook. Analysts preserve cloud audit records, revoke active sessions and newly created keys, temporarily restrict the account, and isolate the laptop after confirming that isolation will not disrupt a safety-critical process. The storage policy is restored from an approved configuration. The organisation identifies accessed records, notifies appropriate privacy and legal functions, and searches for the same indicators across other accounts.

The detection is then improved. Instead of alerting on every first-seen device, the rule assigns greater priority when first-seen device access is followed by credential creation, privilege change or unusual data volume. The exclusion for approved administration is limited to specific tools and change tickets. The playbook is updated to require human approval before broad account suspension.

Outcome

The organisation determines that the account session was hijacked through a malicious browser extension. Rapid correlation limited the period of unauthorised access. The post-incident review identifies gaps in browser governance, token protection, cloud-key lifetime and monitoring of policy changes.

Learning points

• Correlation across identity, endpoint, cloud and network sources is stronger than isolated anomaly detection.
• Multifactor authentication does not eliminate session theft or endpoint compromise.
• Contextual enrichment improves prioritisation.
• Automated containment must consider operational consequences.
• Detection quality depends on post-incident learning and controlled tuning.

10. Contemporary Developments and Best Practice

Security operations increasingly use cloud-native telemetry, identity-centric detection, attack-path analysis, eBPF-based endpoint observation, privacy-preserving analytics and large-language-model assistants. Language models may help summarise cases, translate queries or suggest investigative pivots, but generated content can be inaccurate, omit evidence or expose sensitive data to an unauthorised service. Their outputs require verification and must not be treated as authoritative evidence.

A mature monitoring programme should:

• begin with an asset, identity and threat model;
• prioritise high-value telemetry rather than collecting indiscriminately;
• maintain a documented detection catalogue;
• test detections using simulations and purple-team exercises;
• measure both detection performance and analyst workload;
• protect telemetry as sensitive information;
• establish model and automation governance;
• plan for sensor failure and adversary evasion; and
• retain human accountability for consequential decisions.

Chapter Summary

Security monitoring transforms technical observations into decisions about risk and response. SIEM platforms centralise and correlate events, while SOAR platforms coordinate enrichment and response. Network monitoring provides communication context; endpoint telemetry reveals processes, files and host behaviour. Effective detection engineering begins with explicit threat hypotheses, suitable telemetry, carefully segmented baselines and documented logic.

Statistical measures such as means, variance and standardised scores can identify unusual activity, but unusual does not mean malicious. AI-assisted methods can support classification, clustering and sequence analysis, yet they face drift, poor labels, imbalance, explainability problems and adversarial manipulation. Precision, recall and F1 help evaluate performance, but operational priorities and consequence must guide interpretation.

False positives and false negatives have organisational costs. Tuning should use evidence and precise exclusions. Privacy, proportionality, retention, access control, auditability and human review are essential because monitoring may affect individuals and expose highly sensitive information. The strongest security operations capability combines reliable telemetry, transparent analytics, controlled automation and accountable professional judgement.

Key Terms

Alert: A notification that activity may require security review.

Anomaly detection: Identification of observations that differ from an established reference pattern.

Baseline: A documented representation of expected behaviour for a defined population and period.

Correlation: Linking events using shared attributes, sequence or time relationships.

EDR: Endpoint detection and response technology that collects and analyses host activity.

Event: A recorded occurrence in a system or network.

False negative: Malicious activity incorrectly classified as benign or missed entirely.

False positive: Benign activity incorrectly classified as malicious.

Indicator: An observable artefact or pattern associated with potentially harmful activity.

Machine learning: Methods that infer patterns or decision functions from data.

NDR: Network detection and response technology that analyses network activity and metadata.

Precision: The proportion of raised alerts that are true positives.

Recall: The proportion of actual malicious cases that are detected.

SIEM: Security information and event management platform for collecting, analysing and managing security events.

SOAR: Security orchestration, automation and response platform for coordinating investigative and response actions.

Telemetry: Measurements and records generated by systems, applications, endpoints and networks.

Threat hypothesis: A testable statement about how harmful activity may appear in available data.

True positive: Malicious activity correctly identified by a detection.

Review Questions

Short Answer Questions

1. Distinguish an event, an alert and an incident.
2. Why is time synchronisation important in security investigations?
3. State two benefits and two limitations of endpoint telemetry.
4. Define precision and recall in the context of detection systems.
5. Why should automated response actions be bounded and logged?

Long Answer Questions

1. Explain how a SIEM normalises and correlates events from identity, endpoint and network sources.
2. Discuss how baselines should be designed for different users, devices and business processes.
3. Compare rule-based detection with statistical and machine-learning approaches.
4. Analyse the causes and consequences of false positives and false negatives.
5. Explain the privacy, governance and human-oversight requirements of AI-assisted monitoring.

Analytical / Application Questions

1. Design a detection use case for suspicious privileged-account activity. Specify the threat hypothesis, telemetry, correlation logic, exclusions, severity and validation method.
2. A detection produces 80 true positives, 20 false positives and 40 false negatives. Calculate its precision and recall, and explain which operational issue each metric highlights.
3. An organisation proposes automatically isolating every endpoint with a high anomaly score. Critically evaluate this proposal and design a safer graduated-response model.

Multiple Choice Questions with Answers

Question 1

Which capability is primarily associated with a SIEM?

A. Centralising, correlating and analysing security events

B. Replacing all endpoint operating systems

C. Physically repairing network cables

D. Encrypting every file without key management

Correct Answer: A. Centralising, correlating and analysing security events

Question 2

What does a false positive represent in security monitoring?

A. Malicious activity that was missed

B. Benign activity incorrectly treated as malicious

C. Malicious activity correctly detected

D. A log that was successfully archived

Correct Answer: B. Benign activity incorrectly treated as malicious

Question 3

Which statement about a baseline is most accurate?

A. It is a permanent universal definition of normality

B. It describes expected behaviour for a defined population and period

C. It proves that every deviation is an attack

D. It can be created without selecting data sources

Correct Answer: B. It describes expected behaviour for a defined population and period

Question 4

If a detector produces 90 true positives and 10 false positives, what is its precision?

A. 10%

B. 50%

C. 90%

D. 100%

Correct Answer: C. 90%

Question 5

Which metric measures the proportion of actual malicious cases that were detected?

A. Precision

B. Recall

C. Specificity only

D. Alert volume

Correct Answer: B. Recall

Question 6

Why can a large z-score not, by itself, prove that an event is malicious?

A. A z-score is always measured in bytes

B. Unusual behaviour may have a benign explanation

C. Standard deviation cannot be calculated from data

D. All malicious activity has a z-score of zero

Correct Answer: B. Unusual behaviour may have a benign explanation

Question 7

Which SOAR action generally requires the greatest caution?

A. Adding asset ownership to an alert

B. Searching a reputation service

C. Automatically isolating a critical production device

D. Recording an analyst comment

Correct Answer: C. Automatically isolating a critical production device

Question 8

Which is an example of concept drift?

A. A log field is renamed by a vendor

B. Normal organisational behaviour changes after a major business process change

C. An analyst opens an investigation case

D. A firewall rejects a packet

Correct Answer: B. Normal organisational behaviour changes after a major business process change

Question 9

What is a major advantage of correlating identity, endpoint and cloud telemetry?

A. It guarantees that no false positives occur

B. It provides context across different stages of activity

C. It removes the need for human review

D. It makes privacy obligations irrelevant

Correct Answer: B. It provides context across different stages of activity

Question 10

Which principle is most appropriate when designing monitoring data collection?

A. Collect every possible field indefinitely

B. Use data minimisation and retain information for a justified purpose

C. Allow all employees unrestricted access to raw logs

D. Delete all evidence immediately after an alert

Correct Answer: B. Use data minimisation and retain information for a justified purpose