Security Monitoring, Analytics and Threat Intelligence

Introduction

Security monitoring is the disciplined observation, collection and interpretation of events from information systems in order to identify malicious activity, policy violations, operational failure and emerging risk. Modern organisations generate telemetry from endpoints, networks, cloud services, applications, identity systems and security controls. The central challenge is not merely collecting data; it is converting heterogeneous, incomplete and sometimes contradictory observations into defensible decisions.

This chapter examines how security information and event management (SIEM), intrusion detection systems (IDS), endpoint detection and response (EDR), and threat-intelligence platforms work together. It also explains detection engineering, baselining, correlation, metrics, dashboards and the limitations of automated analysis.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Explain the purpose and relationships of SIEM, IDS, EDR and threat-intelligence platforms.
2. Collect, normalise and correlate logs, network telemetry and indicators of compromise (IOCs).
3. Develop detection use cases and distinguish indicators, behaviours and analytics.
4. Construct useful baselines, metrics and security dashboards.
5. Evaluate false positives, false negatives, data-quality problems and analytical limitations.
6. Apply governance, privacy and ethical principles to monitoring operations.

1. The Monitoring and Detection Ecosystem

A security monitoring capability has four logical activities:

• Collection: obtaining events and telemetry from relevant sources.
• Analysis: interpreting individual events and relationships among events.
• Decision: determining whether activity requires investigation or response.
• Action and learning: containing threats, documenting findings and improving controls.

A log is a recorded statement about an event, such as an authentication attempt. Telemetry is broader: it includes logs, network flows, packet metadata, process trees, file activity, DNS requests, identity changes and cloud audit records. An alert is an analytic conclusion that activity may require attention; it is not proof that an incident has occurred.

1.1 SIEM

A SIEM aggregates data from multiple sources, parses and normalises it, stores it for search and retention, correlates events, and produces alerts, investigations and reports. Its value comes from context. A failed login alone may be routine; a failed login followed by a successful login, privilege elevation and unusual data access is more significant.

A SIEM commonly includes:

• collectors or forwarders;
• parsing and normalisation pipelines;
• time synchronisation and event enrichment;
• rules, queries and statistical analytics;
• case management and investigation workflows;
• dashboards, reporting and retention controls.

SIEMs are not automatically intelligent. Poorly selected data, weak time synchronisation, excessive noise or badly designed rules can produce a high-volume system with little security value.

1.2 Intrusion Detection Systems

An IDS identifies suspicious activity in network or host data. A network IDS (NIDS) analyses traffic at a sensor location, while a host IDS (HIDS) monitors activity on an individual system. Detection approaches include:

• Signature detection: matches known byte patterns, domains, hashes or protocol features. It is efficient for known threats but weak against novel or modified attacks.
• Anomaly detection: compares activity with an expected pattern. It can identify previously unseen behaviour but may generate many false positives.
• Stateful protocol analysis: checks whether protocol behaviour violates expected standards or state transitions.
• Behavioural detection: identifies sequences such as scripting-engine execution followed by credential access and outbound communication.

An IDS often detects; it may not block. Prevention systems add enforcement capability, but blocking requires confidence because erroneous prevention can disrupt legitimate operations.

1.3 Endpoint Detection and Response

EDR agents collect endpoint telemetry such as process creation, command lines, parent-child relationships, file modifications, registry changes, network connections and user context. EDR supports retrospective investigation because analysts can reconstruct process activity and search across endpoints. It may also isolate a host, terminate a process or quarantine a file.

EDR visibility depends on agent health, operating-system support, sensor configuration and the attacker’s ability to tamper with or evade the agent. An endpoint view is powerful but incomplete without identity, network and cloud context.

1.4 Threat-Intelligence Platforms

A threat-intelligence platform (TIP) stores, enriches, evaluates and distributes intelligence. Intelligence may include:

• Tactical intelligence: hashes, IP addresses, domains, URLs and malware artefacts.
• Operational intelligence: campaigns, techniques, targeting and infrastructure relationships.
• Strategic intelligence: trends, sector risks, geopolitical developments and business implications.

An IOC is evidence associated with a potentially malicious activity. An IOC is not necessarily malicious in every context: shared hosting, recycled domains and compromised legitimate services can create ambiguity. Intelligence must therefore include source, confidence, first-seen and last-seen times, context, expected lifetime and handling restrictions.

2. Data Collection, Normalisation and Correlation

2.1 Important Data Sources

A balanced collection strategy may include:

Collection should be risk-based. Retaining every possible event may increase cost and analyst workload without improving detection. Critical systems, privileged identities and high-value data require stronger coverage than low-risk assets.

2.2 Normalisation and Enrichment

Different products represent the same concept differently. One source may use src_ip, another sourceAddress. Normalisation maps fields into a common schema. Useful common fields include timestamp, source and destination addresses, source and destination ports, user, host, action, outcome, process, rule identifier and event category.

Enrichment adds meaning, such as asset criticality, user department, geolocation, vulnerability status, business owner, threat-intelligence reputation and identity risk. Enrichment must be time-aware: an IP address reputation or asset owner can change.

Time is fundamental. Systems should use synchronised clocks, usually through an approved time service. Analysts must distinguish event time from ingestion time because delays can distort sequences.

2.3 Correlation

Correlation links events using shared attributes and temporal or behavioural relationships. Common correlation keys are username, host, session, process identifier, IP address, domain, cloud account and transaction identifier.

A simple sequence might be:

1. A user authenticates from an unfamiliar location.
2. MFA is reset or bypassed.
3. The account creates a new access key.
4. A large volume of cloud data is read.
5. Data is transferred to an unfamiliar destination.

No single event proves compromise. Their combination produces a stronger investigative hypothesis.

Correlation can be rule-based, query-based, statistical or graph-based. Rules are explainable and operationally predictable. Statistical methods can identify deviations but require stable data and careful calibration. Graph analysis can reveal relationships among accounts, hosts, domains and processes, but it introduces complexity and does not remove the need for investigation.

3. Detection Engineering and Use Cases

Detection engineering treats detections as maintained technical products rather than one-time queries. Each use case should document:

• threat behaviour and its business relevance;
• data sources and required fields;
• detection logic and time window;
• severity and confidence;
• expected false positives;
• triage questions and response actions;
• owner, test cases and review date;
• dependencies and known blind spots.

3.1 Indicators, Tactics and Behaviours

An indicator may be precise but short-lived, such as a malware hash. A behaviour, such as suspicious credential dumping, may survive minor changes to tools but can be difficult to observe reliably. Mature programmes combine IOC matching with behaviour and context.

Threat frameworks such as MITRE ATT&CK provide a vocabulary for adversary tactics and techniques. They support coverage mapping and communication, but framework mapping does not demonstrate that an organisation can detect a technique. Coverage claims must be supported by tested data and measurable performance.

3.2 Example Use Case

Use case: suspicious PowerShell execution.

A detection may identify a scripting process launched by an office application, using encoded or hidden commands, followed by an outbound connection. The logic should consider the parent process, signed status, user role, destination reputation and whether the host is an approved administrative workstation. A server-management script may be legitimate; the same behaviour from a finance laptop deserves greater scrutiny.

A robust detection should not depend on one keyword. It should combine process lineage, command-line features, user and asset context, and a time relationship. It should also include exclusions that are narrow, documented and reviewed rather than a broad exception for an entire department.

3.3 Detection Testing

Testing should include known malicious simulations, benign administrative activity, boundary cases and missing-data conditions. Purple-team exercises can validate whether telemetry is collected, whether the rule fires, whether the alert is understandable and whether response procedures work. A detection that fires correctly in a laboratory but fails when an endpoint sensor is offline is not operationally complete.

4. Baselines and Security Analytics

A baseline describes expected activity over a defined period and population. Examples include normal login times, typical DNS volume, usual data-transfer rates, common administrative tools and ordinary process relationships.

A baseline must specify scope, time period, population and treatment of exceptional events. A single average can be misleading. Organisations should consider distributions, seasonality, working patterns and legitimate operational changes.

For a measured value x and a reference mean μ with standard deviation σ, a standardised score is:

Here, z is dimensionless; x and μ have the same units; and σ has those same units. This score is most interpretable when observations are reasonably stable and the mean and standard deviation are not distorted by attacks or extreme outliers. A high score is an investigation signal, not proof of maliciousness. For skewed data, robust measures such as the median and interquartile range may be more appropriate.

Worked Example: Login Volume

Suppose a service account produces a daily mean of 120 authentication events with a standard deviation of 20. On one day it produces 190 events:

z = (190 − 120) / 20 = 3.5

The observation is 3.5 standard deviations above the reference mean. An analyst should examine scheduled jobs, deployment activity, credential misuse and whether the baseline included previous incidents. The result does not establish compromise.

Baselines can be static, rolling or peer-group based. Rolling baselines adapt to change but can learn malicious behaviour. Peer-group baselines compare similar users, hosts or services and may reduce false positives, but poor grouping can conceal abnormal activity.

5. Metrics, Alert Quality and Dashboards

Metrics should support decisions rather than reward superficial activity. Useful measures include:

• Mean time to detect (MTTD): average time between the relevant malicious event and detection.
• Mean time to respond (MTTR): a defined response-duration measure; organisations must state whether it means triage, containment or closure.
• alert volume by source, severity and use case;
• percentage of alerts triaged within service targets;
• detection coverage for priority techniques and assets;
• sensor health and telemetry delay;
• false-positive rate and repeat-alert rate;
• cases escalated, contained and closed with evidence.

For a set of alerts, precision is:

Recall is:

Here, TP means true positives, FP false positives and FN false negatives. Precision describes the proportion of investigated alerts that are confirmed relevant under the chosen definition; recall describes the proportion of known relevant events detected. Both require a defensible ground truth. Incident datasets are rarely complete, so reported values can be estimates rather than objective properties of a detector.

A dashboard should answer operational questions: What requires action now? Which critical assets lack telemetry? Which detections are degrading? Where are response delays occurring? Effective dashboards use clear ownership, time windows, filters, severity definitions and drill-down paths. Attractive charts without decisions or data-quality indicators are not effective security management.

6. An Illustrative Case Study: Cloud Account Compromise

Background

A fictional online retailer operates a cloud environment with central identity, endpoint monitoring and a SIEM. Its security team receives identity, cloud audit, DNS, proxy and EDR data.

Problem

The finance director’s account generates alerts for an unfamiliar login and a new cloud access key. The first alert is low confidence because the user travels frequently.

Analysis

The team correlates the events and finds that the login was followed by an MFA-method change, assignment to a privileged group, enumeration of storage buckets and an unusual download volume. DNS and proxy records show connections to a newly observed external domain. The endpoint associated with the session shows no corresponding travel or approved administrative activity.

The analysis uses several forms of evidence: identity context, cloud audit events, data-access volume, network destinations and asset information. The unfamiliar login alone would be weak evidence; the sequence is materially stronger.

Approach and Solution

The team disables active sessions, removes the unauthorised access key, restores approved MFA settings, restricts the account temporarily and preserves relevant logs. It investigates whether downloaded data contained regulated information, searches for the same destination across tenants, reviews other accounts and checks for persistence mechanisms. A new detection is developed for privileged-group changes followed by high-volume storage access, with exclusions for documented migration accounts.

Outcome

The organisation identifies unauthorised access to a limited dataset, confirms that broader administrative control was not obtained, and reports the event according to its legal and contractual obligations. It improves cloud audit coverage, shortens log-ingestion delay and introduces a review process for high-risk identity changes.

Learning Points

1. Context and sequence are more informative than isolated indicators.
2. A low-confidence alert can become significant after enrichment.
3. Containment must preserve evidence and respect legal requirements.
4. Use cases should be improved from incidents, not merely added to a catalogue.

7. Limitations, Governance and Ethics

7.1 Analytical Limitations

Monitoring systems face several unavoidable limitations:

• Visibility gaps: encrypted traffic, unmanaged devices, unsupported applications and missing cloud logs reduce observability.
• Adversarial adaptation: attackers alter infrastructure, timing, tools and behaviour in response to detections.
• Data-quality defects: duplicate events, parsing errors, clock drift, dropped telemetry and inconsistent identities corrupt analysis.
• Concept drift: normal business activity changes after mergers, remote-work shifts, software deployments or seasonal demand.
• Alert fatigue: excessive low-value alerts cause delayed triage and human error.
• Attribution uncertainty: an IOC may identify infrastructure without proving who operated it.
• Automation risk: machine-learning models can encode bias, be manipulated by poisoning or produce explanations too weak for high-impact decisions.

Absence of an alert is not evidence of absence of compromise. Detection performance is conditional on the data, threat, environment and time period.

7.2 Privacy and Governance

Monitoring may process personal data, communications metadata and employee activity. Organisations should define a lawful purpose, collect proportionately, restrict access, protect data at rest and in transit, establish retention periods, document legitimate use and provide appropriate oversight. Access to sensitive investigations should be logged and reviewed.

Governance should define who may authorise monitoring, who may access raw data, how evidence is preserved, how cases are escalated and when legal, privacy or regulatory specialists must be consulted. Cross-border data transfer and third-party service processing require particular attention.

Analysts must avoid treating unusual nationality, disability, protected characteristic or personal circumstance as a threat indicator. Risk should be based on relevant technical and behavioural evidence, not discriminatory proxies.

8. Contemporary Practice and Best Practices

Current monitoring programmes increasingly combine identity telemetry, cloud-native logs, endpoint data, network detection and response, attack-surface information and automated enrichment. Security orchestration and automation can enrich an alert, open a case, revoke a token or isolate a host. High-impact actions should use approval gates, rollback procedures and audit trails.

Artificial intelligence can assist with summarisation, clustering, anomaly discovery and query generation. It should not be treated as an authoritative investigator. Outputs require source verification, protection against sensitive-data leakage, testing for bias, monitoring for drift and clear human accountability.

A practical improvement programme is to:

1. identify critical business services and realistic threats;
2. map required telemetry and test its availability;
3. prioritise a small number of high-value detection use cases;
4. normalise and enrich data consistently;
5. measure alert quality, coverage and response outcomes;
6. test detections with red-team, purple-team or simulation activities;
7. review exceptions, baselines and data retention regularly;
8. feed incident lessons into engineering and governance.

Chapter Summary

Security monitoring converts system observations into decisions about risk and response. SIEM provides aggregation, correlation and investigation; IDS identifies suspicious network or host activity; EDR supplies detailed endpoint evidence and response capabilities; and threat-intelligence platforms organise indicators and contextual intelligence. Effective detection combines indicators, behaviours, identity, asset and temporal context.

Reliable operations depend on normalisation, time accuracy, enrichment, tested use cases and explicit assumptions. Baselines and statistical scores can prioritise investigation but do not prove maliciousness. Metrics such as precision, recall, detection and response time are meaningful only when definitions and ground truth are clear. Dashboards should support decisions, not merely display activity.

Finally, monitoring is limited by incomplete visibility, changing environments, adversarial behaviour, data-quality defects and human judgement. Privacy, proportionality, access control, evidence handling and accountable use of automation are essential components of professional security analytics.

Key Terms

SIEM: A platform that collects, normalises, correlates and presents security events.

IDS: A system that detects suspicious activity in network or host data.

EDR: Endpoint technology that records endpoint activity and supports investigation and response.

Threat intelligence: Contextual information about threats, adversaries, capabilities and infrastructure.

IOC: An artefact associated with potentially malicious activity.

Telemetry: Observational data generated by systems, networks, applications or security tools.

Correlation: Linking events using shared attributes, time or behavioural relationships.

Normalisation: Mapping heterogeneous data into a consistent representation.

Enrichment: Adding contextual information to an event.

Detection use case: A documented analytic purpose, logic, data requirement and response workflow.

Baseline: A description of expected activity for a defined population and period.

False positive: An alert classified as relevant when the suspected condition is not present.

False negative: A relevant event that the detection fails to identify.

Precision: The proportion of flagged results that are confirmed relevant.

Recall: The proportion of known relevant events that are detected.

Alert fatigue: Reduced analyst effectiveness caused by excessive or repetitive alerts.

Concept drift: Change in data or behaviour that makes an established model less reliable.

Threat hunting: Proactive investigation for evidence of malicious activity.

Review Questions

Short Answer Questions

1. What are the four logical activities in a security monitoring capability?
2. Distinguish an IOC from behavioural detection.
3. Why are normalisation and time synchronisation important?
4. What is the purpose of enrichment in a SIEM?
5. Why does an alert not prove that an incident has occurred?

Long Answer Questions

1. Compare SIEM, IDS, EDR and threat-intelligence platforms, explaining how they complement one another.
2. Describe a professional process for developing, testing and maintaining a detection use case.
3. Explain how baselines can support anomaly detection and discuss their limitations.
4. Discuss the meaning and limitations of precision, recall, MTTD and MTTR in security operations.
5. Evaluate the privacy, governance and ethical issues associated with employee and customer monitoring.

Analytical / Application Questions

1. Design a correlation sequence for detecting compromise of a privileged cloud account. Identify data sources, enrichment fields and response actions.
2. A detection has high alert volume but few confirmed incidents. Analyse possible causes and propose a measurement-led improvement plan.
3. An organisation’s baseline learns a major attack as normal activity. Explain how this can happen and design controls to reduce the risk.

Multiple Choice Questions with Answers

Question 1

Which capability primarily provides detailed process and endpoint activity for investigation?

A. EDR

B. DNS

C. A vulnerability scanner

D. A network cable monitor

Correct Answer: A. EDR

Question 2

What is the main purpose of normalisation in a SIEM?

A. To delete all duplicate users

B. To map different event formats into a consistent representation

C. To block every external connection

D. To encrypt all endpoint files

Correct Answer: B. To map different event formats into a consistent representation

Question 3

Which statement best describes an IOC?

A. A guaranteed attribution of an attack to a person

B. An artefact associated with potentially malicious activity

C. A legal authorisation to monitor employees

D. A measure of analyst productivity

Correct Answer: B. An artefact associated with potentially malicious activity

Question 4

A login count has mean 120 and standard deviation 20. What is the z-score for 190 logins?

A. 2.5

B. 3.0

C. 3.5

D. 4.0

Correct Answer: C. 3.5

Question 5

Which event sequence gives the strongest evidence of possible cloud-account compromise?

A. One failed login

B. A routine successful login from a known device

C. An unfamiliar login followed by MFA change, privilege elevation and unusual data access

D. A scheduled backup completing successfully

Correct Answer: C. An unfamiliar login followed by MFA change, privilege elevation and unusual data access

Question 6

In the formula Precision = TP ÷ (TP + FP), what does FP represent?

A. Failed processes

B. False positives

C. Forwarding paths

D. Firewall policies

Correct Answer: B. False positives

Question 7

Why can signature-based detection miss a modified or previously unknown attack?

A. It depends on matching known patterns or artefacts

B. It cannot process timestamps

C. It always requires endpoint isolation

D. It ignores all network traffic

Correct Answer: A. It depends on matching known patterns or artefacts

Question 8

Which dashboard measure most directly indicates a telemetry visibility problem?

A. Number of screen colours

B. Sensor health and ingestion delay

C. Number of analyst emails

D. Average password length

Correct Answer: B. Sensor health and ingestion delay

Question 9

Why should a high baseline z-score not automatically be treated as proof of malicious activity?

A. A deviation may result from legitimate activity or a distorted baseline

B. z-scores are always qualitative

C. The score has units of seconds

D. Baselines cannot use numerical values

Correct Answer: A. A deviation may result from legitimate activity or a distorted baseline

Question 10

Which is an appropriate governance practice for security monitoring?

A. Allow unrestricted access to raw employee data

B. Retain all data forever

C. Define purpose, access controls, retention and oversight

D. Use protected characteristics as threat indicators

Correct Answer: C. Define purpose, access controls, retention and oversight