Security Operations, Threat Intelligence and Analytics

Introduction

Security operations is the disciplined practice of continuously monitoring, detecting, investigating and responding to events that may threaten the confidentiality, integrity or availability of information systems. Modern operations centres must process data from endpoints, identity systems, networks, cloud services, applications and business platforms. The challenge is not simply collecting more data. It is converting heterogeneous observations into trustworthy decisions while controlling false alarms, privacy risks, bias and automation failures.

This chapter examines the operational security lifecycle, log management, baselining, SIEM, SOAR, EDR, threat intelligence, detection engineering, alert prioritisation and security analytics. It also considers how artificial intelligence can support analysts without replacing governance, human judgement or accountability.

Learning Objectives

After studying this chapter, a learner should be able to:

1. Explain the role and architecture of a security operations capability.
2. Design a practical log collection, normalisation and retention strategy.
3. Establish behavioural baselines and identify meaningful deviations.
4. Distinguish the functions of SIEM, SOAR, EDR and threat-intelligence platforms.
5. Develop, test and maintain detection rules.
6. Prioritise alerts using evidence, asset criticality and likely impact.
7. Apply security analytics and AI while managing privacy, bias and model risks.
8. Evaluate operational performance using appropriate metrics and governance.

1. The Security Operations Function

A security operations centre (SOC) coordinates people, processes and technologies to identify and manage cyber risk. Its work is cyclical rather than linear:

• Prepare: define assets, logging requirements, playbooks, roles and escalation paths.
• Monitor: collect and inspect telemetry continuously or at scheduled intervals.
• Detect: identify suspicious patterns using rules, statistical methods or analyst judgement.
• Analyse: establish scope, timeline, affected identities and probable cause.
• Respond: contain, eradicate and recover from an incident.
• Learn: improve controls, detections, documentation and user awareness.

A mature SOC integrates technical investigation with business context. A failed login on a public test system is not equivalent to a successful privileged login to a payment database from an unfamiliar country. Effective operations therefore require an asset inventory, identity context, vulnerability information, ownership data and an agreed incident classification scheme.

A useful division of responsibility is:

2. Telemetry and Log Management

A log is a time-stamped record of an event. Telemetry is broader: it includes logs, process activity, network flows, endpoint observations, identity events, configuration states and sensor measurements. Useful telemetry should be sufficiently complete, accurate, timely and protected from unauthorised alteration.

2.1 Important data sources

Common sources include:

• authentication and identity-provider logs;
• firewalls, proxies, VPNs, DNS and network-flow systems;
• operating-system security logs;
• endpoint detection and response agents;
• cloud control-plane and storage-access logs;
• email security gateways;
• web servers, databases and business applications;
• vulnerability scanners and configuration-management systems.

Collection should be risk-based. High-value assets, privileged identities and internet-facing services usually deserve more detailed and longer-retained telemetry than low-risk endpoints. The organisation should document the purpose of each source, its owner, fields collected, sensitivity, retention period and expected ingestion rate.

2.2 Log quality and normalisation

Before correlation, events should be normalised into consistent fields such as timestamp, source, destination, user, asset, action, result, process, event type and confidence. Time synchronisation is essential. If systems use inconsistent clocks, an investigator may construct an incorrect attack sequence. Network Time Protocol should be monitored, and the chosen time zone and timestamp format should be documented.

A robust pipeline commonly contains collection agents, transport, parsing, enrichment, storage and search. Enrichment may add asset criticality, geographic information, identity risk, vulnerability status and known-indicator matches. Original evidence should be retained where feasible so that parser errors do not destroy forensic value.

Logs can contain personal data, credentials in error messages, document names, IP addresses or employee activity patterns. Collection must therefore follow data minimisation, purpose limitation, access control, encryption and retention requirements. Security teams should not treat unrestricted visibility as automatically legitimate visibility.

2.3 Availability, integrity and retention

Operational logs should be protected against deletion and tampering. Controls include write-restricted storage, cryptographic integrity mechanisms, separated administrative roles, monitored retention policies and tested restoration. Retention should reflect legal duties, investigation needs, storage cost and the time required to discover compromise. A short retention period may make an investigation impossible; indefinite retention may create unnecessary privacy and legal exposure.

3. Baselines and Behavioural Deviation

A baseline describes expected activity for a defined entity and context. Examples include normal login hours for a service account, usual data-transfer volume for an application, or the ordinary parent process of a workstation application. Baselines must account for seasonality, maintenance windows, shifts, holidays and business change.

A simple statistical baseline for observations x₁, x₂, …, x_n is the arithmetic mean:

Here, μ is the sample mean, x_i is observation i and n is the number of observations. The observations must measure the same quantity in comparable units. The mean is sensitive to extreme values, so a median or trimmed mean may be more appropriate for highly skewed security data.

The sample standard deviation is:

The denominator n−1 is used when estimating population variability from a sample. A z-score for a new observation x is:

The z-score is dimensionless. It is informative when the baseline is reasonably stable and approximately suitable for mean-and-variance interpretation. It should not be treated as proof of malicious activity. A legitimate monthly backup may be statistically unusual but expected.

For streaming data, an exponentially weighted moving average can adapt gradually:

where x_t is the current observation, m_t is the updated baseline and 0 < α ≤ 1 controls responsiveness. A large α reacts quickly but may absorb an attack into the baseline; a small value is stable but slower to reflect legitimate change.

Baselines should be segmented by user, asset role, geography, application and time period where justified. Overly broad baselines hide anomalies; overly narrow baselines generate noise.

4. SIEM, SOAR and EDR

4.1 Security information and event management

A SIEM platform aggregates, normalises, indexes and correlates security events. Its capabilities often include dashboards, search, rule-based detection, case management, alerting and reporting. Correlation can combine events that are individually weak. For example, an unfamiliar login, a new authentication token and a large download may collectively indicate account compromise.

SIEM does not automatically guarantee detection. Its performance depends on source coverage, field quality, time accuracy, parsing, rule design and analyst workflow. Storage and ingestion costs also require careful architecture, such as tiered hot, warm and archival storage.

4.2 Security orchestration, automation and response

SOAR connects tools and automates repeatable workflows. A playbook might receive an alert, enrich an IP address, check asset ownership, request analyst approval, disable a token and open an incident record. Automation should be graduated:

1. Observe: collect and enrich information.
2. Recommend: present a proposed action.
3. Approve: require human authorisation for consequential actions.
4. Contain automatically: perform narrowly scoped, reversible actions.
5. Remediate automatically: use only where confidence, testing and rollback are strong.

A poorly designed playbook can amplify an erroneous detection, disable legitimate accounts or create an outage. Every high-impact action needs authorization boundaries, audit logs, failure handling and rollback.

4.3 Endpoint detection and response

EDR records endpoint processes, command lines, files, network connections, persistence mechanisms and security events. It supports investigation through process trees and timelines, and may provide isolation, process termination or file quarantine. EDR is especially valuable when network visibility is limited, but it introduces agent deployment, performance, privacy and evasion considerations. Endpoint evidence should be correlated with identity, cloud and network data rather than interpreted alone.

5. Threat Intelligence

Threat intelligence is analysed information about threats that supports a decision. An isolated indicator is not necessarily intelligence. Useful intelligence explains who or what may be involved, how activity occurs, which assets are at risk, how reliable the information is and what action is recommended.

Intelligence is often classified as:

• Strategic: trends, campaigns and business risk for leadership.
• Operational: adversary objectives, targeting and campaign activity.
• Tactical: techniques, procedures and defensive priorities.
• Technical: indicators such as hashes, domains, addresses and file paths.

The intelligence lifecycle includes direction, collection, processing, analysis, dissemination and feedback. Indicators have limited value without context. A domain may be shared hosting, a hash may represent a repackaged legitimate file, and an IP address may be reassigned. Feeds should therefore be scored for source reliability, indicator confidence, age, relevance and observed behaviour.

Structured frameworks such as MITRE ATT&CK can map observations to adversary tactics and techniques. This supports coverage analysis and threat hunting, but a framework mapping is an analytical hypothesis, not conclusive attribution.

6. Detection Engineering

Detection engineering treats detections as maintained technical products. A detection should specify its purpose, data dependencies, logic, expected false positives, severity, owner, test cases, response guidance and retirement conditions.

A sound development process is:

1. Identify a threat scenario and required outcome.
2. Describe observable behaviours rather than relying only on fragile indicators.
3. Confirm telemetry availability and field meaning.
4. Write and test the rule against known benign and malicious examples.
5. Estimate alert volume and operational cost.
6. Deploy in a monitored or low-impact mode.
7. Tune exclusions using documented business evidence.
8. Measure performance and review it after environmental changes.

Detections may be signature-based, threshold-based, sequence-based, behaviour-based or statistical. A rule for repeated failed logins may be useful, but a sequence involving a successful login, privilege change and unusual data access may be more informative.

Detection quality can be described using a confusion matrix:

Precision, also called positive predictive value, is:

Recall, also called sensitivity, is:

Here, TP, FP and FN denote true positives, false positives and false negatives. Precision answers “How many alerts were useful?”; recall answers “How much of the relevant malicious activity was detected?” These measures require a trustworthy labelled set. In security, undetected attacks are difficult to count, so reported recall may be an estimate rather than a complete truth.

A rule with high recall but overwhelming false positives may be operationally ineffective. Conversely, a highly precise rule that misses most attacks may create dangerous confidence. Detection selection should reflect asset criticality, response capacity and threat consequences.

7. Alert Triage and Risk-Based Prioritisation

Alert severity should not be based solely on the detection name. A practical triage decision considers:

• confidence that the activity is malicious;
• criticality and exposure of the affected asset;
• privilege and sensitivity of the identity;
• evidence of successful execution or persistence;
• scope, propagation and business impact;
• containment urgency;
• quality and recency of supporting evidence.

A conceptual risk model is:

This is a decision model, not a universal physical law. Likelihood and impact must use defined scales, and the result should not be presented as objectively precise when inputs are subjective. For example, an apparently medium-confidence alert involving a domain administrator on a payment server may outrank a high-confidence malware alert on an isolated laboratory workstation because potential impact and urgency differ.

Analysts should enrich before closing. They should record the hypothesis, evidence examined, decision, confidence, affected scope and recommended follow-up. Closure codes should distinguish false positive, benign true positive, duplicate, policy violation, confirmed incident and insufficient evidence.

8. Security Analytics and AI

Security analytics includes search, aggregation, correlation, anomaly detection, graph analysis, sequence analysis and supervised classification. Machine learning can assist with clustering unusual behaviour, ranking alerts, identifying related entities and summarising investigations. It should augment, not obscure, the evidential basis of a decision.

Important risks include:

• Data drift: normal activity changes after migration, restructuring or new work patterns.
• Concept drift: the relationship between features and threats changes as attackers adapt.
• Training contamination: malicious or incorrect data influences the model.
• Class imbalance: confirmed attacks are much rarer than benign events.
• Adversarial manipulation: attackers deliberately evade or poison analytics.
• Opacity: analysts cannot understand why a score was produced.
• Automation bias: humans accept model recommendations without challenge.

A model must have a defined purpose, population, decision threshold, owner and review schedule. Performance should be assessed on temporally separated data, not only randomly divided historical data. Evaluation should include precision, recall, alert volume, analyst workload, latency, subgroup performance and the consequences of errors.

Fairness is relevant even in cybersecurity. A model that assigns higher suspicion to a geographic region, language group, work pattern or contractor population may reproduce organisational bias or create discriminatory surveillance. Features should have a legitimate security rationale, sensitive attributes should be governed carefully, and outcomes should be tested across relevant groups.

Privacy-preserving practices include collecting only necessary features, pseudonymising data where operationally possible, restricting access, documenting automated decisions, applying retention limits and separating security investigation from employee performance monitoring. Legal review is necessary when monitoring crosses jurisdictions or processes personal data.

9. Governance, Metrics and Assurance

Operational metrics should measure outcomes rather than celebrate activity. Useful measures include mean time to detect, mean time to contain, mean time to recover, alert-to-case conversion, false-positive rate, detection coverage, telemetry availability, percentage of critical assets monitored and playbook success rate.

Mean time to detect can be defined as:

where t_detected,i and t_occurred,i are detection and occurrence times for incident i, and n is the number of incidents included. The metric is meaningful only when occurrence time is estimated consistently. Median and percentile values should often accompany the mean because incident times can be highly skewed.

Governance should define authority to isolate systems, disclose incidents, preserve evidence and accept residual risk. Detection content, parsers, playbooks and models require version control, peer review, change management and documented testing. Exercises should test not only technical controls but also communications, legal decisions, supplier dependencies and recovery.

10. Illustrative Case: Unusual Cloud Data Access

Background

A fictional online retailer operates cloud applications, a central identity provider and a SOC using SIEM, SOAR and EDR. Customer records are classified as highly sensitive. Analysts have observed occasional credential-phishing attempts but have no confirmed breach.

Problem

The SIEM generates an alert for a privileged administrator account: a successful login from a new device, followed by creation of an access token and an unusual download from a customer-data store. The account normally operates during local business hours from managed devices.

Analysis

The analyst validates time synchronisation and correlates identity, cloud, endpoint and network events. The login used multifactor authentication, but the second factor was approved after an unusual sequence of failed attempts. The device is unmanaged, the token was created shortly after authentication, and the download volume is several standard deviations above the account’s recent baseline. The statistical anomaly alone is not proof; the sequence and asset sensitivity provide stronger context.

Approach and solution

The SOC assigns high priority because confidence, privilege, data sensitivity and potential impact are all high. A SOAR playbook gathers evidence and requests approval to revoke the token, suspend the account and isolate the associated endpoint. The incident manager authorises those reversible containment actions. Cloud audit logs are preserved, access keys are rotated, and legal and privacy officers are notified according to the incident plan. Threat-intelligence analysts compare infrastructure indicators with current campaigns, while detection engineers create a rule for the sequence of unusual authentication, token creation and sensitive-data access.

Outcome

Investigation shows that the administrator’s password was obtained through a phishing site. The attacker passed multifactor authentication through social engineering but did not complete bulk exfiltration before containment. The organisation resets credentials, strengthens phishing-resistant authentication for privileged users and improves monitoring of token creation.

Learning points

The case demonstrates why single indicators are weak, why identity and cloud telemetry must be correlated, why asset context affects prioritisation, and why automation should be constrained by approval and rollback. It also shows that a baseline is an investigative aid, not a verdict.

11. Contemporary Developments and Best Practice

Security operations increasingly address cloud-native workloads, software supply chains, identity attacks, container environments, operational technology and deepfake-enabled social engineering. Extended detection and response platforms attempt to unify endpoint, identity, email, network and cloud evidence. Generative AI can summarise cases or translate analyst queries, but generated content may be inaccurate, disclose sensitive information or invent supporting evidence.

Recommended practice includes:

1. Start with critical business services and threat scenarios rather than purchasing tools first.
2. Maintain an authoritative asset and identity inventory.
3. Measure telemetry quality, not merely ingestion volume.
4. Normalise timestamps and preserve original evidence.
5. Build detections from behaviours and sequences where possible.
6. Test rules and playbooks against benign and malicious cases.
7. Use confidence, impact and urgency to prioritise alerts.
8. Require human control for destructive or difficult-to-reverse actions.
9. Monitor model drift, subgroup outcomes and analyst overreliance.
10. Apply privacy, retention, access-control and audit requirements throughout the lifecycle.
11. Review detections, intelligence feeds and automation after major system changes.
12. Conduct exercises and feed lessons into engineering and governance.

Chapter Summary

Security operations converts system observations into risk-informed action. Effective capability depends on reliable telemetry, time synchronisation, normalisation, protected retention and contextual enrichment. Baselines help identify deviation, but statistical unusualness is not equivalent to maliciousness. SIEM provides aggregation and correlation; EDR supplies detailed endpoint visibility; SOAR coordinates repeatable workflows; and threat intelligence adds context about adversaries and campaigns.

Detection engineering requires explicit data dependencies, testing, ownership and continuous tuning. Precision and recall describe different aspects of detection performance and must be interpreted with awareness of incomplete labels and operational cost. Alert prioritisation should combine confidence, asset criticality, identity privilege, impact and urgency. Analytics and AI can improve scale and consistency, but drift, bias, privacy loss, adversarial manipulation, opacity and automation bias require governance. The strongest SOCs combine technology with skilled analysts, documented decisions, controlled automation, measurable outcomes and continual learning.

Key Terms

Security operations centre: Organisational function that monitors, investigates and responds to cyber threats.

Telemetry: Observations collected from systems, networks, identities and applications.

Log normalisation: Conversion of varied event formats into consistent fields and meanings.

Security information and event management (SIEM): Platform for collecting, correlating and analysing security events.

Security orchestration, automation and response (SOAR): Technology and processes for coordinating and automating response workflows.

Endpoint detection and response (EDR): Endpoint telemetry and response capability focused on process, file and connection activity.

Baseline: A model of expected activity for a defined entity and context.

Anomaly: An observation that differs from an established expectation.

Threat intelligence: Analysed threat information that supports a security decision.

Indicator: Observable technical information associated with possible malicious activity.

Detection rule: Logic that identifies an event or sequence requiring investigation.

False positive: An alert for activity that is not the condition of interest.

False negative: Relevant malicious activity that does not generate an alert.

Precision: Proportion of generated alerts that are true positives.

Recall: Proportion of relevant positive conditions that are detected.

Playbook: Documented procedure for investigating or responding to a condition.

Data drift: Change in the statistical properties of input data over time.

Automation bias: Tendency to accept automated recommendations without adequate scrutiny.

Asset criticality: Importance of an asset to organisational operations or obligations.

Mean time to detect: Average elapsed time between occurrence and detection for included incidents.

Review Questions

5 Short Answer Questions

1. What are the principal functions of a security operations centre?
2. Why is time synchronisation important in log analysis?
3. Distinguish SIEM, SOAR and EDR.
4. What is the difference between precision and recall?
5. Identify two privacy risks associated with security telemetry.

5 Long Answer Questions

1. Explain how an organisation should design a risk-based log collection and retention strategy.
2. Discuss the strengths and limitations of statistical baselines in security monitoring.
3. Describe a complete detection-engineering lifecycle, including testing and maintenance.
4. Explain how threat intelligence can improve detection and incident response.
5. Evaluate the governance controls required before deploying automated SOAR actions.

3 Analytical / Application Questions

1. A privileged user logs in from an unmanaged device, creates a token and downloads sensitive data at an unusual rate. Explain how you would investigate, prioritise and contain the alert.
2. A machine-learning detector has high recall but generates thousands of daily false positives. Analyse possible causes and propose an improvement plan.
3. Design a monitoring and evaluation plan for a new AI-based alert-ranking model, including privacy, drift, bias, human oversight and performance measures.

Multiple Choice Questions with Answers

Question 1

Which function is primarily associated with a SIEM platform?

A. Physical destruction of compromised hardware

B. Collection, correlation and analysis of security events

C. Replacement of all endpoint operating systems

D. Negotiation of cyber-insurance contracts

Correct Answer: B. Collection, correlation and analysis of security events

Question 2

What is the main limitation of treating a high z-score as proof of an attack?

A. A z-score has units of bytes

B. A statistically unusual event may be legitimate

C. Z-scores can only be calculated for passwords

D. A z-score always produces a false negative

Correct Answer: B. A statistically unusual event may be legitimate

Question 3

Which action is most appropriate for a high-impact SOAR playbook?

A. Execute irreversible changes without logging

B. Disable every account mentioned in an alert

C. Require defined authority, auditability and rollback where feasible

D. Ignore analyst approval in all circumstances

Correct Answer: C. Require defined authority, auditability and rollback where feasible

Question 4

An alert system produces 80 true positives and 20 false positives. What is its precision?

A. 20%

B. 25%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 5

Which statement best describes threat intelligence?

A. Any unverified IP address is intelligence

B. Analysed threat information that supports a decision

C. A replacement for incident response

D. A list that never becomes outdated

Correct Answer: B. Analysed threat information that supports a decision

Question 6

Why should detection rules be tested against benign examples?

A. To increase storage costs

B. To identify and reduce avoidable false positives

C. To remove all human review

D. To guarantee that attackers cannot evade them

Correct Answer: B. To identify and reduce avoidable false positives

Question 7

Which factor should increase the priority of an otherwise similar alert?

A. The affected asset is a critical payment database

B. The alert has a shorter title

C. The event occurred on a weekday

D. The log uses JSON formatting

Correct Answer: A. The affected asset is a critical payment database

Question 8

What is automation bias?

A. A method for encrypting automation scripts

B. A tendency to accept automated recommendations without sufficient scrutiny

C. A technique for increasing network bandwidth

D. A guarantee that a model is fair

Correct Answer: B. A tendency to accept automated recommendations without sufficient scrutiny

Question 9

Which problem occurs when normal system behaviour changes after a major migration?

A. Data drift

B. Perfect precision

C. Cryptographic integrity

D. Time synchronisation

Correct Answer: A. Data drift

Question 10

Why should an isolated threat indicator be interpreted cautiously?

A. Indicators always contain personal data

B. Indicators may be stale, shared, misattributed or lacking context

C. Indicators cannot be stored electronically

D. Indicators are never useful in detection

Correct Answer: B. Indicators may be stale, shared, misattributed or lacking context