Security Monitoring, Logging and Analytics

Introduction and Context

Security monitoring is the continuous observation and interpretation of events occurring across computing and communication environments. It converts technical evidence—such as authentication records, firewall messages, packet captures, flow records and endpoint alerts—into decisions about risk and response. Logging records what happened; monitoring determines what deserves attention; analytics identifies relationships, trends and probable causes.

Modern organisations generate evidence from servers, applications, cloud platforms, wireless networks, identity systems, endpoint protection tools and security appliances. The challenge is not merely collecting more data. It is collecting trustworthy data, preserving its context, reducing irrelevant noise and identifying activity that may indicate compromise. A monitoring programme must therefore combine technology, procedures, skilled analysis and governance.

A security operations team commonly uses a Security Information and Event Management (SIEM) platform to centralise logs, normalise fields, correlate events, generate alerts and present dashboards. However, a SIEM is not a substitute for sound architecture. Poor time synchronisation, incomplete logging, excessive permissions, weak retention policies and badly designed detection rules can make a sophisticated platform ineffective.

Learning Objectives

By the end of this chapter, a learner should be able to:

• distinguish logs, packet captures, flow records and security alerts;
• describe the collection, transport, storage and analysis of security data;
• establish useful behavioural baselines and recognise deviations;
• interpret indicators of compromise (IoCs) and indicators of attack (IoAs);
• explain SIEM components, correlation and dashboard design;
• use threat intelligence in operational decisions;
• investigate a security event while preserving evidence and context; and
• evaluate limitations, ethical issues and practical best practices in monitoring.

1. Evidence Sources in Network Security

1.1 Logs

A log is a time-stamped record generated by a system, application or device. Typical fields include event time, host, user or service account, source and destination addresses, protocol, action, result and a message or event identifier.

Examples include:

• Authentication logs: successful and failed logins, password changes, multifactor authentication events and account lockouts.
• Operating-system logs: process creation, privilege changes, service starts and file-access events.
• Network-device logs: firewall permits and denies, VPN connections, routing changes and configuration modifications.
• Application logs: transactions, errors, administrative actions and API requests.
• Cloud audit logs: changes to identities, storage, virtual machines, security groups and keys.
• Endpoint detection logs: suspicious processes, persistence mechanisms and malware detections.

Logs provide context and attribution, but their reliability depends on configuration. A disabled audit policy, an incorrect clock or an attacker with permission to erase logs creates an evidential gap.

1.2 Packet captures

A packet capture records network packets, usually including headers and sometimes payloads. It can reveal protocol behaviour, session establishment, DNS queries, certificate information and data-transfer patterns. A full capture is valuable for detailed forensic analysis but can consume considerable storage and may contain personal or confidential information.

Capture points matter. A capture taken outside a firewall may differ from one taken inside it because of address translation, filtering or encryption. Encrypted traffic can still provide metadata—such as endpoints, timing, volume and TLS certificate characteristics—even when its payload cannot be read.

Packet capture is especially useful when an analyst must answer questions such as:

1. Did two hosts actually communicate?
2. Which protocol and ports were used?
3. Was the connection interactive, automated or periodic?
4. Did the exchange contain evidence of scanning, exploitation or command and control?

1.3 Flow records

A flow record summarises communication rather than storing every packet. Common fields include source and destination addresses, source and destination ports, protocol, start and end times, packet count and byte count. NetFlow, IPFIX and similar technologies are examples.

Flow data is cheaper to retain than full packet capture and is effective for traffic-volume analysis, scanning detection and identifying unusual destinations. It generally cannot show the exact application content or individual packet sequence. Consequently, flow records and packet captures are complementary rather than interchangeable.

1.4 Alerts

An alert is an interpretation produced by a detection rule or security product. Examples include “malware blocked,” “impossible travel,” “port scan detected” and “multiple failed administrator logins.” An alert is not proof of compromise. It is a prioritised claim requiring validation.

Analysts should assess an alert’s source, confidence, affected asset, severity, supporting evidence and potential business impact. A high-severity label from a poorly tuned rule may be less useful than a medium-severity alert supported by several independent data sources.

2. The Monitoring Pipeline

A practical monitoring architecture has six broad stages:

1. Generation: devices and applications create events.
2. Collection: agents, collectors or sensors receive the events.
3. Transport: records are securely transmitted, preferably with authentication and encryption.
4. Processing: timestamps and fields are normalised, enriched and parsed.
5. Storage: hot data supports rapid searches; archived data supports investigations and compliance.
6. Analysis and action: rules, statistical methods and analysts identify risk and initiate response.

Centralisation improves correlation, but it also creates a high-value target. Collectors should be hardened, access should follow least privilege and logs should be protected against unauthorised alteration. Where feasible, organisations use append-only or immutable storage, cryptographic integrity controls and separate administrative roles for system operation and log review.

Time synchronisation

Accurate time is essential for reconstructing an incident. Systems should use an approved Network Time Protocol hierarchy, monitor clock drift and record timestamps consistently, including time zone and daylight-saving assumptions. Analysts should distinguish event time from ingestion time: a delayed log may arrive after a later event has already occurred.

Normalisation and enrichment

Different products may call the same field src_ip, sourceAddress or client. Normalisation creates a common schema. Enrichment adds context such as asset owner, business criticality, geographic information, identity role, vulnerability status or threat-intelligence reputation. Enrichment improves prioritisation but must be maintained; stale asset ownership can produce incorrect decisions.

3. Baselines and Anomaly Detection

A baseline describes usual behaviour for a system, user, network segment or application. Examples include normal login hours, ordinary outbound destinations, typical DNS volume, expected administrative tools and average data-transfer levels. A baseline is not a fixed universal value. It depends on the asset’s role, season, maintenance schedule and business process.

A useful baseline is:

• sufficiently long to include normal variation;
• segmented by user, device, location or service where necessary;
• updated when legitimate business behaviour changes; and
• documented so that analysts understand its assumptions.

Statistical measures

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here, x̄ is the mean, xᵢ is an observation and n is the number of observations. All xᵢ must represent the same measure and interval, such as daily failed-login counts. The mean is sensitive to unusually large values.

The sample standard deviation is:

The denominator n − 1 is used when the observations are treated as a sample estimating a wider population. Standard deviation has the same units as x. A simple standardised score for a new observation x is:

The score has no units. It indicates how far x lies from the mean in standard-deviation units, assuming s is non-zero. A large absolute z-score may justify investigation, but it is not automatically evidence of attack. Network data is often skewed, seasonal or correlated, so thresholds must be validated rather than adopted mechanically.

For example, suppose a server records daily failed logins of 4, 5, 6, 5 and 5. The mean is 5. If the sample standard deviation is approximately 0.71 and the next day has 9 failures, then:

This is an unusual observation relative to the short baseline. The analyst should examine source addresses, account names, authentication method, maintenance activity and related alerts before concluding that password spraying occurred.

Rates and detection quality

Counts can mislead when observation periods differ. An event rate is:

where r is the event rate, E is the number of events and T is the observation period. If 120 events occur during 2 hours, the rate is 60 events per hour. Units must remain consistent.

Detection systems are also evaluated using true and false outcomes. A true positive is an attack correctly detected; a false positive is benign activity incorrectly alerted; a true negative is benign activity correctly ignored; and a false negative is an attack missed. Precision is:

Recall, also called sensitivity, is:

These measures require labelled cases and can change as the threat environment changes. Improving recall may increase analyst workload through additional false positives. In operational security, the preferred balance depends on asset criticality and the cost of missing an event.

4. Indicators of Compromise and Attack

An indicator of compromise is an observable artefact associated with a compromise, such as a malware hash, malicious domain, known command-and-control address, suspicious registry key or unexpected executable path. An indicator of attack describes behaviour suggesting an attack, such as repeated authentication failures followed by a successful login, unusual remote administration or rapid access to many hosts.

IoCs can be specific but short-lived. A hash may identify one known file but fail when attackers rebuild or modify malware. Behavioural indicators are more adaptable but can be less precise. Strong investigations combine both with asset context and threat intelligence.

Indicators should be handled with lifecycle discipline:

1. validate the source and confidence;
2. record first-seen, last-seen and expiration information;
3. search historical data where retention permits;
4. identify affected assets and users;
5. block or monitor the indicator according to risk; and
6. document the result and retire obsolete indicators.

A match is a lead, not a final verdict. Shared hosting, compromised websites and recycled IP addresses can create false associations.

5. SIEM, Correlation and Dashboards

A SIEM commonly provides ingestion, parsing, indexing, correlation, alerting, search, case management and reporting. Correlation rules combine events that are weak individually but meaningful together. For example, a failed login burst, a successful login from a new country and privileged access to a file server may form a stronger investigation lead than any single event.

Correlation can be:

• Rule-based: explicit conditions such as “five failures followed by success within ten minutes.”
• Sequence-based: events are ordered in a likely attack path.
• Threshold-based: an alert occurs when a count or rate exceeds a threshold.
• Risk-based: event severity is combined with asset value, identity privilege and threat confidence.
• Behavioural: activity is compared with a learned baseline.

Rules require ownership, testing, documentation and periodic tuning. Every important rule should state its purpose, data dependencies, threshold, expected false positives, response action and review date.

Dashboards should support decisions rather than display data for its own sake. An operational dashboard might show alert backlog, mean time to acknowledge, high-risk assets, authentication anomalies, endpoint coverage and ingestion failures. A management dashboard may show trends, control coverage, incident categories and risk reduction. Visual prominence should reflect priority, not merely volume.

6. Threat Intelligence and Operational Decisions

Threat intelligence is information about threats that has been collected, evaluated and made useful for a decision. It may be:

• Strategic: long-term trends, adversary motivations and business risk;
• Operational: campaigns, sectors and attack methods;
• Tactical: techniques, procedures and defensive opportunities; or
• Technical: hashes, domains, addresses, signatures and detection patterns.

The intelligence cycle consists of direction, collection, processing, analysis, dissemination and feedback. An organisation should define the question first—for example, “Are our internet-facing VPN gateways being targeted by this campaign?”—rather than indiscriminately importing every available feed.

Threat feeds differ in accuracy, timeliness, coverage, licensing and potential bias. Automated blocking should normally require confidence, relevance and an approved exception process. Intelligence should be mapped to internal assets and controls; a technically correct indicator that does not affect the organisation may not deserve immediate action.

7. Investigation Workflow

A disciplined investigation reduces premature conclusions:

1. Triage: confirm the alert, identify affected assets and assess urgency.
2. Scope: search related users, hosts, addresses, domains, processes and time periods.
3. Timeline: align logs, flows, captures and endpoint events using trusted timestamps.
4. Validate: compare evidence with expected business activity and known maintenance.
5. Contain: follow approved procedures to isolate accounts, hosts or network paths when necessary.
6. Eradicate and recover: remove persistence, restore systems and monitor for recurrence.
7. Document: preserve queries, evidence, decisions, approvals and limitations.
8. Learn: improve controls, rules, training and asset information.

Evidence should be handled according to organisational policy and applicable law. Analysts should avoid modifying original data, record who accessed evidence and calculate cryptographic hashes when preserving exported files. A hash supports integrity checking; it does not by itself prove that the underlying content is accurate or that the evidence has a complete chain of custody.

Illustrative Case: Suspected Password Spraying

Background

A fictional regional college operates a central identity service, a VPN gateway and several cloud applications. Its SIEM receives authentication logs, VPN records, endpoint alerts and firewall flows.

Problem

The SIEM reports a sudden increase in failed VPN logins. The alert is initially classified as medium severity because the gateway blocked the attempts.

Analysis

The analyst establishes that the baseline is 8–15 failed VPN logins per hour during teaching weeks. In the last hour there were 180 failures from 34 external addresses, targeting 96 usernames. The attempts use a small number of passwords across many accounts, a pattern consistent with password spraying. One account then authenticates successfully from an unfamiliar address. A flow record shows that the session contacted an internal file service, and the identity log records an unusual download.

The analyst checks whether the successful login was expected, verifies multifactor authentication status, examines endpoint and file-access records, and searches for the source addresses and account activity over the previous seven days. Threat intelligence shows that two addresses have been reported in recent credential attacks, but the remaining addresses have no reputation data. This illustrates why an absence of an IoC does not establish safety.

Approach and solution

The response team temporarily disables the affected account, requires a password reset, reviews multifactor authentication, blocks confirmed malicious sources at the gateway and increases monitoring for related accounts. The file server is checked for unauthorised changes and unusual downloads. The SIEM rule is improved to correlate distributed failures by username and password pattern rather than relying only on a single source address.

Outcome

Investigation finds no evidence of lateral movement, but the account’s downloaded files require review. The college retains the incident record, improves VPN rate limiting and adds an alert for a successful login following distributed failures. It also corrects an asset record that had incorrectly marked the file server as low criticality.

Learning points

The incident demonstrates the value of combining counts, identity context, flow records, threat intelligence and asset criticality. Blocking a source address alone would have been insufficient because the activity was distributed. It also demonstrates that detection quality depends on complete, synchronised and correctly classified data.

8. Challenges, Governance and Ethics

Data quality and visibility

Missing logs, duplicate events, inconsistent fields and ingestion delays reduce confidence. Organisations should monitor collector health, event volume, parsing failures and source coverage as security controls in their own right.

Privacy and proportionality

Monitoring may expose employee communications, location, browsing activity or sensitive customer data. Collection should have a defined purpose, lawful basis where required, access restrictions, retention limits and transparent notices. Analysts should avoid collecting payloads when metadata is sufficient and should mask sensitive fields in dashboards and test environments.

Retention and resilience

Retention should reflect investigative needs, legal obligations, storage cost and the likelihood that an incident is discovered late. Logs should be protected from deletion and ransomware, and critical monitoring should continue during outages. Backups must themselves be access-controlled and tested.

Automation and artificial intelligence

Machine-learning systems can identify unusual behaviour, cluster alerts and assist with summarisation. They can also inherit biased training data, generate opaque conclusions, drift as behaviour changes or be manipulated by adversaries. AI-generated findings should be treated as analyst assistance, not unquestionable evidence. Human review, explainable evidence links, testing and an escalation path remain necessary.

9. Best Practices

A practical monitoring programme should:

• define critical assets, important events and investigation objectives;
• enable appropriate audit policies before an incident occurs;
• synchronise clocks and record both event and ingestion times;
• centralise logs while preserving source identity and integrity;
• protect collectors and storage with least privilege and segmentation;
• normalise fields and enrich events with current asset and identity data;
• establish documented, segmented baselines rather than universal thresholds;
• tune rules using observed false positives and missed detections;
• combine IoCs with behaviour, context and threat intelligence;
• design dashboards around operational decisions;
• test alert delivery, retention, recovery and analyst procedures;
• preserve evidence and document investigative reasoning; and
• review monitoring for privacy, fairness, legality and proportionality.

Chapter Summary

Security monitoring transforms technical events into operational decisions. Logs provide structured records, packet captures provide detailed communication evidence, flow records summarise traffic and alerts provide prioritised interpretations. Effective monitoring depends on complete collection, secure transport, accurate time, normalisation, enrichment and resilient storage.

Baselines describe expected behaviour and support anomaly detection, but statistical deviations require contextual validation. IoCs identify artefacts associated with compromise, while IoAs describe suspicious behaviour. SIEM platforms correlate evidence, manage alerts and present dashboards, but their success depends on data quality and well-maintained rules. Threat intelligence is most valuable when it answers a defined organisational question and leads to a proportionate action.

A professional investigation combines triage, scoping, timeline construction, validation, containment, recovery and learning. Privacy, evidence integrity, retention, automation risk and governance must be addressed alongside technical capability. The goal is not maximum data collection; it is trustworthy evidence that enables timely, defensible and risk-informed decisions.

Key Terms

Log: A time-stamped record produced by a system, device or application.

Packet capture: A recording of network packets for protocol or forensic analysis.

Flow record: A summary of communication between network endpoints.

Alert: A notification produced by a detection mechanism that requires assessment.

SIEM: A platform for collecting, correlating, searching and reporting security events.

Normalisation: Converting diverse event formats into a common field structure.

Enrichment: Adding contextual information to an event.

Baseline: A documented description of expected activity.

Anomaly: Activity that differs significantly from an established expectation.

Indicator of compromise: An artefact associated with a suspected or confirmed compromise.

Indicator of attack: A behaviour suggesting an attack in progress.

Threat intelligence: Evaluated threat information prepared for a decision.

Correlation: Combining related events to identify a meaningful pattern.

False positive: A benign event incorrectly classified as malicious.

False negative: Malicious activity that a detection fails to identify.

Chain of custody: A record of evidence handling and access.

Precision: The proportion of positive detections that are correct.

Recall: The proportion of relevant malicious cases that are detected.

Data retention: The period for which records are preserved and accessible.

Review Questions

Short Answer Questions

1. Distinguish a packet capture from a flow record.
2. Why is time synchronisation important during incident investigation?
3. Define an indicator of compromise and give one example.
4. What is the purpose of event normalisation in a SIEM?
5. Why should a security alert not automatically be treated as proof of compromise?

Long Answer Questions

1. Explain the stages of a security-monitoring pipeline and identify controls needed at each stage.
2. Discuss how an organisation can establish, maintain and validate behavioural baselines.
3. Compare rule-based, threshold-based and behavioural correlation in a SIEM.
4. Explain the threat-intelligence lifecycle and how intelligence should influence operational decisions.
5. Discuss privacy, evidence integrity and retention issues in security monitoring.

Analytical / Application Questions

1. A company normally records 10–20 failed administrator logins per hour, but records 140 in one hour from 25 addresses. Design an investigation using logs, flow records, identity data and threat intelligence. Explain what evidence would support or weaken a password-spraying hypothesis.
2. A detection rule generated 80 alerts, of which 20 were confirmed malicious. It missed 5 malicious events that were discovered independently. Calculate its precision and recall, state the assumptions, and recommend how the rule could be improved.
3. A university wants to collect full packet payloads from all student networks for one year. Evaluate the proposal in terms of detection value, storage, privacy, governance and alternative sources of evidence.

Multiple Choice Questions with Answers

Question 1

Which evidence source summarises communication using fields such as endpoints, ports, duration and byte count without normally storing every packet?

A. Flow record

B. Application source code

C. Password database

D. Security policy

Correct Answer: A. Flow record

Question 2

What is the primary purpose of time synchronisation in security monitoring?

A. To increase network bandwidth

B. To reconstruct the order and timing of events

C. To encrypt all log messages

D. To remove false positives automatically

Correct Answer: B. To reconstruct the order and timing of events

Question 3

A baseline has a mean of 5 failed logins and a standard deviation of 0.71. What is the approximate z-score for 9 failed logins?

A. 0.56

B. 1.41

C. 4.00

D. 5.63

Correct Answer: D. 5.63

Question 4

Which statement best describes an indicator of compromise?

A. A documented normal operating range

B. An artefact associated with a suspected compromise

C. A legal requirement for data retention

D. A dashboard display format

Correct Answer: B. An artefact associated with a suspected compromise

Question 5

What does SIEM event normalisation accomplish?

A. It converts different event formats into a common structure

B. It deletes events considered unimportant

C. It guarantees that every alert is correct

D. It replaces the need for network sensors

Correct Answer: A. It converts different event formats into a common structure

Question 6

If a detection system has 30 true positives and 10 false positives, what is its precision?

A. 25%

B. 30%

C. 75%

D. 90%

Correct Answer: C. 75%

Question 7

Which threat-intelligence type focuses most directly on hashes, domains and IP addresses?

A. Strategic intelligence

B. Technical intelligence

C. Financial intelligence

D. Organisational intelligence

Correct Answer: B. Technical intelligence

Question 8

Why is a match against a malicious IP address not automatically proof of compromise?

A. IP addresses cannot appear in logs

B. Shared or reassigned infrastructure can create false associations

C. Threat intelligence never contains useful information

D. Firewalls cannot record destinations

Correct Answer: B. Shared or reassigned infrastructure can create false associations

Question 9

Which investigation activity should normally occur before containment?

A. Deleting the original logs

B. Triage and initial scoping

C. Disabling all organisational accounts

D. Publishing the incident publicly

Correct Answer: B. Triage and initial scoping

Question 10

Which is an appropriate use of AI in security monitoring?

A. Replacing all human review of evidence

B. Making unrecorded account decisions

C. Assisting alert clustering while retaining human validation

D. Ignoring changes in normal behaviour

Correct Answer: C. Assisting alert clustering while retaining human validation