10. Network Monitoring, Detection and Security Analytics

Introduction and Context

Network security depends not only on preventive controls but also on the ability to observe, interpret, and respond to events. Firewalls, access controls, encryption, endpoint protection, and secure architecture reduce exposure, but no preventive mechanism is perfect. Misconfigurations, stolen credentials, insider misuse, newly discovered vulnerabilities, and supply-chain compromise may allow an adversary to operate inside a supposedly protected environment. Network monitoring and security analytics provide the evidence required to discover such activity and assess whether security controls are working as intended.

Network monitoring is the systematic collection and examination of information about communications, devices, services, users, and security controls. Detection converts observations into indications of potentially harmful activity. Security analytics applies statistical, rule-based, behavioural, and contextual methods to determine whether an indication represents a meaningful security event. These activities form a continuous operational cycle: collect, normalize, enrich, detect, investigate, respond, learn, and improve.

At postgraduate level, the subject should be understood as both a technical discipline and a decision-making function. A packet capture may reveal the mechanics of a connection, while a SIEM may reveal that the same connection occurred shortly after an unusual privilege change. Neither data source is sufficient in isolation. Effective monitoring therefore depends on selecting appropriate telemetry, preserving its integrity, understanding its limitations, and relating technical evidence to business risk.

Learning Objectives

After completing this chapter, students should be able to:

1. Distinguish logs, metrics, packet data, flow records, and other forms of security telemetry.

2. Explain the roles and relationships of IDS, IPS, SIEM, SOAR, threat intelligence, and behavioural analytics.

3. Design a monitoring and detection workflow for a complex enterprise network.

4. Evaluate detections using precision, recall, fidelity, coverage, and operational cost.

5. Analyse an incident using network, identity, endpoint, and cloud evidence.

6. Identify common monitoring failures, governance obligations, and ethical risks.

7. Apply detection-engineering principles to create maintainable and testable rules.

From Network Visibility to Security Knowledge

Visibility is not equivalent to surveillance of every byte. It is the purposeful acquisition of evidence relevant to security decisions. The appropriate level of visibility depends on the threat model, architecture, regulatory environment, privacy expectations, and available resources.

Forms of telemetry

Logs are structured or semi-structured records generated by systems and applications. Authentication logs, firewall decisions, DNS queries, VPN events, database audit records, and cloud control-plane events can establish who attempted an action, when it occurred, from where, and whether it succeeded. Logs are usually efficient to store and search, but they depend on correct configuration and may omit important context.

Metrics represent numerical measurements over time, such as connection rates, failed-login counts, bandwidth, CPU usage, or the number of DNS requests. Metrics are particularly useful for identifying deviations, denial-of-service conditions, and capacity-related security events. They generally provide less detail than logs.

Packet data contains network traffic at the packet level. Full packet capture can support protocol analysis, malware investigation, and reconstruction of an intrusion. Its disadvantages include storage volume, encryption, privacy concerns, and the difficulty of searching large datasets. Selective capture, ring buffers, and capture at strategic network locations are common compromises.

Flow records summarize conversations rather than storing complete content. Typical fields include source and destination addresses, ports, protocol, timestamps, packet counts, byte counts, and connection state. NetFlow, IPFIX, and similar technologies are useful for identifying scanning, beaconing, data transfer, and unusual communications. Flow records cannot normally answer what content was exchanged.

Endpoint and identity telemetry extends network analysis. Process creation, command-line arguments, file access, authentication context, group membership, and device posture can distinguish legitimate administration from credential abuse. In cloud environments, control-plane events such as creation of keys, changes to security groups, and modification of storage policies are especially important.

Quality dimensions of telemetry

A monitoring programme should evaluate telemetry according to reliability, completeness, timeliness, precision, provenance, and retention. Time synchronization is fundamental: inconsistent clocks can make a sequence of events appear impossible or conceal the relationship between them. Network Time Protocol should therefore be authenticated or otherwise protected where appropriate, and systems should record time zones and clock corrections clearly.

The meaning of a record also depends on its source. A firewall denial is not evidence that an attack succeeded; an authentication success is not proof that the legitimate account owner acted. Analysts must understand collection points, sampling, dropped records, parser behaviour, and the difference between observed facts and inferred conclusions.

Detection Architecture and Analytical Workflow

A mature architecture usually contains several layers. Sensors collect data from network segments, endpoints, identity providers, applications, and cloud platforms. Collectors securely transmit records to storage or analysis services. A normalization layer maps different formats to a common schema. Enrichment adds asset ownership, user identity, vulnerability status, geographical information, threat-intelligence references, and business criticality. Detection engines evaluate rules or models, while case-management and response systems support investigation and action.

IDS and IPS

An intrusion detection system (IDS) identifies suspicious activity and generates an alert. An intrusion prevention system (IPS) places detection inline or integrates with enforcement mechanisms so that selected traffic can be blocked, reset, or quarantined. Network IDS tools commonly use signature matching, protocol analysis, anomaly detection, or combinations of these approaches.

Signature detection is effective when the relevant pattern is known and observable. It may identify a malicious payload, exploit sequence, or command-and-control protocol. Its limitations include evasion, encrypted traffic, and poor performance against novel threats. Anomaly detection establishes a baseline and identifies deviations, but unusual behaviour is not necessarily malicious. Protocol-aware detection examines whether communications conform to expected protocol semantics, often revealing malformed or abusive traffic even when payload signatures are unavailable.

An IPS introduces a major trade-off: blocking improves containment but creates the possibility of disrupting legitimate operations. Prevention policies should therefore be introduced with carefully chosen confidence levels, exception processes, safe failure modes, and rollback procedures.

SIEM and security operations

A security information and event management (SIEM) platform aggregates, parses, correlates, stores, and searches security events. Its value does not arise merely from collecting large volumes of data. It lies in linking events that have meaning together. For example, a successful login from an unusual location, followed by privilege elevation and access to a sensitive repository, is more significant than any individual event.

A useful SIEM implementation distinguishes raw data from normalized events, alerts from incidents, and evidence from analyst conclusions. Data models and consistent field names support correlation across vendors. Retention should reflect investigative needs, legal obligations, cost, and the likelihood that an intrusion will be discovered after a delay.

Security orchestration, automation, and response (SOAR) platforms can enrich alerts, create cases, disable accounts, isolate hosts, or notify stakeholders. Automation should be proportional to confidence and impact. A low-confidence anomaly should not automatically disable a critical production account, whereas a high-confidence malicious process on an isolated workstation may justify immediate containment.

Detection engineering

Detection engineering treats detections as operational software. A sound detection has a defined purpose, data prerequisites, logic, severity, owner, response guidance, test cases, and review date. The engineer should state the threat behaviour being detected rather than merely copying a vendor rule.

The lifecycle commonly includes:

• translating a threat hypothesis into observable behaviours;
• identifying required data sources and fields;
• writing and testing detection logic;
• tuning exclusions without hiding genuine attacks;
• deploying through controlled change management;
• measuring performance and analyst workload; and
• retiring or revising the detection as systems and adversary methods change.

Detection logic may use threshold rules, sequences, joins, statistical baselines, graph relationships, or supervised and unsupervised models. A sequence rule might identify a password spray followed by a successful login and an unusual administrative action. A graph approach might highlight a new relationship between a service account, a workstation, and a sensitive database.

Interpreting Alerts and Measuring Effectiveness

An alert is a request for investigation, not a final determination. Analysts should establish what happened, whether it was authorized, what assets and identities were involved, whether the activity continued, and what evidence supports each conclusion.

Several metrics help evaluate detection quality. Precision is the proportion of alerts that are genuinely relevant among all alerts generated. Recall is the proportion of relevant malicious events that the detection identifies. Increasing recall may reduce precision, creating analyst fatigue. Fidelity describes how reliably an alert represents the behaviour it claims to identify. Coverage concerns which assets, techniques, protocols, and attack stages are observable. Mean time to detect and mean time to respond measure operational speed, but they should not be interpreted without considering alert severity and investigation complexity.

A detection can have high precision but low coverage, finding one narrow attack reliably while missing most other paths. Conversely, a broad anomaly rule may have high theoretical coverage but low practical value because analysts cannot process its alerts. Control effectiveness should therefore be assessed through balanced evidence: attack simulations, purple-team exercises, incident reviews, telemetry health checks, and measures of response quality.

Threat Intelligence and Behavioural Analytics

Threat intelligence adds context to observations. Strategic intelligence concerns threat actors, motivations, and trends; operational intelligence concerns campaigns and targeting; tactical intelligence describes techniques and procedures; and technical intelligence includes indicators such as domains, addresses, hashes, and certificates. Indicators can be useful, but they decay quickly and may be shared infrastructure. They should not replace behavioural detection.

A responsible intelligence workflow records source reliability, confidence, time validity, collection method, and permitted use. An address listed in a reputation feed should be treated as a clue, not conclusive proof. Enrichment should help analysts prioritize and investigate rather than automatically label all associated activity as malicious.

Behavioural analytics establishes expected patterns for users, devices, services, and applications. Examples include a service account authenticating interactively, a workstation contacting an unusual number of external domains, or a database exporting data at an atypical time. User and entity behaviour analytics (UEBA) can combine peer-group comparisons, historical baselines, and risk scoring.

Behavioural models face the problems of concept drift, biased baselines, and explainability. A new business project may make previously unusual traffic legitimate. A compromised account may gradually adapt to normal patterns. Analysts should therefore be able to inspect the features contributing to a score, challenge model outputs, and update baselines through governed processes.

A Practical Investigation Method

A structured investigation reduces both premature conclusions and uncontrolled data collection. First, validate the alert and its data source. Second, construct a timeline using synchronized events. Third, identify the initiating identity, device, process, or external service. Fourth, examine related authentication, DNS, endpoint, and network activity. Fifth, determine scope, persistence, privilege changes, lateral movement, and possible data access. Sixth, contain according to business impact and confidence. Finally, preserve evidence, document decisions, and improve detections.

The analyst should separate facts, interpretations, and hypotheses. “The account authenticated from two addresses within ten minutes” is an observation. “The account was stolen” is an interpretation requiring supporting evidence. “The attacker may have accessed the finance database” is a hypothesis that directs further investigation.

Illustrative Case Study: Detecting Compromise of a Service Account

Background

A fictional regional logistics company operates a hybrid network containing warehouse systems, a cloud identity provider, and an externally accessible application programming interface. Service accounts are used by applications to access databases and message queues. Network flow, identity logs, endpoint events, DNS data, and cloud audit records are forwarded to a SIEM.

Situation

The monitoring team receives an alert that a service account has authenticated interactively to a warehouse server. Interactive authentication is unusual for that account, but the alert has historically generated false positives during maintenance. Twenty minutes later, flow data shows the server contacting an unfamiliar external host over an uncommon port. A cloud audit event records the creation of a new access key associated with the same account.

Analysis

The analyst validates that the identity event was successful and confirms that the source workstation was not on the approved maintenance list. Endpoint telemetry shows a command interpreter launched by a previously unseen process. DNS records indicate that the external host was resolved shortly before the connection. Threat intelligence provides low-confidence information linking the domain to malicious infrastructure, but the investigation does not rely on that indicator alone.

Correlation of the timeline produces a stronger hypothesis: credentials associated with the service account were used to gain access, execute a process, establish external communication, and create a cloud credential for persistence. The combination of identity, endpoint, flow, DNS, and cloud evidence has higher fidelity than any single alert.

Approach and response

The incident team disables the newly created key, rotates the service-account secret, blocks the external destination after confirming that it is not business-required, and isolates the warehouse server. Investigators preserve relevant logs and endpoint artifacts before rebuilding the host. They review database access and cloud activity to determine whether sensitive information was read or changed. Business owners approve temporary service degradation while the account dependencies are mapped.

The detection team then modifies its rules. The prior rule for interactive service-account login is retained but enriched with asset criticality and maintenance-window context. A new correlation rule links unusual service-account authentication, process creation, external flow, and credential creation. A test dataset is used to verify both malicious and legitimate maintenance scenarios.

Outcome

The investigation establishes unauthorized use of the account and prevents continued access through the new key. No evidence is found of alteration to shipment records, although some directory metadata was queried. The organization introduces managed identities where possible, reduces service-account privileges, requires approval for interactive use, and adds a quarterly detection test.

Learning points

The case demonstrates the value of multi-source correlation, the danger of treating threat-intelligence matches as proof, and the importance of contextual exceptions. It also shows that detection quality depends on identity governance and telemetry health, not solely on SIEM technology.

Challenges, Governance, and Professional Responsibility

Monitoring programmes generate sensitive information about employees, customers, communications, and business operations. Organizations should define a legitimate security purpose, collect proportionate data, restrict access, protect stored records, establish retention limits, and communicate monitoring practices where required. Privacy-preserving techniques may include metadata-first collection, role-based access, masking, aggregation, and controlled access to packet content.

Evidence must be protected against alteration and unauthorized disclosure. Centralized logs should use secure transport, access controls, integrity mechanisms, and monitored administrative activity. Chain-of-custody procedures are important when records may support disciplinary or legal action. Analysts should avoid accessing unrelated personal content merely because technical capability permits it.

Common operational failures include collecting data without an investigation plan, allowing parser changes to silently break detections, ignoring clock drift, creating excessive low-value alerts, using permanent exceptions, failing to monitor the monitoring system, and measuring success only by the number of alerts closed. Third-party feeds can also introduce poisoning, stale indicators, licensing restrictions, or inappropriate attribution.

Automation, AI, and Emerging Practice

Automation is increasingly used for enrichment, alert grouping, entity risk scoring, query generation, and response playbooks. Machine-learning systems can detect subtle relationships across large datasets, but they remain vulnerable to poor training data, adversarial manipulation, distribution changes, and opaque decisions. Generative AI may assist analysts in summarizing timelines or translating queries between platforms, but outputs require verification and must not expose confidential telemetry to unauthorized services.

Modern architectures are increasing the importance of identity-centric and cloud-native monitoring. Zero-trust environments require continuous evaluation of user, device, application, and session context rather than reliance on a network perimeter. Extended detection and response (XDR) integrates endpoint, network, identity, email, and cloud signals. Open schemas and interoperable data formats can reduce dependence on a single platform, while privacy-enhancing analytics may allow useful detection with reduced exposure of raw content.

Best Practices for an Evidence-Based Monitoring Programme

An effective programme begins with a threat-informed inventory of critical assets, identities, data flows, and business processes. It then prioritizes telemetry that can answer specific investigative questions. Detection rules should be version-controlled, documented, tested, and assigned owners. Analysts need playbooks that identify evidence sources, escalation thresholds, containment options, and communication responsibilities.

Organizations should regularly conduct adversary emulation or purple-team exercises, measure both missed detections and unnecessary alerts, review detection performance after incidents, and test restoration of historical data. Governance should connect technical findings to risk: an unusual connection to a low-value test host does not have the same significance as the same behaviour involving a payment system. Finally, monitoring should be treated as a continually improved capability rather than a product installation.

Chapter Summary

Network monitoring transforms technical activity into evidence for security decisions. Logs, metrics, packet captures, flow records, endpoint events, identity records, and cloud audit data each provide different levels of detail and confidence. IDS and IPS identify or block suspicious traffic; SIEM platforms correlate events; SOAR supports governed automation; threat intelligence supplies context; and behavioural analytics identifies deviations from expected activity.

Effective detection engineering begins with a threat hypothesis and ends with tested, measured, and maintained detection logic. Analysts must distinguish observations from conclusions, combine multiple sources, and evaluate alerts using precision, recall, fidelity, coverage, and operational cost. Monitoring also involves privacy, evidence integrity, access governance, and responsible use of automation and AI. The central professional lesson is that visibility has value only when it supports accurate interpretation, timely action, and measurable improvement.

Key Terms

Telemetry: Data collected from systems, networks, identities, and applications for observation or analysis.

Flow record: A summary of a network conversation containing metadata such as addresses, ports, times, and byte counts.

Packet capture: Recorded network packets used for detailed protocol and content analysis.

Intrusion Detection System (IDS): A system that identifies suspicious activity and produces alerts.

Intrusion Prevention System (IPS): A detection system capable of enforcing actions such as blocking or resetting traffic.

SIEM: A platform that aggregates, normalizes, correlates, stores, and searches security events.

SOAR: Technology supporting security orchestration, automated enrichment, and response workflows.

Detection engineering: The disciplined design, testing, deployment, and maintenance of detection logic.

Threat intelligence: Processed information about threats, actors, indicators, and adversary behaviour.

Indicator: An observable artifact associated with activity, such as a domain, address, hash, or certificate.

UEBA: Analysis of user and entity behaviour to identify unusual or risky activity.

Baseline: A representation of expected behaviour used for comparison.

Precision: The proportion of generated alerts that are relevant or true positives.

Recall: The proportion of relevant malicious activity detected by a system.

Fidelity: The reliability with which an alert represents the behaviour it claims to identify.

Telemetry coverage: The extent to which relevant assets, techniques, and events are observable.

Correlation: Combining related events to derive greater meaning than individual records provide.

Threat hypothesis: A stated expectation about adversary behaviour and its observable evidence.

Chain of custody: Documentation showing how evidence was collected, handled, preserved, and accessed.

Concept drift: Change over time in the behaviour or data patterns on which an analytical model relies.

Review Questions

Short Answer Questions

1. Distinguish packet capture from flow records.

2. What is the primary purpose of a SIEM correlation rule?

3. Define precision and recall in detection analysis.

4. Why is time synchronization important during an investigation?

5. What does a threat hypothesis contribute to detection engineering?

Long Answer Questions

1. Explain how logs, identity telemetry, endpoint events, flow records, and cloud audit data can be combined during an investigation.

2. Compare signature-based, anomaly-based, and protocol-aware IDS detection.

3. Describe the lifecycle of a detection from threat hypothesis to retirement.

4. Discuss the benefits and risks of automating response through SOAR.

5. Explain how privacy, evidence integrity, and access governance affect network monitoring.

Analytical / Application Questions

1. A service account authenticates interactively, launches a command interpreter, contacts an unfamiliar external host, and creates a cloud access key. Design an investigation timeline and identify the most important containment actions.

2. A detection generates many alerts during legitimate maintenance but may also identify credential compromise. Propose tuning, enrichment, and measurement methods that improve precision without substantially reducing recall.

3. An organization has extensive firewall logs but no endpoint or identity telemetry. Evaluate what attacks it can and cannot detect, and recommend a prioritized improvement plan.

22. Multiple Choice Questions with Answers

Question 1

Which statement best describes the principal value of correlating identity, endpoint, and network telemetry?

A. It eliminates the need for analyst review.

B. It provides contextual evidence that can be stronger than any individual event.

C. It guarantees that every alert is a true positive.

D. It allows packet capture to be discarded in all investigations.

Correct Answer: B. It provides contextual evidence that can be stronger than any individual event.

Question 2

A flow record can normally show which of the following?

A. The exact contents of an encrypted application message.

B. The process that created every packet.

C. Addresses, ports, timestamps, and byte counts for a conversation.

D. The complete commands typed by a user.

Correct Answer: C. Addresses, ports, timestamps, and byte counts for a conversation.

Question 3

What is the main operational risk of placing an IPS inline?

A. It cannot detect known signatures.

B. It may disrupt legitimate traffic through an incorrect blocking decision.

C. It prevents collection of all network metadata.

D. It makes time synchronization unnecessary.

Correct Answer: B. It may disrupt legitimate traffic through an incorrect blocking decision.

Question 4

A detection has high precision but low recall. What does this most likely mean?

A. Most alerts are relevant, but many relevant events are missed.

B. Most relevant events are detected, but nearly all alerts are false positives.

C. The detection has no data prerequisites.

D. The detection cannot be evaluated.

Correct Answer: A. Most alerts are relevant, but many relevant events are missed.

Question 5

Which action best reflects responsible use of threat-intelligence indicators?

A. Treat every listed IP address as conclusive proof of compromise.

B. Use indicators as contextual clues alongside behavioural and internal evidence.

C. Block all indicators permanently without review.

D. Ignore indicator age and source reliability.

Correct Answer: B. Use indicators as contextual clues alongside behavioural and internal evidence.

Question 6

In the illustrative case, which combination most increased detection fidelity?

A. A single low-confidence domain reputation match.

B. Interactive service-account use, endpoint execution, external flow, and key creation.

C. A high volume of ordinary DNS requests alone.

D. A firewall denial without any related event.

Correct Answer: B. Interactive service-account use, endpoint execution, external flow, and key creation.

Question 7

Why should detection rules be treated as operational software?

A. They require ownership, testing, version control, and maintenance.

B. They must always be written in a programming language.

C. They can function without defined data sources.

D. They should never be changed after deployment.

Correct Answer: A. They require ownership, testing, version control, and maintenance.

Question 8

Which practice best addresses privacy risk in security monitoring?

A. Collect all content indefinitely and allow universal analyst access.

B. Use purpose limitation, proportionate collection, access controls, and retention limits.

C. Disable audit logging for monitoring administrators.

D. Share raw telemetry with all business units.

Correct Answer: B. Use purpose limitation, proportionate collection, access controls, and retention limits.

Question 9

What is concept drift in behavioural analytics?

A. A failure to encrypt a packet capture.

B. A change in normal patterns that can reduce the validity of an existing model.

C. A method for synchronizing clocks.

D. A technique for converting packets into flow records.

Correct Answer: B. A change in normal patterns that can reduce the validity of an existing model.

Question 10

Which response is most appropriate for a low-confidence anomaly involving a critical production account?

A. Automatically disable the account without validation.

B. Ignore the event because it is not a signature match.

C. Enrich and investigate it, applying containment proportional to confidence and impact.

D. Delete the underlying telemetry to reduce alert volume.

Correct Answer: C. Enrich and investigate it, applying containment proportional to confidence and impact.