Security Monitoring, Logging, and Analytics

Introduction

Modern networks generate an enormous quantity of evidence: authentication records, firewall decisions, DNS requests, endpoint events, cloud audit trails, application logs, network flows, packet captures, vulnerability alerts, and threat-intelligence indicators. Security monitoring is the disciplined process of collecting, protecting, analysing, and acting on this evidence.

Monitoring is not simply watching dashboards. Its purpose is to support decisions: Is an event normal or suspicious? Which asset is affected? How serious is the activity? What evidence should be preserved? Should an account be disabled, a host isolated, or an incident escalated?

A monitoring capability therefore combines technology, procedures, people, and governance. A technically advanced Security Information and Event Management (SIEM) platform cannot compensate for incomplete log sources, inaccurate time settings, poorly designed detection rules, or unclear response responsibilities.

This chapter introduces the foundations required to collect and interpret logs, flows, alerts, and indicators of compromise (IOCs). It explains SIEM architecture, packet and flow analysis, baselining, threat intelligence, security metrics, and the limitations of automated analytics.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and characteristics of security logs, flows, alerts, and IOCs.
2. Identify important log sources and design a basic collection strategy.
3. Describe how SIEM platforms normalize, correlate, retain, and analyse events.
4. Distinguish packet analysis from flow analysis and select an appropriate method.
5. Construct a simple behavioural baseline and interpret deviations from it.
6. Apply threat intelligence responsibly to detection and investigation.
7. Calculate and interpret basic security metrics such as detection rate, false-positive rate, and mean time to respond.
8. Recognize privacy, governance, reliability, and operational limitations in security monitoring.

1. Evidence Used in Security Monitoring

1.1 Events and logs

An event is a recorded occurrence, such as a successful login, a blocked connection, a process starting, or a file being modified. A log is a structured or unstructured record of one or more events. Useful security logs normally contain:

• a timestamp;
• the event type and outcome;
• the identity involved;
• source and destination addresses or systems;
• the affected asset;
• a descriptive message or event code; and
• enough context to support investigation.

A log is valuable only when it is trustworthy and interpretable. The organization should synchronize clocks, document time zones, preserve the original event where possible, and record the source system. Network Time Protocol (NTP) is commonly used for clock synchronization, but synchronization itself must be monitored because a manipulated clock can obscure the sequence of an attack.

Logs may be structured, such as JSON or a database record, or semi-structured, such as syslog text. Structured data is easier to search and correlate. However, parsing errors, inconsistent field names, missing values, and changes in vendor formats can reduce analytical quality.

1.2 Common log sources

Collection should be risk-based. An organization need not collect every possible message indefinitely. It should prioritize systems that are business-critical, exposed to the internet, privileged, or likely to contain evidence of compromise.

1.3 Logs, flows, alerts, and IOCs

These evidence types are related but not interchangeable.

• Logs describe events recorded by systems or applications.
• Network flows summarize communication, commonly including source and destination addresses, ports, protocol, byte counts, packet counts, and start and end times. NetFlow and IPFIX are examples of flow technologies.
• Alerts are notifications produced by a security control or analytical rule. An alert is an interpretation, not proof of compromise.
• Indicators of compromise are observable characteristics associated with malicious activity, such as a file hash, domain, IP address, URL, certificate fingerprint, or unusual account behaviour.

An IOC should be evaluated in context. A malicious IP address may be shared by many customers, a hash may change after recompilation, and a domain may be compromised temporarily. Indicators have confidence, scope, age, and expiry considerations.

2. Collection and Log Management

2.1 A basic collection architecture

A typical architecture contains four logical stages:

1. Producers generate events, including endpoints, servers, network devices, and cloud services.
2. Collectors or forwarders receive, buffer, filter, and securely transmit events.
3. The SIEM or data platform parses, normalizes, enriches, indexes, and correlates data.
4. Analysts and response systems investigate alerts and initiate action.

Secure transport, access control, and integrity protection are essential. Logs should not be writable by ordinary users or by an attacker who has compromised the monitored host. Centralized collection helps because an intruder cannot erase every copy by modifying one server.

A useful design separates hot, warm, and archival data. Hot data is rapidly searchable and supports current investigations. Warm data is less immediately accessible but remains available for analysis. Archives support legal, regulatory, or forensic requirements. Retention must be based on risk, cost, contractual duties, and applicable law.

2.2 Data quality

Security analytics depends on data quality. Important controls include:

• clock synchronization and explicit time-zone handling;
• reliable parsing and schema validation;
• monitoring for missing or interrupted sources;
• deduplication where forwarding produces duplicates;
• preservation of raw records alongside normalized fields;
• protection against unauthorized alteration;
• documented retention and deletion schedules; and
• testing after changes to devices, applications, or collectors.

A silent logging failure is itself a security event. A dashboard showing no failed logins may indicate that nobody is failing to log in—or that the identity log source has stopped reporting.

2.3 Centralized logging and its trade-offs

Centralization supports correlation. For example, a failed login on a VPN gateway can be linked to a successful login on a cloud service and a new administrative role assignment. However, centralization creates concentration risk: the SIEM becomes a high-value target, storage costs increase, and excessive data can overwhelm analysts.

A collection policy should specify the source, event categories, minimum fields, sensitivity, transport method, retention period, owner, and intended detections. This makes collection purposeful rather than indiscriminate.

3. SIEM Concepts and Analytical Workflow

A SIEM combines log management with search, correlation, alerting, dashboards, case management, and often automated response. Although products differ, a common workflow is:

collect → normalize → enrich → correlate → prioritize → investigate → respond → learn

3.1 Normalization and enrichment

Normalization maps vendor-specific fields to common concepts. For example, different products may use src, source_ip, or clientAddress; a common schema maps them to a source-address field. Normalization makes cross-platform searches possible.

Enrichment adds context, such as:

• asset owner and business criticality;
• geographic or autonomous-system information;
• identity and department;
• vulnerability status;
• known threat-intelligence associations; and
• whether the asset is a domain controller, database, workstation, or public service.

Enrichment must be maintained. Incorrect asset ownership or stale intelligence can cause analysts to prioritize the wrong event.

3.2 Correlation and detection logic

Correlation combines events that are more meaningful together than separately. Examples include:

• multiple failed logins followed by a success from the same source;
• a new administrative account followed by access to sensitive data;
• an office document spawning a scripting interpreter and making an external connection;
• an endpoint contacting a suspicious domain and then creating a persistence mechanism.

Detection rules should have a clear purpose, data dependency, severity rationale, owner, and test method. They should be tuned using representative benign activity, not merely disabled when noisy.

A rule can be signature-based, behavioural, statistical, or a combination. Signature rules are often precise for known patterns. Behavioural rules can identify novel activity but may produce more false positives. No detection method is complete; attackers can avoid known indicators, use legitimate administration tools, or operate below thresholds.

3.3 Alert triage

A practical triage sequence is:

1. Validate that the alert is based on real and complete data.
2. Identify the asset, account, process, source, and time window.
3. Determine whether the activity is authorized or expected.
4. Search for related events before and after the alert.
5. Assess impact, confidence, and urgency.
6. Preserve evidence and escalate according to procedure.

Severity and confidence should be distinguished. A high-severity alert with low confidence may require rapid validation, while a medium-severity alert with high confidence on a critical server may deserve immediate containment.

4. Packet Analysis and Flow Analysis

4.1 Packet analysis

A packet capture records network packets, often including headers and sometimes payloads. Analysts can inspect protocol fields, TCP flags, DNS content, certificates, application requests, and transmission sequences. Packet analysis is powerful for detailed investigations, protocol troubleshooting, and confirming exploitation.

Its limitations include storage volume, encryption, privacy concerns, and the difficulty of capturing traffic at all relevant locations. Full payload collection may be inappropriate or unlawful in some environments. A capture should have a defined purpose, scope, access control, and retention period.

4.2 Flow analysis

Flow records summarize conversations rather than storing every packet. They are efficient for long-term visibility and can reveal scanning, unusual data transfer, beaconing, and unexpected communication relationships. Flow analysis usually cannot show the exact application content or prove what was transmitted.

The two methods complement each other. Flow data may identify a suspicious host and time interval; targeted packet capture may then clarify the protocol behaviour.

5. Baselines and Anomaly Detection

A baseline describes expected activity for a system, user, network segment, or service. Examples include normal login times, typical DNS volume, usual outbound destinations, average data transfer, and normal administrative actions.

A baseline must account for legitimate variation. A university network may have different patterns during examinations, holidays, and registration periods. Baselines should therefore be segmented by time, asset role, user population, and business process where appropriate.

5.1 Simple statistical measures

For observations x₁, x₂, ..., x_n, the arithmetic mean is:

Here, μ is the mean, x_i is observation i, and n is the number of observations. All observations must use the same unit and comparable measurement conditions.

A sample standard deviation is:

Here, x̄ is the sample mean, s has the same units as the observations, and n − 1 provides the usual sample correction when estimating population variability from a sample.

A simple standardized deviation, or z-score, is:

In this expression, x is a new observation, μ is the baseline mean, and σ is the baseline population standard deviation. The z-score is dimensionless. It is most informative when the baseline is stable and the distribution is approximately suitable for this comparison. A large absolute z-score is a reason to investigate, not automatic proof of an attack.

Worked example. A server normally sends 100 MB of outbound data per hour with a standard deviation of 20 MB. An hour with 180 MB produces z = (180 − 100)/20 = 4. This is notably above the established pattern. The analyst should examine the destination, process, user, and business context before concluding that exfiltration occurred.

Mean-based methods can be distorted by outliers. Median and percentile thresholds may be more robust for highly skewed data. Baselines also become unreliable after major infrastructure or business changes and must be recalibrated carefully rather than immediately treating all new behaviour as malicious.

6. Threat Intelligence and Indicators of Compromise

Threat intelligence transforms information about threats into knowledge that supports decisions. It may be strategic, describing campaigns and risks; operational, describing adversary activity; tactical, describing techniques and procedures; or technical, describing observables such as hashes and domains.

Useful intelligence answers questions such as:

• Who or what is associated with the indicator?
• How was the indicator obtained and when?
• How reliable is the source?
• What infrastructure, malware, or technique is involved?
• What action is appropriate in this environment?
• When should the indicator expire or be reviewed?

Threat-intelligence platforms can distribute indicators to SIEM, endpoint, DNS, firewall, and email controls. Automated blocking should be conservative. A low-confidence or shared indicator may cause availability problems or block legitimate users. Indicators should be scored, scoped, time-limited where appropriate, and linked to an observable investigative hypothesis.

The MITRE ATT&CK knowledge base is commonly used to describe adversary tactics and techniques. Mapping an alert to a technique can improve investigation and identify coverage gaps, but ATT&CK mapping does not itself establish that an intrusion occurred.

7. Metrics for Monitoring and Response

Metrics should support decisions rather than reward superficial activity. Counting alerts alone can encourage excessive, low-quality detection. Useful metrics include coverage, data quality, timeliness, effectiveness, and workload.

7.1 Detection and classification measures

When a detection is compared with a labelled set of events:

Here, TP is true positives, FP is false positives, and FN is false negatives. Precision measures the proportion of generated positives that are correct. Recall measures the proportion of actual positives that are detected. These measures require a meaningful reference set; in real security operations, the number of undetected incidents is often uncertain.

The false-positive rate is:

where TN is true negatives. It is the proportion of actual benign cases incorrectly classified as positive. A low false-positive rate does not guarantee good security if recall is poor.

Worked example. Suppose a reviewed dataset contains 40 confirmed malicious events and 160 benign events. A rule detects 30 malicious events and incorrectly alerts on 20 benign events. Then TP = 30, FN = 10, FP = 20, and TN = 140. Precision is 30/(30+20) = 0.60, or 60%. Recall is 30/(30+10) = 0.75, or 75%. The rule detects three quarters of the malicious events, but two out of every five generated alerts are false positives.

7.2 Operational metrics

Mean time to detect (MTTD) measures the average time between the relevant activity and its detection. Mean time to respond (MTTR) is used in different organizations with different definitions; it may mean time to begin investigation, contain an incident, or restore service. The definition must be stated before comparison.

For a set of n incidents, mean time to respond can be represented as:

where t_i is the response time for incident i, measured in a consistent unit. Averages can hide severe delays, so medians and high percentiles should also be considered.

Other useful measures include the percentage of critical assets sending required logs, percentage of alerts reviewed within service-level targets, rule coverage for important attack techniques, unresolved alert backlog, and the age of data at the time of analysis. Metrics should be interpreted with context, including staffing, event volume, business seasonality, and changes in logging coverage.

8. Applied Case: Illustrative University Account Compromise

Background

This fictional case concerns Northbridge University, which uses a cloud identity provider, VPN, endpoint monitoring, DNS logging, and a SIEM. The security team receives frequent authentication alerts but has not correlated identity events with endpoint and DNS data.

Problem

An attacker obtains a staff member’s password through phishing. The attacker signs in through the VPN, accesses a research file service, and attempts to create a forwarding rule in the university email system. A single failed-login alert is generated, but it is not escalated because the source address has been seen previously.

Analysis

An analyst reviews a six-hour window and finds:

1. Six failed VPN logins followed by a successful login from an unusual country.
2. The account had no previous activity during that local time period.
3. The VPN session accessed a file service that the user had not used previously.
4. DNS logs show requests to a newly registered domain shortly after the login.
5. The cloud audit log records an attempted email-forwarding rule.

No single event proves compromise. Together, the sequence has stronger evidential value. The unusual location is not sufficient by itself because travel and VPN routing can create false positives. The combination of authentication anomalies, new resource access, suspicious DNS activity, and a persistence attempt increases confidence.

Approach and solution

The team disables active sessions, resets the password, requires multifactor authentication re-enrolment, removes the forwarding rule, and contacts the user through a trusted channel. It preserves identity, VPN, DNS, cloud, and endpoint records. The suspicious domain is investigated and shared with relevant controls after confidence and scope are assessed.

The SIEM rule is improved to correlate repeated failures, unusual successful authentication, new resource access, and persistence-related cloud changes. A risk score is used for prioritization, but human review remains required. The team also tests whether the alert still works during legitimate international travel.

Outcome

The organization finds no evidence that the research files were downloaded, but it confirms unauthorized access to the account. The incident is classified as a contained account compromise. The review identifies a logging gap: cloud audit events were retained but not connected to the identity alert workflow.

Learning points

• Alerts must be correlated across controls and time.
• Context such as asset importance and user history affects priority.
• An IOC is evidence to investigate, not an automatic conclusion.
• Response actions should preserve evidence and minimize unnecessary disruption.
• Detection improvements should be tested against both attack and benign scenarios.

9. Limitations, Ethics, and Governance

Monitoring can expose personal information, communications, location data, and employee behaviour. Organizations should establish a lawful purpose, collect only necessary data, restrict access, document retention, and apply role-based permissions. Sensitive fields may require masking or tokenization. Analysts should not inspect unrelated personal content merely because a technical system makes it accessible.

Governance should define who owns each data source, who may search it, how incidents are escalated, how evidence is preserved, and how records are deleted. Legal requirements differ by jurisdiction and sector; privacy, employment, communications, and breach-notification obligations may all apply.

AI-assisted analytics can summarize alerts, identify relationships, and suggest queries. However, AI output may be incorrect, biased by incomplete data, or vulnerable to manipulation. Sensitive logs should not be submitted to an unapproved service. Human analysts must validate important conclusions, retain supporting evidence, and understand how automated actions are triggered.

Operational limitations include encrypted traffic, missing telemetry, log flooding, attacker evasion, shared infrastructure, alert fatigue, and dependence on vendor-specific formats. Monitoring should therefore be treated as one layer of defence, complemented by secure configuration, prevention, identity controls, backups, vulnerability management, and practiced incident response.

10. Best-Practice Implementation Sequence

A certificate-level implementation can proceed in the following order:

1. Identify critical assets, accounts, services, and likely attack paths.
2. Define required events and owners for each important source.
3. Synchronize clocks and establish secure log transport.
4. Centralize priority logs and monitor collection health.
5. Normalize fields and preserve original records.
6. Create a small number of high-value detections before adding broad rules.
7. Enrich events with asset, identity, vulnerability, and intelligence context.
8. Establish normal patterns and document expected exceptions.
9. Test alerts using controlled simulations and benign scenarios.
10. Measure detection quality, response time, coverage, and analyst workload.
11. Review rules, retention, permissions, and privacy controls regularly.

Chapter Summary

Security monitoring uses logs, network flows, alerts, and indicators of compromise to support detection and response decisions. Effective monitoring begins with purposeful collection, synchronized time, reliable parsing, secure transport, appropriate retention, and protection against tampering. SIEM systems add value by normalizing, enriching, correlating, and prioritizing evidence, but alerts require human interpretation.

Packet analysis provides detailed protocol evidence, whereas flow analysis provides efficient long-term visibility. Baselines describe expected behaviour and can support anomaly detection, but statistical deviations are investigative signals rather than proof of maliciousness. Threat intelligence becomes useful when its source, confidence, age, scope, and recommended action are understood.

Metrics such as precision, recall, false-positive rate, MTTD, and MTTR help evaluate monitoring, provided their definitions and limitations are explicit. Finally, privacy, access control, retention, data quality, AI governance, and legal requirements are integral to responsible security analytics.

Key Terms

Log: A recorded representation of one or more system or application events.

Event: An observable occurrence, such as a login or process start.

Network flow: A summary of communication between network endpoints.

Alert: A notification generated by a security control or analytical rule.

Indicator of compromise: An observable associated with suspected malicious activity.

SIEM: A platform that collects, searches, correlates, and manages security events.

Normalization: Mapping different source formats into common fields and meanings.

Enrichment: Adding contextual information to an event.

Correlation: Combining related events to produce a more meaningful detection.

Baseline: A representation of expected activity for a system, user, or service.

Anomaly: Activity that differs significantly from an established pattern.

Packet capture: A recording of network packets for analysis.

Threat intelligence: Evaluated information about threats used to support decisions.

Precision: The proportion of generated positive classifications that are correct.

Recall: The proportion of actual positive cases that are detected.

False positive: A benign event incorrectly identified as suspicious.

False negative: A malicious or relevant event that is not detected.

MTTD: Mean time to detect relevant activity or an incident.

MTTR: Mean time to respond, subject to an explicitly defined response endpoint.

Alert fatigue: Reduced analyst effectiveness caused by excessive or low-quality alerts.

Review Questions

Short Answer Questions

1. What is the difference between a log, a flow, an alert, and an IOC?
2. Why is time synchronization important for security investigations?
3. State two advantages of centralized logging.
4. What is the main difference between packet analysis and flow analysis?
5. Why should a large z-score be treated as an investigation signal rather than proof of an attack?

Long Answer Questions

1. Explain the stages of a SIEM workflow from collection through response and learning.
2. Describe how an organization should design a risk-based log-collection strategy.
3. Compare signature-based and behavioural detections, including their principal strengths and limitations.
4. Explain how threat intelligence can be integrated with identity, endpoint, DNS, and firewall controls.
5. Discuss the privacy, governance, and operational risks associated with extensive security monitoring.

Analytical / Application Questions

1. A rule produces 25 true positives, 15 false positives, 5 false negatives, and 155 true negatives. Calculate its precision, recall, and false-positive rate, and explain what the results imply for an analyst team.
2. A workstation normally generates 50 DNS requests per hour with a standard deviation of 10. It generates 95 requests in one hour. Calculate the z-score and identify at least three contextual checks that should be performed before escalation.
3. Design a correlation rule for detecting possible account compromise using authentication, VPN, cloud audit, and DNS data. Specify the events, useful enrichment, likely false positives, and an appropriate response process.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a network flow from a packet capture?

A. A flow summarizes communication, while a packet capture can contain detailed packet headers and payloads.

B. A flow always contains more application content than a packet capture.

C. A packet capture records only source and destination IP addresses.

D. A flow can be created only from encrypted traffic.

Correct Answer: A. A flow summarizes communication, while a packet capture can contain detailed packet headers and payloads.

Question 2

What is the primary purpose of normalization in a SIEM?

A. To delete all raw logs after parsing

B. To map different source formats into common fields and meanings

C. To encrypt every network packet

D. To replace analysts with automated decisions

Correct Answer: B. To map different source formats into common fields and meanings

Question 3

A baseline has a mean of 100 MB and a standard deviation of 20 MB. What is the z-score for an observation of 160 MB?

A. 2

B. 3

C. 4

D. 8

Correct Answer: B. 3

Question 4

If a detection system has 30 true positives and 20 false positives, what is its precision?

A. 40%

B. 50%

C. 60%

D. 75%

Correct Answer: C. 60%

Question 5

Which action is most appropriate when a low-confidence IP indicator matches a shared cloud service?

A. Block it permanently without investigation

B. Treat the match as proof of compromise

C. Evaluate context, confidence, scope, and age before taking action

D. Delete the related logs to prevent false positives

Correct Answer: C. Evaluate context, confidence, scope, and age before taking action

Question 6

Which log source is most directly useful for investigating cloud privilege changes?

A. Cloud control-plane audit log

B. Printer status log

C. Local monitor brightness log

D. Keyboard layout log

Correct Answer: A. Cloud control-plane audit log

Question 7

What does recall measure in a labelled detection dataset?

A. The proportion of generated alerts that are correct

B. The proportion of actual positive cases that are detected

C. The number of analysts assigned to an alert

D. The storage required for archived logs

Correct Answer: B. The proportion of actual positive cases that are detected

Question 8

Why can a centralized SIEM become a security risk?

A. It prevents any correlation between events

B. It creates a high-value target and concentration of sensitive data

C. It makes time synchronization impossible

D. It removes the need for access controls

Correct Answer: B. It creates a high-value target and concentration of sensitive data

Question 9

Which sequence best represents a useful monitoring workflow?

A. Respond, collect, delete, normalize, ignore

B. Collect, normalize, enrich, correlate, investigate, respond

C. Block, archive, collect, publish, disable

D. Encrypt, forget, alert, reinstall, collect

Correct Answer: B. Collect, normalize, enrich, correlate, investigate, respond

Question 10

Why should MTTR be defined explicitly before comparing organizations?

A. MTTR has no relationship to incident handling

B. Different organizations may use different response endpoints in the metric

C. MTTR can be calculated only from packet payloads

D. MTTR always means time to restore every affected system

Correct Answer: B. Different organizations may use different response endpoints in the metric