13. Network Management, Performance and Analytics

Introduction and Context

A computer network is successful only when it delivers reliable, secure, and appropriately responsive communication. Designing routers, switches, wireless systems, and links is therefore only the beginning of network administration. Once infrastructure is deployed, an organisation must determine whether services are available, whether performance meets user and application requirements, where failures originate, and when additional capacity or architectural change is justified. These activities constitute network management and performance engineering.

Modern networks produce evidence through many channels: device counters, Simple Network Management Protocol (SNMP) measurements, streaming telemetry, system logs, packet captures, flow records, application traces, and synthetic tests. The challenge is not merely collecting data. It is selecting suitable measurements, interpreting them in context, correlating events across layers, distinguishing symptoms from causes, and converting technical findings into operational and managerial decisions.

For a data science professional, network management provides an important analytical domain. Network observations are time-series, event, graph, and high-volume transactional data. Their analysis supports anomaly detection, capacity planning, service-level management, incident response, and cost optimisation. This chapter presents an integrated approach to those activities.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the purposes and architectural principles of network management.

2. Compare SNMP, telemetry, logs, packet captures, flow records, and observability data.

3. Interpret key availability, performance, traffic, and reliability metrics.

4. Apply structured methods for troubleshooting and root-cause analysis.

5. Model traffic, forecast capacity, and assess quality of service.

6. Design meaningful service-level indicators and service-level objectives.

7. Evaluate dashboards, alerts, automation, and analytics in operational environments.

8. Discuss governance, privacy, security, and limitations of network monitoring.

From Device Monitoring to Network Observability

Traditional network management focused on polling devices for status and counters. A management station requested information from routers, switches, firewalls, and other devices, then displayed values such as interface utilisation, errors, temperature, and CPU load. This remains useful, especially for infrastructure with mature SNMP support, but polling alone can miss short-lived events and provides limited context about user experience.

Observability extends monitoring by asking whether the available evidence is sufficient to explain the internal state of a system. It combines metrics, logs, traces, events, topology, configuration state, and user or application experience. Monitoring commonly answers a predefined question—such as whether packet loss exceeds a threshold—whereas observability supports investigation of less predictable questions, such as why only one class of users experiences slow access to a particular service.

A useful operational model has four layers. The first is the data plane, where packets and flows are forwarded. The second is the control and management plane, where routing, configuration, and device state are maintained. The third is the service layer, consisting of applications and business transactions. The fourth is the experience layer, representing latency, availability, and quality perceived by users. An interface counter may be correct at the data-plane layer while a service remains unusable because of application delay or a failing dependency.

Network management is often described through the FCAPS framework: fault, configuration, accounting, performance, and security management. Fault management detects and resolves abnormal conditions. Configuration management controls device and service settings. Accounting management measures resource usage, often for chargeback or cost allocation. Performance management evaluates efficiency and service quality. Security management identifies unauthorised access, policy violations, and suspicious behaviour. Contemporary platforms integrate these functions rather than treating them as isolated activities.

The Evidence Sources and What They Reveal

SNMP and Device Counters

SNMP uses a manager, agents, and a Management Information Base (MIB). Each managed object is identified through an object identifier (OID). The manager may poll values using requests such as GET, while devices can send unsolicited notifications, traditionally called traps or informs.

SNMP is effective for stable, low-frequency measurements: interface octets, packet counts, error counters, link state, memory, environmental readings, and routing information. Rates must be calculated from counter differences over time. If an octet counter increases by 600 million bytes over 60 seconds, the average bit rate is approximately 80 Mb/s. Counter discontinuities, device restarts, counter wrapping, and inconsistent polling intervals must be handled carefully.

SNMP has limitations. Polling intervals may hide microbursts, high-cardinality data is inconvenient, and vendor-specific MIBs can complicate integration. An interface showing low average utilisation may nevertheless experience transient queues and packet loss. SNMP should therefore be regarded as one evidence source, not a complete representation of network health.

Streaming Telemetry

Streaming telemetry sends measurements from devices to a collector at configured intervals or when state changes occur. It can offer higher frequency, richer structured data, and lower dependence on repeated polling. Common implementations use model-driven schemas and transport protocols such as gNMI, often with data encoded using formats such as JSON or Protocol Buffers.

Telemetry is valuable for queue depth, per-class traffic, optical power, routing state, interface statistics, and rapidly changing operational conditions. Its advantages are accompanied by design responsibilities: sampling frequency, timestamp accuracy, retention, schema versioning, secure transport, and collector scalability. Excessive telemetry can itself consume device, link, and storage resources.

Logs and Events

Logs record discrete events, including interface transitions, authentication failures, routing adjacency changes, configuration commits, firewall decisions, and system errors. They are indispensable for reconstructing sequences of events. A useful log system preserves timestamps, source identity, severity, event type, and correlation identifiers where possible.

Logs are not automatically reliable evidence. Clock differences can produce a false sequence; duplicate messages may inflate event counts; and unstructured text complicates analysis. Network Time Protocol (NTP) or Precision Time Protocol (PTP), consistent time zones, structured logging, and central collection substantially improve their value.

Packet Captures

A packet capture records packets or selected packet metadata at an observation point. It can reveal protocol exchanges, retransmissions, TCP window behaviour, DNS responses, malformed frames, fragmentation, handshake failures, and application-layer interactions when payload access is lawful and technically possible.

Captures provide deep evidence but are expensive to store and analyse. They are usually collected selectively through filters, short time windows, sampling, or triggered capture. A capture point matters: observing a packet before a firewall, after a load balancer, or at an endpoint may produce different interpretations. Encryption limits payload inspection, although timing, sizes, addresses, and protocol metadata remain informative.

Flow Records

Flow records summarise conversations rather than storing every packet. A record may contain source and destination addresses, ports, protocol, start and end time, packet count, byte count, and interface information. NetFlow, IPFIX, and related technologies support traffic engineering, top-talker analysis, security investigation, and capacity planning.

Flows offer a useful middle ground between counters and packet captures. They are less costly than full captures and preserve communication relationships, but they cannot normally explain individual retransmissions or packet-level protocol errors. Sampling must be considered when estimating total traffic: a sampled flow system requires an appropriate expansion factor and careful validation.

Synthetic Tests, Traces, and User Experience

Active probes generate controlled transactions such as ping, DNS lookup, TCP connection, HTTP request, or application login. They measure availability and latency from defined locations. Distributed probes can identify regional or path-specific problems.

Application traces and real-user measurements connect network conditions to service outcomes. For example, a web transaction may decompose total response time into DNS, connection establishment, TLS negotiation, server processing, and content transfer. This prevents the network from being blamed for every slow application and enables joint analysis with application and platform teams.

Performance Measurement and Interpretation

Performance analysis should begin with a precise question. “Is the network slow?” is too vague. A better question specifies service, population, location, time period, and metric: “What caused the 95th-percentile transaction latency for remote users to exceed the objective during the 10:00–10:30 interval?”

Important metrics include:

• Availability: the proportion of time a service is usable. A simple form is available time divided by total measured time.
• Latency: elapsed time between defined events. Mean latency may conceal user impact, so median and tail percentiles are often more informative.
• Jitter: variation in packet delay, particularly significant for voice and interactive media.
• Packet loss: packets not delivered or discarded. Loss may arise from congestion, faulty interfaces, wireless interference, filtering, or route instability.
• Throughput: successfully delivered data per unit time. It is distinct from link capacity.
• Utilisation: the proportion of configured capacity being used. High utilisation is not automatically failure, while low average utilisation does not rule out microbursts.
• Error and discard rates: physical, data-link, queue, and policy-related abnormalities.
• Retransmission rate: a symptom of loss, reordering, congestion, or endpoint behaviour.
• Saturation and queue depth: evidence that demand temporarily exceeds forwarding or scheduling capacity.

Percentiles are central to performance analysis. If the 50th percentile latency is 20 ms and the 99th percentile is 800 ms, the typical transaction is acceptable but a significant tail experiences severe delay. Averages alone would obscure this distinction. Analysts should also examine distributions, time series, seasonality, correlations, and changes after configuration or topology events.

Correlation is not causation. High CPU and packet loss may both be consequences of a traffic spike, or CPU may be unrelated because forwarding is hardware-accelerated. A disciplined analyst aligns timestamps, compares control and data paths, establishes a baseline, and tests competing explanations.

A Structured Troubleshooting Method

Effective troubleshooting moves from impact to evidence rather than from favourite device to favourite explanation. First define the symptom: who is affected, which service, from where, when, and how severely. Next verify whether the issue is reproducible and distinguish a complete outage from degradation.

The second stage establishes scope. Compare affected and unaffected users, paths, protocols, access technologies, sites, and service instances. Scope often reveals whether a fault is local, regional, application-specific, or systemic.

The third stage maps the service path. A logical path may include endpoint, wireless access point, access switch, distribution layer, firewall, WAN circuit, load balancer, server, database, and external dependency. Measurements should be collected at points that allow hypotheses to be separated. For example, packet loss before a firewall suggests a different investigation from loss only after the firewall.

The fourth stage correlates evidence. A routing adjacency event, an interface error increase, a flow shift, and an application-latency spike occurring within the same interval provide stronger evidence than any single alarm. Change records are particularly important: many incidents begin immediately after software, policy, route, or configuration changes.

The fifth stage tests the least disruptive hypothesis. Analysts may compare paths, perform controlled traces, inspect queue statistics, reproduce a transaction, or temporarily move traffic under an approved change procedure. Remediation should be reversible where possible. The final stage documents the root cause, contributing conditions, impact, detection quality, and preventive actions. Root cause is not simply “the link failed”; it may include insufficient redundancy, an ineffective alert, or an unsafe change process.

Traffic Modelling and Capacity Planning

Capacity planning links observed demand to future requirements and acceptable risk. The process begins by identifying demand drivers: users, sites, applications, backups, cloud services, replication, video, and seasonal events. Historical flows and counters are then cleaned for missing intervals, counter resets, sampling bias, and unit inconsistencies.

A basic time-series model can represent traffic as:

Traffic(t) = trend(t) + seasonality(t) + residual(t).

Trend captures long-term growth, seasonality captures recurring daily or weekly patterns, and residual represents irregular variation. Forecasts should be accompanied by uncertainty intervals rather than a single number. Capacity decisions should consider peak and tail behaviour, not only the mean. A link with a comfortable monthly average may be unsuitable if predictable peaks cause queueing.

A practical planning calculation estimates required capacity as forecast peak demand multiplied by a growth factor and a resilience or operating margin. The margin must reflect business criticality, burstiness, failover conditions, and provider commitments. Capacity must also be evaluated under failure scenarios: if one link or device is lost, can the remaining path carry essential traffic within the required performance objective?

Little’s Law, L = λW, relates average number of items in a system to arrival rate and average time in the system. In networking, it can support reasoning about queues and in-flight work, although its assumptions and measurement boundaries must be respected. Queueing models further show why latency can increase sharply as utilisation approaches saturation. Consequently, planning for 100 percent average utilisation is generally unsafe for interactive services.

Quality of Service and Service-Level Management

Quality of Service (QoS) controls how packets are classified, queued, scheduled, shaped, policed, and marked. A QoS policy should begin with business and application requirements, not with arbitrary device commands. Voice may require low delay and jitter; transactional systems may require predictable latency; bulk transfers may tolerate delay but consume substantial bandwidth.

Classification identifies traffic using addresses, ports, application identity, or trusted markings. Marking communicates treatment across administrative boundaries, but markings should not be trusted blindly. Queuing mechanisms such as priority, weighted fair queuing, and class-based scheduling allocate service. Policing limits traffic by discarding or remarking excess packets, while shaping buffers and smooths transmission. Both can cause side effects if configured without understanding burst size, interface rates, and application behaviour.

QoS cannot create bandwidth. It allocates scarcity and protects selected traffic at the possible expense of other classes. Poorly designed priority queues can starve ordinary traffic, and inconsistent markings across sites can create unpredictable treatment. Measurements must therefore include per-class latency, loss, queue drops, utilisation, and the effect on business transactions.

Service-level management translates technical measures into commitments. An SLA may specify availability, latency, packet loss, restoration time, or incident response. Internally, an SLI is the measured indicator, an SLO is the desired target, and an SLA is a formal commitment whose breach may have contractual or financial consequences. The measurement method, eligible time, exclusions, observation points, and calculation formula must be explicit. An availability target without a clearly defined service boundary is not operationally meaningful.

Analytics, Dashboards, and Automation

A useful dashboard is organised around decisions rather than around every available metric. An executive view may show service availability, major incidents, capacity risk, and SLA status. An operations view may show topology, active alerts, interface health, queue drops, and recent changes. An investigative view should support drill-down from service to site, device, interface, flow, and packet evidence.

Alert design requires attention to signal, noise, and context. Static thresholds are simple but may be unsuitable for seasonal traffic. Baselines, dynamic thresholds, rate-of-change rules, and anomaly detection can improve detection, but statistical novelty is not necessarily a service failure. Alerts should have an owner, severity, runbook, suppression logic, and escalation path. Alert fatigue is an organisational failure as much as a tooling failure.

Data science methods can support clustering of traffic patterns, classification of incident types, forecasting, graph-based dependency analysis, and anomaly detection. Machine learning should augment, not replace, operational reasoning. Models can be affected by missing data, concept drift, configuration changes, biased training sets, and feedback loops. Explainability and confidence indicators are essential when automated recommendations affect production systems.

Automation may collect diagnostics, enrich incidents with topology and change data, open tickets, or execute approved remediation. Closed-loop control—where a system detects a condition and changes configuration automatically—requires safeguards, validation, rate limits, rollback, audit trails, and human override. Automation that is fast but wrong can enlarge an incident dramatically.

Illustrative Case: Diagnosing Intermittent Slowness in a Regional Service

Background. A fictional financial-services organisation operates a central application accessed by branch offices over an SD-WAN environment. The service-level objective is that 99 percent of authenticated transactions complete within two seconds during business hours.

Situation. Users at three branches report intermittent slowness. The central dashboard shows normal average WAN utilisation and no sustained interface outage. The application team initially suspects overloaded servers.

Analysis. Real-user measurements show that the 95th-percentile transaction time increased from 1.1 to 3.8 seconds only for the three branches. Synthetic tests from the data centre remain normal. Flow records reveal short, intense bursts from a backup process at the affected sites. Telemetry shows queue drops on the WAN egress class during those bursts, while SNMP’s five-minute utilisation average remains below 60 percent. Packet captures from one branch show TCP retransmissions during the same periods. Server CPU and database latency are unchanged.

The evidence separates the problem from a server fault. Average utilisation concealed microbursts; queue drops and retransmissions explain the long transaction tail.

Approach. The organisation reschedules backups, assigns them to a lower-priority QoS class, applies appropriate shaping, and reserves capacity for interactive traffic. It also changes monitoring to include per-class queue drops, 95th- and 99th-percentile transaction latency, and burst-aware telemetry. A controlled test confirms that transaction latency returns within the objective while backup completion remains acceptable.

Outcome. The immediate user impact is removed without purchasing a larger circuit. The capacity review nevertheless identifies that failover capacity would be insufficient during a major site loss, leading to a separate resilience investment.

Learning points. Average utilisation is an inadequate sole capacity metric; application experience must be correlated with network evidence; QoS should reflect business priorities; and a local fix should not replace failure-scenario planning.

Risks, Governance, and Professional Practice

Monitoring creates risks as well as knowledge. Packet captures, DNS records, flow data, usernames, and logs may reveal personal information, confidential business relationships, or regulated data. Collection should follow data minimisation, purpose limitation, retention schedules, access control, encryption, and documented authorisation. Payload inspection requires particular care and may be prohibited or restricted by law, policy, or contract.

Management systems are high-value targets. SNMP community strings, telemetry credentials, log collectors, and automation APIs must be protected through secure protocols, least privilege, network segmentation, key rotation, and audit logging. Monitoring data must be integrity-protected because altered timestamps or counters can mislead investigations.

Common mistakes include monitoring only devices rather than services, relying on averages, ignoring clock synchronisation, collecting data without a retention purpose, treating every alert as equally urgent, changing production systems without evidence, and declaring root cause from correlation alone. Another frequent error is measuring what is easy rather than what matters to users. Governance should define ownership, escalation, evidence handling, access rights, and review of monitoring effectiveness.

Current Developments

Network operations are increasingly integrating telemetry with cloud platforms, application-performance monitoring, digital experience monitoring, and security analytics. Intent-based management expresses desired outcomes and evaluates whether network state conforms to policy. Programmable interfaces and infrastructure-as-code improve repeatability and enable configuration testing before deployment.

The boundary between network performance management and security operations is also becoming less distinct. A sudden flow change may indicate congestion, a routing event, or exfiltration. Correlating network, identity, endpoint, and application evidence can improve detection, but it increases governance and data-integration complexity. Edge computing, encrypted traffic, high-speed links, and distributed cloud services further increase the importance of metadata, endpoint instrumentation, and carefully designed sampling.

Best-Practice Principles

A mature network analytics practice should:

• define service-oriented indicators and measurable objectives;
• combine counters, telemetry, logs, flows, captures, and user experience data;
• maintain accurate time, topology, inventory, and change records;
• use percentiles, distributions, baselines, and failure scenarios rather than averages alone;
• design alerts with ownership, severity, runbooks, and review processes;
• validate sampling, counter behaviour, missing data, and measurement boundaries;
• protect monitoring data and collect no more sensitive information than necessary;
• automate repetitive, reversible tasks with approval, rollback, and audit controls; and
• perform post-incident learning that addresses both technical and organisational causes.

Chapter Summary

Network management transforms operational data into decisions about reliability, performance, security, and investment. SNMP provides established device measurements; streaming telemetry supplies richer and more frequent state; logs explain events; packet captures expose protocol detail; flow records describe communication patterns; and synthetic tests and application traces reveal service experience. No source is sufficient in isolation.

Performance analysis depends on appropriate metrics, percentiles, baselines, timestamp alignment, and correlation across layers. Structured troubleshooting moves from impact and scope to path mapping, evidence correlation, hypothesis testing, remediation, and learning. Traffic modelling separates trend, seasonality, and residual variation, while capacity planning considers peaks, uncertainty, growth, margins, and failures. QoS manages scarce resources but cannot create bandwidth. SLIs, SLOs, and SLAs connect measurement with organisational commitments. Dashboards, analytics, and automation improve operations only when they are explainable, governed, secure, and aligned with service outcomes.

Key Terms

SNMP: A protocol for monitoring and managing network devices through managed objects.

MIB: A structured collection of objects that a management system can read or modify.

Streaming telemetry: Continuous or event-driven transmission of structured operational measurements.

Network observability: The ability to infer system state and investigate causes using correlated operational evidence.

Flow record: A summary of a communication session containing endpoints, timing, protocol, and traffic quantities.

Packet capture: A recorded set of packets or packet metadata observed at a network point.

Latency: Time taken for a packet, transaction, or operation to move between defined events.

Jitter: Variation in packet or transaction delay.

Packet loss: Failure of packets to reach their intended destination.

Throughput: Successfully delivered data per unit time.

Utilisation: The proportion of available capacity being used.

Microburst: A short-duration traffic burst that may cause queueing despite acceptable averages.

QoS: Techniques for classifying, scheduling, shaping, policing, and prioritising traffic.

SLA: A formal service commitment with defined measures and consequences.

SLO: A target level for a service indicator.

SLI: A quantitative measurement of service performance.

Capacity planning: Forecasting demand and determining resources needed to meet performance and resilience goals.

FCAPS: A framework covering fault, configuration, accounting, performance, and security management.

Root-cause analysis: A systematic investigation of the underlying causes of an incident.

Alert fatigue: Reduced attention and response quality caused by excessive or low-value alerts.

Review Questions

Short Answer Questions

1. What information can flow records provide that interface counters usually cannot?

2. Why can average interface utilisation conceal a performance problem?

3. Distinguish an SLI, an SLO, and an SLA.

4. State two limitations of packet captures.

5. What is the purpose of clock synchronisation in network analysis?

Long Answer Questions

1. Compare SNMP, streaming telemetry, logs, packet captures, and flow records in terms of evidence, cost, and use cases.

2. Explain a structured troubleshooting workflow from reported symptom to verified root cause.

3. Discuss how traffic modelling and capacity planning should account for seasonality, uncertainty, burstiness, and failures.

4. Explain how classification, queuing, policing, and shaping operate within a QoS strategy.

5. Evaluate the role of dashboards, anomaly detection, and automation in network operations, including their risks.

Analytical / Application Questions

1. A network reports 45 percent average utilisation, but users experience periodic delays and queue drops. Identify the additional evidence you would collect and explain your analysis.

2. A service meets its monthly availability target but violates its latency objective for a small geographic region. Design an investigation using at least four evidence sources.

3. An organisation wants to inspect packet payloads continuously to improve troubleshooting. Assess the technical, operational, privacy, and governance implications and propose a safer monitoring design.

22. Multiple Choice Questions with Answers

Question 1

Which evidence source is most suitable for identifying the communicating endpoints and approximate volume of a large number of conversations without storing every packet?

A. Flow records

B. A temperature sensor

C. A configuration backup

D. A single interface status poll

Correct Answer: A. Flow records

Question 2

Why might five-minute average utilisation fail to reveal the cause of intermittent application slowness?

A. It cannot measure link capacity at all

B. It can conceal short traffic bursts that fill queues

C. It always exaggerates packet loss

D. It records only application-layer payloads

Correct Answer: B. It can conceal short traffic bursts that fill queues

Question 3

Which statement correctly distinguishes an SLI from an SLO?

A. An SLI is a contractual penalty and an SLO is a packet capture

B. An SLI is a measured indicator and an SLO is the target for that indicator

C. An SLI is a routing protocol and an SLO is a firewall rule

D. An SLI is a forecast and an SLO is a device counter

Correct Answer: B. An SLI is a measured indicator and an SLO is the target for that indicator

Question 4

A packet capture shows TCP retransmissions at the same time that telemetry reports queue drops on a WAN egress interface. What is the most reasonable interpretation?

A. The evidence is consistent with congestion-related loss

B. The server must be overheating

C. The routing table is necessarily empty

D. SNMP counters have become encrypted

Correct Answer: A. The evidence is consistent with congestion-related loss

Question 5

Which action best improves the reliability of event correlation across devices?

A. Increasing payload inspection on every packet

B. Disabling all low-severity logs

C. Synchronising clocks and preserving accurate timestamps

D. Replacing flow records with CPU measurements

Correct Answer: C. Synchronising clocks and preserving accurate timestamps

Question 6

What is the principal purpose of QoS shaping?

A. To smooth transmission by buffering traffic

B. To increase the physical capacity of a link

C. To encrypt all application payloads

D. To eliminate the need for capacity planning

Correct Answer: A. To smooth transmission by buffering traffic

Question 7

In the traffic model Traffic(t) = trend(t) + seasonality(t) + residual(t), what does seasonality represent?

A. Permanent equipment failure

B. Recurring patterns such as daily or weekly changes

C. The security classification of a packet

D. The physical length of a cable

Correct Answer: B. Recurring patterns such as daily or weekly changes

Question 8

Which dashboard design is most aligned with service-oriented network management?

A. Displaying every available counter without prioritisation

B. Showing only device temperatures

C. Linking service impact to sites, paths, devices, and recent changes

D. Displaying configuration commands without performance data

Correct Answer: C. Linking service impact to sites, paths, devices, and recent changes

Question 9

Why should machine-learning anomaly detection not be treated as proof of an incident?

A. Anomalies can result from legitimate changes, missing data, or model drift

B. Machine learning can measure only cable length

C. Anomaly detection always produces identical results

D. Models cannot process time-series data

Correct Answer: A. Anomalies can result from legitimate changes, missing data, or model drift

Question 10

Which practice is most appropriate when collecting packet captures that may contain sensitive information?

A. Capture all payloads indefinitely

B. Use authorised, minimised, access-controlled, and time-limited collection

C. Publish captures to all operations staff

D. Disable encryption before every capture

Correct Answer: B. Use authorised, minimised, access-controlled, and time-limited collection