Network Architecture, Protocols and Traffic Analysis

Introduction and Context

Modern organisations depend on networks to deliver applications, exchange information, operate cloud services and support remote work. A network is therefore both an enabling infrastructure and a major security boundary. Understanding how devices communicate is essential for identifying misconfiguration, diagnosing faults and recognising malicious activity.

This chapter examines network architecture, the OSI and TCP/IP models, addressing and subnetting, switching and routing, common infrastructure protocols, and packet analysis. The emphasis is analytical: a security practitioner should be able to move from an observed packet to an explanation of the service, device, protocol behaviour and possible security significance.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the OSI and TCP/IP models and map protocols to their layers.
2. Interpret IPv4 and IPv6 addresses, prefixes and subnet masks.
3. Calculate basic IPv4 subnet capacity and address ranges.
4. Distinguish switching, routing, ports and transport connections.
5. Explain the operation and security implications of DNS, DHCP, ARP, HTTP/S and email protocols.
6. Capture and filter traffic using tools such as Wireshark and tcpdump.
7. Identify normal, anomalous and potentially malicious traffic while recognising the limits of packet-based evidence.

1. Network Architecture and Layered Communication

A network architecture describes how end systems, intermediary devices, media, addressing schemes and services work together. A small office may contain hosts, wireless access points, switches, a router or firewall, a DHCP service and an internet connection. An enterprise commonly adds VLANs, redundant links, identity services, load balancers, proxies, intrusion detection systems and cloud connectivity.

Communication is usually represented as a layered process. An application creates data; transport protocols provide delivery between processes; network protocols provide delivery between hosts; and link and physical technologies move frames and signals across each local segment. At the destination, the process is reversed. Each layer adds a header or other control information. This process is called encapsulation; removal at the receiver is decapsulation.

Layering provides modularity. Web applications can operate over different physical media, while network engineers can replace a link technology without redesigning the application. It also supports troubleshooting: a failed DNS resolution, for example, should not immediately be treated as a physical-layer failure.

The OSI Reference Model

The seven-layer Open Systems Interconnection (OSI) model is a conceptual reference model:

The OSI model is useful for reasoning, but real protocol stacks do not always correspond to one layer exactly. TLS, for example, is often placed between the application and transport layers, although its functions span presentation and session concerns.

The TCP/IP Model

The TCP/IP model is more closely aligned with deployed internet protocols. Its commonly used four-layer form is:

The TCP/IP application layer combines several OSI layers. The Internet layer provides best-effort packet delivery; reliability, ordering and flow control, when required, are normally provided by TCP rather than IP.

A protocol stack should not be confused with a security boundary. Attackers can exploit weaknesses at any layer, and a single event can involve several layers. For example, a malicious web request uses an application protocol, a TCP connection, an IP route and an Ethernet or wireless frame.

2. Addressing, Naming and Subnetting

MAC, IP and Port Addresses

A MAC address identifies a network interface at the local link layer. An IP address identifies a logical interface for communication across interconnected networks. A port identifies an application endpoint associated with a transport protocol. These identifiers serve different purposes:

• A MAC address supports local-frame delivery.
• An IP address supports host-to-host routing.
• A port supports process-to-process delivery.

A connection is often described by a five-tuple: source IP, destination IP, transport protocol, source port and destination port. This tuple is useful in firewall rules, flow records and incident investigations.

IPv4 Addressing

IPv4 uses 32 bits, commonly written as four decimal octets. A prefix length, such as /24, identifies how many leading bits represent the network. The remaining bits identify addresses within that network.

For a conventional IPv4 subnet, the total number of addresses is:

where h is the number of host bits. In a traditional subnet with h ≥ 2, the usable host count is:

The subtraction excludes the all-zero network address and the all-one directed-broadcast address. This rule is not applied blindly: /31 point-to-point links and /32 host routes are important special cases, and modern network designs may use other mechanisms.

For prefix length p in IPv4:

The subnet mask contains p one-bits followed by h zero-bits. For /24, h = 8, so there are 256 total addresses and normally 254 usable host addresses.

#### Worked Example: Subnetting

An organisation owns 192.168.10.0/24 and requires four equal-sized subnets. Borrowing two host bits creates a /26 prefix because /24 + 2 = /26. Each subnet has six host bits:

Normally, usable hosts per subnet are:

The ranges are:

The block size is 64 addresses, so each network boundary occurs at multiples of 64 in the final octet. Subnetting reduces broadcast scope and can support security segmentation, but it does not itself replace firewall policy or identity-based access control.

IPv6 Addressing

IPv6 uses 128-bit addresses written in hexadecimal groups. It provides a vastly larger address space, hierarchical prefixes, link-local addresses and mechanisms such as neighbour discovery. IPv6 has no broadcast; multicast and neighbour-discovery procedures perform related functions. Security analysis must therefore include IPv6 even when an organisation believes it operates only IPv4, because hosts may have automatically configured IPv6 connectivity.

Public, Private and Special Addresses

Common private IPv4 ranges are 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. They are not globally routed and often use network address translation (NAT) for internet access. NAT can reduce direct exposure but is not a complete security control. Logging, segmentation and explicit filtering remain necessary.

3. Switching, Routing and Transport

Switching and VLANs

An Ethernet switch learns source MAC addresses and associates them with ports. When a destination MAC is known, it forwards a frame to the corresponding port; otherwise, it floods within the relevant broadcast domain. A switch usually separates collision domains, but ordinary Layer 2 switching does not necessarily separate broadcast domains.

A VLAN logically divides a physical switching infrastructure into broadcast domains. Traffic between VLANs requires a Layer 3 device and can be filtered by a router or firewall. Security risks include VLAN hopping, unauthorised trunk configuration, rogue devices and poorly controlled management interfaces.

Routing

A router forwards packets between IP networks using a routing table. Routes may be connected, statically configured or learned through routing protocols. When multiple routes match, the longest-prefix match is generally preferred; administrative preference and metric then influence selection among comparable routes.

Routing permits scale, but an incorrect route can cause outage, interception or traffic leakage. Security controls include route authentication where supported, filtering of route advertisements, anti-spoofing controls and monitoring for unexpected path changes.

TCP and UDP

TCP is connection-oriented and provides ordered, reliable byte-stream delivery using sequence numbers, acknowledgements, retransmission and flow control. A simplified TCP connection begins with a three-way handshake: SYN, SYN-ACK and ACK. A large number of incomplete handshakes may indicate scanning, a denial-of-service attempt or a poorly functioning client, although context is essential.

UDP is connectionless and provides datagrams without built-in delivery, ordering or congestion recovery. DNS, DHCP, streaming and many real-time applications use UDP, while some protocols can use either UDP or TCP. UDP traffic is not inherently unsafe; its security significance depends on the service, source, destination and observed behaviour.

Ports range from 0 to 65,535. Well-known ports are conventionally associated with services, but a port number is not proof of the application running there. Malware can use port 443, and legitimate administrators can operate services on non-standard ports. Identification should combine port information with payload metadata, TLS information, process data and endpoint context.

4. Core Network and Application Protocols

ARP and Neighbour Discovery

Address Resolution Protocol (ARP) maps an IPv4 address to a MAC address on a local network. A host broadcasts an ARP request and the owner of the address replies, usually unicast. ARP has no built-in authentication, making poisoning possible. An attacker may send false replies that associate the attacker’s MAC address with a gateway IP, enabling interception or disruption.

Defences include switch inspection features, static bindings in limited environments, segmentation, encrypted application protocols and monitoring for conflicting IP-to-MAC mappings. IPv6 uses neighbour discovery rather than ARP, with different but related attack considerations.

DHCP

Dynamic Host Configuration Protocol automatically supplies configuration such as IP address, subnet mask, default gateway and DNS server. The common IPv4 exchange is DORA: Discover, Offer, Request and Acknowledgement. DHCP is normally local-segment traffic and may use relay agents when the server is on another network.

A rogue DHCP server can provide a malicious gateway or DNS server. Controls include DHCP snooping, trusted switch ports, port security and monitoring of lease and option changes. DHCP logs are valuable for mapping an IP address observed in a packet capture to a device at a particular time.

DNS

The Domain Name System maps names to records, including A and AAAA address records, MX mail-exchange records, NS name-server records and TXT text records. A client commonly queries a recursive resolver, which obtains answers from authoritative servers and caches them according to time to live (TTL).

DNS commonly uses UDP port 53, while TCP may be used for large responses, zone transfers and some operational conditions. DNS over HTTPS (DoH) and DNS over TLS (DoT) encrypt resolver traffic, improving privacy but potentially reducing visibility for traditional monitoring systems.

Security issues include cache poisoning, unauthorised zone transfers, domain generation algorithms, DNS tunnelling and malicious resolution. High query volume, unusually long labels, high entropy and repeated requests for non-existent domains can be indicators, but none is conclusive alone.

HTTP and HTTPS

HTTP is an application protocol based on requests and responses. A request includes a method, target, headers and possibly a body; a response includes a status code, headers and content. Common methods include GET, POST, PUT and DELETE. Status classes include 2xx success, 3xx redirection, 4xx client errors and 5xx server errors.

HTTPS is HTTP protected by TLS. TLS provides server authentication when certificates are correctly validated, confidentiality and integrity in transit. It does not guarantee that the website is trustworthy, that the endpoint is uncompromised or that the requested content is safe. Analysts can still examine IP addresses, ports, certificate metadata, timing, traffic volume and, in authorised environments, decrypted traffic.

Email Protocols

SMTP transfers email between clients and servers or between mail servers. Submission commonly uses authenticated, encrypted service ports such as 587; server-to-server SMTP commonly uses port 25. POP3 downloads mail, while IMAP synchronises mailboxes and folders. Exact deployment varies, and port numbers alone are not sufficient for identification.

Email security involves sender authentication and policy mechanisms such as SPF, DKIM and DMARC, alongside filtering, user awareness and endpoint protection. Phishing analysis should consider domains, authentication results, URLs, attachments, headers and business context.

5. Packet Capture and Traffic Analysis

Packet capture records frames observed by an interface or monitoring sensor. A capture point matters: a host capture may show traffic after local processing, while a switch mirror port, network tap or firewall capture provides a different perspective. Encryption and capture position limit what can be inferred.

Common tools include Wireshark for graphical inspection and tcpdump for command-line capture. Typical authorised commands include:

tcpdump -i eth0 -nn 'host 192.168.1.20 and port 53'

The options select an interface, suppress name resolution and apply a capture filter. In Wireshark, display filters are applied after capture; examples include:

dns
 tcp.flags.syn == 1 && tcp.flags.ack == 0
 ip.addr == 192.168.1.20
 http.request.method == "POST"

Capture filters and display filters are not interchangeable. A capture filter reduces collected traffic, whereas a display filter changes what is shown from an existing capture. Filters should be documented so that analysis is reproducible.

A Structured Analysis Method

1. Establish scope and authorisation. Confirm the system, time zone, interfaces and legal authority.
2. Build a baseline. Identify normal hosts, services, protocols, volumes and periodic behaviour.
3. Validate timestamps. Check clock synchronisation and account for capture delays.
4. Inspect conversations. Group packets into flows and review source, destination, ports, flags and byte counts.
5. Examine protocol behaviour. Look for failed handshakes, abnormal DNS patterns, suspicious HTTP methods, certificate anomalies or repeated authentication failures.
6. Correlate evidence. Compare packets with DHCP leases, DNS logs, firewall records, endpoint telemetry and authentication logs.
7. Form and test hypotheses. An unusual connection is a lead, not a verdict.
8. Preserve evidence. Record hash values, acquisition details, filters and analyst actions.

Normal, Anomalous and Malicious Traffic

Normal traffic conforms to expected architecture, timing, endpoints and service behaviour. Anomalous traffic deviates from an established baseline. Malicious traffic is traffic associated with unauthorised or harmful activity. These categories are not equivalent: unusual traffic may be legitimate, and sophisticated malicious traffic may resemble normal activity.

Useful indicators include:

• A workstation contacting an unfamiliar external country or autonomous system.
• Repeated connections to many sequential ports, suggesting scanning.
• A sudden increase in DNS queries or long encoded subdomains, suggesting possible tunnelling.
• ARP replies that repeatedly change the gateway’s MAC address.
• A server sending large volumes of outbound data at an unusual time.
• TCP retransmissions caused by congestion, faulty links or interference—not necessarily attack activity.

Analysts should avoid overreliance on signatures. Encryption, tunnelling, shared infrastructure, NAT and cloud services make attribution difficult. Statistical baselines can assist, but threshold selection creates false positives and false negatives. Machine-learning systems may prioritise flows, but they require representative data, careful validation and human review.

Applied Case: Illustrative University Network Investigation

Background

A university security team receives reports that several laboratory computers are slow and that the internal DNS service is generating unusually high query volume.

Problem

The team must determine whether the event is a network fault, a misconfigured application or a compromise involving command-and-control communication.

Analysis

A packet capture from a switch mirror port shows that three laboratory hosts generate thousands of DNS requests per hour. Many queried names contain long, apparently random labels under the same newly registered domain. The requests are distributed over UDP port 53 and occur at regular intervals. DHCP logs associate the IP addresses with specific devices, while endpoint logs show an unfamiliar executable launched shortly before the increase.

The analysts compare ordinary laboratory traffic with the affected hosts. The affected hosts have unusually high query entropy and repeated non-existent-domain responses. However, they do not treat these characteristics alone as proof of tunnelling: software updates and security tools can also create repetitive DNS patterns.

Approach and Solution

The team isolates the hosts through the access-control system, preserves packet captures and endpoint images, and blocks the domain at the authorised DNS resolver and web gateway. Malware analysis identifies a compromised browser extension using DNS queries to exchange encoded data. Credentials are reset, systems are rebuilt from trusted images and the extension is removed from the approved software catalogue. Monitoring is added for long labels, excessive NXDOMAIN responses and unexpected outbound DNS resolvers.

Outcome

The investigation combines packet evidence, DHCP records, endpoint telemetry and containment actions. No single packet proves compromise, but the correlated evidence supports a defensible conclusion and response.

Learning Points

1. A capture point determines what can be observed.
2. DNS anomalies require contextual analysis rather than automatic classification.
3. DHCP and endpoint logs can convert an IP address into an accountable device and user context.
4. Evidence preservation and documented filters improve investigative quality.
5. Segmentation and controlled DNS egress reduce the impact of similar incidents.

Challenges, Governance and Best Practices

Traffic monitoring can reveal personal communications, credentials, health information and employee behaviour. Organisations should define a lawful purpose, minimise collection, restrict analyst access, protect captures, establish retention periods and communicate monitoring policies. Where applicable, privacy and data-protection requirements may regulate interception and storage.

Best practices include using least privilege, synchronising clocks, protecting management protocols, disabling unnecessary services, applying secure configuration baselines, segmenting sensitive systems, using encrypted protocols, validating certificates, maintaining current asset inventories and regularly testing detection rules. Analysts must distinguish observation from attribution and report uncertainty honestly.

Contemporary networks increasingly use cloud services, software-defined networking, zero-trust architectures, encrypted DNS, HTTP/3 over QUIC and IPv6. These developments improve performance or privacy but alter visibility. Security teams should combine endpoint, identity, DNS, proxy, firewall, cloud-flow and packet data rather than assuming that a traditional perimeter sensor sees the complete transaction.

Chapter Summary

Network security analysis begins with understanding layered communication. The OSI model provides a conceptual vocabulary, while the TCP/IP model reflects deployed internet protocols. MAC addresses support local delivery, IP addresses support routing and ports identify transport endpoints. Subnetting divides address space and broadcast domains; switching forwards frames within local networks, while routing connects different IP networks.

ARP, DHCP and DNS support local addressing and naming but introduce spoofing, rogue-service and tunnelling risks. HTTP, HTTPS and email protocols must be interpreted through their methods, headers, encryption and authentication controls rather than port numbers alone. Packet capture tools enable detailed investigation, but capture location, encryption, time accuracy and incomplete context limit conclusions. Reliable analysis establishes a baseline, examines flows and protocol behaviour, correlates independent evidence and preserves records. Ethical monitoring requires authorisation, proportionality, privacy protection and controlled retention.

Key Terms

OSI model: Seven-layer conceptual model for network communication.

TCP/IP model: Practical protocol architecture used by the internet.

Encapsulation: Addition of protocol control information as data moves down a stack.

MAC address: Link-layer identifier of a network interface.

IP address: Logical address used for host-to-host communication and routing.

Port: Transport-layer identifier for an application endpoint.

Subnet: A logically divided IP network.

Prefix length: Number of leading bits identifying an IP network.

VLAN: Logical Layer 2 broadcast domain.

Switch: Device that forwards frames using link-layer information.

Router: Device that forwards packets between IP networks.

ARP: Protocol that resolves an IPv4 address to a local MAC address.

DHCP: Protocol that dynamically supplies host configuration.

DNS: Distributed system for resolving names and storing service records.

TCP: Reliable, ordered, connection-oriented transport protocol.

UDP: Connectionless datagram transport protocol.

TLS: Cryptographic protocol providing confidentiality and integrity in transit.

Packet capture: Recorded network frames or packets used for analysis.

Traffic baseline: Characterisation of expected network behaviour.

DNS tunnelling: Use of DNS queries or responses to transport concealed data.

Review Questions

Short Answer Questions

1. What is the main difference between a MAC address and an IP address?
2. Why does a traditional /26 IPv4 subnet normally provide 62 usable host addresses?
3. What is the purpose of a VLAN?
4. Why can a port number not conclusively identify an application?
5. Name two security risks associated with DHCP or ARP.

Long Answer Questions

1. Compare the OSI and TCP/IP models and explain why protocol-layer mapping is useful in incident analysis.
2. Explain how a router and a switch make forwarding decisions.
3. Describe the normal operation and security risks of DNS and DHCP.
4. Explain how TLS changes the evidence available to a packet analyst when investigating HTTPS traffic.
5. Develop a structured procedure for investigating suspected DNS tunnelling.

Analytical / Application Questions

1. A /24 network must be divided into eight equal-sized IPv4 subnets. Calculate the new prefix length, total addresses and normal usable hosts per subnet, and list the network addresses.
2. A workstation sends SYN packets to ports 22, 23, 80, 443 and 3389 on hundreds of hosts but receives few responses. Analyse at least three possible explanations and identify additional evidence required.
3. During an investigation, the gateway IP is repeatedly associated with different MAC addresses, while users report intermittent connectivity. Explain the likely protocol-level concern, propose containment actions and identify evidence that would distinguish spoofing from a legitimate network change.

Multiple Choice Questions with Answers

Question 1

Which TCP/IP layer contains IPv4 and IPv6?

A. Application

B. Transport

C. Internet

D. Link

Correct Answer: C. Internet

Question 2

A traditional IPv4 /26 subnet contains how many total addresses?

A. 32

B. 64

C. 128

D. 256

Correct Answer: B. 64

Question 3

Which device primarily forwards frames using MAC-address information?

A. Router

B. Switch

C. DNS resolver

D. DHCP server

Correct Answer: B. Switch

Question 4

What is the normal purpose of ARP?

A. Resolve an IPv4 address to a local MAC address

B. Assign a domain name to a mail server

C. Encrypt an HTTP session

D. Select an internet route

Correct Answer: A. Resolve an IPv4 address to a local MAC address

Question 5

Which sequence represents the common DHCP IPv4 exchange?

A. Request, Discover, Acknowledge, Offer

B. Discover, Offer, Request, Acknowledge

C. Offer, Discover, Acknowledge, Request

D. Acknowledge, Request, Offer, Discover

Correct Answer: B. Discover, Offer, Request, Acknowledge

Question 6

Which statement about HTTPS is correct?

A. It guarantees that the website owner is trustworthy

B. It encrypts HTTP traffic using TLS when correctly configured

C. It always uses UDP port 53

D. It prevents endpoint compromise

Correct Answer: B. It encrypts HTTP traffic using TLS when correctly configured

Question 7

Which observation is most suggestive of possible DNS tunnelling?

A. One normal A-record lookup during login

B. Long, high-entropy labels queried repeatedly at regular intervals

C. A DHCP acknowledgement after a lease request

D. A single TCP retransmission during congestion

Correct Answer: B. Long, high-entropy labels queried repeatedly at regular intervals

Question 8

What does a TCP SYN packet normally indicate?

A. A request to begin a TCP connection

B. A completed DNS transfer

C. A DHCP lease expiration

D. An encrypted email attachment

Correct Answer: A. A request to begin a TCP connection

Question 9

What is the main difference between a capture filter and a display filter in Wireshark?

A. A capture filter reduces collected traffic; a display filter limits displayed packets

B. A capture filter decrypts TLS; a display filter encrypts it

C. A capture filter applies only to DNS; a display filter applies only to HTTP

D. There is no difference

Correct Answer: A. A capture filter reduces collected traffic; a display filter limits displayed packets

Question 10

Which evidence most directly helps map an observed historical IP address to a device?

A. A DHCP lease record

B. A web browser bookmark

C. A screen resolution setting

D. A keyboard layout

Correct Answer: A. A DHCP lease record