Network Protocols, Traffic and Security Models

Introduction

Computer networks function through cooperating protocols. A protocol is a documented set of rules that defines how devices format, transmit, receive and interpret data. Security weaknesses often arise not because a protocol is “broken” in isolation, but because protocols were designed for connectivity and interoperability before confidentiality, authentication and attack resistance became central requirements.

A network-security practitioner must therefore understand both normal communication and abnormal communication. This requires relating the OSI reference model to the TCP/IP model, identifying addresses and ports, interpreting routing and name-resolution activity, and recognising how protocols such as ARP, DHCP, DNS, HTTP and SSH create opportunities for attack or control.

This chapter develops a practical foundation for analysing traffic. It treats security as a property of a complete communication system: endpoints, protocols, infrastructure, configuration, users and monitoring controls all contribute to the final security outcome.

Learning Objectives

After studying this chapter, you should be able to:

1. Relate the OSI layers to the TCP/IP model and explain their security relevance.
2. Distinguish MAC addresses, IP addresses, ports, sockets and protocol identifiers.
3. Explain how switching, routing, ARP, DHCP and DNS support network communication.
4. Describe common application protocols and their principal security weaknesses.
5. Analyse a basic traffic flow from source application to destination application.
6. Select appropriate controls, including encryption, authentication, segmentation, filtering and monitoring.
7. Explain how security models such as CIA, defence in depth and zero trust guide network protection.

1. Layered Communication and Security

1.1 The OSI reference model

The Open Systems Interconnection (OSI) model divides communication into seven conceptual layers:

The OSI model is a teaching and analysis framework rather than a precise description of every modern protocol. For example, encryption may be implemented at several layers: link encryption at Layer 2, IPsec at Layer 3, TLS above the transport layer, or application-level encryption at Layer 7.

1.2 The TCP/IP model

The TCP/IP model is closer to the protocol suite used in operational networks. A common four-layer representation is:

Some textbooks use a five-layer TCP/IP teaching model by separating the physical and data-link functions. Neither representation changes the operational principles.

Layering provides modularity. An application can use TCP without knowing how Ethernet encodes bits, while a router can forward an IP packet without understanding the application data. The same modularity also creates security boundaries: a firewall may filter transport ports, a web application firewall may inspect HTTP, and a switch may enforce local segmentation.

1.3 Encapsulation and decapsulation

When an application sends data, each lower layer adds control information. Application data becomes a transport segment or datagram, which becomes an IP packet, which is carried in a link-layer frame. At the destination, headers are removed in reverse order. This process is called decapsulation.

A simplified flow is:

Figure: Simplified encapsulation from application data to a link-layer frame.

Encapsulation helps traffic analysis. If a packet reaches the correct host but the expected application port is closed, the problem is likely above the network layer. If the host has no route, an application change will not solve the problem. Security analysts use this separation to locate faults and identify the layer at which an attack occurs.

2. Addressing, Ports and Traffic Identity

2.1 MAC addresses and local delivery

A Media Access Control (MAC) address identifies a network interface at the data-link layer. Ethernet switches use destination MAC addresses to select an outgoing interface within a local broadcast domain. A MAC address is not a universal identity proof: it can often be changed or impersonated by software.

MAC addresses are useful for local inventory, switch-port controls and incident investigation, but they should not be treated as strong authentication. Wireless networks and virtual machines further complicate the relationship between a device, its interface and its observed MAC address.

2.2 IP addresses and logical delivery

An IP address identifies a logical interface in an internetwork. IPv4 addresses contain 32 bits and are commonly written in dotted-decimal form, such as 192.0.2.10. IPv6 addresses contain 128 bits and are written in hexadecimal notation, such as 2001:db8::10. The ranges 192.0.2.0/24 and 2001:db8::/32 are documentation ranges and should not be used as real public addresses.

A subnet prefix determines which destinations are local and which require a router. For an IPv4 address and prefix length, the number of addresses in the full binary subnet is:

Here, N is the total number of IPv4 addresses in the subnet and p is the prefix length, from 0 to 32. In traditional IPv4 subnetting, two addresses are normally reserved for the network and directed broadcast addresses, so usable host addresses are:

This subtraction applies to ordinary IPv4 subnets where network and broadcast reservations are used; special cases such as /31 point-to-point links and /32 host routes require separate treatment.

For example, a /26 subnet contains 2^(32−26) = 64 total addresses and normally provides 62 usable host addresses. Security teams use subnet boundaries to define broadcast domains, firewall zones and access-control policies.

2.3 Ports, sockets and protocols

A port identifies a transport-layer service endpoint. TCP and UDP each have port numbers from 0 through 65,535. A socket is commonly described by an IP address, transport protocol and port; a connection is more precisely identified by a combination of source address, source port, destination address and destination port.

Well-known ports are conventionally associated with services: DNS commonly uses UDP or TCP port 53, HTTP uses TCP port 80, HTTPS uses TCP port 443, SSH uses TCP port 22, and DHCP uses UDP ports 67 and 68. Port numbers are conventions, not proof of service identity. Malware can operate on a conventional port, and legitimate software can use a non-standard port.

A listening service increases attack surface. Security controls should therefore restrict unnecessary services, enforce authentication, patch exposed software and monitor unexpected port use. Port scanning is not automatically malicious, but it is a common reconnaissance activity.

3. Switching, Routing and Traffic Flow

A switch forwards frames within a local network using a learned MAC-address table. A router forwards IP packets between networks using a routing table. The router normally removes the incoming link-layer frame and creates a new frame for the next link; the source and destination IP addresses generally remain unchanged, although network address translation may modify them.

A host decides whether a destination is local by comparing the destination address with its own address and subnet prefix. If the destination is remote, the host sends the frame to its default gateway. The gateway then makes a routing decision. Routing security depends on accurate route information, authenticated routing exchanges where supported, and controls against unauthorised route advertisement.

Network Address Translation (NAT) maps addresses or transport endpoints, often allowing private addresses to share public addresses. NAT can reduce direct inbound reachability, but it is not a replacement for a firewall: it does not guarantee authentication, correct application behaviour or protection from malicious outbound traffic.

3.1 Traffic direction and state

Security devices commonly classify traffic as inbound, outbound, lateral or local. A stateful firewall records connection state and can allow return traffic for an established outbound session while denying unsolicited inbound traffic. Stateless filtering evaluates individual packets without maintaining such context.

A basic flow description should identify:

• source and destination IP addresses;
• source and destination ports;
• transport protocol;
• direction and network zone;
• application protocol and expected behaviour;
• authentication and encryption status; and
• relevant DNS, DHCP and ARP events.

This information supports troubleshooting and detection. For example, a workstation contacting an unfamiliar external IP over TCP 443 may be legitimate web traffic, but the analyst should correlate the destination, DNS name, process, certificate, timing and user activity before drawing a conclusion.

4. ARP, DHCP and DNS

4.1 ARP and local address resolution

Address Resolution Protocol (ARP) maps an IPv4 address to a MAC address on a local network. A host broadcasts an ARP request asking which device owns an IP address; the owner replies with its MAC address. The result is stored temporarily in an ARP cache.

ARP has no built-in cryptographic authentication. An attacker can send forged ARP replies, causing victims to associate the gateway’s IP address with the attacker’s MAC address. This can support man-in-the-middle interception or denial of service.

Controls include switch inspection features, static or carefully managed bindings for critical systems, network segmentation, monitoring for inconsistent mappings and encryption at higher layers. Encryption such as HTTPS limits the usefulness of intercepted traffic, but it does not prevent disruption or all forms of manipulation.

IPv6 uses Neighbor Discovery rather than ARP. It has related local-link security concerns, although the messages and controls differ.

4.2 DHCP and configuration delivery

Dynamic Host Configuration Protocol (DHCP) automatically supplies configuration such as an IP address, subnet mask or prefix, default gateway and DNS server. In a common IPv4 exchange, a client broadcasts DHCP Discover, a server responds with Offer, the client sends Request, and the server returns Acknowledgement. This is often remembered as DORA.

DHCP normally uses UDP port 67 for servers and port 68 for clients. Because initial clients may not yet have an IP address, the early exchange uses broadcast or relay mechanisms. A rogue DHCP server may provide a malicious gateway or DNS server. DHCP starvation attempts to consume available address leases and can prepare the ground for a rogue server.

Controls include DHCP snooping on managed switches, trusted switch ports, port-security limits, relay protection, monitoring for duplicate or unexpected servers, and validation of assigned configuration. DHCP itself should not be considered a strong identity system.

4.3 DNS and name resolution

The Domain Name System translates names such as www.example.org into records containing addresses and other information. A recursive resolver obtains answers for clients, possibly using authoritative name servers. DNS commonly uses UDP port 53 for ordinary queries and may use TCP 53 for large responses, zone transfers or reliability requirements. Encrypted DNS transports, including DNS over TLS and DNS over HTTPS, protect portions of the communication path but introduce policy and visibility considerations.

DNS weaknesses include cache poisoning, unauthorised zone transfers, malicious domains, typosquatting, tunnelling and dependence on compromised resolvers. DNSSEC can provide origin authentication and integrity for signed DNS data, but it does not encrypt ordinary DNS queries and is not universally deployed.

Useful controls include protected recursive resolvers, response-policy filtering, restricted zone transfers, DNS logging, DNSSEC validation where appropriate, domain monitoring and endpoint detection. Analysts should remember that a domain name is not equivalent to a trustworthy organisation; domains, addresses and certificates must be assessed in context.

5. Common Application Protocols and Their Security Properties

TLS provides confidentiality, integrity and server authentication when correctly configured and when the client validates the certificate chain and name. It does not make a compromised endpoint trustworthy, prevent denial of service or guarantee that the user is interacting with a benevolent application.

Remote administration deserves particular attention. SSH is generally preferable to Telnet because it protects the session when properly configured. However, an exposed SSH service can still be attacked through password guessing, stolen keys or software vulnerabilities. Controls should include strong authentication, multi-factor authentication where feasible, restricted source networks, rate limiting, logging and timely updates.

6. Security Models for Network Protection

6.1 The CIA triad

The confidentiality, integrity and availability model provides a concise way to classify security objectives.

• Confidentiality prevents unauthorised disclosure. Encryption and access control support it.
• Integrity prevents unauthorised alteration. Digital signatures, hashes, authenticated protocols and change control support it.
• Availability ensures authorised users can obtain service. Redundancy, capacity planning, filtering and recovery procedures support it.

One control may support several objectives. TLS can protect confidentiality and integrity, while redundancy primarily supports availability. Security design should avoid protecting one objective by unnecessarily damaging another; excessive filtering, for example, may improve confidentiality but reduce availability.

6.2 Defence in depth

Defence in depth uses independent or partially independent controls at multiple layers. A typical design may combine physical security, switch segmentation, firewall policy, secure protocols, endpoint protection, identity controls, logging and tested recovery. Layered controls reduce dependence on one boundary and improve the chance that an attack is detected or contained.

6.3 Least privilege and zero trust

Least privilege grants only the access needed for an authorised task. In networking, this means limiting administrative interfaces, restricting east-west traffic, using narrowly defined firewall rules and avoiding shared credentials.

Zero trust is a security approach in which access is not granted solely because a user or device is inside a network perimeter. Requests are evaluated using identity, device condition, context, policy and resource sensitivity. Zero trust does not mean that every packet must be individually inspected or that traditional firewalls are obsolete. It means that implicit trust should be reduced and access decisions should be explicit, observable and continually reassessed.

7. Analysing a Complete Traffic Flow

Consider a user opening an HTTPS web application from a laptop. The sequence may be:

1. The laptop obtains configuration through DHCP or uses a previously assigned lease.
2. The user enters a domain name; the resolver returns an address through DNS.
3. The laptop checks whether the destination is local. If not, it resolves the gateway’s MAC address through ARP.
4. The laptop establishes a TCP connection to the server’s TCP port 443, unless another transport is used by the application.
5. The client and server negotiate TLS and authenticate the server certificate.
6. The browser sends an HTTP request inside the protected session.
7. The gateway and routers forward packets according to routing tables and policy.
8. Responses return through the network, and the browser validates and renders them.

A security analyst can inspect this flow at several points. DHCP logs may show an unexpected DNS server. DNS logs may show a newly registered or suspicious domain. ARP monitoring may show a gateway mapping change. Firewall logs may show denied or unusual connections. Endpoint telemetry may identify the process that created the connection. Packet inspection may reveal whether encryption is present, although encrypted payloads limit content visibility.

Traffic analysis must distinguish evidence from inference. A connection to TCP 443 indicates a port, not necessarily legitimate HTTPS. A large volume of DNS queries may indicate normal software updates, malware command-and-control, or DNS tunnelling. Context and correlation are essential.

8. Illustrative Case: Suspected Man-in-the-Middle Activity

Background

A small college network reports that several users receive certificate warnings when accessing the learning-management system. Some users also experience intermittent connectivity.

Problem

The network team must determine whether the issue is a server certificate problem, a routing fault, an ARP attack or user error.

Analysis

The team first confirms that the learning system’s certificate is valid when accessed from an independent trusted network. On the college LAN, ARP tables on affected computers show that the gateway IP is associated with different MAC addresses at different times. A switch log shows that one MAC address appears on a student-access port and later claims the gateway’s address. Packet captures show gratuitous ARP replies and intermittent TCP resets.

The evidence supports an ARP-spoofing incident. The certificate warnings occur because the attacker is attempting to intercept or redirect traffic; TLS validation is exposing the abnormal condition. The incident is not conclusively explained by a simple DNS error because the gateway mapping itself is unstable.

Approach and solution

The team disables the suspicious switch port, preserves logs and captures, and checks affected endpoints for malware. It enables DHCP snooping and dynamic ARP inspection on supported switches, separates student and administrative networks, restricts management access and verifies that administrative services require strong authentication. The team also reviews whether sensitive services enforce HTTPS and whether users have been trained not to bypass certificate warnings.

Outcome

Connectivity returns to normal, and certificate warnings cease. Monitoring is configured to alert on conflicting gateway mappings and unexpected switch-port changes. The team documents that encryption protected the confidentiality of properly validated sessions, but local interception still caused disruption and could have affected unprotected protocols.

Learning points

1. A protocol weakness may be reduced, but not eliminated, by controls at another layer.
2. Certificate validation is a valuable detection and protection mechanism.
3. ARP observations, switch logs and endpoint evidence are stronger together than in isolation.
4. Segmentation and access-layer controls are important parts of network security.

9. Challenges, Limitations and Governance

Protocol analysis is limited by encryption, incomplete logs, address translation, shared infrastructure and cloud services. NAT can obscure the original internal host; virtualisation can create rapidly changing interfaces; encrypted DNS can reduce traditional resolver visibility; and legitimate services may use common ports in unexpected ways.

Organisations should establish lawful monitoring policies. Packet capture may expose passwords, personal data, health information or confidential business content. Collection should have a defined purpose, appropriate authorisation, retention limits, access controls and secure disposal. In many jurisdictions, privacy and data-protection laws regulate monitoring and processing of identifiable network information. Incident responders should preserve evidence with documented procedures and restrict access to authorised personnel.

Artificial intelligence can assist with traffic classification, anomaly detection and alert prioritisation, but it does not replace protocol knowledge. Models can generate false positives, miss novel attacks, inherit biased training data or expose sensitive telemetry. Human review, explainable decisions, controlled data use and continuous validation remain necessary.

10. Best Practices for Diploma-Level Network Security Work

• Document normal addressing, naming, routing and service behaviour.
• Use segmentation to limit broadcast scope and lateral movement.
• Disable unnecessary services and restrict management interfaces.
• Prefer authenticated and encrypted protocols such as HTTPS, SSH and appropriately configured secure monitoring protocols.
• Apply patches and verify configurations rather than relying on port changes for security.
• Use least privilege, strong authentication and multi-factor authentication for sensitive access.
• Protect DHCP and ARP functions with managed-switch controls where available.
• Log DNS, DHCP, firewall, VPN, authentication and endpoint events, then synchronise time across systems.
• Investigate traffic by correlating addresses, ports, processes, identities and time.
• Test backups, incident procedures and recovery plans.
• Treat certificate warnings, unexpected DNS changes and unusual outbound traffic as signals requiring investigation.

Chapter Summary

Network protocols divide communication into layers that simplify design but create distinct security considerations. The OSI model provides a conceptual seven-layer framework, while the TCP/IP model groups the protocols used in operational networks. MAC addresses support local delivery, IP addresses support routed delivery, and transport ports identify service endpoints; none of these alone proves identity or trust.

Switches forward frames using MAC addresses, routers forward packets using IP routes, and NAT modifies address reachability without replacing security controls. ARP resolves local IPv4 addresses but is vulnerable to spoofing. DHCP automates configuration but can be abused by rogue servers. DNS supports name resolution but requires protection against poisoning, malicious domains, tunnelling and unauthorised transfers.

Application protocols differ in confidentiality, integrity and authentication properties. Secure versions and appropriate controls reduce risk, but encryption cannot correct compromised endpoints or guarantee availability. The CIA triad, defence in depth, least privilege and zero trust provide useful frameworks for selecting controls. Effective traffic-flow analysis correlates protocol behaviour with infrastructure, endpoint and identity evidence while respecting privacy and legal requirements.

Key Terms

ARP: Protocol that resolves an IPv4 address to a local MAC address.

Address: An identifier used for local, logical or service-level communication.

CIA triad: Confidentiality, integrity and availability security objectives.

DHCP: Protocol that supplies hosts with network configuration automatically.

DNS: Distributed system for resolving names and storing related records.

Encapsulation: Addition of protocol control information as data moves down layers.

Firewall: System that permits or blocks traffic according to policy.

IP address: Logical address used for communication across interconnected networks.

MAC address: Link-layer address associated with a network interface.

NAT: Translation of network addresses or transport endpoints.

OSI model: Seven-layer reference model for network communication.

Port: Transport-layer identifier for a service endpoint.

Routing: Selection of a path for packets between networks.

Segmentation: Division of a network into zones with controlled communication.

Socket: An endpoint described by addressing and transport information.

TCP/IP model: Practical layered model based on the Internet protocol suite.

TLS: Protocol framework providing cryptographic protection for network sessions.

Traffic analysis: Examination of communication metadata and content to understand behaviour.

Zero trust: Approach that avoids implicit trust and evaluates access explicitly.

Review Questions

Short Answer Questions

1. What is the principal difference between a MAC address and an IP address?
2. Why is a port number not reliable proof of the application using a connection?
3. State two security weaknesses of ARP.
4. What information does DHCP commonly provide to a host?
5. Distinguish confidentiality from integrity in the CIA triad.

Long Answer Questions

1. Compare the OSI and TCP/IP models and explain how layering assists security analysis.
2. Explain the complete process by which a host communicates with a remote HTTPS server, including DNS, ARP, routing, TCP and TLS.
3. Describe DNS threats and evaluate controls that can reduce them.
4. Explain defence in depth, least privilege and zero trust in a segmented enterprise network.
5. Discuss why encryption is important but insufficient as a complete network-security strategy.

Analytical / Application Questions

1. A workstation receives a gateway address through DHCP, but users later observe certificate warnings and changing gateway MAC addresses. Analyse the likely attack, identify evidence to collect and recommend controls.
2. A firewall log shows repeated outbound connections from one workstation to many external IP addresses over TCP 443. Explain why the port alone is insufficient for a conclusion and design a correlation-based investigation.
3. Design a secure network-monitoring approach for a college that balances DNS, DHCP, firewall and endpoint visibility with privacy, retention and access-control requirements.

Multiple Choice Questions with Answers

Question 1

Which TCP/IP layer most directly contains TCP and UDP?

A. Application

B. Transport

C. Internet

D. Link

Correct Answer: B. Transport

Question 2

What is the main function of ARP in an IPv4 local network?

A. Resolve a domain name to an IP address

B. Assign a default gateway automatically

C. Map an IPv4 address to a local MAC address

D. Encrypt an HTTP session

Correct Answer: C. Map an IPv4 address to a local MAC address

Question 3

How many total IPv4 addresses are in a /26 subnet?

A. 32

B. 64

C. 128

D. 256

Correct Answer: B. 64

Question 4

Which protocol commonly uses UDP ports 67 and 68 for IPv4 configuration?

A. DNS

B. DHCP

C. SSH

D. HTTP

Correct Answer: B. DHCP

Question 5

Why is a rogue DHCP server dangerous?

A. It can provide a malicious gateway or DNS server

B. It changes the physical cable signal

C. It guarantees stronger encryption

D. It prevents all local broadcasts

Correct Answer: A. It can provide a malicious gateway or DNS server

Question 6

Which statement about NAT is correct?

A. NAT is a replacement for firewall policy

B. NAT guarantees that an endpoint is authenticated

C. NAT translates addresses or transport endpoints

D. NAT prevents all malicious outbound traffic

Correct Answer: C. NAT translates addresses or transport endpoints

Question 7

Which security objective concerns preventing unauthorised alteration of data?

A. Availability

B. Integrity

C. Confidentiality

D. Redundancy

Correct Answer: B. Integrity

Question 8

What is a principal security advantage of HTTPS over unprotected HTTP?

A. It eliminates denial-of-service attacks

B. It provides cryptographic protection for the session when correctly configured

C. It guarantees that the endpoint is free of malware

D. It prevents DNS queries from occurring

Correct Answer: B. It provides cryptographic protection for the session when correctly configured

Question 9

Which observation most strongly suggests ARP spoofing?

A. A stable DNS response from an authorised resolver

B. A gateway IP repeatedly associated with different MAC addresses

C. A correctly validated server certificate

D. A workstation receiving a normal subnet mask

Correct Answer: B. A gateway IP repeatedly associated with different MAC addresses

Question 10

What does zero trust primarily reject?

A. All use of encryption

B. The need for network monitoring

C. Implicit trust based solely on network location

D. The use of application protocols

Correct Answer: C. Implicit trust based solely on network location