8. Linux Networking and Distributed Services

Introduction and Context

Linux networking is the foundation on which modern computing services communicate. A Linux host may act simultaneously as a client, web server, database server, router, firewall, container host, virtual machine host, or authentication gateway. These roles are not isolated. They depend on correctly configured interfaces, addresses, routes, name resolution, transport protocols, access-control rules, and service processes.

At postgraduate level, networking should be understood as both a protocol system and an operational discipline. A configuration can be syntactically correct yet operationally unsuitable: a route may create asymmetric traffic, a DNS resolver may introduce unacceptable latency, a firewall may block legitimate service dependencies, or a namespace may provide isolation without providing the connectivity required by an application. Effective administration therefore combines conceptual models with systematic observation and diagnosis.

This chapter examines Linux networking from the host perspective. It explains interface and routing configuration, DNS and DHCP, secure remote administration with SSH, firewall design, network namespaces, and distributed services. It also develops a practical diagnostic method using contemporary tools such as `ip`, `ss`, `dig`, `resolvectl`, `tcpdump`, `nft`, and `systemd-analyze`.

Learning Objectives

After studying this chapter, the reader should be able to:

• explain how Linux represents interfaces, addresses, routes, sockets, and network namespaces;
• configure and inspect IPv4 and IPv6 connectivity using modern tools;
• distinguish name resolution, address assignment, routing, and service discovery;
• configure and secure SSH-based administration;
• design host and gateway firewall policies using nftables concepts;
• construct simple network-namespace topologies with virtual Ethernet pairs;
• diagnose failures by moving systematically through the networking stack;
• evaluate operational, security, governance, and reliability implications of distributed services.

A Working Model of Linux Networking

A useful abstraction is the layered path followed by an application request. An application first uses a socket API. The kernel associates the socket with a protocol, local address, local port, and remote endpoint. The routing subsystem then selects an output interface and next hop. Link-layer processing places the packet on the appropriate medium, while the receiving host reverses this process.

Several Linux objects are central to this model. A network interface represents a packet-transmission endpoint, such as an Ethernet device, wireless adapter, bond, bridge, VLAN, tunnel, or virtual interface. An IP address identifies an interface within a network prefix. A route maps a destination prefix to an interface and, when necessary, a gateway. A socket is an endpoint used by a process for network communication. A network namespace is an isolated instance of networking state containing its own interfaces, routes, firewall rules, and sockets.

The distinction between a link and a route is important. An interface can be operationally up while no usable route exists. Conversely, a route can be present while the link is disconnected or the next hop is unreachable. Similarly, successful DNS resolution does not prove that the destination service is listening, and a listening service does not prove that a firewall permits access.

Interfaces and Addresses

The modern inspection tool is `ip`, supplied by the `iproute2` suite. Typical commands include:

```bash

ip link show

ip addr show

ip -br addr

ip link set dev eth0 up

ip addr add 192.0.2.20/24 dev eth0

```

The last command adds an address temporarily. Persistent configuration is normally managed by a distribution-specific system such as NetworkManager, systemd-networkd, or another declarative network manager. Operational practice should distinguish runtime state from persistent state; changing one does not necessarily change the other.

IPv4 uses 32-bit addresses, whereas IPv6 uses 128-bit addresses. IPv6 introduces link-local addresses, normally beginning with `fe80::/10`, which support communication on the local link and neighbor discovery. An IPv6 host can therefore have useful connectivity even when it lacks a globally routable address. Administrators must also understand address scope, prefix length, duplicate-address detection, and the possibility that applications select IPv6 before IPv4.

The loopback interface, usually `lo`, is internally significant. Services bound to `127.0.0.1` or `::1` accept only local connections. Binding to `0.0.0.0` or `::` requests listening on all suitable addresses, although firewall and application policies may further restrict access. A service should not listen broadly merely because doing so is convenient.

Routing and Forwarding

The kernel routing table determines where packets go. It may be inspected with:

```bash

ip route show

ip -6 route show

ip route get 198.51.100.40

```

A route generally contains a destination prefix, an optional gateway, a device, and sometimes a metric or source preference. When multiple routes match, the most specific prefix is normally preferred; among routes with comparable specificity, metrics and policy rules influence selection. `ip route get` is particularly useful because it asks the kernel to calculate the path for a specific destination rather than merely displaying the table.

A Linux system forwards packets between interfaces only when forwarding is enabled. IPv4 forwarding is commonly controlled by `net.ipv4.ip_forward`, while IPv6 forwarding has its own setting. Forwarding alone does not create a complete router: the system also needs appropriate routes, firewall rules, and often address translation or a return route on downstream networks.

Policy routing extends this model. Instead of one single route table, rules can select tables based on source address, interface, packet mark, or other attributes. This supports multihoming, virtual private networks, and traffic engineering, but it also increases diagnostic complexity. The effective path is then a function of rules, tables, and route metrics rather than the main table alone.

Ports, Sockets, and Services

TCP provides reliable, ordered byte-stream communication; UDP provides datagrams without TCP’s delivery guarantees; and QUIC provides encrypted, multiplexed transport over UDP. A service usually binds to a port and address. The command `ss` exposes socket state more directly and efficiently than older tools:

```bash

ss -lntup

ss -s

ss -tan state established

```

A listening socket demonstrates that a process has requested a local endpoint. It does not demonstrate successful application-level operation. For example, a web server may listen on port 443 while failing to access its database. Diagnosis must therefore proceed beyond port reachability.

Configuration Services: DHCP, DNS, and Name Resolution

Address configuration and name resolution solve different problems. DHCP supplies configuration parameters such as an address, prefix, default gateway, DNS servers, and lease duration. DNS maps names to records such as IPv4 (`A`), IPv6 (`AAAA`), mail (`MX`), service (`SRV`), and reverse-mapping (`PTR`) records.

A DHCP lease is a negotiated assignment, not merely an address handed out permanently. Clients renew leases, and administrators must consider reservations, pools, relay agents, authentication, and rogue DHCP servers. In IPv6, address configuration may involve DHCPv6, router advertisements, or both. Router advertisements communicate prefixes and default-router information; DHCPv6 may supply additional configuration.

Linux name resolution is commonly mediated by a resolver library and a local service such as `systemd-resolved`, NetworkManager, or a locally installed caching resolver. `/etc/hosts` provides static local mappings, while `/etc/resolv.conf` may be a file, a symbolic link, or a generated configuration. Inspecting the file alone may not reveal the full resolution path.

Useful diagnostic commands include:

```bash

getent hosts server.example.org

dig server.example.org A

dig @192.0.2.53 server.example.org

resolvectl status

resolvectl query server.example.org

```

`getent` tests the system’s configured name-service path, whereas `dig` directly examines DNS behavior and records. A successful lookup may be cached, split-horizon, or obtained from a different resolver than the application uses. Troubleshooting should therefore ask: which resolver was queried, which record was returned, what is its time to live, and is the returned address reachable?

DNS operational failures include stale records, incorrect delegation, missing reverse records, excessive TTLs during migration, and split-DNS policies that expose different answers internally and externally. DNSSEC can provide authenticity for validated records, but encryption of DNS transport and authenticity of DNS data are distinct concerns. DNS over TLS or HTTPS protects transport characteristics differently from DNSSEC.

Secure Remote Administration with SSH

SSH combines encrypted transport, server authentication, user authentication, and secure channels for interactive shells, commands, forwarding, and file transfer. A typical connection is:

```bash

ssh -p 22 [email protected]

```

The client verifies the server’s host key using locally stored known-host information. Users may authenticate with passwords, public keys, certificates, hardware-backed credentials, or multi-factor mechanisms. Public-key authentication is generally preferable to reusable passwords for administrative access, but key management becomes a governance problem when keys are shared, untracked, or never revoked.

A hardened SSH deployment commonly limits direct privileged login, disables unnecessary authentication methods, restricts users or groups, uses modern cryptographic defaults, and records authentication events. Configuration changes should be tested in a second session before an existing administrative session is closed. This simple practice prevents a syntax or access-control error from causing lockout.

SSH forwarding is powerful but potentially dangerous. Local forwarding exposes a remote service locally; remote forwarding exposes a local service remotely; dynamic forwarding can create a proxy. These features should be governed because they may bypass network segmentation and monitoring. File transfer through SFTP or secure copy should similarly respect data classification and authorization requirements.

Firewall Design with nftables

A firewall enforces a policy about traffic, not merely a list of ports. A sound policy begins with trust boundaries, required flows, administrative paths, service dependencies, and logging requirements. nftables is the contemporary Linux packet-filtering framework. Its conceptual objects include tables, chains, rules, sets, maps, counters, and connection tracking.

A simplified illustrative policy might be expressed as:

```bash

nft add table inet filter

nft 'add chain inet filter input { type filter hook input priority 0; policy drop; }'

nft add rule inet filter input iifname "lo" accept

nft add rule inet filter input ct state established,related accept

nft add rule inet filter input tcp dport 22 ip saddr 192.0.2.0/24 accept

nft add rule inet filter input counter drop

```

This is an example of a host policy, not a universally safe production script. A production design must account for IPv6, required ICMP or ICMPv6 traffic, configuration management access, service ports, logging limits, and rollback. Blindly dropping all ICMP can damage path MTU discovery and IPv6 neighbor operation.

Connection tracking allows rules to treat established flows differently from new flows. Stateful filtering is not the same as application security: it does not validate authorization, input, or business logic. Network address translation may be useful where address conservation or topology requirements justify it, but it can complicate troubleshooting and does not replace firewall policy.

Firewalls should be managed as code when possible. Version-controlled rules, automated validation, staged deployment, explicit ownership, and recovery procedures reduce the risk of accidental outage. Logs should be selective; logging every packet can overwhelm storage and obscure important events.

Network Namespaces and Virtual Topologies

Network namespaces provide process-level isolation of networking resources. Containers use this mechanism, but namespaces are also valuable for laboratories, multi-tenant services, routing experiments, and controlled testing. A process in one namespace cannot directly see interfaces or sockets in another.

A virtual Ethernet pair creates two linked interfaces. One end can remain in the root namespace while the other is moved into a test namespace:

```bash

ip netns add ns1

ip link add veth-host type veth peer name veth-ns

ip link set veth-ns netns ns1

ip addr add 10.10.0.1/24 dev veth-host

ip link set veth-host up

ip netns exec ns1 ip addr add 10.10.0.2/24 dev veth-ns

ip netns exec ns1 ip link set veth-ns up

ip netns exec ns1 ip link set lo up

ip netns exec ns1 ping -c 2 10.10.0.1

```

The namespace has an independent loopback, routing table, firewall context, and socket space. To connect several namespaces, an Ethernet bridge can act as a virtual switch. To provide outside access, an administrator may configure forwarding, routes, firewall rules, and possibly NAT. Namespaces alone provide isolation, not automatic internet access.

This model helps explain container networking. An orchestration platform may combine namespaces with bridges, virtual Ethernet devices, overlay networks, service discovery, policy engines, and load balancing. The resulting path can cross several abstraction layers, so diagnosis requires identifying the namespace, interface, route, and policy at each stage.

A Systematic Method for Diagnosing Connectivity

Random command execution is less effective than a hypothesis-driven sequence. Begin by defining the failed operation precisely: source, destination name or address, protocol, port, time, and expected result. Then move from the local layer outward.

1. Inspect link and address state. Use `ip link`, `ip addr`, and, where appropriate, `ethtool`. Confirm that the expected interface exists, is up, and has an appropriate address.

2. Test local routing. Use `ip route get destination`. Confirm that the selected interface, gateway, and source address make sense.

3. Test neighbor and gateway reachability. Examine `ip neigh`; use carefully selected ping or equivalent tests. Lack of ICMP response is not conclusive because filtering may be intentional.

4. Test DNS independently. Compare `getent`, `dig`, and `resolvectl`. Verify both the answer and the resolver path.

5. Test transport reachability. Use `ss` locally and tools such as `nc` or `curl` appropriately. A TCP connection test should identify whether failure occurs before connection, during TLS, or at the application protocol.

6. Inspect filtering and services. Review nftables counters, service status, listening sockets, and logs. `journalctl` can correlate events with time.

7. Capture packets when uncertainty remains. `tcpdump` can reveal whether packets leave, whether replies return, and whether resets, retransmissions, or ICMP errors occur:

```bash

tcpdump -ni eth0 host 198.51.100.40 and port 443

```

Packet capture should be interpreted carefully. Seeing an outbound SYN but no response suggests a path, filtering, or destination problem; seeing a SYN followed by RST indicates that some host actively rejected the connection; a completed TCP handshake followed by TLS failure moves the problem into a higher layer.

Modern observability may add eBPF-based tools, flow logs, exporters, and distributed tracing. These tools can associate kernel events with processes and provide low-overhead visibility, but they do not eliminate the need for protocol knowledge. Measurements must have a defined question and an understood collection point.

Distributed Services and Operational Design

A distributed service consists of cooperating components that communicate over a network and may fail independently. Common examples include web applications with reverse proxies, application servers, databases, caches, message brokers, directory services, and monitoring systems. Network configuration is therefore part of application architecture.

Service design should identify dependencies explicitly. A web server may require DNS, certificate validation, an identity provider, a database, a cache, and an external API. Availability is constrained by critical dependencies, while security depends on least-privilege communication between components. Network segmentation, service accounts, mutual TLS, secrets management, health checks, and controlled retries are architectural concerns rather than merely command-line tasks.

Retries deserve particular attention. A retry can improve resilience against transient failure but can amplify load during an outage. Timeouts, bounded retries, exponential backoff, circuit breakers, and idempotent operations should be designed together. DNS caching and load balancing also affect failure recovery; a changed address does not become effective everywhere at the same time.

Operational teams should define service-level indicators such as successful request rate, latency, connection errors, DNS resolution time, and packet loss. Logs, metrics, traces, and packet observations answer different questions. A mature incident process preserves evidence, records changes, avoids premature blame, and uses post-incident analysis to improve controls.

Illustrative Case Study: A Dual-Stack Research Portal

Background

A university research group operates a portal on Linux. The portal consists of a reverse proxy, an application service, and a database on separate virtual machines. Students report intermittent failure when accessing the portal by name, although administrators can sometimes access it by IPv4 address.

Situation and Initial Analysis

The DNS name has both `A` and `AAAA` records. The application server is healthy, and the reverse proxy is listening on TCP port 443. Initial testing with `curl` shows that some clients attempt IPv6 first and experience long delays. `dig` confirms that the `AAAA` record is present. On the proxy, `ip -6 route get` shows a route through an upstream gateway, but packet capture shows outbound IPv6 SYN packets without replies.

The team then discovers that the upstream firewall permits IPv4 but does not permit or correctly route IPv6. The proxy’s local nftables policy also includes IPv4 rules that were not mirrored for IPv6. The problem is therefore not DNS alone: DNS exposed an address for which the complete network path was unavailable.

Approach and Solution

The team first documents the intended architecture and decides whether IPv6 will be supported as a production path. Rather than deleting the record as a temporary concealment, they either complete the IPv6 design or remove the record through a controlled change process. They add required IPv6 routing and firewall rules, verify ICMPv6 operation and path MTU behavior, and test the proxy-to-application path separately.

Testing includes `dig` for both record types, `curl -4` and `curl -6`, `ss` on the proxy, `nft` counters, and packet captures at relevant interfaces. Monitoring is updated to test both address families. The proxy is configured with explicit upstream timeouts and health checks so that an unavailable backend does not appear as an unexplained client timeout.

Outcome

After the network path and policy are corrected, IPv6 clients complete TLS negotiation and receive the same application response as IPv4 clients. The organization retains dual-stack service with documented ownership of DNS, routing, firewalling, and application health.

Learning Points

The case demonstrates that name resolution, routing, filtering, transport, and application health are separate diagnostic dimensions. It also shows why dual-stack operation must be tested deliberately, why a DNS record is an operational commitment, and why monitoring should exercise the paths users actually take.

Risks, Limitations, and Professional Considerations

Common mistakes include modifying only runtime configuration, confusing a listening port with a working service, ignoring IPv6, relying on ping as the sole test, using broad firewall exceptions, exposing administrative SSH to untrusted networks, and changing network policy without an out-of-band recovery path. Another frequent error is applying NAT to conceal a missing route; this may make a demonstration work while creating long-term observability and security problems.

Network data can contain usernames, addresses, queries, credentials accidentally transmitted by applications, and sensitive organizational information. Packet captures and logs must therefore follow access-control, retention, and privacy requirements. Administrative actions should be attributable, authorized, and auditable. SSH keys, firewall policies, DNS changes, and service credentials require lifecycle management, including ownership, rotation, revocation, and emergency recovery.

Automation improves consistency but can also distribute a faulty configuration rapidly. Infrastructure-as-code should include validation, peer review, staged rollout, health checks, and rollback. Configuration management should avoid storing secrets in plain text and should distinguish secrets from ordinary network parameters.

Current Developments and Best Practices

Linux networking increasingly operates across physical hosts, virtual machines, containers, cloud networks, and software-defined overlays. eBPF is being used for observability, traffic control, security enforcement, and programmable networking. QUIC and HTTP/3 alter assumptions based exclusively on TCP. Zero-trust architectures emphasize authenticated, policy-controlled communication rather than broad network location as a proxy for trust. These developments increase capability but also demand stronger identity, telemetry, and policy design.

A practical professional baseline is to:

• document interfaces, addresses, routes, dependencies, and ownership;
• use declarative, version-controlled configuration where feasible;
• prefer least privilege for services, users, firewall rules, and forwarding;
• test IPv4 and IPv6 intentionally;
• separate DNS diagnosis from transport and application diagnosis;
• protect and audit SSH access;
• validate firewall changes and maintain recovery access;
• use namespaces and disposable environments for experiments;
• collect meaningful metrics, logs, traces, and packet evidence;
• rehearse failure and rollback procedures rather than assuming configuration is correct.

Chapter Summary

Linux networking is an interaction among interfaces, addresses, routes, sockets, protocols, resolvers, firewall policies, and service processes. The `ip` and `ss` tools expose kernel state; DHCP supplies configuration; DNS resolves names and publishes service-related records; SSH enables controlled remote administration; and nftables expresses stateful traffic policy. Network namespaces isolate networking state and form the basis of many container and laboratory topologies.

Reliable diagnosis proceeds from link and address state through routing, name resolution, transport, filtering, and application behavior. Distributed services add dependencies, independent failure modes, retries, health checks, and operational ownership. Secure and reliable administration requires least privilege, documented change control, privacy-aware observability, tested automation, and deliberate support for both IPv4 and IPv6.

Key Terms

Interface: A Linux networking endpoint capable of sending and receiving packets.

IP address: A numerical identifier assigned to an interface within an address space.

Prefix length: The number of leading address bits defining a network prefix.

Route: A kernel instruction mapping a destination prefix to a path and interface.

Default route: A route used when no more specific destination route matches.

Policy routing: Routing controlled by rules that may select tables using packet attributes.

Socket: An operating-system communication endpoint associated with a process.

Port: A transport-layer identifier used to distinguish services or conversations.

DHCP: A protocol for dynamically supplying hosts with network configuration parameters.

DNS: A distributed naming system that maps domain names to resource records.

Resolver: Software or a service that obtains answers to name-resolution requests.

SSH: An encrypted protocol for remote login, command execution, forwarding, and file transfer.

Firewall: A policy enforcement system that permits, rejects, or drops network traffic.

nftables: The modern Linux framework for packet filtering and related network policy.

Connection tracking: Kernel state maintenance describing flows and their connection status.

Network namespace: An isolated Linux networking environment with independent network state.

Virtual Ethernet pair: Two linked virtual interfaces commonly used to connect namespaces.

NAT: Translation of network addresses, often at a routing boundary.

Dual stack: Simultaneous operation of IPv4 and IPv6.

Service discovery: Mechanisms by which clients locate service instances and their endpoints.

Review Questions

Short Answer Questions

1. Why does an interface being administratively up not guarantee network connectivity?

2. Distinguish the diagnostic purposes of `getent hosts` and `dig`.

3. What information does `ip route get` provide that a general route-table listing may not?

4. Why should ICMP and ICMPv6 not be blocked indiscriminately?

5. What resources are isolated by a Linux network namespace?

Long Answer Questions

1. Explain the relationship among interfaces, addresses, routes, sockets, and services in a Linux host.

2. Describe a secure SSH administration design, including authentication, access restriction, auditing, and forwarding risks.

3. Explain how a stateful nftables firewall can be designed for a multi-service Linux server.

4. Describe a systematic workflow for diagnosing a failed HTTPS connection from a client to a distributed service.

5. Explain how DHCP, DNS, routing, and service discovery contribute different functions to distributed application operation.

Analytical / Application Questions

1. A host resolves a service name to an IPv6 address, but `curl -6` times out while `curl -4` succeeds. Design an evidence-based diagnostic plan and identify at least two possible corrective strategies.

2. Design a namespace-based topology containing two isolated application environments that share a controlled database service. Specify the interfaces, routes, firewall boundaries, and tests you would use.

3. An organization wants to expose SSH to the public internet because remote administration is required. Evaluate the security and operational trade-offs and propose a layered access architecture.

22. Multiple Choice Questions with Answers

Question 1

Which command asks the Linux kernel to determine the effective route for a particular destination?

A. ip route get 198.51.100.40

B. ss -lntup

C. dig 198.51.100.40

D. ip neigh flush all

Correct Answer: A. ip route get 198.51.100.40

Question 2

A successful DNS lookup proves which of the following?

A. The destination service is listening

B. A name-resolution request produced an answer

C. The firewall permits the application protocol

D. The selected address has a working return route

Correct Answer: B. A name-resolution request produced an answer

Question 3

What is the principal purpose of connection tracking in a stateful firewall?

A. Assigning addresses to clients

B. Translating every hostname into an IP address

C. Maintaining state about network flows

D. Creating virtual Ethernet devices

Correct Answer: C. Maintaining state about network flows

Question 4

Which situation best explains why a service bound to 127.0.0.1 is unreachable from another host?

A. The service is listening only on the local loopback address

B. The service has selected the default route

C. DNS has returned an AAAA record

D. The host has enabled IPv4 forwarding

Correct Answer: A. The service is listening only on the local loopback address

Question 5

What is required in addition to creating a network namespace for it to reach an external network?

A. Only a hostname in /etc/hosts

B. A separate SSH host key

C. Connectivity through interfaces, routes, and appropriate policy

D. A second DNS root server

Correct Answer: C. Connectivity through interfaces, routes, and appropriate policy

Question 6

Which comparison is most accurate?

A. DHCP resolves names, while DNS assigns leases

B. DHCP supplies configuration, while DNS maps names to records

C. DHCP filters packets, while DNS encrypts SSH

D. DHCP creates sockets, while DNS creates namespaces

Correct Answer: B. DHCP supplies configuration, while DNS maps names to records

Question 7

A packet capture shows an outbound TCP SYN but no returning packet. Which conclusion is most justified?

A. The application protocol definitely rejected the request

B. The TLS certificate is invalid

C. The response is absent at or before the return path, requiring further investigation

D. The local service completed the connection

Correct Answer: C. The response is absent at or before the return path, requiring further investigation

Question 8

Why is testing a firewall policy only with IPv4 insufficient on a dual-stack host?

A. IPv6 never uses ports

B. IPv6 has separate addressing and filtering behavior that must also be configured

C. IPv4 packets cannot be captured

D. DNS cannot publish IPv6 records

Correct Answer: B. IPv6 has separate addressing and filtering behavior that must also be configured

Question 9

Which SSH practice most directly reduces the risk of losing administrative access during a configuration change?

A. Close the current session before testing

B. Test the change in a second session before closing the first

C. Enable every forwarding feature

D. Replace public-key authentication with shared passwords

Correct Answer: B. Test the change in a second session before closing the first

Question 10

In the illustrative dual-stack case, what was the fundamental problem with retaining the AAAA record without further work?

A. AAAA records cannot be used by web browsers

B. The published IPv6 endpoint did not have a complete working route and policy path

C. IPv4 clients cannot resolve names with AAAA records

D. The reverse proxy could not listen on TCP 443

Correct Answer: B. The published IPv6 endpoint did not have a complete working route and policy path