4. Cloud-Native Networking and Service Delivery

Introduction and Context

Kubernetes networking is the communication fabric that allows workloads, control components, users, and external systems to interact. Unlike traditional server environments, Kubernetes does not assume that applications are permanently attached to particular machines or fixed IP addresses. Pods are created, replaced, rescheduled, and scaled dynamically. Consequently, networking must provide stable service identities while accommodating constantly changing endpoints.

Cloud-native networking is therefore more than the assignment of IP addresses. It includes pod-to-pod communication, service discovery, load balancing, ingress and egress control, policy enforcement, observability, multi-cluster connectivity, and integration with cloud-provider networks. These capabilities are implemented through cooperation among the Linux kernel, container runtime, Kubernetes control plane, Container Network Interface (CNI) plugins, kube-proxy or equivalent dataplanes, DNS services, ingress controllers, gateways, and cloud load balancers.

For DevOps professionals, networking is also a troubleshooting discipline. An application may be healthy while requests fail because a Service selector is incorrect, DNS returns an unexpected record, a NetworkPolicy blocks traffic, an ingress controller lacks a route, or a cloud load balancer has unhealthy targets. Effective diagnosis requires a layered model rather than isolated command execution.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain Kubernetes’ networking model and the role of CNI plugins.

2. Distinguish pods, Services, endpoints, DNS records, ingress, and Gateway API resources.

3. Select appropriate service-delivery and load-balancing patterns.

4. Design NetworkPolicy and egress-control strategies.

5. Explain service discovery, IPv4/IPv6 operation, and multi-cluster connectivity.

6. Troubleshoot connectivity, routing, name resolution, policy, and latency problems systematically.

7. Evaluate governance, security, operational, and organizational implications of networking choices.

The Kubernetes Networking Model

Kubernetes adopts several important networking assumptions. Every pod receives its own IP address, and pods should be able to communicate directly with one another without requiring address translation. Nodes should be able to reach pods, and pods should be able to reach nodes. These assumptions give applications a relatively consistent network experience even when containers move between nodes.

A pod is the smallest deployable network unit in Kubernetes. All containers in a pod share the same network namespace, IP address, and port space. Containers communicate with one another through localhost, while communication with other pods uses the pod IP and port. This arrangement is useful for tightly coupled containers such as an application process and a local proxy, but it also means that containers in the same pod cannot bind the same port.

Pod IP addresses are normally ephemeral. A replacement pod may receive a different address, and therefore clients should not use pod IPs as durable application identities. A Service provides a stable virtual IP, DNS name, and logical policy boundary in front of a changing set of pods.

The actual implementation of pod networking is delegated to a CNI plugin. At pod creation time, the runtime invokes the plugin to create a network interface, connect it to a bridge, virtual Ethernet pair, overlay, routed fabric, or cloud-native network, and assign addresses and routes. CNI implementations differ significantly. Some use overlay encapsulation, while others integrate directly with the underlying cloud or data-centre network. Some provide only connectivity; others also implement NetworkPolicy, encryption, observability, eBPF-based routing, or multi-cluster functions.

The choice between overlay and routed networking is an architectural decision. Overlays simplify operation across heterogeneous node networks because pod traffic is encapsulated over an underlay. Their costs may include additional headers, MTU complexity, and encapsulation overhead. Routed or cloud-integrated networking can improve performance and visibility, but it may require careful IP planning, cloud permissions, route propagation, and integration with provider limits.

Services, Endpoints, and Stable Delivery

A Service is an abstraction that exposes a logical set of pods. Its selector identifies matching pods, and the control plane maintains endpoint information representing the currently usable backends. In modern Kubernetes, EndpointSlices provide a scalable representation of endpoints and their topology information.

The principal Service types are:

• ClusterIP: Provides an internal virtual address reachable within the cluster. It is the default type and is appropriate for internal application-to-application communication.
• NodePort: Exposes a port on each node and forwards traffic to Service backends. It is simple but exposes node-level ports and is often used as a building block for external load balancers.
• LoadBalancer: Requests an external load-balancing capability, commonly from a cloud provider. The exact implementation and cost depend on the platform.
• ExternalName: Returns a DNS alias for an external name rather than selecting pods. It should not be confused with a proxy or health-checking load balancer.

A Service selector is a frequent source of failure. If labels do not match, the Service can exist correctly while having no endpoints. A disciplined diagnostic sequence checks the Service definition, selector, EndpointSlices, pod readiness, target ports, and network policy. The `port` exposed by a Service and the pod’s `targetPort` are separate values; confusing them causes connection failures even when endpoints exist.

Service routing may be implemented through kube-proxy using iptables or IPVS, or through an alternative dataplane such as eBPF. The dataplane translates traffic addressed to a virtual Service IP toward a selected backend. Session affinity, topology-aware routing, external traffic policy, and internal traffic policy influence backend selection and source-IP preservation.

`externalTrafficPolicy: Local` can preserve the original client IP and avoid an additional cross-node hop, but it may produce uneven load when only some nodes have local endpoints. `externalTrafficPolicy: Cluster` generally distributes traffic more broadly but may obscure the original source address through address translation. The choice should follow application, observability, and security requirements rather than a universal rule.

DNS and Service Discovery

Kubernetes normally deploys a cluster DNS service, commonly based on CoreDNS. A Service receives a DNS record based on its name and namespace. Within the same namespace, applications can often use the short name; across namespaces, the fully qualified form follows a pattern such as `service.namespace.svc.cluster-domain`.

A headless Service, created by setting `clusterIP: None`, does not provide a single virtual IP. Instead, DNS returns records for the selected pod endpoints. Headless Services are useful for stateful systems, direct endpoint discovery, and client-side load balancing. They transfer more responsibility to the client, which must handle endpoint changes, retries, and balancing.

DNS failures should be separated into three categories: failure to resolve a name, successful resolution to an incorrect address, and successful resolution followed by connection failure. Useful evidence includes the pod’s resolver configuration, DNS query results from inside the same namespace, CoreDNS logs and metrics, and the Service and EndpointSlice definitions. Search domains can make a short name resolve differently than expected, particularly when applications construct names incorrectly.

DNS is also a performance component. Excessive retries, long search lists, negative caching behaviour, overloaded DNS pods, or applications performing a lookup for every request can create latency. Caching must be designed with awareness of endpoint churn and failover requirements.

Ingress and the Gateway API

A Service provides reachability to an application, but external HTTP and HTTPS delivery usually requires an ingress or gateway layer. An Ingress resource describes host- and path-based HTTP routing, TLS termination, and sometimes redirects or traffic annotations. The Ingress object itself does not process traffic; an ingress controller watches it and configures a proxy, load balancer, or cloud integration.

This separation is important operationally. A syntactically valid Ingress may have no effect if the required controller is absent, the class is incorrect, the controller cannot obtain a certificate, or the external load balancer is unhealthy. Ingress annotations are powerful but often controller-specific, reducing portability and making configuration governance important.

The Gateway API provides a more expressive and role-oriented model. A `GatewayClass` identifies an implementation, a `Gateway` represents an infrastructure listening point, and route resources such as `HTTPRoute` express application routing. The model separates platform responsibilities from application-team responsibilities and supports explicit attachment rules. This separation can improve multi-team governance, although it introduces additional resources and requires platform standardisation.

Ingress and Gateway API should be treated as delivery planes rather than merely YAML objects. Design questions include where TLS is terminated, how certificates are issued and rotated, whether traffic is re-encrypted to workloads, how client identity is preserved, how rate limiting is applied, and how routes are audited.

Load Balancing and Traffic Engineering

Load balancing occurs at several layers. A cloud load balancer may distribute traffic to nodes or directly to pod addresses. A Service distributes traffic among endpoints. An ingress controller or gateway makes application-layer decisions based on hosts, paths, headers, or weights. A service mesh may add retries, timeouts, circuit breaking, and telemetry.

Each additional layer can provide useful control but also increases the number of possible failure points. A request may traverse a public DNS record, a cloud load balancer, a node or pod target, an ingress proxy, a Service virtual IP, and finally an application container. The architecture should avoid unnecessary proxy chains and define clear ownership of timeouts, retries, and health checks.

Retries require particular caution. If a client, gateway, mesh, and application each retry, one failed request can generate a burst of duplicate work. Retries should normally be limited, bounded by deadlines, and restricted to operations that are safe to repeat. Timeouts should reflect the expected latency budget across all hops.

Topology-aware routing can reduce cross-zone traffic and improve latency, but strict locality may reduce resilience when a zone has insufficient healthy endpoints. Traffic policies should therefore balance cost, performance, capacity, and failure-domain tolerance.

NetworkPolicy and Egress Governance

NetworkPolicy expresses permitted traffic between pods and, depending on implementation, between pods and external networks. Policies are additive: traffic is allowed if at least one applicable policy permits it, while traffic not permitted by an active policy becomes denied in the relevant direction. A namespace with no policies is not automatically isolated.

Policies may select pods by namespace, labels, or IP blocks and may constrain ports and protocols. A robust design begins with application communication mapping: which workloads communicate, on which ports, in which direction, and for what purpose. Policies should then be tested in stages, beginning with observation and narrowly scoped rules rather than an immediate cluster-wide deny that disrupts operations.

Ingress and egress are independent dimensions. Allowing inbound traffic to a database does not necessarily allow the database to initiate outbound traffic. Egress controls are valuable for limiting data exfiltration, enforcing approved destinations, and requiring external access through an inspection gateway or proxy. However, external destinations can be difficult to identify when they use changing IP addresses. DNS-aware egress mechanisms or controlled proxies may be preferable to maintaining fragile IP allowlists.

NetworkPolicy is not a complete security architecture. It may not provide identity-aware authorization, payload inspection, certificate management, or protection against a compromised application using an already permitted connection. It should complement workload identity, encryption, admission controls, image security, and application-level authorization.

IPv6, Dual-Stack, and Multi-Cluster Connectivity

IPv6 adoption addresses address-space constraints and can simplify direct addressing in large environments, but it introduces operational requirements involving routing, DNS, firewall rules, observability, and application compatibility. Kubernetes dual-stack environments assign IPv4 and IPv6 addresses according to cluster configuration. Applications and libraries must correctly handle address selection, listener binding, and fallback behaviour.

Dual-stack testing should include Services, pod-to-pod communication, external load balancers, DNS records, NetworkPolicy, and monitoring. A system that works over IPv4 alone may fail because a client prefers an unreachable IPv6 address or because a policy covers only one address family.

Multi-cluster architectures are used for regional resilience, regulatory separation, environment isolation, and organisational boundaries. Connectivity models include routed pod networks, shared service meshes, DNS-based federation, gateway-mediated access, and dedicated inter-cluster networks. The design must define identity, naming, failover, data consistency, and trust boundaries. Extending a flat network across clusters can simplify discovery but also expands the blast radius and complicates troubleshooting. Gateway-mediated connectivity is often more explicit and governable, though it introduces additional proxies and operational dependencies.

A Systematic Method for Troubleshooting Traffic and Latency

Troubleshooting should proceed from the observed symptom toward the narrowest failing layer. Begin by recording the complete request path, source and destination, protocol, port, namespace, timestamps, and whether the failure is total, intermittent, or latency-related.

A practical sequence is:

1. Confirm application readiness. Check pod status, readiness probes, listening ports, container logs, and resource saturation.

2. Verify Service selection. Inspect the Service type, selector, ports, EndpointSlices, and readiness state.

3. Test name resolution. Query the relevant name from a diagnostic pod in the same namespace and network context.

4. Test direct connectivity. Where appropriate, connect to a pod IP or endpoint to distinguish application failure from Service routing failure.

5. Inspect policy. Determine whether ingress or egress policies, cloud firewalls, security groups, or gateway rules deny the traffic.

6. Trace the delivery plane. Examine Ingress or Gateway status, controller events, listener configuration, certificates, and load-balancer health.

7. Measure the path. Use flow logs, packet captures, proxy metrics, DNS latency, connection establishment time, and application timing.

8. Compare failure domains. Test from different nodes, zones, clients, and address families to expose topology or locality issues.

Latency should be decomposed into DNS lookup, TCP or QUIC connection establishment, TLS negotiation, proxy processing, queueing, service routing, application execution, and response transmission. Percentile measurements, especially p95 and p99, are more informative than averages for user-facing services. A latency increase may be caused by retries or queue growth rather than by the network itself.

Illustrative Case Study: Regional Checkout Service

Background

A fictional online retailer operates a Kubernetes platform across two availability zones. Its checkout API is exposed through a cloud load balancer, an HTTP gateway, and an internal Service leading to checkout pods. Payment processing occurs through an external provider.

Situation

Following a deployment, some customers experience intermittent checkout timeouts. Monitoring shows that average latency is nearly unchanged, but p99 latency has doubled. The problem is more common in one zone and is not reproduced from inside the checkout pods.

Analysis

The operations team first verifies that checkout pods are ready and that the Service has healthy EndpointSlices in both zones. DNS resolution is normal. Direct requests to local pod IPs are successful. Gateway logs show repeated upstream attempts, and flow telemetry reveals that requests entering one zone are frequently forwarded to endpoints in the other zone. The deployment’s topology-aware routing configuration was ineffective because the Service had insufficient local endpoints after a rolling update. In addition, gateway retries were configured without a sufficiently strict deadline. Some payment requests were repeated after the upstream connection had already reached the payment provider.

Approach and Solution

The team increases zone-balanced replica placement, configures topology-aware routing with an explicit fallback strategy, and reviews `externalTrafficPolicy` and gateway target selection. It replaces indefinite retries with one bounded retry for clearly idempotent operations and adds a request deadline. Payment operations receive an idempotency key. Dashboards are expanded to show DNS, gateway, Service, cross-zone, and application timing separately.

Outcome

Tail latency returns to its previous range, cross-zone traffic falls, and duplicate payment attempts are eliminated. The incident also leads to a standard review checklist for topology, retries, and idempotency during releases.

Learning Points

The case demonstrates that a network symptom may arise from endpoint distribution and retry policy rather than packet loss. It also shows why average latency is inadequate, why topology policies require sufficient local capacity, and why traffic engineering must consider business semantics.

Design Risks, Governance, and Best Practices

Common mistakes include exposing pod IPs as permanent endpoints, using broad firewall exceptions, assuming an Ingress controller exists, overlooking empty EndpointSlices, creating overlapping policies without documentation, and allowing every component to define controller-specific annotations. Another mistake is treating encryption as a substitute for authorization; encrypted traffic can still be sent to an unintended destination.

Platform teams should establish approved CNI and gateway patterns, IP address-management procedures, naming conventions, ownership boundaries, policy templates, certificate processes, and observability standards. Changes to routes, policies, gateways, and load balancers should be reviewed like production code. Admission policies can enforce required labels, approved classes, restricted hostnames, and security defaults.

Governance should also address cost and accountability. Cross-zone transfer, public load balancers, gateway instances, and high-volume logging can create significant expenditure. Teams need service-level objectives that connect technical indicators—availability, latency, DNS success, connection errors, and policy denials—to user outcomes.

Emerging developments include eBPF-based dataplanes, gateway-oriented platform APIs, ambient or sidecar-reduced service-mesh models, workload identity integrated with network enforcement, IPv6 expansion, and automated network verification. These technologies can improve performance and abstraction, but they do not remove the need for sound address planning, clear ownership, and disciplined diagnosis.

Chapter Summary

Kubernetes networking combines pod connectivity, CNI implementation, Service abstraction, DNS-based discovery, ingress and Gateway API delivery, load balancing, policy enforcement, and external integration. Pods provide ephemeral network identities, while Services provide stable logical access. CNI plugins implement the underlying network and may also deliver policy, encryption, observability, and multi-cluster capabilities.

Reliable service delivery depends on correct selectors, endpoints, ports, DNS, gateway configuration, health checks, and load-balancing policies. NetworkPolicy provides important segmentation but must be combined with identity, encryption, authorization, and governance. IPv6 and multi-cluster designs require deliberate treatment of routing, naming, trust, failure domains, and observability. Troubleshooting is most effective when engineers trace the complete request path and decompose latency into measurable stages.

Key Terms

Pod IP: An address assigned to a pod and normally treated as ephemeral.

CNI: The Container Network Interface specification and plugin model for configuring container networking.

Service: A stable Kubernetes abstraction that exposes a logical set of endpoints.

ClusterIP: An internal virtual IP assigned to a Service.

EndpointSlice: A scalable Kubernetes resource representing Service endpoints.

Headless Service: A Service without a virtual cluster IP that exposes endpoint records through DNS.

kube-proxy: A component or equivalent dataplane that implements Service traffic forwarding.

CoreDNS: A commonly deployed DNS server providing Kubernetes service discovery.

Ingress: A Kubernetes API resource describing external HTTP or HTTPS routing.

Ingress controller: The implementation that translates Ingress resources into operational routing configuration.

Gateway API: A role-oriented Kubernetes networking API for gateways and routes.

NetworkPolicy: A resource expressing permitted pod ingress and egress traffic.

Egress: Traffic leaving a workload or network boundary.

Dual-stack: Operation using both IPv4 and IPv6 address families.

Topology-aware routing: Traffic selection that considers node, zone, or region locality.

ExternalTrafficPolicy: A Service setting controlling external traffic routing and source-IP behaviour.

Service discovery: The process of locating a service through names, addresses, and endpoint information.

CNI overlay: A network implementation that encapsulates pod traffic across an underlay network.

Tail latency: High-percentile response time, such as p95 or p99 latency.

Multi-cluster networking: Connectivity and service access spanning two or more Kubernetes clusters.

Review Questions

Short Answer Questions

1. Why are pod IP addresses generally unsuitable as permanent application endpoints?

2. What is the relationship between a Service selector and its EndpointSlices?

3. How does a headless Service differ from a standard ClusterIP Service?

4. What is the purpose of a CNI plugin?

5. Why can average latency conceal an operational problem?

Long Answer Questions

1. Explain the Kubernetes networking model and discuss the architectural trade-offs between overlay and routed CNI implementations.

2. Compare ClusterIP, NodePort, LoadBalancer, and ExternalName Services, including appropriate use cases.

3. Explain how DNS, Ingress, Gateway API, and load-balancing layers cooperate to deliver an external application.

4. Discuss NetworkPolicy design, including ingress, egress, additive rules, and limitations.

5. Describe a systematic method for diagnosing an intermittent service timeout in a multi-zone cluster.

Analytical / Application Questions

1. A Service has no reachable backends although its pods are running. Construct a diagnostic sequence and identify at least four possible causes.

2. A company wants low latency across regions but must prevent one regional failure from spreading to others. Compare a flat multi-cluster network with gateway-mediated connectivity.

3. A payment API shows high p99 latency after a rollout, while average latency remains stable. Analyse how endpoint locality, retries, deadlines, and idempotency could interact to create the symptom.

22. Multiple Choice Questions with Answers

Question 1

A Kubernetes Service exists, but requests fail because no backend endpoints are listed. Which investigation is most appropriate first?

A. Replace the CNI plugin immediately

B. Compare the Service selector with pod labels and inspect EndpointSlices

C. Change the cluster DNS domain

D. Enable IPv6 on all nodes

Correct Answer: B. Compare the Service selector with pod labels and inspect EndpointSlices

Question 2

What is the principal characteristic of a headless Service?

A. It always exposes the application publicly

B. It assigns a fixed IP to every pod

C. It returns endpoint information through DNS without a virtual cluster IP

D. It disables readiness checking

Correct Answer: C. It returns endpoint information through DNS without a virtual cluster IP

Question 3

Which statement best describes the role of a CNI plugin?

A. It stores application logs

B. It configures container network interfaces, addresses, and routes

C. It defines HTTP host-based routes only

D. It replaces all application authentication

Correct Answer: B. It configures container network interfaces, addresses, and routes

Question 4

A platform team wants application teams to define routes while retaining infrastructure control over listeners and gateways. Which model best supports this separation?

A. Gateway API

B. ExternalName Services only

C. Pod IP addressing

D. NodePort without controllers

Correct Answer: A. Gateway API

Question 5

What is a likely trade-off of externalTrafficPolicy: Local?

A. It always removes source-IP visibility

B. It can preserve source IP but may cause uneven load when local endpoints are sparse

C. It prevents all cross-zone traffic under every condition

D. It disables Service health checks

Correct Answer: B. It can preserve source IP but may cause uneven load when local endpoints are sparse

Question 6

Which statement about NetworkPolicy is correct?

A. A policy automatically grants all traffic not explicitly denied

B. Policies are additive, and applicable rules determine permitted traffic

C. NetworkPolicy replaces application-level authorization

D. A namespace with one policy is necessarily isolated in every direction

Correct Answer: B. Policies are additive, and applicable rules determine permitted traffic

Question 7

A DNS query succeeds, but the application still cannot connect. Which conclusion is most accurate?

A. The problem must be CoreDNS

B. Name resolution worked, so routing, policy, ports, endpoints, or the application may still be failing

C. The Service cannot have endpoints

D. IPv6 is definitely misconfigured

Correct Answer: B. Name resolution worked, so routing, policy, ports, endpoints, or the application may still be failing

Question 8

Why should retries in gateways and clients be bounded and carefully designed?

A. Retries always reduce traffic

B. Multiple retry layers can amplify load and duplicate non-idempotent operations

C. Retries prevent all timeout failures

D. Retries eliminate the need for health checks

Correct Answer: B. Multiple retry layers can amplify load and duplicate non-idempotent operations

Question 9

Which measurement is generally most useful for detecting user-impacting tail-latency degradation?

A. Only average CPU utilisation

B. Only total pod count

C. p95 or p99 request latency broken down by network and application stages

D. The number of Kubernetes namespaces

Correct Answer: C. p95 or p99 request latency broken down by network and application stages

Question 10

What is a central architectural concern in multi-cluster networking?

A. Whether every pod must use the same hostname

B. Identity, naming, trust boundaries, failover, and the resulting troubleshooting complexity

C. Removing all gateway components

D. Avoiding all DNS use

Correct Answer: B. Identity, naming, trust boundaries, failover, and the resulting troubleshooting complexity