8. Kubernetes Networking, Services and Traffic Management

Introduction and Context

A Kubernetes application is rarely a single container running on one machine. It is usually a collection of independently deployed workloads: APIs, web front ends, databases, message brokers, workers, and observability components. These workloads must communicate reliably despite pod creation, deletion, rescheduling, scaling, and movement between nodes. Kubernetes networking provides the logical connectivity model that makes this possible, while Services and traffic-management resources provide stable access, discovery, routing, and control.

Networking is therefore more than assigning IP addresses. It is an architectural concern involving identity, reachability, load distribution, security boundaries, name resolution, external exposure, and operational governance. A poorly designed network can make a healthy application unreachable; an overly permissive network can expose sensitive services; and an unsuitable ingress or routing strategy can produce outages during deployments.

This chapter develops the Kubernetes network model and then examines the Container Network Interface (CNI), Services, DNS, Ingress, Gateway API, NetworkPolicy, load balancing, and advanced traffic-routing strategies. The emphasis is on design reasoning: selecting an appropriate mechanism, understanding its scope, and recognizing its operational and security consequences.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Explain the Kubernetes pod and service networking model.

2. Describe the role of CNI plugins and distinguish pod, node, and service networks.

3. Select and configure appropriate Service types.

4. Explain Kubernetes DNS and service discovery behavior.

5. Compare Ingress and Gateway API for HTTP and other traffic patterns.

6. Design NetworkPolicy rules using least-privilege principles.

7. Evaluate load-balancing and traffic-routing strategies.

8. Diagnose common connectivity failures and design safer deployment patterns.

9. Assess governance, security, observability, and operational implications of networking choices.

The Kubernetes Network Model

Kubernetes adopts several important networking assumptions. Each pod receives its own IP address, and containers within the same pod share the pod network namespace. Consequently, containers in one pod communicate through localhost and can share ports only if those ports do not conflict. Communication between pods should normally occur directly through their pod IP addresses without requiring address translation. Nodes must also be able to reach pods, and pods must be able to reach other pods and usually external destinations, subject to policy and infrastructure controls.

This model gives every pod a routable identity within the cluster network. It avoids the need to publish a port on every node merely for one pod-to-pod interaction. However, pod IP addresses are ephemeral. A pod replacement may receive a different address, so applications should not use pod IPs as durable endpoints. Services solve this stability problem.

Three logical networks are commonly distinguished:

• The pod network provides addresses and routes for pod-to-pod communication.
• The node network connects worker nodes and control-plane infrastructure through the underlying physical or virtual network.
• The service network is a virtual address space used by stable Service IPs and implemented through dataplane rules, proxies, eBPF, or equivalent mechanisms.

These networks are logical rather than necessarily separate physical networks. Their implementation depends on the cloud provider, cluster distribution, CNI plugin, routing mode, and policy engine.

A pod’s network namespace is created when the pod is scheduled. The CNI plugin is then invoked to attach the pod to the network, allocate an address, configure interfaces and routes, and possibly apply security or encryption controls. When the pod terminates, the plugin releases the network resources. This lifecycle explains why networking failures may result from either Kubernetes objects or lower-level infrastructure such as route tables, security groups, IP exhaustion, or encapsulation settings.

CNI as the Networking Extension Point

The Container Network Interface is a specification and ecosystem for configuring network interfaces in Linux containers. Kubernetes delegates much of pod networking to a CNI implementation rather than prescribing a single networking technology. Examples of CNI-oriented solutions include Calico, Cilium, Flannel, and cloud-provider plugins, although their capabilities and architectures differ.

A CNI solution may provide basic connectivity, IP address management, network policy enforcement, encryption, observability, and service dataplane functions. Some use overlay networks, encapsulating pod traffic over the node network. Others use direct routing, cloud-native secondary interfaces, or eBPF-based forwarding. Overlay designs simplify operation across networks that do not natively route pod CIDRs, but introduce encapsulation overhead and troubleshooting considerations. Direct-routing designs can improve efficiency but require compatible routes and infrastructure integration.

Important selection criteria include:

• support for the required Kubernetes and cloud environment;
• IPv4, IPv6, or dual-stack capability;
• NetworkPolicy completeness and enforcement model;
• encryption and identity-based security features;
• performance, scale, and failure behavior;
• observability and troubleshooting tools;
• integration with the chosen Service and ingress dataplane;
• operational maturity and upgrade procedures.

A CNI plugin is not automatically a complete security architecture. Administrators must verify whether policy is enforced for ingress, egress, host traffic, DNS, and external destinations. They must also understand default behavior: in many clusters, if no NetworkPolicy selects a pod, traffic is broadly permitted.

Services: Stable Application Identity

A Service is an abstraction that exposes a logical set of pods, usually selected by labels. It provides a stable virtual IP, a DNS name, and a mechanism for distributing traffic to eligible endpoints. The Service decouples clients from pod lifecycle events. When replicas change, the control plane updates endpoint information, commonly represented through EndpointSlices.

The principal Service types are as follows.

ClusterIP

ClusterIP is the default type. It exposes the application inside the cluster through a virtual IP and DNS name. It is appropriate for internal APIs, databases, and microservice communication. The address is not normally reachable directly from outside the cluster.

NodePort

NodePort allocates a port on each node and forwards traffic toward the Service. It can be useful for development, bare-metal environments, or as a building block for external load balancers. It exposes a relatively broad infrastructure surface and may complicate firewall control, so it is not always the preferred production interface.

LoadBalancer

LoadBalancer requests an external load-balancing capability from the infrastructure provider. In a cloud environment, this commonly creates or configures a provider-managed load balancer. Behavior differs between providers, including source-IP preservation, health checks, address allocation, and integration with security groups. A LoadBalancer Service is often suitable for a small number of direct external services, whereas an ingress or gateway can consolidate many HTTP routes behind fewer front ends.

ExternalName

ExternalName maps a Service name to an external DNS name through a DNS alias rather than forwarding traffic through Kubernetes. It is useful in selected migration and abstraction scenarios, but it does not provide a normal cluster proxy, health checking, or endpoint management. Its behavior should be tested carefully with clients that handle DNS aliases differently.

A Service selects pods using labels. A common mistake is a selector that is syntactically valid but semantically wrong, such as selecting a staging application from a production Service. Endpoint inspection is therefore essential during diagnosis. A Service with no endpoints may indicate a selector mismatch, unready pods, a port mismatch, or a readiness condition that excludes all replicas.

Service ports have several related concepts: the port clients use on the Service, the targetPort exposed by the selected pod, and, where relevant, a nodePort. Named ports improve readability and reduce coupling to numerical changes. Readiness probes are particularly important because endpoints should represent pods capable of serving requests, not merely processes that exist.

DNS and Service Discovery

Kubernetes commonly deploys a DNS service such as CoreDNS. A Service normally receives a DNS record based on its name and namespace. Within the same namespace, clients can often use the short name; across namespaces, a qualified form such as `service-name.namespace` is used. Fully qualified service names follow the cluster domain convention, commonly ending in `svc.cluster-domain`.

DNS makes service discovery dynamic. Applications resolve a stable name while the set of backing pods changes. However, DNS is not instantaneous state synchronization. Client-side caching, JVM or language runtime behavior, negative caching, and connection pooling can delay the visibility of changes. Long-lived connections may continue using an old endpoint even after DNS data changes.

Headless Services, created with `clusterIP: None`, do not allocate a conventional virtual IP. DNS can return the individual pod addresses. They are useful for stateful systems, peer discovery, and applications that need endpoint-level awareness. They transfer more responsibility to the client, which must handle selection, failure, and often ordering or quorum behavior.

DNS troubleshooting should proceed systematically: verify the pod’s resolver configuration, query the expected name from within the cluster, confirm that the Service exists in the correct namespace, inspect EndpointSlices, and test connectivity separately from name resolution. A successful DNS lookup does not prove that the application is listening or that policy permits the connection.

From Ingress to Gateway API

Ingress provides an HTTP and HTTPS routing abstraction. An Ingress can route requests by host and path to Services and can define TLS termination. The Ingress resource is only a specification; an Ingress controller implements it using a reverse proxy, cloud load balancer, or another dataplane. Therefore, annotations and supported features vary substantially between controllers.

Ingress is effective for common web routing, but its original API has limited expressiveness and relies heavily on controller-specific annotations. It also tends to combine several concerns—routing, infrastructure ownership, and policy—into one resource model.

The Gateway API is a newer, role-oriented approach to traffic management. It separates infrastructure ownership from application routing through resources such as GatewayClass, Gateway, HTTPRoute, and related route types. A platform team may manage a Gateway while application teams manage route objects subject to explicit attachment and authorization rules. Gateway API also offers a more extensible model for protocols, cross-namespace references, traffic weighting, and policy attachment.

Gateway API does not eliminate the need to understand the implementation. Feature support varies by controller, and organizations must define who may attach routes, which namespaces are trusted, how certificates are managed, and how external addresses are governed. Migration from Ingress should be driven by operational requirements rather than novelty.

TLS may be terminated at an external load balancer, ingress controller, or gateway. End-to-end encryption to the workload may still be required for regulatory, trust-boundary, or zero-trust reasons. Certificate rotation should be automated, and private keys should be protected through appropriate secret-management controls.

NetworkPolicy and Workload Isolation

NetworkPolicy expresses permitted traffic for selected pods. It can restrict ingress, egress, or both. Policies are additive: traffic is allowed if it is permitted by at least one applicable policy, and a policy does not create a global deny rule for unrelated pods. Once a pod is isolated for a direction, only explicitly permitted traffic in that direction is allowed.

A practical policy design begins with flows rather than namespaces. For example, a web front end may accept traffic from an ingress gateway, call an API on a defined port, and query DNS. The API may accept traffic only from the web tier and connect to a database on its database port. Egress to arbitrary destinations should be justified rather than assumed.

Selectors may identify pods, namespaces, or IP blocks, depending on the policy expression. Namespace labels are safer than relying on ambiguous names when expressing cross-namespace trust. DNS access must be explicitly permitted when egress isolation is enabled. Administrators must also account for control-plane callbacks, metrics scraping, image retrieval, cloud metadata services, and external APIs.

A mature rollout commonly follows this sequence:

1. Document expected communication flows.

2. Observe traffic where tooling permits.

3. Apply policies in a test namespace.

4. Establish baseline DNS and essential infrastructure access.

5. Introduce default-deny policies carefully.

6. Add narrowly scoped allow rules.

7. Test both permitted and forbidden paths.

8. Monitor policy drops and review exceptions.

NetworkPolicy implementations are not identical in advanced features. Some support host endpoints, global policies, logging, denial rules, or layer-seven conditions beyond the standard API. Such features can be valuable but may reduce portability.

Load Balancing and Traffic-Management Strategies

Kubernetes distributes Service traffic among eligible endpoints, but the exact algorithm and connection behavior depend on the dataplane. Load balancing is often connection-oriented: a long-lived connection may remain with one endpoint while new connections are distributed elsewhere. This matters for WebSockets, streaming, gRPC, and uneven request costs.

Several traffic strategies are useful in delivery and operations:

• Rolling replacement gradually changes replicas while maintaining availability.
• Blue-green deployment maintains two environments and switches traffic between them.
• Canary deployment sends a controlled percentage or selected user segment to a new version.
• Header, cookie, or path routing directs particular requests to a chosen backend.
• Weighted routing distributes traffic according to configured proportions.
• Session affinity attempts to send a client repeatedly to the same endpoint, although it should not replace proper stateless design.

For canary releases, percentages alone are insufficient. A sound strategy combines routing control with indicators such as error rate, latency, saturation, business conversion, and rollback thresholds. Automated progressive delivery systems may adjust weights based on metrics, but automation can amplify a faulty signal. Human approval, bounded changes, and clear rollback behavior remain important for high-impact services.

Service meshes can add retries, timeouts, circuit breaking, mutual TLS, and detailed telemetry. These features improve control but add proxies, configuration, resource use, and operational complexity. Retries must be bounded and coordinated; otherwise, a failing dependency may receive more traffic during an incident. Timeouts should reflect the application’s actual latency budget.

Applied Case: A Multi-Tenant Retail Platform

Background

This illustrative case concerns a retail platform operated by a company with separate development, staging, and production namespaces. The platform contains a public web application, an internal order API, a payment adapter, a PostgreSQL database, and asynchronous workers. The company uses a cloud load balancer and wants to introduce canary releases without exposing the database or payment adapter to unnecessary traffic.

Situation and Problem

The first production deployment used separate LoadBalancer Services for the web application, API, and payment adapter. Costs and firewall rules grew rapidly. A deployment then exposed an API Service with no endpoints because its selector used `app: order-api` while the pods used `app: orders-api`. In addition, a permissive namespace configuration allowed a compromised test pod to connect to production-facing services.

Analysis

The architecture had three problems. First, stable access and external exposure were mixed: every externally visible component received its own infrastructure load balancer. Second, endpoint readiness and selector correctness were not included in deployment verification. Third, the network trust model was implicit rather than expressed through policy.

Approach

The platform team adopted one external Gateway for approved web entry points and defined HTTP routes for the web and API. The payment adapter remained internal and was called only by the order API. Services used consistent labels, named ports, and readiness probes. EndpointSlice checks were added to deployment validation.

Network policies implemented default-deny behavior for production application namespaces, followed by explicit rules. The gateway could reach the web tier; the web tier could reach the API; the API could reach the payment adapter and database on required ports; and workloads could query the cluster DNS service. Cross-namespace access required controlled namespace labels and review.

A canary route directed a small percentage of API traffic to the new version. Metrics for five-minute error rate, p95 latency, and payment failures controlled promotion. The team documented rollback as a route-weight change rather than a pod deletion operation.

Outcome

The design reduced the number of public entry points, made application ownership clearer, and limited lateral movement. The selector validation prevented a repeat of the empty-endpoint failure. During a later canary, elevated latency halted promotion and restored traffic to the stable version without replacing all workloads.

Learning Points

The case demonstrates that networking is a system of contracts. Services contract to provide stable identity; readiness contracts to expose only capable endpoints; policies contract to restrict communication; and gateway routes contract to define external behavior. Each contract requires validation, observability, and ownership.

Failure Modes, Risks, and Professional Governance

Common mistakes include using pod IPs in application configuration, assuming a Service automatically creates an external endpoint, confusing a DNS failure with a network-policy failure, exposing databases through NodePort, and applying default-deny policies without permitting DNS or monitoring traffic. Another frequent error is treating an Ingress manifest as portable when it depends on controller-specific annotations.

Operational risks include IP exhaustion, overlapping pod and corporate CIDRs, asymmetric routing, MTU problems caused by encapsulation, cloud load-balancer quota limits, stale DNS caches, and inconsistent controller versions. Changes to CNI or kube-proxy-like dataplanes should be rehearsed because a networking failure can affect the whole cluster.

Governance should assign ownership for IP ranges, gateways, certificates, namespaces, route attachment, and policy exceptions. External exposure must be reviewed for data classification and regulatory requirements. Network logs may contain client addresses, identifiers, or sensitive paths; retention and access should follow organizational privacy rules. Administrative access to change routes or disable policies should be strongly authenticated, audited, and separated from routine application deployment permissions.

Observability should cover DNS latency and failures, Service endpoint health, gateway response codes, connection counts, policy drops, packet loss, node routes, and application-level latency. Packet captures and flow logs are valuable, but they should be used under approved procedures because payloads may contain sensitive information.

Current Developments and Best Practices

Kubernetes networking is increasingly influenced by eBPF dataplanes, which can provide efficient forwarding, visibility, and policy enforcement in the kernel. Gateway API is developing as a more expressive alternative to the original Ingress model. Dual-stack deployments, encrypted pod traffic, identity-aware authorization, and platform-managed multi-cluster connectivity are also receiving greater attention.

These developments do not remove the need for foundational discipline. Recommended practices include:

• use Services rather than pod IPs for stable communication;
• keep internal and external exposure separate;
• standardize labels, named ports, readiness probes, and ownership metadata;
• treat NetworkPolicy as code and test it in CI and staging;
• use least privilege for route attachment and cross-namespace access;
• automate certificate management and secret protection;
• monitor endpoints, DNS, policy decisions, and gateway behavior;
• design deployment routing with measurable promotion and rollback criteria;
• document CNI assumptions, CIDRs, MTU, encryption, and failure recovery;
• verify controller-specific behavior before promising portability.

Chapter Summary

Kubernetes networking gives pods logical connectivity while recognizing that pod addresses are temporary. CNI plugins implement address allocation, interfaces, routing, and often policy and observability. Services provide stable identity and distribute traffic to ready endpoints; DNS makes those Services discoverable. Ingress supports common HTTP exposure, while Gateway API provides a more role-oriented and extensible traffic model. NetworkPolicy expresses least-privilege communication but requires careful treatment of DNS, infrastructure, egress, and cross-namespace trust. Load balancing and progressive delivery strategies must account for connection behavior, application metrics, rollback, and operational complexity. Effective networking combines technical configuration with governance, security, testing, and observability.

Key Terms

Pod network: Logical network that provides addresses and connectivity for pods.

CNI: Standard interface through which container runtimes and orchestrators configure container networking.

Service: Stable Kubernetes abstraction exposing a selected set of pods.

ClusterIP: Virtual, normally internal IP assigned to a Service.

NodePort: Service mode exposing a port on each node.

LoadBalancer: Service mode requesting an external infrastructure load balancer.

EndpointSlice: Kubernetes resource representing backend endpoints for a Service.

Headless Service: Service without a virtual ClusterIP that exposes individual endpoints through DNS.

CoreDNS: Common Kubernetes DNS server used for service discovery.

Ingress: API resource for HTTP and HTTPS routing to Services.

Ingress controller: Component that implements Ingress behavior.

Gateway API: Kubernetes networking API offering role-oriented, extensible traffic routing.

NetworkPolicy: Resource defining permitted ingress and egress flows for selected pods.

Default deny: Policy posture in which traffic is blocked unless explicitly allowed.

Session affinity: Mechanism attempting to keep a client connected to the same backend.

Canary deployment: Progressive release sending controlled traffic to a new version.

Blue-green deployment: Release strategy switching traffic between two environments.

Service mesh: Infrastructure layer providing service-to-service traffic controls and telemetry.

MTU: Maximum transmission unit, or largest packet size supported on a network path.

Review Questions

Short Answer Questions

1. Why should applications avoid using pod IP addresses as permanent endpoints?

2. What is the role of a CNI plugin in a Kubernetes cluster?

3. Distinguish ClusterIP from LoadBalancer.

4. Why are readiness probes important to Service traffic?

5. What is the purpose of a headless Service?

Long Answer Questions

1. Explain the Kubernetes pod, node, and service networks and their relationships.

2. Compare Ingress and Gateway API in terms of ownership, extensibility, and implementation.

3. Describe a systematic method for designing NetworkPolicy rules for a three-tier application.

4. Discuss the operational and security factors involved in selecting a CNI plugin.

5. Explain how canary, blue-green, and rolling deployment strategies use traffic management differently.

Analytical / Application Questions

1. A Service exists, but clients receive connection failures and EndpointSlices are empty. Analyze the likely causes and propose diagnostic steps.

2. Design a network policy model for a web tier, API tier, database, and DNS service. State which flows should be allowed and why.

3. An organization has ten externally exposed HTTP applications and wants controlled canary releases. Recommend an exposure and routing architecture, including governance and rollback controls.

22. Multiple Choice Questions with Answers

Question 1

Which statement best explains why Kubernetes Services are needed?

A. They guarantee that pods never restart

B. They provide stable discovery and access to changing pod endpoints

C. They replace the CNI plugin

D. They encrypt every application connection automatically

Correct Answer: B. They provide stable discovery and access to changing pod endpoints

Question 2

A Service has no endpoints even though pods appear to be running. Which is the most likely first investigation?

A. Check whether the Service selector matches pod labels

B. Increase the node CPU limit

C. Delete the cluster DNS deployment

D. Change the pod CIDR

Correct Answer: A. Check whether the Service selector matches pod labels

Question 3

What is a defining characteristic of a headless Service?

A. It is always externally exposed

B. It allocates a unique public IP to every pod

C. It does not provide a conventional virtual ClusterIP

D. It disables Kubernetes DNS

Correct Answer: C. It does not provide a conventional virtual ClusterIP

Question 4

When egress isolation is enabled by NetworkPolicy, which commonly overlooked flow may need explicit permission?

A. Access to the cluster DNS service

B. Access to an unused NodePort

C. Access to every namespace automatically

D. Access to stopped pods

Correct Answer: A. Access to the cluster DNS service

Question 5

What is the principal architectural distinction of Gateway API compared with traditional Ingress?

A. Gateway API eliminates Services

B. Gateway API separates infrastructure and route responsibilities more explicitly

C. Gateway API works only for databases

D. Gateway API prevents TLS termination

Correct Answer: B. Gateway API separates infrastructure and route responsibilities more explicitly

Question 6

A pod replacement receives a new IP address. Which design remains resilient without application reconfiguration?

A. Clients connect directly to the old pod IP

B. Clients use the Service DNS name

C. Clients connect to the node's loopback address

D. Clients disable readiness probes

Correct Answer: B. Clients use the Service DNS name

Question 7

Which deployment strategy sends a controlled proportion of requests to a new version while monitoring its behavior?

A. Canary deployment

B. Headless discovery

C. Default-deny isolation

D. NodePort allocation

Correct Answer: A. Canary deployment

Question 8

What does a default-deny NetworkPolicy posture require operators to do?

A. Permit all traffic through the gateway

B. Add explicit rules for required communication flows

C. Remove all Service selectors

D. Disable DNS caching

Correct Answer: B. Add explicit rules for required communication flows

Question 9

Which issue is particularly associated with overlay networking?

A. Encapsulation-related MTU and troubleshooting concerns

B. Inability to assign pod IP addresses

C. Automatic exposure of all pods to the Internet

D. Removal of the node network

Correct Answer: A. Encapsulation-related MTU and troubleshooting concerns

Question 10

Why can percentage-based traffic splitting alone be insufficient for a canary release?

A. Percentages cannot be configured in Kubernetes

B. It provides no evidence about errors, latency, or business impact

C. It always disables load balancing

D. It prevents rollback through route changes

Correct Answer: B. It provides no evidence about errors, latency, or business impact