Transport Protocols and Congestion Control

Introduction

The transport layer provides process-to-process communication above the network layer. Whereas IP delivers packets between hosts without guaranteeing delivery, ordering, or rate compatibility, transport protocols establish the communication model required by applications. They may provide reliable byte streams, message-oriented delivery, encryption, multiplexing, flow control, congestion control, or some combination of these services.

For data-science systems, transport behavior is a performance concern rather than merely an implementation detail. Distributed training, cloud storage, streaming analytics, database replication, telemetry, and large-scale model serving all depend on how endpoints react to loss, delay, limited buffers, and changing bandwidth. A protocol that performs well on a short, stable local-area path may perform poorly on a long, wireless, or highly shared path.

This chapter examines TCP, UDP, QUIC, reliability mechanisms, socket behavior, flow control, congestion-control algorithms, and optimization strategies for data-intensive applications.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Explain the services and limitations of the transport layer.
2. Compare TCP, UDP, and QUIC in terms of reliability, ordering, security, multiplexing, and deployment.
3. Distinguish flow control from congestion control.
4. Analyze sliding-window protocols, retransmission, acknowledgements, and timeout estimation.
5. Compare major TCP congestion-control approaches.
6. Explain socket states and common transport-layer failure modes.
7. Select and optimize transport protocols for data-intensive systems.
8. Evaluate a transport design using throughput, latency, fairness, and operational constraints.

1. Transport-Layer Service Models

A transport protocol accepts data from an application and divides it into transport segments or datagrams. At the receiver, it validates, reorders, reassembles, and delivers data to the appropriate process. The receiving process is identified by a port number, while an endpoint is commonly represented by an IP address and port pair.

Two broad service models dominate modern networks:

• Connection-oriented, reliable byte-stream service: TCP establishes state at both endpoints and presents an ordered stream of bytes. It retransmits lost data and suppresses duplicates.
• Connectionless datagram service: UDP sends independent datagrams with minimal transport-layer machinery. It does not inherently guarantee delivery, ordering, duplicate suppression, or congestion control.

QUIC combines reliable transport features with user-space deployment and mandatory cryptographic protection. It runs over UDP but is not equivalent to raw UDP; it implements its own packet numbering, acknowledgements, streams, retransmission logic, and congestion control.

Transport protocols must address several distinct problems:

2. TCP: Reliable Byte-Stream Transport

2.1 Connection establishment and termination

TCP uses a three-way handshake. The client sends a SYN containing an initial sequence number. The server replies with SYN-ACK, acknowledging the client’s sequence number and announcing its own. The client then sends ACK. This exchange confirms bidirectional reachability and synchronizes sequence-number spaces.

TCP is full duplex: each direction has an independent sequence space, acknowledgement state, and flow-control window. A connection is identified by a four-tuple:

(source IP, source port, destination IP, destination port)

A server normally performs bind, listen, and accept; a client performs connect. The listening socket is distinct from each established connection socket. Consequently, one server port can support many simultaneous clients.

TCP termination commonly uses FIN and ACK segments in each direction. Because each direction closes independently, termination may involve four logical messages. An endpoint in TIME_WAIT retains state for a period so that delayed segments from an old connection do not interfere with a new connection using the same four-tuple. Excessive short-lived connections can therefore exhaust ephemeral ports or create many TIME_WAIT entries.

2.2 Reliability and ordering

TCP numbers bytes rather than application messages. If an application writes three records, the receiver may read them as one, several, or partial records. Applications requiring message boundaries must implement framing, for example by using a length prefix, delimiter, or fixed-size record format.

A TCP receiver acknowledges received data, usually cumulatively. An ACK for sequence number k indicates that all bytes before k have arrived in order and that k is the next expected byte. Selective acknowledgements (SACK) allow the receiver to report noncontiguous blocks that have already arrived, helping the sender retransmit only missing ranges.

TCP detects corruption with a checksum. A corrupted segment is discarded, and loss is inferred through duplicate acknowledgements, an expiry of the retransmission timer, or other mechanisms. Retransmission provides eventual recovery, subject to connection failure and application timeouts.

2.3 TCP performance model

A useful approximation for long-lived TCP throughput is the bandwidth-delay product (BDP):

Here, C is bottleneck capacity in bits per second, RTT is round-trip time in seconds, and BDP is in bits. It represents the amount of data that can be in transit when a path is fully utilized. A 1 Gb/s path with a 40 ms RTT has:

Thus, approximately 5 MB of outstanding data may be needed to fill the path. This calculation assumes a stable bottleneck rate and ignores protocol headers, competing traffic, receiver limitations, and loss.

A widely used loss-based approximation is:

MSS is the maximum segment size in bytes, p is the steady-state segment-loss probability, RTT is in seconds, and K is a model-dependent constant. This is an approximation for particular congestion-avoidance conditions; it is not a universal TCP law. It does not accurately predict short transfers, modern model-based algorithms, application-limited traffic, or severe retransmission timeout behavior.

3. UDP and QUIC

3.1 UDP

UDP supplies source and destination ports, length, and a checksum. It preserves datagram boundaries: one send operation corresponds conceptually to one datagram, although the network may discard, duplicate, or reorder it. UDP has no handshake, retransmission, ordering, receiver flow control, or standardized congestion-control behavior.

UDP is suitable when the application can tolerate loss or must define its own semantics. Examples include DNS queries, real-time media, telemetry, online games, and protocols that implement reliability above UDP. The application remains responsible for sequence numbers, acknowledgements, retransmission, rate control, and security when needed.

UDP is not automatically faster than TCP. If an application requires reliable delivery, adding reliability, congestion control, and security may recreate much of TCP’s complexity. An uncontrolled UDP sender can also harm other traffic and may violate network policy.

3.2 QUIC

QUIC is a secure, multiplexed transport protocol standardized for use with HTTP/3. It runs over UDP to facilitate deployment and allows rapid evolution in user space. QUIC integrates TLS 1.3 handshakes, so encryption and endpoint authentication are fundamental rather than optional add-ons.

Important QUIC properties include:

• Streams: One connection carries multiple independently ordered streams.
• Reduced head-of-line blocking: Loss on one stream need not prevent delivery of already available data on other streams, although packet loss still consumes capacity and can affect congestion control.
• Connection identifiers: A connection can survive changes in network path, such as movement between Wi-Fi and cellular networks.
• Encrypted transport metadata: Most transport information is protected from passive observers.
• User-space evolution: Implementations can be updated without changing the operating system kernel.
• Integrated congestion control: QUIC requires congestion control and acknowledgement mechanisms comparable in purpose to TCP’s, while allowing implementation variation.

QUIC does not eliminate latency, loss, or congestion. Encryption can also complicate network monitoring, middlebox inspection, and troubleshooting. Its advantages are strongest for applications needing secure multiplexing, connection migration, or rapid protocol deployment.

4. Flow Control and Congestion Control

These mechanisms are related but solve different problems.

Flow control prevents a sender from overwhelming a receiver whose application is reading slowly or whose receive buffer is full. TCP advertises a receive window, commonly denoted rwnd.

Congestion control prevents a sender from overwhelming the network path. The sender maintains a congestion window, denoted cwnd, based on signals such as loss, delay, or explicit congestion notification.

The effective TCP sending window is approximately:

Both windows are measured in bytes or, in some algorithmic descriptions, segments. This relationship assumes the sender is governed by both receiver capacity and network capacity. If rwnd is small, improving the network does not increase throughput; if cwnd is small, increasing the receiver buffer alone does not help.

The advertised receive window may be scaled during the handshake to support windows larger than 65,535 bytes. Window scaling is important on high-BDP paths. However, excessively large buffers can create bufferbloat: queues become deep, latency increases, and loss may occur after long delays.

5. Congestion-Control Algorithms

5.1 Slow start

TCP begins cautiously because the available path capacity is unknown. In slow start, the congestion window grows rapidly, approximately doubling each RTT when acknowledgements arrive promptly. The growth is limited by a slow-start threshold, ssthresh, and by receiver capacity.

The name is historical: the algorithm is initially conservative, but exponential growth can become aggressive on high-capacity paths. When congestion is detected, TCP reduces its sending rate and enters congestion avoidance or recovery.

5.2 Congestion avoidance and additive increase

In congestion avoidance, classic TCP increases cwnd more gradually, approximately one MSS per RTT. If a loss is interpreted as congestion, the sender reduces its congestion window. This additive-increase, multiplicative-decrease (AIMD) behavior encourages sharing among competing flows under certain assumptions.

A simplified congestion-avoidance update is:

The update is applied as acknowledgements arrive; over one RTT, the aggregate increase is approximately one MSS. The expression assumes a congestion window measured in bytes and an MSS measured in bytes. Implementations use more precise byte-counting rules and must account for delayed acknowledgements and acknowledged data.

5.3 Loss recovery

Three duplicate ACKs traditionally trigger fast retransmit, because the receiver has evidence that a segment is missing while later data arrived. Fast recovery reduces the sending rate without returning to the initial slow-start state. A retransmission timeout indicates a more serious uncertainty and generally causes a stronger reduction.

Modern TCP uses SACK, retransmission timers, and increasingly sophisticated loss-detection logic. The retransmission timeout is based on smoothed estimates of RTT and variation rather than a fixed constant. A common form is:

SRTT is the smoothed round-trip time and RTTVAR is the smoothed mean deviation. Implementations impose minimum and maximum bounds and apply exponential backoff after timeout. The formula is meaningful only with carefully defined update rules and timestamp samples; it should not be implemented as an isolated equation.

5.4 Major TCP variants

• Tahoe: Uses slow start, congestion avoidance, and fast retransmit; a loss causes a strong reduction, traditionally resetting the congestion window to one MSS.
• Reno/NewReno: Adds fast recovery and handles multiple losses more effectively than Tahoe, but may perform poorly with many losses in one window.
• SACK-based TCP: Reports received blocks, reducing unnecessary retransmission and improving recovery from multiple losses.
• CUBIC: Uses a cubic growth function based on elapsed time since the last congestion event. It is designed for high-speed, long-distance networks and is widely deployed.
• BBR: Estimates bottleneck bandwidth and round-trip propagation time, attempting to operate near the path’s delivery rate while limiting queue growth. It is model-based rather than purely loss-based.

No variant is best in every environment. Loss-based algorithms may fill buffers and produce high delay, while delay-sensitive algorithms may underutilize paths with noisy RTT measurements. Model-based algorithms can have fairness and coexistence differences when competing with loss-based flows. Evaluation should therefore include throughput, latency, loss, fairness, and workload completion time.

5.5 Explicit Congestion Notification

ECN allows routers to mark packets instead of dropping them when queues begin to build, provided that endpoints and network devices support it. The receiver echoes the congestion indication, and the sender reduces its rate. ECN can reduce loss and latency, but deployment requires correct configuration and operational monitoring.

6. Multiplexing, Socket Behavior, and Application Framing

Transport multiplexing maps many processes to network traffic. A TCP listening socket accepts connections, while each accepted socket represents one client-server flow. A UDP socket may receive datagrams from many peers unless the application uses connected-UDP behavior to restrict the peer.

Important socket concerns include:

• Blocking versus nonblocking I/O: Blocking calls wait for data or buffer space; nonblocking calls return immediately and commonly use select, poll, epoll, or similar mechanisms.
• Partial writes and reads: A successful write does not guarantee that the peer has received the bytes. A read may return fewer bytes than requested.
• Backpressure: When send buffers fill, a sender must slow down or risk memory growth and timeout.
• Nagle’s algorithm and delayed ACKs: These can improve efficiency for small writes but may add latency. Applications with interactive, small messages may use appropriate socket options after measurement.
• Keepalives and application heartbeats: They detect idle failures but consume traffic and do not replace meaningful request deadlines.
• Half-close: A TCP endpoint can signal that it will send no more data while continuing to receive.
• Error handling: Connection reset, timeout, unreachable destination, and orderly EOF have different meanings and should not be collapsed into one generic error.

TCP’s byte-stream abstraction makes application framing essential. For data pipelines, length-prefixed binary records are usually more robust than delimiter-only framing, especially when payloads may contain arbitrary bytes.

7. Worked Example: Window Limits and Throughput

Consider a data-transfer service with capacity C = 200 Mb/s and RTT = 80 ms. The receiver advertises rwnd = 1 MB, while the sender’s congestion window is currently cwnd = 4 MB.

The bandwidth-delay product is:

The effective sending window is:

If the sender is continuously limited by this window, an approximate window-limited throughput is:

This is approximately 100 Mb/s, ignoring protocol overhead and implementation effects. The path could support about 2 MB in flight at 200 Mb/s, but the receiver allows only 1 MB. Increasing cwnd will not solve the bottleneck; the receive buffer and advertised window must be increased, subject to memory and latency considerations.

8. Transport Optimization for Data-Intensive Applications

Optimization begins with measurement. Record connection setup time, RTT distribution, retransmissions, congestion-window behavior, goodput, CPU utilization, queueing delay, and application-level completion time. Average throughput alone can hide tail latency and repeated retries.

8.1 Large transfers

For bulk data, use persistent connections, sufficiently large buffers, parallelism only where justified, and a protocol with effective loss recovery. Parallel TCP connections may increase aggregate throughput but can harm fairness and increase congestion. Application-level chunking supports checkpointing, retrying failed ranges, and distributed storage, but chunk sizes should balance scheduling overhead against recovery cost.

8.2 Interactive and latency-sensitive traffic

Use small, bounded messages, explicit deadlines, connection reuse, and prioritization. QUIC streams can separate independent requests, while UDP may be appropriate for media or state updates that tolerate loss. Reliability should be selective: retransmitting obsolete sensor data may be less useful than sending the newest value.

8.3 Distributed data science workloads

Distributed training and shuffle systems generate many simultaneous flows. Poor transport configuration can produce incast, in which many senders transmit toward one receiver and overflow a switch or host buffer. Mitigations include receiver-aware scheduling, bounded concurrency, pacing, partition sizing, ECN, and application-level backpressure.

Data transfers across regions require attention to BDP, cloud egress cost, encryption overhead, and failure recovery. Compression reduces bytes transmitted but consumes CPU and may increase latency. Encryption is normally mandatory for sensitive datasets; optimization must not weaken authentication or confidentiality.

8.4 Pacing and fairness

Pacing spreads packets over time rather than emitting an entire congestion window in a burst. It can reduce queue oscillation and loss, particularly after an ACK burst or application stall. Fairness should be evaluated across flows, tenants, and traffic classes. A transport optimization that improves one job by starving other users is not operationally successful.

9. Case Study: A Distributed Feature-Store Ingestion Service

This is an illustrative case.

Background

A company ingests feature data from 300 regional workers into a central feature store. Each worker sends compressed batches over long-lived secure connections. The initial implementation uses one TCP connection per worker and reports low average network utilization.

Problem

Training jobs experience highly variable completion times. Monitoring shows that some transfers achieve near-link speed while others stall after packet loss. The central receiver also experiences bursts of memory consumption.

Analysis

The path RTT is approximately 60 ms, and the central link is 500 Mb/s. Its BDP is:

Several workers use receive buffers below 1 MB, making them receiver-window limited. In addition, all workers flush batches simultaneously, causing bursty traffic and queue overflow. Retries occur at the application layer without coordination with TCP, so a lost batch may be retransmitted while the original TCP transfer is still recovering.

Approach

The engineering team applies the following changes:

1. Increase receive and send buffers within measured memory limits.
2. Use persistent connections and length-prefixed batch framing.
3. Add application-level deadlines and idempotent batch identifiers.
4. Stagger worker flush times using randomized scheduling.
5. Limit concurrent uploads per worker and apply backpressure when receiver queues grow.
6. Enable SACK and ECN where supported.
7. Compare the deployed TCP algorithm with CUBIC and a model-based alternative in a controlled test.
8. Record p50, p95, and p99 batch completion time rather than only mean throughput.

Outcome

The principal improvement comes not from selecting a new protocol, but from matching windows to BDP and eliminating synchronized bursts. Tail completion time decreases, receiver memory becomes predictable, and retries become less disruptive. A QUIC prototype is retained for mobile workers that change networks, while TCP remains suitable for stable data-center paths.

Learning Points

Transport optimization is a systems problem. Protocol selection, buffer sizing, application batching, scheduling, congestion behavior, and observability interact. A protocol should be selected according to workload and path characteristics rather than fashion.

10. Limitations, Governance, and Contemporary Developments

Transport measurements can be misleading when encryption hides packet details, when middleboxes alter behavior, or when traffic is application-limited. Laboratory results may not generalize to public networks with asymmetric paths, wireless loss, or competing flows. Reproducible testing should document RTT, capacity, loss, queue policy, endpoint versions, and workload shape.

Security considerations include SYN-flood resistance, resource exhaustion, spoofing, amplification, certificate validation, and protection of sensitive metadata. Servers should limit connection rates, enforce request and idle deadlines, validate message sizes, and avoid unbounded buffering. UDP-based services require particular care because unauthenticated requests can enable amplification attacks.

QUIC and HTTP/3 continue to influence transport design by moving functionality into user space and integrating security. Multipath transport, improved congestion-control research, programmable network interfaces, and application-aware scheduling are active areas of development. Machine-learning-based congestion control is promising but must be assessed for stability, fairness, explainability, and safe behavior under distribution shift; a model that maximizes local throughput can damage shared-network performance.

Best-Practice Checklist

• Choose TCP, UDP, or QUIC according to reliability, ordering, security, mobility, and latency requirements.
• Separate receiver flow control from network congestion control.
• Size buffers with BDP, memory, and latency objectives in mind.
• Use framing over byte streams and validate lengths before allocation.
• Reuse connections where appropriate and handle partial I/O correctly.
• Apply deadlines, bounded retries, and idempotent operation identifiers.
• Measure tail latency, retransmission, goodput, fairness, and CPU cost.
• Use pacing, ECN, and backpressure when supported and justified by tests.
• Protect UDP and QUIC services against spoofing, amplification, and resource exhaustion.
• Benchmark under realistic RTT, loss, concurrency, and workload conditions.

Chapter Summary

Transport protocols convert host-to-host packet delivery into application-level communication. TCP provides a reliable, ordered byte stream using sequence numbers, acknowledgements, retransmission, flow control, and congestion control. UDP provides lightweight datagrams but leaves reliability and rate control to the application. QUIC uses UDP as a deployment substrate while providing encrypted, multiplexed, reliable streams, connection migration, and integrated congestion control.

Flow control protects the receiver; congestion control protects the shared network. The effective sending window is constrained by the smaller of the receive window and congestion window. Slow start, congestion avoidance, fast recovery, SACK, CUBIC, BBR, ECN, pacing, and accurate RTT estimation are central to modern transport performance. Socket semantics, framing, partial I/O, deadlines, and backpressure are equally important at the application boundary.

For data-intensive systems, optimization requires BDP-aware buffering, controlled concurrency, persistent connections, careful batching, realistic measurement, and security-conscious operations. No single transport variant is universally superior; the correct design depends on workload, path, failure tolerance, and governance requirements.

Key Terms

Transport layer: Network layer responsible for process-to-process communication.

TCP: Reliable, connection-oriented byte-stream protocol.

UDP: Connectionless datagram protocol with minimal transport services.

QUIC: Encrypted, multiplexed transport protocol implemented over UDP.

Socket: Operating-system interface representing an endpoint for network communication.

Port: Transport-layer identifier for an application process or service.

Sequence number: Number used to identify the position of data in a stream.

Acknowledgement: Feedback indicating data received or expected.

Retransmission: Sending data again after suspected loss.

Flow control: Regulation that prevents receiver-buffer exhaustion.

Congestion control: Regulation that limits load imposed on a network path.

Receive window (rwnd): Receiver-advertised available buffer capacity.

Congestion window (cwnd): Sender estimate of permissible in-flight network data.

BDP: Bandwidth-delay product, approximately the path capacity multiplied by RTT.

MSS: Maximum TCP payload size in one segment.

RTT: Round-trip time between endpoints.

SACK: Selective acknowledgement of noncontiguous received blocks.

AIMD: Additive-increase, multiplicative-decrease control strategy.

ECN: Explicit Congestion Notification through packet marking.

Head-of-line blocking: Delay caused when missing earlier data prevents later data delivery.

Review Questions

Short Answer Questions

1. What services distinguish TCP from UDP?
2. Why is application framing required over TCP?
3. Define flow control and congestion control.
4. What does the bandwidth-delay product represent?
5. State two important advantages of QUIC.

Long Answer Questions

1. Explain TCP connection establishment, reliable delivery, flow control, and termination.
2. Compare TCP, UDP, and QUIC for a distributed data-science application.
3. Describe slow start, congestion avoidance, fast retransmit, and fast recovery.
4. Compare loss-based and model-based congestion-control approaches.
5. Discuss socket behavior and application practices required for robust high-throughput services.

Analytical / Application Questions

1. A path has capacity 400 Mb/s and RTT 50 ms. Calculate its BDP and explain how the result should influence buffer sizing.
2. A service has high throughput but severe p99 latency after increasing its TCP buffers. Diagnose the likely mechanism and propose tests and remedies.
3. Design a transport strategy for a mobile analytics client that sends both urgent status updates and large historical batches. Justify protocol, multiplexing, reliability, congestion, and security choices.

Multiple Choice Questions with Answers

Question 1

Which mechanism primarily protects a receiver from a sender that transmits faster than the receiver can process data?

A. Congestion control

B. Flow control

C. Slow start

D. ECN

Correct Answer: B. Flow control

Question 2

A path has capacity 100 Mb/s and RTT 40 ms. What is its approximate bandwidth-delay product?

A. 50,000 bytes

B. 250,000 bytes

C. 500,000 bytes

D. 4,000,000 bytes

Correct Answer: C. 500,000 bytes

Question 3

What does a TCP cumulative ACK for byte number k conventionally indicate?

A. Only byte k was corrupted

B. All bytes before k arrived and k is next expected

C. The receiver has no buffer space

D. The connection must terminate

Correct Answer: B. All bytes before k arrived and k is next expected

Question 4

Which characteristic is native to QUIC but not to a traditional single TCP byte stream?

A. IP addressing

B. Checksums

C. Independent multiplexed streams

D. Packet transmission

Correct Answer: C. Independent multiplexed streams

Question 5

If rwnd is 1 MB and cwnd is 4 MB, which value approximately limits the sender’s in-flight data?

A. 256 KB

B. 1 MB

C. 4 MB

D. 5 MB

Correct Answer: B. 1 MB

Question 6

Which protocol preserves application datagram boundaries without providing inherent retransmission or ordering?

A. TCP

B. UDP

C. QUIC stream transport

D. TLS

Correct Answer: B. UDP

Question 7

What is the main purpose of TCP SACK?

A. Encrypting packet payloads

B. Reporting received noncontiguous blocks

C. Increasing the IP address space

D. Closing idle sockets

Correct Answer: B. Reporting received noncontiguous blocks

Question 8

Which event traditionally causes TCP fast retransmit?

A. One successful ACK

B. Three duplicate acknowledgements

C. A larger receive window

D. A completed DNS lookup

Correct Answer: B. Three duplicate acknowledgements

Question 9

Why can very large buffers increase application latency?

A. They eliminate all retransmissions

B. They can create bufferbloat and long queues

C. They disable port numbers

D. They prevent multiplexing

Correct Answer: B. They can create bufferbloat and long queues

Question 10

Which design is most appropriate for a TCP service carrying variable-length binary records?

A. Assume each read returns one complete record

B. Use explicit length-prefixed framing

C. Disable all acknowledgements

D. Send records without boundaries

Correct Answer: B. Use explicit length-prefixed framing