3. Information Theory and Communication Limits

Introduction and Context

Digital communication systems are ultimately constrained by two resources: the information that must be represented and the physical channel through which that information must pass. Modulation, multiplexing, equalization, antennas, and digital signal processing improve the use of a channel, but they do not eliminate fundamental limits imposed by uncertainty, noise, bandwidth, and energy. Information theory provides the mathematical framework for understanding these limits.

The central question is not merely whether a signal can be transmitted, but whether information can be transmitted reliably at a specified rate and quality. This distinction is important. A system may operate with a low probability of error at one data rate and fail completely at a slightly higher rate. Similarly, compression can reduce the number of bits needed to describe a source, while channel coding can add carefully structured redundancy to protect those bits from noise. These operations appear opposite, but together they form the conceptual basis of modern digital communications.

This chapter develops entropy, mutual information, channel capacity, source and channel coding theorems, rate-distortion theory, and Shannon limits. It also interprets these ideas for system design, including bandwidth-versus-power trade-offs, spectral efficiency, energy efficiency, coding choices, and the role of contemporary coded communication systems.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Define self-information, entropy, conditional entropy, mutual information, and relative entropy.

2. Calculate and interpret the entropy of discrete sources and the capacity of common communication channels.

3. Distinguish lossless source coding, lossy source coding, and channel coding.

4. Apply the source coding and channel coding theorems to communication-system decisions.

5. Explain rate-distortion theory and the meaning of distortion constraints.

6. Interpret the additive white Gaussian noise capacity formula and Shannon limits.

7. Analyse bandwidth, power, spectral efficiency, and energy-per-bit trade-offs.

8. Identify practical limitations, implementation risks, and emerging developments in information-theoretic communication systems.

Information as Reduction of Uncertainty

Suppose a source produces symbols from a random variable X. Before observing a particular symbol, the receiver has uncertainty about the outcome. Information theory measures the information associated with an outcome x as its self-information,

I(x) = log₂(1/P(x)) = −log₂ P(x) bits.

A rare event carries more information than a common event. If a symbol has probability one, its occurrence is fully predictable and its information content is zero. The logarithm makes information additive for independent events: the information in two independent outcomes is the sum of their individual information values.

The average information per source symbol is entropy,

H(X) = −Σₓ P(x) log₂ P(x).

Entropy is therefore an average uncertainty, not a measure of the semantic importance of a message. A highly meaningful sentence may have low entropy if it is statistically predictable, while a random sequence may have high entropy even though it carries no human meaning. This distinction is essential in engineering: compression exploits statistical predictability, not meaning.

For a binary source with probabilities p and 1 − p,

H(X) = −p log₂ p − (1 − p) log₂(1 − p).

Entropy is zero when the source is deterministic and reaches one bit per symbol when the two binary outcomes are equally likely. More generally, an alphabet of M equally likely symbols has maximum entropy log₂ M bits per symbol.

Joint and Conditional Entropy

When two random variables X and Y are considered together, their joint entropy is

H(X,Y) = −ΣₓΣᵧ P(x,y) log₂ P(x,y).

Conditional entropy measures the uncertainty remaining in one variable after the other is known:

H(X|Y) = −ΣₓΣᵧ P(x,y) log₂ P(x|y).

The chain rule states that

H(X,Y) = H(X) + H(Y|X).

This relation is useful because it separates total uncertainty into uncertainty about X and the additional uncertainty about Y once X is known. Conditioning cannot increase average uncertainty, so H(X|Y) ≤ H(X).

For a memoryless source, symbols are statistically independent and the entropy of a sequence of n symbols is nH(X). Real sources, however, often contain memory. Speech, images, sensor measurements, and text exhibit correlation, so their entropy rate is generally lower than the entropy calculated by treating every symbol as independent. Effective compression algorithms exploit this memory through prediction, transformation, dictionary construction, or probabilistic modelling.

Mutual Information and the Information Carried by a Channel

A communication channel accepts an input random variable X and produces an output random variable Y. Noise makes Y uncertain even when X is known, and the channel output may not reveal X completely. The mutual information is

I(X;Y) = H(X) − H(X|Y)

= H(Y) − H(Y|X).

It measures the reduction in uncertainty about X obtained by observing Y. Mutual information is symmetric: I(X;Y) = I(Y;X), although the engineering interpretation usually concerns information about the transmitted input obtained from the received output.

Another useful identity is

I(X;Y) = H(X) + H(Y) − H(X,Y).

If X and Y are statistically independent, mutual information is zero. If Y determines X without ambiguity, mutual information equals H(X). In a noisy channel, it lies between these extremes.

The data-processing inequality states that processing cannot create information about an original source. If X → Y → Z forms a Markov chain, then

I(X;Z) ≤ I(X;Y).

Thus, an intermediate detector, quantizer, or digital processing stage cannot increase the information that the received waveform contains about the transmitted data. It may preserve information, or lose some through noise and irreversible processing, but it cannot manufacture channel information.

From Mutual Information to Channel Capacity

For a specified channel, the mutual information depends on the probability distribution used at the input. Channel capacity is the maximum reliable information rate permitted by the channel:

C = maxₚ(x) I(X;Y).

For a discrete memoryless channel, capacity is measured in bits per channel use. The maximizing distribution is not always uniform. A channel with unequal error probabilities, constrained input symbols, or asymmetric costs may achieve its greatest mutual information with a nonuniform input distribution.

The capacity of a noiseless binary channel is one bit per channel use. For a binary symmetric channel with crossover probability p,

C = 1 − H₂(p),

where H₂(p) is the binary entropy function. When p = 0, capacity is one bit per use. When p = 0.5, the output is statistically independent of the input and capacity is zero. If p exceeds 0.5, the receiver can invert the output, so the capacity again follows 1 − H₂(p), with the relevant uncertainty determined by the crossover probability.

Capacity is an asymptotic concept. It does not claim that every finite block can operate exactly at capacity with zero error. Rather, the channel coding theorem states that rates below capacity can, in principle, be made reliable using sufficiently long codes, while rates above capacity cannot be made reliable regardless of coding sophistication. Practical systems operate below capacity because of finite block length, decoding complexity, latency, channel variation, imperfect channel knowledge, and hardware constraints.

Source Coding: Removing Redundancy

Source coding represents information using as few bits as possible. In lossless source coding, the original source sequence must be recovered exactly. The source coding theorem states that for a stationary memoryless source, the average length of a uniquely decodable lossless code cannot be less than the source entropy, while codes can be constructed with average length arbitrarily close to entropy for sufficiently long blocks.

For a binary source with probabilities 0.9 and 0.1, the entropy is approximately 0.469 bits per symbol, even though a fixed-length representation requires one bit per symbol. Variable-length codes such as Huffman coding exploit this unequal probability. Arithmetic coding and modern entropy coders can approach the entropy limit more closely for large, structured alphabets.

The distinction between entropy and average codeword length is important. Entropy is a theoretical lower bound, whereas a particular code has overhead caused by integer codeword lengths, finite blocks, framing, model mismatch, and error-protection requirements. Compression ratio must therefore be evaluated against the actual source statistics and implementation conditions.

Lossless compression is appropriate when any error changes the meaning or validity of the information, as in software, financial records, legal documents, and many scientific data sets. It is not always optimal for perceptual media, where a controlled amount of distortion may be acceptable.

Rate-Distortion Theory: Controlled Loss

Lossy source coding permits the reconstruction to differ from the original source according to a specified distortion measure. Let X be the source and X̂ its reconstruction. A distortion function d(x,x̂) quantifies the cost of representing x by x̂. Common choices include squared error, absolute error, symbol error, perceptual metrics, and application-specific quality measures.

The rate-distortion function is

R(D) = min I(X;X̂),

where the minimization is over all conditional distributions P(x̂|x) satisfying E[d(X,X̂)] ≤ D. It gives the minimum number of bits per source symbol required to achieve average distortion no greater than D.

R(D) generally decreases as the allowed distortion D increases. At zero distortion, it may approach the lossless rate; with sufficiently large distortion, the required rate can become zero because a constant reconstruction may satisfy the distortion constraint. The curve provides a principled way to compare compression operating points rather than choosing a codec only by nominal bit rate.

In image and video systems, distortion may be measured by mean-squared error, peak signal-to-noise ratio, structural similarity, or learned perceptual measures. These metrics are not interchangeable. A mathematically small error can be visually objectionable, while a larger numerical error may be perceptually insignificant. Consequently, rate-distortion optimisation in practical codecs often combines mathematical models with perceptual and application-level criteria.

Channel Coding: Adding Redundancy for Reliability

Channel coding deliberately adds redundancy so that the receiver can detect and correct errors caused by noise, interference, fading, and hardware imperfections. This redundancy increases the transmitted bit rate relative to the source-coded information rate, but it improves reliability.

A code with k information bits mapped into n transmitted bits has code rate R_c = k/n. Lower code rate means more redundancy and generally stronger protection, although it reduces useful information throughput. Block codes process finite blocks, while convolutional and trellis-based codes operate continuously or with memory. Modern systems commonly use turbo codes, low-density parity-check (LDPC) codes, and polar codes, selected according to block length, latency, decoding complexity, and target operating conditions.

The channel coding theorem establishes a remarkable result: reliable communication is possible at any information rate below channel capacity, provided sufficiently long and appropriately designed codes are used. Above capacity, the probability of decoding error cannot be driven to zero. The theorem does not provide a single practical code or decoder; it defines an attainable boundary against which implementations are judged.

Source and channel coding serve different purposes. Source coding removes redundancy introduced by the source, whereas channel coding adds redundancy to combat the channel. In an ideal architecture, compression is performed before error protection. If a compressed stream is highly sensitive to bit errors, unequal error protection, packet-level coding, or joint source-channel techniques may be used.

The Gaussian Channel and Shannon Limits

A widely used model for a band-limited communication link is the additive white Gaussian noise (AWGN) channel. For bandwidth B hertz, average signal power S, and noise power N, its capacity is

C = B log₂(1 + S/N) bits per second.

If the noise is represented by a one-sided power spectral density N₀ over bandwidth B, then N = N₀B and

C = B log₂(1 + S/(N₀B)).

The formula reveals two important design freedoms. Increasing power raises capacity, but with diminishing returns because of the logarithm. Increasing bandwidth also raises capacity and can reduce the power spectral density of the transmitted signal, but it may require more spectrum, higher sampling rates, and wider front-end components.

Spectral efficiency is defined as η = C/B, measured in bits per second per hertz:

η = log₂(1 + S/N).

Energy efficiency is often described through E_b/N₀, the energy per information bit divided by noise spectral density. Since E_b = S/C,

E_b/N₀ = (S/N₀B)/(C/B).

As spectral efficiency approaches zero, the theoretical minimum required E_b/N₀ for reliable communication over the ideal AWGN channel approaches −1.59 dB. This is the ultimate Shannon energy limit under the stated assumptions, not a guarantee achievable by an arbitrary practical system. At higher spectral efficiencies, the required E_b/N₀ increases, expressing the fundamental trade-off between bandwidth efficiency and power efficiency.

For example, if S/N = 15, the ideal spectral efficiency is log₂(16) = 4 bits/s/Hz. A practical modulation and coding scheme operating near this point must account for implementation loss, finite block length, synchronization errors, nonlinear amplification, channel estimation error, and target error probability.

A Design Framework for Communication Decisions

Information-theoretic analysis is most useful when translated into a sequence of engineering decisions.

First, characterize the source. Estimate symbol probabilities, temporal or spatial correlation, acceptable distortion, and traffic variability. This determines whether lossless or lossy compression is appropriate and provides an estimate of the source entropy or rate-distortion requirement.

Second, characterize the channel. Determine bandwidth, noise level, fading statistics, interference, outage requirements, latency, and available transmit power. A nominal signal-to-noise ratio is insufficient when the channel is time-varying or frequency-selective.

Third, compare the required information rate with channel capacity. If the source rate exceeds capacity, no coding adjustment can solve the problem under the same channel conditions. The designer must reduce source rate, increase bandwidth, increase power, improve antenna gain, reduce interference, or relax reliability and distortion requirements.

Fourth, select a coding and modulation operating point below capacity. The margin should reflect finite block length, channel uncertainty, implementation loss, and service-level requirements. A design that is theoretically feasible but has no operational margin is fragile.

Finally, evaluate system-level effects. Compression can increase sensitivity to residual errors; strong coding can increase latency and energy consumption; higher-order modulation can improve throughput but demands better SNR and linearity. The best operating point is therefore a constrained engineering choice rather than the point of maximum nominal capacity.

Applied Case Study: Designing a Low-Power Environmental Sensor Link

Background

Consider an illustrative wireless environmental sensor that reports temperature, humidity, and air-quality measurements to a gateway. The sensor is battery powered and must operate over a constrained radio channel. Measurements are generated every minute, but occasional alarm events require prompt delivery.

Situation and Problem

The raw measurement representation requires 160 bits per report. Statistical analysis shows that predictive lossless compression reduces the average report to 80 bits. The radio channel has an available bandwidth of 20 kHz and an estimated SNR of 7, corresponding to an ideal AWGN capacity of

C = 20,000 log₂(1 + 7) = 60,000 bits/s approximately.

The application requires a reliability of at least 99.99 percent for ordinary reports and stronger protection for alarm messages. The challenge is to select compression, coding, and transmission parameters without unnecessarily exhausting the battery.

Analysis

The compressed source rate is only 80 bits per minute, so average throughput is not the limiting factor. The relevant issues are energy per transmission, latency, burst errors, and reliability. A channel code with rate 1/2 would produce approximately 160 coded bits per report. Although this doubles the transmitted bits, it creates substantial error-correction capability and remains far below the channel capacity.

Using a very high-rate modulation would reduce transmission time but require higher SNR and potentially greater peak power. Since the sensor has low average traffic and values energy conservation, a robust low-order modulation with coding may be preferable. Alarm reports can be assigned a lower code rate or repeated transmission, while ordinary reports use a less redundant code. This is unequal error protection based on message importance.

Approach and Solution

The proposed design uses predictive lossless compression for ordinary measurements, because exact reconstruction is required for later analysis. A systematic error-correcting code protects the compressed packet, and a cyclic redundancy check detects residual undetected errors. Alarm packets receive stronger coding and acknowledgement-based retransmission. The gateway estimates channel quality and can select among a small set of modulation-and-coding modes.

The design also includes framing and sequence numbers. These do not increase information-theoretic capacity, but they are essential for detecting loss, duplication, and reordering. If the channel becomes severely degraded, the sensor stores measurements and transmits them when conditions improve rather than repeatedly transmitting at an inefficient operating point.

Outcome

The link has a large average capacity margin, but the design does not interpret that margin as permission to eliminate coding. Coding provides reliability against short-term noise and fading, while adaptive operation prevents excessive energy expenditure. Separating ordinary and alarm traffic allows reliability requirements to be met without applying the highest redundancy to every packet.

Learning Points

The case illustrates that capacity is a feasibility boundary, not a complete design. Source compression reduces the information burden, channel coding protects the reduced representation, and system constraints determine the practical operating point. It also demonstrates why average data rate alone is inadequate for evaluating low-power links.

Limitations, Risks, and Common Misinterpretations

A frequent error is to treat entropy as the amount of meaningful content. Entropy describes statistical uncertainty under a specified probability model. If the model is wrong, calculated entropy and predicted compression performance may be misleading.

A second error is to regard capacity as an achievable rate with zero delay and zero error. Shannon’s results are asymptotic. Short packets, strict latency limits, and rapidly changing channels can produce a significant gap from capacity. Finite-blocklength information theory gives more realistic rate-reliability-latency trade-offs for such systems.

A third error is to compare SNR values without specifying bandwidth, coding rate, modulation, and error criterion. Capacity depends on the channel model and constraints. A high SNR does not automatically imply high capacity if bandwidth is narrow or interference is structured rather than Gaussian.

Compression may also create a vulnerability: a single bit error in a compressed stream can affect many reconstructed symbols. Error-resilient syntax, packet segmentation, resynchronisation markers, and unequal protection reduce this risk. Conversely, excessive channel coding can consume energy and latency that the application cannot tolerate.

The AWGN model is analytically powerful but idealised. Real channels exhibit fading, interference, impulsive noise, nonlinear distortion, channel estimation errors, and spectrum regulations. Capacity calculations should therefore be treated as benchmarks, followed by link-level simulation, measurement, and margin analysis.

Professional and Regulatory Considerations

Engineers must not present theoretical capacity as a performance guarantee in product specifications. Claims should state assumptions, operating conditions, target error probability, packet size, and implementation margins. Spectrum usage is subject to national regulations, licensing conditions, emission masks, and coexistence requirements. Increasing bandwidth is not simply a technical choice when spectrum is scarce or shared.

Reliability decisions can also have safety implications. In industrial control, medical monitoring, and transportation systems, a communication failure may cause physical harm. The system should define acceptable loss, delay, integrity, and availability requirements, and should use independent monitoring and fail-safe behaviour where appropriate. Security is related but distinct from reliability: channel coding can correct accidental errors, but it does not provide confidentiality, authentication, or protection against deliberate manipulation.

Contemporary Developments

Modern communication systems increasingly operate close to theoretical limits through iterative decoding, soft information, adaptive modulation and coding, and multiple-antenna processing. LDPC and polar codes are prominent in contemporary standards because they offer strong performance with structured decoding algorithms. Massive multiple-input multiple-output systems exploit spatial dimensions in addition to time, frequency, and power.

Finite-blocklength theory is increasingly important for ultra-reliable low-latency communication, where asymptotically long codewords are impossible. Joint source-channel coding and semantic communication research investigate whether source meaning, task performance, or inference accuracy should replace conventional bit-perfect fidelity in selected applications. Machine learning is being applied to channel estimation, decoding, compression, and rate control, although learned systems must still be evaluated for generalisation, interpretability, computational cost, and robustness.

Best practice remains to use information theory as a design compass rather than as a substitute for engineering validation. A sound workflow combines source measurement, realistic channel modelling, capacity benchmarking, coding and modulation simulation, hardware testing, and continuous monitoring of field performance.

Chapter Summary

Information theory quantifies uncertainty and establishes limits on communication. Entropy measures the average uncertainty of a source, while conditional entropy measures residual uncertainty after side information is available. Mutual information quantifies the information transferred through a noisy channel, and channel capacity is the maximum mutual information over permissible input distributions.

Lossless source coding removes statistical redundancy and approaches the entropy rate. Lossy source coding trades reconstruction fidelity for rate according to the rate-distortion function. Channel coding adds redundancy so that information can be recovered reliably despite noise. Shannon’s channel coding theorem separates feasible rates from impossible rates, while the AWGN capacity formula exposes the trade-off among bandwidth, power, and spectral efficiency.

In practice, capacity is only a benchmark. Finite block lengths, latency, fading, interference, implementation loss, energy limits, and regulatory constraints determine the operating point. Effective system design therefore combines compression, error protection, adaptive modulation, realistic channel models, and application-specific reliability and distortion requirements.

Key Terms

Entropy: Average uncertainty or information produced by a random source, measured in bits when logarithms are base two.

Self-information: Information associated with an individual outcome, equal to −log₂ of its probability.

Joint entropy: Uncertainty in two or more random variables considered together.

Conditional entropy: Uncertainty remaining in one random variable after another is known.

Mutual information: Reduction in uncertainty about one variable obtained from observing another.

Channel capacity: Maximum information rate that can be transmitted with arbitrarily low error under a specified channel model and constraints.

Binary entropy function: Entropy of a binary random variable with a specified probability of either outcome.

Source coding: Representation of source information using a smaller or more suitable number of bits.

Lossless coding: Coding in which the original source is recovered exactly.

Lossy coding: Coding that permits controlled reconstruction error to reduce bit rate.

Rate-distortion function: Minimum rate required to achieve a specified average distortion.

Channel coding: Addition of structured redundancy to detect or correct channel errors.

Code rate: Ratio of information symbols to transmitted coded symbols.

AWGN channel: Channel model with additive white Gaussian noise.

Spectral efficiency: Information rate per unit bandwidth, measured in bits per second per hertz.

Energy per bit: Transmitted energy allocated to each information bit, commonly normalised by noise spectral density as E_b/N₀.

Shannon limit: Theoretical boundary on reliable communication performance under a specified model.

Data-processing inequality: Principle that processing cannot increase mutual information about an original variable.

Unequal error protection: Allocation of different error-control strengths to information of different importance.

Finite-blocklength regime: Communication regime in which latency and short codewords prevent asymptotic capacity assumptions from being exact.

Review Questions

Short Answer Questions

1. Define entropy and explain why a deterministic source has zero entropy.

2. What is the relationship between mutual information and conditional entropy?

3. Distinguish source coding from channel coding.

4. Define spectral efficiency and state its unit.

5. Why is channel capacity not normally achieved exactly in a practical system?

Long Answer Questions

1. Derive and interpret the capacity of a binary symmetric channel.

2. Explain the source coding theorem and discuss the practical roles of Huffman and arithmetic coding.

3. Describe rate-distortion theory and explain how distortion measures affect lossy compression design.

4. Derive the AWGN channel capacity expression and discuss the bandwidth-power trade-off.

5. Explain the channel coding theorem and analyse the effect of code rate, latency, and decoding complexity on practical design.

Analytical / Application Questions

1. A binary source produces symbol 1 with probability 0.9. Calculate its entropy and explain the implication for lossless compression.

2. A channel has bandwidth 100 kHz and SNR 15. Calculate its ideal AWGN capacity and spectral efficiency. Discuss at least two reasons why a real system may achieve less.

3. A sensor system permits a small reconstruction error but requires very low energy consumption. Decide whether lossless or lossy source coding is more suitable, and propose how channel coding and unequal error protection could be incorporated.

22. Multiple Choice Questions with Answers

Question 1

A binary source produces symbols with probabilities 0.5 and 0.5. What is its entropy?

A. 0 bits per symbol

B. 0.5 bits per symbol

C. 1 bit per symbol

D. 2 bits per symbol

Correct Answer: C. 1 bit per symbol

Question 2

Which quantity measures the reduction in uncertainty about X obtained by observing Y?

A. Joint entropy

B. Mutual information

C. Code rate

D. Distortion

Correct Answer: B. Mutual information

Question 3

For a binary symmetric channel with crossover probability 0.5, the capacity is:

A. 0 bits per channel use

B. 0.5 bits per channel use

C. 1 bit per channel use

D. Dependent only on bandwidth

Correct Answer: A. 0 bits per channel use

Question 4

The principal purpose of channel coding is to:

A. Remove all source redundancy

B. Increase source entropy

C. Add structured redundancy for error protection

D. Guarantee confidentiality

Correct Answer: C. Add structured redundancy for error protection

Question 5

Which statement best describes the rate-distortion function R(D)?

A. Maximum channel rate at a fixed SNR

B. Minimum source-coding rate for a specified distortion

C. Probability of a channel decoding error

D. Ratio of coded bits to information bits

Correct Answer: B. Minimum source-coding rate for a specified distortion

Question 6

An AWGN channel has bandwidth B and signal-to-noise ratio S/N. Its ideal capacity is:

A. B(S/N)

B. log₂(1+S/N)

C. B log₂(1+S/N)

D. S log₂(1+B/N)

Correct Answer: C. B log₂(1+S/N)

Question 7

If a communication system operates above the channel capacity under the same channel constraints, then:

A. A longer code can always achieve zero error

B. Increasing modulation order guarantees reliability

C. Reliable communication with arbitrarily small error is impossible

D. Source compression is irrelevant

Correct Answer: C. Reliable communication with arbitrarily small error is impossible

Question 8

A code maps 200 information bits into 400 transmitted bits. Its code rate is:

A. 0.25

B. 0.5

C. 2

D. 400

Correct Answer: B. 0.5

Question 9

Which design decision is most appropriate when alarm messages are more important than routine sensor reports?

A. Use identical protection for every message

B. Use unequal error protection

C. Remove all channel coding

D. Increase source distortion for alarms

Correct Answer: B. Use unequal error protection

Question 10

The data-processing inequality implies that an intermediate processing stage:

A. Can create information about the transmitted source

B. Cannot increase mutual information about the original source

C. Always eliminates all channel noise

D. Changes channel capacity into source entropy

Correct Answer: B. Cannot increase mutual information about the original source