Classical Information and Mathematical Foundations

Introduction

Quantum information theory extends information science from classical bits to quantum systems. However, quantum concepts are not a replacement for classical foundations. Probability, complex numbers, vector spaces, linear operators, tensor products, entropy, coding, and communication channels remain essential. A quantum state is described mathematically using many of the same structures that appear in classical signal processing and information theory, but with additional rules governing measurement and physical evolution.

This chapter develops the mathematical and informational language required for later study of qubits, quantum gates, quantum measurement, entanglement, and quantum communication. The emphasis is on interpretation as well as calculation: a learner should understand not only how to compute a probability or entropy, but also what the result means for a communication system.

Learning Objectives

By the end of this chapter, you should be able to:

1. Describe sample spaces, events, random variables, conditional probability, and independence.
2. Work with complex numbers, vectors, inner products, norms, and orthonormal bases.
3. Explain Hilbert spaces and linear operators as mathematical models of physical systems.
4. Construct tensor-product spaces and explain why they represent composite systems.
5. Calculate Shannon entropy, conditional entropy, mutual information, and channel capacity in elementary cases.
6. Distinguish source coding from channel coding and identify common channel models.
7. Apply these ideas to a classical communication problem and explain their relevance to quantum information.

1. Information as Representation and Uncertainty

Information concerns the resolution of uncertainty. A message may be represented by symbols, electrical signals, bits, or physical states. The same abstract message can have different physical implementations. For example, the binary symbols 0 and 1 may be represented by low and high voltages, two optical intensities, or two distinguishable quantum states.

A bit is a classical variable that can take one of two values, conventionally 0 or 1. A string of n bits can represent up to 2ⁿ distinct messages. If all messages are equally likely, n bits are required to identify one message without ambiguity. If some messages are more likely than others, average information can be less than n bits, which motivates entropy and compression.

Information must be distinguished from data. Data are recorded symbols or measurements; information is related to the uncertainty reduced by receiving or interpreting those data. Noise, ambiguity, and limited resources affect how reliably information can be stored and transmitted.

2. Probability Foundations

2.1 Sample spaces and events

A probability model begins with a sample space Ω, the set of possible outcomes. An event A is a subset of Ω. A probability measure P assigns each event a number between 0 and 1 and satisfies:

for mutually exclusive events A and B. The last relationship generalizes through additivity for any finite collection of pairwise disjoint events.

The complement of A is denoted Aᶜ and represents the event that A does not occur:

For arbitrary events, the addition rule is:

The subtraction avoids counting the intersection twice.

2.2 Conditional probability and independence

The conditional probability of A given B is:

It measures the probability of A when B is known to have occurred. Rearranging gives the multiplication rule:

Events A and B are independent when learning B does not change the probability of A. Thus:

and equivalently:

Independence is an assumption, not a synonym for events being different. In communication systems, repeated noise errors are often modelled as independent, but real channels may exhibit correlated errors.

2.3 Random variables and expectation

A random variable X maps outcomes to numerical values. Its probability mass function for a discrete variable is pₓ(x) = P(X = x). The expected value is:

where the sum covers all possible values of X. The variance measures spread around the mean μ = E[X]:

Expectation is an average over repeated trials; it need not be a value that occurs in a particular trial.

Worked Example 1: Conditional probability

Suppose a binary sensor reports 1 with probability 0.1 when the true input is 0 and reports 1 with probability 0.9 when the true input is 1. Let P(X = 1) = 0.4, so P(X = 0) = 0.6. The probability of receiving 1 is

The probability that the input was 1 given that the output was 1 is Bayes' rule:

Thus, a positive report indicates a true 1 with probability approximately 85.7%, not 90%, because the prior probabilities and false-positive events also matter.

3. Complex Numbers and Vectors

3.1 Complex numbers

A complex number has the form z = a + ib, where a and b are real numbers and i² = −1. The real part is a and the imaginary part is b. Its complex conjugate is z* = a − ib, and its magnitude is:

A nonzero complex number can also be written in polar form:

where r = |z| and θ is the phase. In quantum mechanics, complex amplitudes are used, but measurable probabilities are real and nonnegative. An amplitude α contributes probability |α|² when it is associated with a particular measurement outcome.

3.2 Vectors, inner products, and norms

A vector in ℂⁿ is an ordered list of n complex components. The column vector

has conjugate transpose v† = [v₁, v₂, …, vₙ*]. The inner product of u and v is:

The norm is:

A vector is normalized if ||v|| = 1. Two vectors are orthogonal if ⟨u,v⟩ = 0. An orthonormal set consists of mutually orthogonal unit vectors. A basis is a set of vectors that can represent every vector in the space as a linear combination of basis vectors.

The distinction between amplitudes and probabilities is important. If a normalized state is written as α|0⟩ + β|1⟩, then the normalization condition is:

The quantities |α|² and |β|² are probabilities for the corresponding standard-basis outcomes, subject to the measurement rule used.

Worked Example 2: Normalization

Consider v = [1, i]ᵀ. Its squared norm is

Therefore the normalized vector is

The global phase of a vector does not change ordinary measurement probabilities: multiplying every component by e^{iφ}, where φ is real, leaves each magnitude squared unchanged. Relative phase between components, however, can affect interference and is physically significant.

4. Hilbert Spaces and Linear Operators

A Hilbert space is a complete inner-product vector space. For diploma-level quantum information, the most important examples are finite-dimensional spaces such as ℂ² and ℂⁿ. Completeness means that sequences that should converge according to the norm have limits within the space; this supports rigorous treatment of limits and expansions.

A qubit is modelled by a unit vector in a two-dimensional complex Hilbert space. The computational basis is conventionally written |0⟩ and |1⟩. A general pure qubit state is:

A linear operator A maps vectors to vectors and satisfies A(cu + dv) = cA(u) + dA(v). In a chosen basis, A is represented by a matrix. Operators can represent transformations, observables, or projectors.

The adjoint A† is the conjugate transpose of A. An operator U is unitary when:

where I is the identity operator. Unitary evolution preserves inner products and norms, so it preserves total probability. Classical reversible logic and quantum gates are closely related through this preservation principle, although quantum systems allow continuous complex superpositions.

An operator P is a projector if P² = P. Projectors associated with mutually exclusive measurement outcomes satisfy orthogonality conditions. In a projective measurement, probabilities are obtained from squared norms of projected components. More general quantum measurements use positive operators, but projective measurement provides the essential introductory model.

5. Tensor Products and Composite Systems

If system A has vector space ℋₐ and system B has vector space ℋᵦ, the joint system is described by the tensor product:

If dimensions are dₐ and dᵦ, then the composite dimension is dₐdᵦ. Thus two qubits require a four-dimensional state space, not merely two dimensions.

For vectors u and v, u ⊗ v forms a vector whose components are pairwise products. For example:

The computational basis for two qubits is {|00⟩, |01⟩, |10⟩, |11⟩}. A separable state can be written |ψ⟩ ⊗ |φ⟩. Some joint states cannot be expressed in this form; they are entangled. For example:

is entangled. Tensor products also describe operators: applying A to system A and B to system B is represented by A ⊗ B. If A and B are unitary, then A ⊗ B is unitary.

Figure: The tensor product combines subsystem spaces and multiplies their dimensions.

6. Shannon Information Theory

6.1 Entropy

For a discrete random variable X with probabilities p(x), Shannon entropy is:

Entropy is measured in bits because the logarithm is base 2. The convention 0 log₂ 0 = 0 is used by continuity. Entropy is largest for a uniform distribution over a fixed number of outcomes. If X has n equally likely outcomes, then H(X) = log₂ n. For a fair binary variable, H(X) = 1 bit; for a deterministic variable, H(X) = 0 bits.

Entropy is an average uncertainty, not necessarily the length of every encoded message. It provides a theoretical lower bound on the average number of bits needed by an efficient lossless code under suitable long-block assumptions.

6.2 Joint and conditional entropy

For variables X and Y, joint entropy is:

Conditional entropy is:

The chain rule states:

Knowing X cannot increase uncertainty about Y, so H(Y|X) ≤ H(Y). This is a classical information-theoretic statement; in quantum information, analogous quantities require density operators and von Neumann entropy.

6.3 Mutual information

Mutual information measures the reduction in uncertainty about one variable obtained from another:

Equivalently:

It is always nonnegative and equals zero exactly when X and Y are statistically independent. Mutual information is not the same as correlation: it captures any statistical dependence, including nonlinear dependence.

Worked Example 3: Binary entropy

Let X = 1 with probability 0.25 and X = 0 with probability 0.75. Then

This source has less uncertainty than a fair bit but more than a deterministic source.

7. Coding and Communication Channels

7.1 Source coding

Source coding represents messages efficiently, usually by removing statistical redundancy without losing information. A uniquely decodable prefix code assigns no codeword as the prefix of another. This permits instantaneous decoding. Huffman coding is a practical method that constructs an optimal binary prefix code for known symbol probabilities, although its average length may be above the entropy for finite symbol sets.

The source coding theorem states, informally, that for a memoryless source, sufficiently long blocks can be encoded with average length approaching H(X) bits per source symbol, while reliable lossless compression below the entropy is impossible under the standard assumptions.

7.2 Channel models

A communication channel maps an input random variable X to an output Y according to transition probabilities p(y|x). A binary symmetric channel (BSC) flips each transmitted bit with probability ε and leaves it unchanged with probability 1 − ε. The standard memoryless assumption means errors in different uses are independent.

For a BSC with crossover probability ε, the capacity is:

where the binary entropy function is:

The formula applies to 0 ≤ ε ≤ 1 and assumes a noiseless receiver model apart from independent bit flips. At ε = 0, C = 1 bit per use. At ε = 0.5, C = 0 because the output gives no information about the input. When ε > 0.5, relabelling the output gives an equivalent channel with crossover probability 1 − ε, so the capacity expression remains symmetric.

A binary erasure channel instead either transmits the bit correctly or produces an erasure symbol. With erasure probability ε, its capacity is C = 1 − ε bits per use under the usual memoryless model.

7.3 Channel coding

Channel coding adds controlled redundancy so that the receiver can detect or correct errors. A code maps k information bits into n transmitted bits. Its rate is:

A code is useful when R is below the channel capacity and its block length and decoding method permit sufficiently small error probability. Shannon's noisy-channel coding theorem establishes that rates below capacity can, in principle, achieve arbitrarily low error probability with suitable codes and sufficiently long blocks. It does not say that every practical code achieves this result, nor does it provide a universally efficient decoder.

Source coding and channel coding have different purposes: source coding removes redundancy, while channel coding adds redundancy. A practical communication system commonly performs source compression followed by channel protection.

8. Applied Case: Reliable Sensor Telemetry

Background

Consider an illustrative industrial monitoring system in which a sensor reports whether a machine is operating normally. The source variable X is 1 for normal operation and 0 for an alert. Suppose P(X = 1) = 0.8 and P(X = 0) = 0.2. The report is sent through a BSC with bit-flip probability ε = 0.1.

Problem

The engineering team wants to estimate the source uncertainty, the channel's theoretical limit, and whether a simple repetition code is appropriate.

Analysis

The source entropy is:

The channel capacity is:

A raw report uses one channel use and has rate 1 bit per use, which is above capacity. Reliable communication at that rate is impossible in the asymptotic idealized model. A protected code must reduce the information rate below 0.531 bits per channel use. For example, a block code carrying 1 information bit in 3 channel uses has rate 1/3, below capacity, although its actual reliability depends on the code and decoder.

A threefold repetition code sends 000 for 0 and 111 for 1, then uses majority vote. Under independent errors, the decoded bit is wrong if at least two of the three transmissions flip:

Thus the repetition code reduces the error probability from 0.1 to 0.028, but it uses three channel transmissions per information bit and is not generally the most efficient code.

Outcome and learning points

The system illustrates three distinct quantities: source entropy describes uncertainty in the data; channel capacity describes the maximum reliable information rate; and code performance describes what a particular finite implementation achieves. A professional design should not confuse a theoretical capacity with guaranteed performance. It should also account for latency, energy, memory, changing error statistics, and the consequences of missed alerts.

9. Connections to Quantum Information

Classical probability assigns a distribution p(x) to mutually exclusive alternatives. Quantum theory assigns a state vector or, more generally, a density operator to a Hilbert space. Complex amplitudes can interfere, so probabilities cannot usually be treated as independent classical alternatives before measurement.

Nevertheless, several classical ideas carry over directly. Normalization ensures total probability one. Inner products determine distinguishability and measurement probabilities. Unitary operators preserve normalization. Tensor products describe composite systems. Entropy and mutual information quantify uncertainty and dependence, although quantum theory replaces Shannon entropy with von Neumann entropy for general quantum states:

Here ρ is a density operator, Tr denotes the trace, and the logarithm is base 2, so entropy is measured in bits. For a classical probability distribution represented by a diagonal density matrix, von Neumann entropy reduces to Shannon entropy.

Quantum channels are physical transformations of quantum states. Unlike a classical channel, a quantum channel may alter superpositions and entanglement and must preserve the mathematical conditions required of physical states. Classical communication theory remains indispensable because quantum systems are eventually prepared, manipulated, measured, and interpreted through physical devices with classical control and readout.

10. Limitations, Professional Practice, and Governance

Mathematical models simplify physical systems. Independence assumptions may fail because of correlated noise, drifting hardware, environmental interference, or repeated calibration errors. Entropy estimates can be biased when data are scarce. Capacity formulas often assume stationary memoryless channels, ideal coding, and unlimited block length, none of which is literally available in an embedded device.

Good practice includes stating assumptions, retaining units, distinguishing probability from amplitude, checking normalization, testing models against measured error data, and reporting confidence or uncertainty where parameters are estimated. Security-sensitive systems should not rely on obscurity of a coding scheme; confidentiality, authentication, and integrity require explicit cryptographic analysis.

Quantum communication also raises governance issues. Quantum technologies may affect privacy, cryptographic transition planning, export controls, and access to specialized infrastructure. Ethical deployment requires evaluating who benefits, who bears failure risks, and whether performance claims have been independently verified. Artificial intelligence can assist in code design, channel estimation, and error diagnosis, but its outputs require validation because an apparently accurate model may conceal biased training data or unrecognized distribution changes.

Chapter Summary

Classical information theory provides the conceptual and mathematical starting point for quantum information. Probability describes uncertainty and conditional dependence. Complex vectors and inner products provide the language of amplitudes and distinguishability. Hilbert spaces contain states, while linear operators model transformations and measurements. Tensor products construct the spaces of composite systems and make entanglement possible.

Shannon entropy quantifies average uncertainty, conditional entropy quantifies remaining uncertainty, and mutual information quantifies shared information. Source coding seeks efficient lossless representation; channel coding adds redundancy to combat noise. Channel capacity is a theoretical limit, not a promise about a particular finite code. The sensor-telemetry case showed how source statistics, channel noise, rate, and error correction interact. These foundations prepare the learner for qubits, quantum gates, measurement, density operators, entanglement, and quantum channels.

Key Terms

Bit: A classical information unit with two possible values.

Sample space: The set of all possible outcomes of a random experiment.

Conditional probability: The probability of an event given that another event is known.

Independence: A condition in which knowledge of one event does not change the probability of another.

Random variable: A mapping from outcomes to numerical values.

Complex number: A number with real and imaginary components.

Complex conjugate: The number obtained by changing the sign of the imaginary component.

Inner product: A product that measures geometric relation, including orthogonality.

Norm: The length or magnitude of a vector.

Hilbert space: A complete inner-product vector space.

Linear operator: A transformation that preserves vector addition and scalar multiplication.

Unitary operator: An operator that preserves inner products and norms.

Tensor product: A construction used to represent composite systems.

Entropy: Average uncertainty in a random variable.

Mutual information: Information shared by two random variables.

Source coding: Compression that removes statistical redundancy.

Channel coding: Addition of redundancy to detect or correct errors.

Channel capacity: The maximum reliable communication rate under a specified channel model.

Binary symmetric channel: A channel that independently flips each bit with a fixed probability.

Amplitude: A complex coefficient whose squared magnitude can determine a quantum probability.

Review Questions

Short Answer Questions

1. What is the difference between a sample space and an event?
2. State the condition for two events to be independent.
3. What does normalization mean for a complex state vector?
4. Why does the tensor product of two qubit spaces have dimension four?
5. Distinguish source coding from channel coding.

Long Answer Questions

1. Explain conditional probability, Bayes' rule, and their importance in interpreting noisy sensor outputs.
2. Describe complex vectors, inner products, orthogonality, and unitary operators in the context of quantum information.
3. Derive the relationship between joint entropy, marginal entropy, and conditional entropy using the chain rule.
4. Explain Shannon entropy and discuss why it is a lower bound on ideal average lossless code length.
5. Compare a binary symmetric channel with a binary erasure channel, including their capacity interpretations.

Analytical / Application Questions

1. A binary source produces 1 with probability 0.2 and is transmitted through a BSC with crossover probability 0.05. Calculate the source entropy and channel capacity, stating the assumptions used.
2. A proposed code carries 4 information bits in 10 channel uses. Determine its rate and explain what comparison with channel capacity would be needed before claiming reliable communication.
3. Explain why the state (|00⟩ + |11⟩)/√2 cannot be represented as a tensor product of two single-qubit states, and discuss why this matters for quantum information processing.

Multiple Choice Questions with Answers

Question 1

Which expression gives the conditional probability of A given B, assuming P(B) > 0?

A. P(A) + P(B)

B. P(A ∩ B)/P(B)

C. P(A)P(B)

D. P(A ∪ B) − P(A)

Correct Answer: B. P(A ∩ B)/P(B)

Question 2

What is the squared norm of the vector [1, i]ᵀ?

A. 0

B. 1

C. 2

D. √2

Correct Answer: C. 2

Question 3

Which condition defines a unitary operator U?

A. U² = U

B. U†U = I

C. U + U† = 0

D. U = 0

Correct Answer: B. U†U = I

Question 4

What is the dimension of the tensor-product space of two qubits?

A. 2

B. 3

C. 4

D. 8

Correct Answer: C. 4

Question 5

What is the Shannon entropy of a deterministic classical variable?

A. −1 bit

B. 0 bits

C. 1 bit

D. log₂2 bits only if it is quantum

Correct Answer: B. 0 bits

Question 6

Which quantity measures the reduction in uncertainty about Y obtained from knowing X?

A. Variance

B. Norm

C. Mutual information

D. Channel crossover probability

Correct Answer: C. Mutual information

Question 7

For a binary symmetric channel with crossover probability ε = 0.5, what is its capacity?

A. 0 bits per use

B. 0.5 bits per use

C. 1 bit per use

D. 2 bits per use

Correct Answer: A. 0 bits per use

Question 8

What is the primary purpose of channel coding?

A. To remove all source information

B. To add redundancy for error detection or correction

C. To increase the physical speed of light

D. To convert every channel into a noiseless channel

Correct Answer: B. To add redundancy for error detection or correction

Question 9

A code carries 1 information bit in 3 channel uses. What is its rate?

A. 1/3 bit per channel use

B. 2 bits per channel use

C. 3 bits per channel use

D. 4 bits per channel use

Correct Answer: A. 1/3 bit per channel use

Question 10

For a three-use repetition code on a BSC with ε = 0.1 and majority decoding, what is the decoding error probability?

A. 0.001

B. 0.010

C. 0.028

D. 0.100

Correct Answer: C. 0.028