4. Signal-Space Representation and Optimum Detection

Introduction and Context

Digital communication is fundamentally a decision problem. A transmitter selects one of several possible messages, converts the selected message into a waveform, and sends it through a physical channel. The receiver observes a noisy and possibly distorted waveform and must decide which message was transmitted. Signal-space methods provide a systematic geometric and statistical framework for making this decision.

Instead of treating signals only as functions of time, signal-space analysis represents them as vectors in a finite-dimensional Euclidean space. This transformation makes it possible to interpret modulation, filtering, detection, distance, noise, and probability of error using geometry and linear algebra. The central result is that, in additive white Gaussian noise (AWGN), optimum detection can be implemented using correlators or matched filters followed by a suitable decision rule.

The chapter develops orthonormal signal expansions, geometric distance measures, matched-filter and correlator receivers, maximum-likelihood and maximum a posteriori detection, Bayesian decision theory, and error-probability evaluation. These ideas form the theoretical foundation for coherent digital receivers and remain important in modern wireless, optical, satellite, and high-speed wired communication systems.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Represent finite-energy communication waveforms as vectors in signal space.

2. Construct an orthonormal basis using Gram–Schmidt orthogonalization.

3. Explain the relationship between signal distance, noise, and detection performance.

4. Derive correlator and matched-filter receiver structures.

5. Formulate maximum-likelihood, MAP, and minimum-error Bayesian decision rules.

6. Evaluate error probabilities for binary and multidimensional signals in AWGN.

7. Distinguish coherent, noncoherent, and mismatched detection.

8. Apply optimum detection principles to practical communication scenarios.

From Waveforms to Vectors

Consider a set of M possible signals, denoted by

s_i(t), 0 ≤ t ≤ T, i = 1, 2, …, M.

Assume that each signal has finite energy,

E_i = ∫₀ᵀ |s_i(t)|² dt.

The inner product of two real signals is defined as

⟨s_i,s_j⟩ = ∫₀ᵀ s_i(t)s_j(t)dt.

For complex signals, the appropriate definition is

⟨s_i,s_j⟩ = ∫₀ᵀ s_i(t)s_j*(t)dt.

The corresponding norm is ||s_i|| = √⟨s_i,s_i⟩. Two signals are orthogonal if their inner product is zero. Orthogonality is especially useful because it allows independent measurement of signal components.

Suppose the signals span an N-dimensional subspace of the space of finite-energy waveforms. An orthonormal basis {φ₁(t), φ₂(t), …, φ_N(t)} satisfies

⟨φ_k,φ_l⟩ = δ_kl,

where δ_kl is one when k = l and zero otherwise. Each transmitted signal can then be written as

s_i(t) = Σₖ₌₁ᴺ s_ik φ_k(t),

where

s_ik = ⟨s_i,φ_k⟩.

The signal s_i(t) is therefore represented by the vector

s_i = [s_i1, s_i2, …, s_iN]ᵀ.

Because the basis is orthonormal, waveform energy becomes vector energy:

E_i = Σₖ₌₁ᴺ s_ik² = ||s_i||².

Similarly, the squared Euclidean distance between two waveforms is

d_ij² = ∫₀ᵀ |s_i(t) − s_j(t)|²dt = ||s_i − s_j||².

This equality is central: detection performance depends on the separation of signal vectors, not merely on their individual energies.

Constructing a Signal-Space Basis

A basis may be obtained by applying the Gram–Schmidt procedure to the distinct signals. First choose a nonzero signal and normalize it:

φ₁(t) = s₁(t)/||s₁||.

For a second signal, remove its projection onto φ₁:

u₂(t) = s₂(t) − ⟨s₂,φ₁⟩φ₁(t),

and normalize u₂(t) to obtain φ₂(t). The procedure continues until all linearly independent directions have been extracted. Signals that are linear combinations of previous signals do not increase the dimension.

The dimension of signal space is thus determined by the number of linearly independent waveform components, not by the number of possible messages. A modulation scheme may have many signal points but occupy only a low-dimensional space.

Noise in Signal Space

The received waveform is modeled as

r(t) = s_m(t) + n(t),

where s_m(t) is the transmitted signal and n(t) is AWGN. The noise has zero mean and two-sided power spectral density N₀/2. Projecting the received waveform onto each basis function gives

r_k = ⟨r,φ_k⟩ = s_mk + n_k.

For an orthonormal basis, the noise coordinates n_k are independent Gaussian random variables with

n_k ~ N(0,N₀/2)

for real signals. Consequently, the receiver observes the vector

r = s_m + n,

where the noise vector has independent Gaussian components. This converts a continuous-time detection problem into a finite-dimensional statistical decision problem.

The conditional probability density of r given signal s_i is

p(r|s_i) = (1/(πN₀)^{N/2}) exp[−||r−s_i||²/N₀]

for the real-vector convention used here. The exact normalizing constant is less important than the exponential dependence on squared distance. The density is largest when r is closest to s_i in Euclidean distance.

Geometric Interpretation of Detection

Each possible signal is represented by a point in signal space. The received vector is a random point displaced from the transmitted point by a Gaussian noise vector. A decision region is the set of received vectors for which the receiver selects a particular signal.

For two equally likely signals, the optimum boundary is the perpendicular bisector of the line joining the two signal points. A received vector is assigned to whichever point is closer. For several equally likely signals, the decision regions are Voronoi regions. Their boundaries consist of points equidistant from two or more signal points.

The pairwise distance d_ij determines how easily signals i and j can be distinguished. Larger distance reduces the probability that noise will move a received point across a decision boundary. This observation explains why signal constellations are designed to maximize minimum Euclidean distance subject to energy and bandwidth constraints.

Correlator and Matched-Filter Receivers

A correlator receiver computes the inner products

r_k = ∫₀ᵀ r(t)φ_k(t)dt.

The resulting coordinates are sufficient statistics for detection because the components of r(t) orthogonal to the signal space contain no information about which signal was transmitted under the AWGN model. The receiver can therefore discard those components without loss in optimum detection performance.

An equivalent implementation uses matched filters. A filter matched to φ_k(t) has impulse response, over the observation interval,

h_k(t) = φ_k(T−t).

Sampling its output at t = T produces

y_k(T) = ∫₀ᵀ r(τ)φ_k(τ)dτ,

which is precisely the correlator output. Thus, a bank of matched filters and a bank of correlators are mathematically equivalent.

The matched filter has a particularly important property: for a known signal in AWGN, it maximizes the output signal-to-noise ratio at the sampling instant. If the filter is matched to s(t), its peak output signal component is proportional to signal energy, while the output-noise variance is proportional to N₀E/2. The resulting maximum SNR is

SNR_max = 2E/N₀

under the real-signal convention.

The matched filter is not necessarily optimum for every channel or noise model. Colored noise, uncertain timing, fading, intersymbol interference, and model mismatch require modified receivers such as whitened matched filters, equalizers, or sequence detectors.

Maximum-Likelihood Detection

The maximum-likelihood (ML) receiver selects the signal having the largest conditional density:

ŝ_ML = arg max_i p(r|s_i).

For AWGN with equal noise variance, maximizing p(r|s_i) is equivalent to minimizing ||r−s_i||². Expanding the distance gives

||r−s_i||² = ||r||² − 2rᵀs_i + ||s_i||².

Since ||r||² is common to all hypotheses, the ML decision selects the signal maximizing

Λ_i(r) = rᵀs_i − 1/2||s_i||².

For equal-energy signals, the second term is common as well, so the receiver selects the signal with the largest correlation rᵀs_i. This explains the widespread use of correlation receivers in phase-shift keying and orthogonal signaling.

For two signals s₁ and s₂, the ML rule selects s₁ if

||r−s₁||² < ||r−s₂||².

Equivalently,

rᵀ(s₁−s₂) > (||s₁||²−||s₂||²)/2.

The right side is the threshold. For equal-energy signals it is zero after projection onto the difference vector.

MAP and Bayesian Detection

The maximum a posteriori (MAP) receiver incorporates prior probabilities P_i = Pr(s_i). It selects

ŝ_MAP = arg max_i p(s_i|r).

Using Bayes’ rule and omitting the common factor p(r), this becomes

ŝ_MAP = arg max_i P_i p(r|s_i).

Taking logarithms for the AWGN model gives the decision metric

Λ_i(r) = rᵀs_i − 1/2||s_i||² + (N₀/2)ln P_i.

Compared with ML detection, MAP detection shifts the decision boundaries toward less probable signals. A more probable signal is assigned a larger decision region because selecting it is statistically more likely to minimize average error.

ML and MAP detection coincide when all signals are equally likely. They differ when source probabilities are unequal. This distinction matters in systems with nonuniform symbol probabilities, coded sequences, control messages, or heavily unbalanced hypotheses.

Bayesian decision theory provides a still more general formulation. Let C_ji denote the cost of deciding signal j when signal i was transmitted. The conditional risk of decision j is

R(j|r) = Σ_i C_ji Pr(s_i|r).

The optimum Bayesian receiver selects the decision with minimum conditional risk. Minimum-error detection is the special case in which C_ji = 0 for a correct decision and C_ji = 1 for an error. In safety-critical systems, the costs of missed detection and false alarm may be unequal, making a risk-based detector preferable to a simple minimum-error rule.

Binary Detection and Error Probability

Consider two signals s₁ and s₂ with equal prior probabilities. Define the difference vector

d = s₁ − s₂,

and let d = ||d||. The decision depends only on the noise component along the unit vector d/d. Under transmission of s₁, an error occurs when the noise projection is sufficiently negative to cross the decision boundary. The resulting probability of error is

P_e = Q(d/√(2N₀)),

where

Q(x) = (1/√(2π))∫ₓ^∞ exp(−u²/2)du.

For antipodal binary signaling, s₂ = −s₁ and each signal has energy E_b. The distance is d = 2√E_b, yielding

P_e = Q(√(2E_b/N₀)).

This is the familiar coherent binary phase-shift keying result and is also the result for coherent binary antipodal signaling in general.

For orthogonal binary signals, the distance is √(2E_b), giving

P_e = Q(√(E_b/N₀)).

Thus antipodal signaling has a larger distance than orthogonal signaling for the same energy and produces a lower error probability under coherent detection.

For M-ary signal sets, exact error probabilities may be difficult to obtain because decision regions can have complicated geometries. A common analytical tool is the union bound:

P_e ≤ Σ_i P_i Σ_{j≠i} Pr(decide j | transmit i).

For equally likely signals, the pairwise error probability between i and j is often bounded or approximated using

Q(d_ij/√(2N₀)).

The nearest-neighbor terms usually dominate at moderate or high signal-to-noise ratios, which is why minimum distance is such an important design metric.

Coherent, Noncoherent, and Differential Detection

Coherent detection assumes that the receiver knows the relevant carrier phase, frequency, timing, and channel parameters. This knowledge allows direct correlation with reference waveforms and generally provides the best performance for a specified signal set.

Noncoherent detection does not require explicit carrier phase knowledge. Instead, it may compare energies, envelopes, or statistics that are insensitive to unknown phase. The reduction in receiver complexity is accompanied by a performance penalty in many cases.

Differential detection encodes information in changes between successive symbols. It avoids absolute phase recovery but introduces error propagation and generally performs worse than ideal coherent detection. The choice among these methods depends on synchronization capability, mobility, oscillator stability, complexity, and required reliability.

An Applied Design Example: Binary Signaling over an AWGN Link

Consider an illustrative telemetry link that transmits one binary symbol during each interval T. The transmitter uses antipodal waveforms

s₁(t) = +√(2E_b/T) cos(2πf_ct),

s₂(t) = −√(2E_b/T) cos(2πf_ct),

over 0 ≤ t ≤ T, with the carrier chosen so that the cosine has unit normalized energy after scaling. The two signals are represented by points at +√E_b and −√E_b on a one-dimensional signal axis.

Background and Problem

The telemetry receiver must decide which symbol was transmitted in the presence of AWGN. The design objective is to minimize the probability of bit error while keeping the transmitted energy fixed.

Analysis

The signal distance is

d = |√E_b − (−√E_b)| = 2√E_b.

The received signal is projected onto the normalized carrier using a correlator. Under the two hypotheses, the correlator output has means +√E_b and −√E_b, with Gaussian noise variance N₀/2. Equal prior probabilities imply a zero decision threshold.

Detection Strategy

The receiver samples the correlator at T. A positive output produces decision s₁; a negative output produces decision s₂. The rule is equivalent to ML and MAP detection because the hypotheses are equally likely and have equal energy.

Outcome

The bit-error probability is

P_b = Q(√(2E_b/N₀)).

Increasing E_b/N₀ increases the distance relative to the noise standard deviation and therefore decreases error probability. If the receiver instead used orthogonal signals with the same energy, the distance would be smaller and the error probability would be Q(√(E_b/N₀)).

Learning Points

The example demonstrates four general principles: the receiver needs only a sufficient statistic; matched filtering maximizes the relevant output SNR; equal-prior equal-energy ML detection is a nearest-neighbor rule; and error performance is controlled by normalized signal distance.

Practical and Industry Applications

Signal-space methods are used in the design and analysis of QPSK, QAM, PSK, FSK, optical intensity modulation, satellite links, software-defined radios, modem standards, and coherent fiber-optic receivers. In QAM, the in-phase and quadrature components form a two-dimensional space, and the receiver estimates the two coordinates before selecting the closest constellation point. In multiple-input multiple-output systems, signal-space ideas extend to higher-dimensional vector models in which channel matrices transform transmitted constellations.

In receiver implementation, the theoretical correlator may be realized digitally through finite-impulse-response filters, fast convolution, polyphase structures, or numerical inner products. Digital signal processors and field-programmable gate arrays can perform these operations in parallel. Channel estimation, synchronization, and adaptive equalization provide the reference information needed for practical approximation of the ideal receiver.

Limitations, Risks, and Common Mistakes

Several errors frequently arise in applying signal-space theory. First, energy and amplitude are not interchangeable: doubling amplitude quadruples energy. Second, the noise variance depends on the normalization of the basis and on whether a one-sided or two-sided PSD convention is used. Third, Euclidean nearest-neighbor detection is optimum only under appropriate assumptions, particularly AWGN and known signal parameters.

A receiver can also be degraded by timing offset, carrier-frequency offset, phase noise, fading, nonlinear amplification, quantization, and inaccurate channel estimates. In colored Gaussian noise, the ordinary inner product is not generally optimum; the noise should first be whitened, or the decision metric should use the inverse noise covariance matrix.

For large constellations, evaluating exact error probability may require multidimensional integration or simulation. Union bounds are useful but can be loose at low signal-to-noise ratios. Simulation results should therefore be interpreted alongside analytical bounds and assumptions.

Professional and Ethical Considerations

Receiver design affects reliability in applications such as emergency communications, aircraft navigation, industrial control, and medical telemetry. Engineers must state assumptions clearly, validate error estimates under realistic channel conditions, and avoid presenting ideal AWGN performance as a guarantee of field performance. Design decisions should also account for electromagnetic compatibility, lawful spectrum use, secure handling of transmitted information, and the consequences of missed or false decisions.

Current Developments

Modern receivers increasingly combine classical optimum detection with data-driven estimation. Machine-learning methods may assist synchronization, channel estimation, constellation classification, or approximate detection in high-dimensional systems. Nevertheless, the signal-space model remains valuable because it supplies interpretable likelihood metrics, performance bounds, and physical insight.

In massive MIMO, millimeter-wave communication, optical communication, and joint communication-sensing systems, detection may involve very large signal spaces and structured noise. Approximate message passing, sphere decoding, factor-graph methods, and neural detectors are active research areas. Their performance is still commonly benchmarked against ML, MAP, or minimum-distance principles derived from the framework developed in this chapter.

Best Practices for Receiver Analysis

A sound analysis should begin by defining the signal set, observation interval, basis normalization, noise PSD, prior probabilities, and decision costs. The next step is to reduce the waveform observation to sufficient coordinates, derive the likelihood or risk metric, and identify the decision boundaries. Error performance should then be expressed in terms of distance and noise variance, checked against limiting cases, and validated through simulation when the geometry is complicated.

It is also good practice to distinguish transmitter energy, energy per bit, and energy per symbol; to state whether detection is coherent; and to separate ideal theoretical performance from implementation loss. These habits prevent many apparently conflicting results in digital communication analysis.

Chapter Summary

Signal-space representation converts finite-energy waveforms into vectors using an orthonormal basis. Energies, inner products, and waveform distances then become Euclidean quantities. Projection onto the signal-space basis produces sufficient statistics for detection in AWGN, and these projections can be implemented using correlators or matched filters.

The ML receiver chooses the signal with the greatest conditional likelihood and, in AWGN, becomes a minimum-distance detector. MAP detection additionally incorporates prior probabilities, while Bayesian detection minimizes conditional decision risk and can include unequal error costs. Geometric separation, especially minimum Euclidean distance, determines error performance. For binary equal-prior signals, the pairwise error probability is expressed using the Gaussian Q-function. Antipodal signaling provides greater distance than orthogonal signaling for equal energy. Practical receivers must also consider synchronization, colored noise, fading, channel estimation, implementation loss, and application-specific reliability requirements.

Key Terms

Signal space: Finite-dimensional vector space spanned by the possible transmitted waveforms.

Orthonormal basis: A set of unit-energy mutually orthogonal functions.

Signal vector: The coordinate vector of a waveform with respect to a chosen basis.

Inner product: An integral measure of similarity between two waveforms.

Signal energy: The integral of squared waveform magnitude over the observation interval.

Euclidean distance: The norm of the difference between two signal vectors.

AWGN: Additive white Gaussian noise with constant spectral density and Gaussian samples after projection.

Sufficient statistic: An observation that retains all information needed for a specified detection task.

Correlator: A receiver that computes inner products between the received waveform and reference functions.

Matched filter: A filter designed to maximize output SNR at a specified sampling instant.

Maximum likelihood: A rule that selects the hypothesis with the greatest conditional observation density.

MAP detection: A rule that selects the hypothesis with the greatest posterior probability.

Bayesian detection: Decision-making based on posterior probabilities and specified decision costs.

Decision region: The set of observations assigned to one particular hypothesis.

Voronoi region: A nearest-neighbor decision region surrounding a signal point.

Minimum distance: The smallest pairwise distance between points in a signal constellation.

Pairwise error probability: The probability of confusing one particular signal with another.

Q-function: The tail probability of a standard Gaussian random variable.

Coherent detection: Detection using known carrier phase and other reference parameters.

Noncoherent detection: Detection that does not require explicit carrier-phase knowledge.

Review Questions

Short Answer Questions

1. Why does an orthonormal basis preserve signal energy in vector representation?

2. What is the relationship between a correlator and a matched-filter receiver?

3. Define a sufficient statistic in the context of AWGN detection.

4. Under what condition do ML and MAP receivers have the same decision rule?

5. What physical feature of a signal constellation most directly affects pairwise error probability?

Long Answer Questions

1. Derive the ML decision metric for a set of signals transmitted over AWGN.

2. Explain the Gram–Schmidt procedure and its role in constructing signal space.

3. Derive the error probability for equal-prior antipodal binary signaling.

4. Compare coherent, noncoherent, and differential detection in terms of assumptions and performance.

5. Explain Bayesian detection with unequal decision costs and give a communication example in which it is useful.

Analytical / Application Questions

1. Two equal-energy signals have Euclidean distance d and are transmitted with equal prior probabilities through AWGN of two-sided PSD N₀/2. Derive the decision threshold and error probability.

2. A four-point constellation has unequal symbol probabilities. Explain how its MAP decision boundaries differ from its ML Voronoi boundaries.

3. A receiver operates in colored Gaussian noise rather than AWGN. Explain why an ordinary correlator is not generally optimum and propose an appropriate modification.

22. Multiple Choice Questions with Answers

Question 1

For an orthonormal signal-space basis, the energy of a waveform is equal to:

A. The sum of its coordinate magnitudes

B. The sum of the squares of its coordinates

C. The product of its coordinates

D. The largest coordinate only

Correct Answer: B. The sum of the squares of its coordinates

Question 2

Under AWGN, projection of the received waveform onto the signal-space basis produces:

A. Insufficient statistics

B. Sufficient statistics for optimum detection

C. Only phase information

D. A noise-free signal

Correct Answer: B. Sufficient statistics for optimum detection

Question 3

The ML detector for equal-energy signals in AWGN selects the signal vector having:

A. The smallest correlation with the received vector

B. The largest Euclidean norm

C. The smallest Euclidean distance from the received vector

D. The lowest prior probability

Correct Answer: C. The smallest Euclidean distance from the received vector

Question 4

MAP detection differs from ML detection primarily because MAP detection uses:

A. Prior probabilities

B. A different noise bandwidth

C. Only signal energy

D. No statistical model

Correct Answer: A. Prior probabilities

Question 5

For two equal-prior signals separated by distance d in AWGN with PSD N₀/2, the pairwise error probability is:

A. Q(d/√(2N₀))

B. Q(√(2N₀)/d)

C. Q(d²/N₀)

D. Q(2d/N₀)

Correct Answer: A. Q(d/√(2N₀))

Question 6

For equal transmitted energy, antipodal binary signals generally outperform orthogonal binary signals because they have:

A. A smaller signal-space dimension

B. A larger Euclidean separation

C. No receiver noise

D. Unequal prior probabilities

Correct Answer: B. A larger Euclidean separation

Question 7

A matched filter is designed to maximize:

A. Bandwidth at every frequency

B. Output SNR at a specified sampling instant

C. The number of possible messages

D. The prior probability of a signal

Correct Answer: B. Output SNR at a specified sampling instant

Question 8

If signal priors are unequal, the MAP detector generally changes the decision regions by:

A. Ignoring the likelihood functions

B. Giving larger regions to less probable signals

C. Favoring more probable signals through the decision metric

D. Making all boundaries perpendicular to the time axis

Correct Answer: C. Favoring more probable signals through the decision metric

Question 9

In a one-dimensional antipodal constellation, the optimum equal-prior decision threshold is located:

A. At the midpoint between the two signal points

B. At the larger signal point

C. At zero noise variance

D. At the signal with lower energy

Correct Answer: A. At the midpoint between the two signal points

Question 10

When the noise is colored Gaussian noise, an appropriate optimum-receiver modification is to:

A. Discard the signal-space coordinates

B. Use noise whitening or a covariance-weighted metric

C. Assume the noise is absent

D. Replace MAP detection with random decisions

Correct Answer: B. Use noise whitening or a covariance-weighted metric