9. Statistical Detection and Decision Theory

Introduction and Context

Digital signal processing systems rarely observe signals in isolation. A radar receiver must decide whether a target is present, a communication receiver must decide which symbol was transmitted, and a biomedical monitor must distinguish a physiological event from noise and interference. In each case, the available data are uncertain and the decision must be made from incomplete information. Statistical detection and decision theory provide the mathematical framework for making such decisions rationally.

Detection theory is concerned primarily with deciding between hypotheses, usually the absence or presence of a signal. Decision theory extends this framework by incorporating consequences, costs, prior probabilities, and alternative actions. The central question is not simply whether a signal exists, but how evidence should be converted into a decision when observations are random and errors are unavoidable.

This chapter develops likelihood-ratio tests, Bayesian and Neyman–Pearson criteria, matched-filter detectors, receiver operating characteristic analysis, sequential decisions, and methods for handling noise and model uncertainty. The discussion assumes familiarity with probability, random processes, linear systems, and discrete-time signal processing.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Formulate binary and multiple hypothesis-testing problems.

2. Derive likelihood-ratio and log-likelihood-ratio tests.

3. Distinguish Bayesian, maximum-likelihood, maximum a posteriori, and Neyman–Pearson decisions.

4. Explain the operation and optimality of matched filters.

5. Calculate and interpret probability of detection, probability of false alarm, and ROC curves.

6. Describe sequential testing and the sequential probability-ratio test.

7. Analyze detector performance under colored noise, unknown parameters, and model mismatch.

8. Apply statistical detection principles to engineering systems and evaluate practical trade-offs.

9.1 The Detection Problem as a Statistical Experiment

Let the observation be a vector

\[

\mathbf{x}=[x_0,x_1,\ldots,x_{N-1}]^T.

\]

A binary detection problem is represented by two hypotheses:

\[

H_0: \mathbf{x}\sim p(\mathbf{x}|H_0), \qquad

H_1: \mathbf{x}\sim p(\mathbf{x}|H_1).

\]

Typically, \(H_0\) denotes noise alone and \(H_1\) denotes signal plus noise. A detector computes a statistic \(T(\mathbf{x})\) and compares it with a threshold \(\eta\):

\[

T(\mathbf{x})\mathop{\gtrless}_{H_0}^{H_1}\eta.

\]

The detector may also include a third action, such as “reject” or “request another observation,” but binary decisions are the fundamental case.

Four outcomes are possible. A correct decision under \(H_0\) is a correct rejection, while a correct decision under \(H_1\) is a detection. Choosing \(H_1\) when \(H_0\) is true produces a false alarm. Choosing \(H_0\) when \(H_1\) is true produces a miss. These outcomes are summarized by

\[

P_{FA}=P(\text{decide }H_1|H_0),

\qquad

P_D=P(\text{decide }H_1|H_1),

\]

and

\[

P_M=1-P_D.

\]

A detector cannot generally reduce both false alarms and misses simultaneously. Improving one usually worsens the other unless additional information or a better observation is obtained. Threshold selection therefore expresses an engineering or operational preference.

Important terminology

A hypothesis is a statistical explanation of the observation. A test is a rule that maps observations to decisions. A test statistic is a reduced quantity used for comparison with a threshold. A sufficient statistic preserves all information in the observation relevant to the hypotheses. The decision region is the set of observations leading to a particular decision.

9.2 Likelihood Ratios and the Fundamental Test Structure

The likelihood functions under the two hypotheses are

\[

p_0(\mathbf{x})=p(\mathbf{x}|H_0), \qquad p_1(\mathbf{x})=p(\mathbf{x}|H_1).

\]

Their ratio is the likelihood ratio

\[

\Lambda(\mathbf{x})=\frac{p_1(\mathbf{x})}{p_0(\mathbf{x})}.

\]

The likelihood ratio measures how much more plausibly the observation is explained by \(H_1\) than by \(H_0\). Large values support \(H_1\), whereas small values support \(H_0\). A broad class of optimal tests has the form

\[

\Lambda(\mathbf{x})\mathop{\gtrless}_{H_0}^{H_1}\gamma.

\]

The logarithmic likelihood ratio,

\[

\ell(\mathbf{x})=\ln \Lambda(\mathbf{x}),

\]

is often more convenient numerically and analytically. Products of sample likelihoods become sums of log-likelihoods, which is particularly useful in sequential detection.

The likelihood-ratio principle is powerful because many apparently different detectors reduce to this structure. The precise threshold depends on the decision criterion, prior probabilities, costs, or constraints.

9.3 Decision Criteria

Bayesian decision theory

Suppose the prior probabilities are \(P(H_0)\) and \(P(H_1)\), and the cost of deciding action \(a_i\) when hypothesis \(H_j\) is true is \(C_{ij}\). The conditional risk of an action is

\[

R(a_i|\mathbf{x})=\sum_j C_{ij}P(H_j|\mathbf{x}).

\]

The Bayes rule selects the action with minimum conditional risk.

For two decisions, the rule can be expressed as a likelihood-ratio test. In the common case in which correct decisions have zero cost, the threshold is

\[

\gamma_B=\frac{[C_{10}-C_{00}]P(H_0)}{[C_{01}-C_{11}]P(H_1)},

\]

provided the denominator is positive. Thus, a rare but costly event may justify a detector threshold different from that suggested by equal priors or equal error costs.

Maximum-likelihood and MAP decisions

The maximum-likelihood rule selects the hypothesis with the larger likelihood. For equal costs and equal priors, it is equivalent to choosing \(H_1\) when \(\Lambda(\mathbf{x})>1\). The maximum a posteriori rule selects the hypothesis with the larger posterior probability and therefore includes prior probabilities. MAP and Bayesian decisions coincide under zero-one loss, while maximum likelihood is a special case of MAP with equal priors.

Neyman–Pearson detection

In many engineering systems, the probability of false alarm must not exceed a prescribed value, while the probability of detection is maximized. The Neyman–Pearson formulation is

\[

\max P_D \quad \text{subject to} \quad P_{FA}\leq \alpha.

\]

The Neyman–Pearson lemma states that the likelihood-ratio test is the most powerful test for a specified false-alarm probability. The threshold is selected so that \(P_{FA}=\alpha\), when a continuous threshold can achieve the constraint.

This criterion is especially appropriate in radar, spectrum monitoring, and safety systems where false alarms consume resources but missed events may be unacceptable. Unlike Bayesian detection, it does not require prior probabilities or explicit costs.

9.4 Gaussian Detection and the Matched Filter

Consider the deterministic known-signal problem

\[

H_0:\mathbf{x}=\mathbf{n}, \qquad H_1:\mathbf{x}=\mathbf{s}+\mathbf{n},

\]

where \(\mathbf{n}\) is zero-mean Gaussian noise with covariance matrix \(\mathbf{C}\). Under the two hypotheses,

\[

p_i(\mathbf{x})\propto \exp\left[-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_i)^T\mathbf{C}^{-1}(\mathbf{x}-\boldsymbol{\mu}_i)\right],

\]

with \(\boldsymbol{\mu}_0=0\) and \(\boldsymbol{\mu}_1=\mathbf{s}\).

Taking the logarithm of the likelihood ratio gives

\[

\ell(\mathbf{x})=\mathbf{s}^T\mathbf{C}^{-1}\mathbf{x}-\frac{1}{2}\mathbf{s}^T\mathbf{C}^{-1}\mathbf{s}.

\]

Therefore the essential statistic is

\[

T(\mathbf{x})=\mathbf{s}^T\mathbf{C}^{-1}\mathbf{x}.

\]

If the noise is white with covariance \(\sigma^2\mathbf{I}\), this becomes

\[

T(\mathbf{x})=\frac{1}{\sigma^2}\mathbf{s}^T\mathbf{x}.

\]

The detector correlates the received data with the known signal. In continuous time, the corresponding filter has impulse response proportional to a time-reversed, conjugated version of the signal:

\[

h(t)=k s^*(T-t).

\]

This is the matched filter.

The matched filter maximizes the output signal-to-noise ratio at the sampling instant when the signal waveform and noise statistics are known. In colored noise, prewhitening is required. The factor \(\mathbf{C}^{-1}\) in the statistic performs the appropriate whitening and weighting: noisy dimensions receive less weight than reliable dimensions.

For white Gaussian noise, if the filter output is normalized, the detection performance depends on the signal energy \(E_s=\mathbf{s}^T\mathbf{s}\) relative to the noise variance. Increasing observation time or signal energy generally improves detectability, although practical systems may face bandwidth, latency, and nonstationarity constraints.

9.5 ROC Analysis and Threshold Selection

An ROC curve plots \(P_D\) against \(P_{FA}\) as the threshold varies. A detector with a curve closer to the upper-left region generally provides better discrimination. The diagonal line corresponds to a detector with no useful information, for which \(P_D=P_{FA}\).

ROC analysis is more informative than quoting a single operating point because it reveals the trade-off across thresholds. A system designed for surveillance may accept a larger false-alarm rate to avoid missing weak targets, whereas an automated alarm system may require a very low false-alarm rate.

The area under the ROC curve is a threshold-independent performance measure, although it should not replace operating-point analysis. Performance can also be represented using detection probability at a specified false-alarm probability, or through plots on probability scales for Gaussian models.

Thresholds should be selected using system-level requirements rather than numerical convenience. The threshold may depend on processing gain, observation duration, environmental interference, available operator attention, and the cost of downstream actions. Adaptive thresholding is often required when noise power changes over time.

9.6 Composite Hypotheses and Unknown Parameters

A simple hypothesis completely specifies the probability distribution. A composite hypothesis contains unknown parameters, such as unknown signal amplitude, phase, delay, or noise variance. For example,

\[

H_1:\mathbf{x}=A\mathbf{s}(\tau,\phi)+\mathbf{n},

\]

where amplitude \(A\), delay \(\tau\), or phase \(\phi\) is unknown.

One practical approach is the generalized likelihood-ratio test (GLRT). Unknown parameters are estimated under each hypothesis and substituted into the likelihood ratio:

\[

\Lambda_{GLRT}(\mathbf{x})=

\frac{\max_{\boldsymbol{\theta}_1}p(\mathbf{x}|H_1,\boldsymbol{\theta}_1)}

{\max_{\boldsymbol{\theta}_0}p(\mathbf{x}|H_0,\boldsymbol{\theta}_0)}.

\]

The GLRT is not universally optimal, but it is often tractable and performs well asymptotically or under well-specified models.

Unknown phase frequently leads to a quadrature detector. Instead of correlating with one template, the receiver correlates with in-phase and quadrature templates and uses the magnitude of the two outputs. Unknown delay may require a bank of matched filters or a correlation search. Unknown amplitude may lead to energy-like statistics, especially when signal direction is known but scale is not.

9.7 Sequential Decisions

A fixed-sample detector collects a predetermined number of observations. A sequential detector can stop early when the evidence is sufficiently strong. The sequential probability-ratio test (SPRT) accumulates

\[

L_n=\sum_{k=1}^{n}\ln \frac{p(x_k|H_1)}{p(x_k|H_0)}.

\]

Two thresholds are used. If \(L_n\) exceeds the upper threshold, decide \(H_1\); if it falls below the lower threshold, decide \(H_0\); otherwise collect another sample.

The thresholds are related approximately to desired error probabilities \(\alpha\) and \(\beta\), with upper and lower boundaries near

\[

\ln\frac{1-\beta}{\alpha},

\qquad

\ln\frac{\beta}{1-\alpha}.

\]

Sequential testing can reduce average observation time when the evidence is strong. It is valuable in adaptive sensing, communications, medical monitoring, and spectrum occupancy detection. However, variable decision time may complicate synchronization, scheduling, and fairness in resource allocation.

9.8 Detection Under Noise and Model Uncertainty

Theoretical detectors often assume known Gaussian noise, but practical noise may be colored, impulsive, non-Gaussian, time-varying, or contaminated by interference. A detector optimized for an incorrect model can suffer severe performance loss and may produce unreliable false-alarm rates.

For colored noise, covariance estimation and whitening are central. If \(\mathbf{C}\) is estimated from reference data, errors in the estimate affect both the test statistic and its threshold. In non-Gaussian environments, robust methods may use heavy-tailed models, rank statistics, or empirical thresholds. In radar and wireless systems, clutter and interference may require spatial, temporal, or frequency-domain suppression before detection.

Model uncertainty can be addressed through robust detection. A robust detector seeks acceptable performance over a set of plausible distributions rather than optimizing for one assumed distribution. Minimax formulations, uncertainty sets, and distributionally robust methods formalize this idea. The price is often reduced nominal performance in exchange for better protection against mismatch.

Data-driven methods, including machine-learning classifiers, can complement likelihood-based detectors. However, a learned score is not automatically a calibrated probability or a valid likelihood ratio. Training-data bias, distribution shift, adversarial inputs, and unrepresentative noise can alter the operating characteristics. A sound engineering design therefore combines statistical validation, calibration, monitoring, and fallback procedures.

9.9 Practical Applications

In digital communications, symbol detection compares likelihoods associated with possible transmitted symbols. In binary phase-shift keying over additive white Gaussian noise, the optimum detector correlates the received waveform with the symbol waveform and selects the larger likelihood. In radar, a matched filter followed by a threshold detector identifies echoes, while CFAR methods adjust the threshold using neighboring cells to control false alarms under changing clutter. In spectrum sensing, a receiver decides whether a licensed signal is present, balancing missed detection against false occupancy declarations.

Medical signal processing uses detection theory to identify arrhythmias, seizures, or weak evoked responses. Here, false alarms can cause alarm fatigue, while missed events may threaten patient safety. The threshold must therefore be considered together with clinical workflow and the cost of human review. Industrial monitoring applies similar ideas to vibration-based fault detection, where changes in machinery noise and operating conditions create composite hypotheses.

At the organizational level, detection systems should be evaluated not only by ROC curves but also by processing cost, latency, explainability, maintenance requirements, and consequences of errors. A technically optimal detector may be unsuitable if its assumptions cannot be monitored or if its decisions cannot be audited.

9.10 Illustrative Case Study: Adaptive Spectrum Occupancy Detection

Background

Consider an illustrative cognitive-radio receiver that must determine whether a narrowband primary-user transmission is present in a channel. The receiver samples the channel during short sensing intervals. Thermal noise varies with temperature and receiver gain, while nearby channels occasionally contribute interference.

Situation and problem

An energy detector is initially selected because the primary waveform is unknown. The detector computes the average received energy over \(N\) samples and compares it with a threshold. A fixed threshold performs well in laboratory conditions but produces excessive false alarms in the field when the noise floor rises.

Analysis

Under \(H_0\), the samples contain noise and interference; under \(H_1\), they contain the primary signal as well. The two distributions are not perfectly known. A fixed Bayesian threshold is therefore inappropriate unless priors and costs remain stable. The operational requirement is to maintain a false-alarm probability near a specified limit while retaining useful detection probability.

Approach

The receiver estimates the local noise level from neighboring time-frequency cells judged unlikely to contain the primary signal. It then normalizes the energy statistic and uses a CFAR-style threshold. A calibration stage checks the empirical false-alarm rate. When the primary waveform and timing become available, the receiver switches to a correlation or matched-filter detector, which provides better sensitivity than energy detection.

The design also records threshold, noise estimate, and decision confidence. If the estimated noise statistics become unreliable, the system declares an uncertain state rather than presenting a confident binary decision.

Outcome

Adaptive normalization reduces false alarms caused by slow noise-floor changes. The matched-filter mode improves detection when waveform information is reliable, while the energy-detector mode remains available when the signal model is unknown. The system accepts a modest loss of nominal performance in exchange for robustness and diagnosability.

Learning points

The case illustrates that detector selection depends on available prior knowledge, not merely on computational sophistication. It also shows why threshold control, model monitoring, and an explicit uncertainty state are important in deployed systems.

9.11 Limitations, Risks, and Common Errors

A common error is to confuse a high test statistic with a high probability of detection. The statistic must be interpreted under both hypotheses and at a specified threshold. Another error is to report detector performance without stating the noise model, signal strength, observation length, or false-alarm operating point.

Using a matched filter with an inaccurate waveform can reduce performance. Estimating noise power from data contaminated by the signal biases the threshold. Treating correlated samples as independent exaggerates the effective amount of information. In sequential testing, repeatedly checking a fixed-sample test can invalidate nominal error probabilities unless the stopping rule is incorporated into the design.

A detector may also be statistically sound but operationally poor. Excessive false alarms can overload operators, while an opaque machine-learning detector may be difficult to validate. Performance must therefore be tested under nominal conditions, stress conditions, distribution shifts, and adversarial or interference-rich environments.

9.12 Professional and Ethical Considerations

Detection decisions can affect safety, privacy, access to services, and allocation of communication resources. Engineers should document assumptions, error rates, calibration procedures, and known failure modes. Thresholds should not be tuned solely on a convenient test set. Data used to estimate distributions must be representative and handled according to privacy and security requirements.

In safety-critical applications, the detector should expose uncertainty and support independent monitoring. Automated decisions should be traceable, and human intervention should be available when the consequences of error are substantial. Professional responsibility requires recognizing that statistical optimality under a mathematical model does not guarantee fairness, safety, or reliability in the field.

9.13 Emerging Directions and Best Practices

Current research combines classical detection theory with adaptive sensing, robust statistics, distributed sensor networks, and neural signal representations. Distributed detectors exchange local statistics rather than raw data to reduce communication cost. Differentiable signal-processing architectures seek to learn parameters while retaining interpretable filtering and decision stages. Conformal and calibration methods are being studied to quantify uncertainty under changing conditions.

Best practice is to begin with an explicit hypothesis model, identify the operational error constraint, derive or select an appropriate statistic, and validate the threshold empirically. Noise and interference should be measured continuously where possible. Performance should be reported using ROC curves or detection probability at clearly stated false-alarm rates. Finally, deployment should include drift detection, recalibration, logging, and a safe response to model uncertainty.

Chapter Summary

Statistical detection converts uncertain observations into decisions by comparing evidence for competing hypotheses. The likelihood ratio is the central test statistic, and Bayesian, MAP, maximum-likelihood, and Neyman–Pearson rules differ mainly in their assumptions about priors, costs, and constraints. For known signals in Gaussian noise, the matched filter or its covariance-weighted form is optimal for maximizing output signal-to-noise ratio. ROC curves describe the trade-off between detection and false alarms, while sequential tests make decisions with a variable number of observations. Unknown parameters motivate GLRTs, detector banks, and energy-based methods. Real systems must also address colored and non-Gaussian noise, model mismatch, adaptive thresholds, calibration, and operational consequences. Effective detection engineering combines mathematical optimality with robustness, validation, and responsible deployment.

Key Terms

Hypothesis: A statistical statement about the process that generated an observation.

Simple hypothesis: A hypothesis specifying a completely known probability distribution.

Composite hypothesis: A hypothesis containing one or more unknown parameters.

Likelihood: The probability model for observed data viewed as a function of the hypothesis or its parameters.

Likelihood ratio: The ratio of the likelihood under one hypothesis to that under another.

Test statistic: A function of the observation used to make a decision.

False alarm: Deciding that a signal is present when it is absent.

Miss: Deciding that a signal is absent when it is present.

Probability of detection: The probability of deciding for the signal-present hypothesis when it is true.

Neyman–Pearson test: A test maximizing detection probability subject to a false-alarm constraint.

Bayesian detector: A detector minimizing expected decision cost using priors and a loss function.

MAP detector: A detector selecting the hypothesis with maximum posterior probability.

Matched filter: A filter matched to a known signal waveform that maximizes output SNR in specified noise.

ROC curve: A plot of detection probability against false-alarm probability as the threshold varies.

GLRT: A generalized likelihood-ratio test that substitutes parameter estimates into a likelihood ratio.

Sequential probability-ratio test: A test that accumulates evidence and can stop when evidence crosses a boundary.

Colored noise: Noise with nonconstant power spectral density or nonzero correlation between samples.

CFAR: Constant false-alarm rate processing that adapts the threshold to local interference or noise conditions.

Model mismatch: Performance degradation caused by differences between assumed and actual statistical models.

Robust detection: Detection designed to maintain acceptable performance over a set of plausible models.

Review Questions

Short Answer Questions

1. Define probability of detection and probability of false alarm.

2. What is the role of the likelihood ratio in binary hypothesis testing?

3. State the optimization objective of the Neyman–Pearson criterion.

4. Why does a matched filter use a time-reversed conjugate waveform?

5. Distinguish a simple hypothesis from a composite hypothesis.

Long Answer Questions

1. Derive the Gaussian likelihood-ratio test for a known deterministic signal in colored Gaussian noise.

2. Compare Bayesian, maximum-likelihood, MAP, and Neyman–Pearson detection criteria.

3. Explain ROC analysis and discuss how an operating point should be selected in an engineering system.

4. Describe the sequential probability-ratio test, including its stopping rules and practical benefits.

5. Explain why model uncertainty affects detector thresholds and discuss robust responses to uncertain noise statistics.

Analytical / Application Questions

1. A detector must maintain a false-alarm probability below 0.01 while maximizing detection probability. Which decision framework is appropriate, and how would its threshold be selected?

2. A receiver observes colored Gaussian noise and a known signal waveform. Explain how the detector should be modified relative to the white-noise matched filter.

3. An energy detector experiences a sudden rise in false alarms because the noise floor changes. Propose a revised detection architecture and identify the assumptions that must be validated.

22. Multiple Choice Questions with Answers

Question 1

Which decision criterion maximizes probability of detection subject to a specified upper bound on probability of false alarm?

A. Bayesian criterion

B. Neyman–Pearson criterion

C. Minimum mean-square error criterion

D. Maximum-entropy criterion

Correct Answer: B. Neyman–Pearson criterion

Question 2

For a known deterministic signal in zero-mean colored Gaussian noise with covariance C, the essential linear detection statistic is

A. sᵀx

B. xᵀx

C. sᵀC⁻¹x

D. det(C)x

Correct Answer: C. sᵀC⁻¹x

Question 3

A false alarm occurs when the detector

A. decides H0 when H1 is true

B. decides H1 when H0 is true

C. decides H1 when H1 is true

D. collects too few samples

Correct Answer: B. decides H1 when H0 is true

Question 4

The matched filter is optimal, under its standard assumptions, because it

A. minimizes sampling frequency

B. maximizes output signal-to-noise ratio at the sampling instant

C. eliminates all interference

D. requires no knowledge of the signal waveform

Correct Answer: B. maximizes output signal-to-noise ratio at the sampling instant

Question 5

A ROC curve represents the relationship between

A. signal energy and bandwidth

B. noise variance and sampling rate

C. probability of detection and probability of false alarm

D. latency and memory usage only

Correct Answer: C. probability of detection and probability of false alarm

Question 6

Which detector explicitly incorporates prior probabilities and decision costs?

A. Bayesian detector

B. Unnormalized correlator

C. Fixed-window Fourier detector

D. Uniform quantizer

Correct Answer: A. Bayesian detector

Question 7

In a sequential probability-ratio test, sampling can stop when

A. the first sample is always positive

B. the accumulated log-likelihood crosses either decision boundary

C. the noise variance becomes zero

D. the ROC becomes linear

Correct Answer: B. the accumulated log-likelihood crosses either decision boundary

Question 8

When the signal amplitude or phase is unknown, a generalized likelihood-ratio test typically

A. ignores the unknown parameters

B. estimates parameters under the hypotheses and substitutes them into the likelihood ratio

C. assumes that all observations are deterministic

D. sets the threshold to zero

Correct Answer: B. estimates parameters under the hypotheses and substitutes them into the likelihood ratio

Question 9

The main purpose of CFAR processing is to

A. make the signal waveform known

B. adapt the threshold to local noise or interference conditions

C. increase the sampling frequency indefinitely

D. remove the need for hypothesis testing

Correct Answer: B. adapt the threshold to local noise or interference conditions

Question 10

If a detector optimized for white Gaussian noise is used in strongly colored noise without compensation, the most likely consequence is

A. guaranteed improvement in detection probability

B. unchanged false-alarm probability under all conditions

C. performance degradation caused by model mismatch

D. automatic conversion to a Bayesian detector

Correct Answer: C. performance degradation caused by model mismatch