Estimation, Detection and Array Signal Processing

Introduction and Context

Digital signal processing (DSP) systems rarely observe a desired quantity directly. Instead, they receive noisy, incomplete, and often indirect measurements. The engineering tasks are therefore to infer unknown parameters or signals, decide between competing hypotheses, and exploit measurements from multiple sensors. These tasks are studied through estimation, detection, and array signal processing.

Estimation produces a numerical description of an unknown quantity, such as a target range, carrier phase, direction of arrival, or channel coefficient. Detection produces a decision, such as whether a target is present or absent. Array signal processing uses spatially distributed sensors to improve inference, suppress interference, and determine source locations.

A common observation model is

where x is the observed data vector, s(θ) is a signal depending on an unknown parameter vector θ, and n is measurement noise. In a linear model,

where H is a known observation matrix. The theory developed in this chapter supports radar, sonar, wireless communications, medical imaging, seismic processing, audio localization, and sensor networks.

Learning Objectives

After studying this chapter, the reader should be able to:

1. distinguish estimation, detection, and classification problems;
2. formulate likelihood, prior, posterior, and hypothesis-testing models;
3. derive and interpret maximum-likelihood, maximum a posteriori, and minimum mean-square-error estimators;
4. evaluate estimator bias, variance, consistency, efficiency, and the Cramér–Rao lower bound;
5. design and interpret Neyman–Pearson and Bayesian detectors;
6. explain matched filtering, generalized likelihood-ratio testing, and constant-false-alarm-rate detection;
7. model uniform linear arrays and derive beamforming and direction-of-arrival methods;
8. assess practical limitations including calibration errors, coherent sources, finite samples, and correlated noise;
9. apply these methods in a realistic signal-processing case.

1. Statistical Foundations of Inference

1.1 Random observations and probability models

Let X denote a random observation and x a particular observed value. A statistical model specifies the probability density p(x;θ), where θ is an unknown deterministic parameter, or p(x|θ), where the parameter is treated conditionally.

For independent observations x₁, x₂, …, x_N, the likelihood is

The likelihood is a function of θ for fixed data; it is not itself a probability distribution over the parameter unless a prior is introduced. The log-likelihood is often more convenient:

For complex vectors, probability densities are defined with respect to the appropriate real dimensions. A proper complex Gaussian vector with mean μ and covariance R has density

where H denotes conjugate transpose, R is positive definite, and N is the vector dimension.

1.2 Estimation versus detection

In estimation, the unknown quantity may be continuous, for example delay τ or amplitude A. An estimator is a function θ̂ = g(x). In detection, the unknown condition is represented by discrete hypotheses, commonly

where H₀ denotes absence and H₁ presence of a signal. A detector maps the observations to a decision. It may also produce a soft statistic that is later thresholded.

1.3 Bayes and frequentist viewpoints

A frequentist model treats θ as fixed but unknown and evaluates performance over repeated data sets. A Bayesian model assigns a prior density p(θ), giving the posterior

where p(x) = ∫ p(x|θ)p(θ)dθ. The posterior combines prior information and measurements. Neither viewpoint is universally superior: prior information can improve performance at low signal-to-noise ratio, but an inaccurate prior can produce systematic errors.

2. Point Estimation

2.1 Maximum-likelihood estimation

The maximum-likelihood (ML) estimator is

It is especially useful when no reliable prior is available. ML is not automatically unbiased or optimal at finite sample sizes. Under regularity conditions, it is often asymptotically efficient and approximately Gaussian as the number of observations increases.

Example: estimating the mean of Gaussian noise. Suppose x_k = μ + n_k, where n_k are independent, zero-mean Gaussian variables with known variance σ². The log-likelihood, apart from constants, is

Differentiating and setting the result to zero gives

The estimator is unbiased and has variance σ²/N. Its units are the same as those of the observations, and its uncertainty decreases with sample size.

2.2 Maximum a posteriori estimation

The maximum a posteriori (MAP) estimator maximizes the posterior density:

For a Gaussian likelihood and a Gaussian prior, MAP estimation becomes a regularized least-squares problem. For example, if

then the MAP estimate minimizes

and is

The matrices must be dimensionally compatible, and positive-definiteness ensures a unique solution. The second term penalizes deviations from prior knowledge.

2.3 Minimum mean-square-error estimation

The minimum mean-square-error (MMSE) estimator minimizes expected squared error:

For the linear Gaussian model above, the posterior mean is

The MMSE criterion is Bayesian and differs from MAP when the posterior is not symmetric or Gaussian. For Gaussian models, posterior mean and MAP coincide.

2.4 Error criteria and estimator quality

Important properties include:

• Bias: b(θ) = E[θ̂] − θ.
• Variance: Var(θ̂) = E[(θ̂ − E[θ̂])²].
• Mean-square error:

• Consistency: θ̂ converges in probability to θ as sample size increases.
• Efficiency: an estimator attains the smallest variance within a specified class, often the Cramér–Rao bound.

For vector parameters, scalar variance is replaced by the error covariance matrix

2.5 Fisher information and the Cramér–Rao lower bound

For a scalar parameter, the Fisher information is

provided the derivative and expectation conditions are valid. For any unbiased estimator,

For a vector parameter, the Fisher information matrix is

and the covariance satisfies C_θ̂ ≥ J^(−1)(θ), in the positive-semidefinite sense. The bound assumes regularity conditions and unbiasedness; it is a lower bound, not a guarantee that an estimator exists that attains it.

3. Detection Theory

3.1 Binary hypothesis testing

A binary detector compares a test statistic T(x) with a threshold γ:

There are four outcomes: detection, miss, false alarm, and correct rejection. Their probabilities are

and

with miss probability P_M = 1−P_D. A receiver operating characteristic (ROC) plots P_D against P_FA as the threshold varies.

3.2 Likelihood-ratio tests

The likelihood-ratio test (LRT) uses

and decides H₁ when Λ(x) > η. The threshold η depends on the desired false-alarm probability, prior probabilities, and decision costs.

The Neyman–Pearson criterion maximizes P_D subject to P_FA ≤ α, where α is a specified false-alarm limit. The Neyman–Pearson lemma establishes that the LRT is optimal for this simple-versus-simple problem.

A Bayesian detector minimizes expected risk. If C_ij is the cost of deciding H_i when H_j is true, the decision depends on posterior probabilities and costs. Thus a safety-critical detector may use a different threshold from a consumer system even with identical sensor data.

3.3 Matched-filter detection

Suppose the signal s is known and the noise is zero-mean Gaussian with covariance R. The optimum linear statistic, up to a positive scale factor, is

This is a whitened matched filter. If noise is white, R = σ²I, so the statistic reduces to s^H x after scaling. Whitening is important when noise samples are correlated; correlating directly with s is generally not optimum.

For a known deterministic signal in additive white Gaussian noise, the matched filter maximizes output signal-to-noise ratio. Its implementation may be a convolution with a time-reversed conjugate template, followed by sampling at the expected arrival time.

3.4 Generalized likelihood-ratio test

When unknown parameters occur under one or both hypotheses, they can be replaced by their ML estimates:

This generalized likelihood-ratio test (GLRT) is widely used because it is practical, although it is not universally optimal. Its threshold is often calibrated by simulation or by an analytical approximation.

3.5 CFAR detection

In radar and sonar, background power may vary with range or time. A fixed threshold then produces unacceptable false alarms. A constant-false-alarm-rate (CFAR) detector estimates local noise power from reference cells and scales the threshold accordingly. Guard cells prevent target energy from contaminating the noise estimate. Cell-averaging CFAR is effective in homogeneous environments; greatest-of and smallest-of variants are useful near clutter transitions but have different detection behavior.

4. Array Signal Processing

4.1 Spatial sampling model

An array consists of sensors at known positions. For a narrowband plane wave from direction θ, the signal arrives at different sensors with phase delays. For a uniform linear array (ULA) of M sensors, spacing d, and wavelength λ, a common steering vector is

where k = 2π/λ is the wavenumber and the angle convention is measured from broadside. Reversing the angle convention changes the sign or trigonometric factor, so the convention must be stated.

For K narrowband sources,

where A(θ) = [a(θ₁),…,a(θ_K) is the steering matrix, s(t) contains source signals, and n(t) is sensor noise.

The spatial covariance matrix is

where R_s and R_n are source and noise covariance matrices. In practice, the expectation is replaced by the sample covariance

assuming snapshots are comparable and sufficiently stationary.

Figure: A uniform linear array observes direction-dependent phase differences between sensors.

4.2 Conventional beamforming

A beamformer forms a weighted sum

with spatial response

For steering toward θ₀, conventional delay-and-sum beamforming chooses w = a(θ₀)/M. The normalization gives unity response in the look direction because a^H(θ₀)a(θ₀)=M for unit-magnitude steering-vector entries.

For an unweighted ULA, the approximate half-power beamwidth decreases as the array aperture increases. However, exact beamwidth depends on angle convention, weighting, and the definition used. Element spacing greater than approximately λ/2 can produce grating lobes over a broad scan range, creating ambiguous directions.

4.3 Minimum-variance distortionless response

The MVDR or Capon beamformer minimizes output power while maintaining unit gain in the look direction:

The solution, assuming R_x is nonsingular, is

It can place nulls toward strong interferers, but it is sensitive to covariance-estimation errors and steering mismatch. Diagonal loading replaces R_x by R_x + δI, where δ > 0 improves conditioning and robustness, at the cost of reduced adaptivity.

4.4 Subspace direction-of-arrival estimation

If the noise is spatially white, R_n = σ²I, the eigenvectors of R_x divide into a signal subspace and a noise subspace. For sufficiently many snapshots and distinct sources, steering vectors of true sources are orthogonal to the noise subspace.

The MUSIC pseudospectrum is

where E_n contains noise-subspace eigenvectors. Peaks indicate estimated directions. MUSIC requires model-order knowledge or estimation, adequate array calibration, and conditions such as uncorrelated sources and a suitable noise model. Coherent multipath sources can reduce covariance rank and require spatial smoothing or other remedies.

ESPRIT exploits rotational invariance between two overlapping subarrays. It avoids a full angular search but requires a suitable array geometry and paired subarrays. Maximum-likelihood DOA estimation can provide excellent statistical performance but is computationally more demanding and can involve multidimensional nonlinear optimization.

5. Worked Application: Direction Finding in a Wireless Base Station

Background

Consider an illustrative base-station receiver with an eight-element ULA operating at a carrier frequency of 3 GHz. The wavelength is approximately

The elements are spaced at d = 0.05 m = λ/2, which limits grating lobes for broadside scanning.

Problem

A desired user is located near 20° from broadside, while an interfering transmitter is near −35°. The receiver collects N = 2,000 snapshots. Thermal noise and calibration errors are present, and the interference is considerably stronger than the desired signal.

Analysis and approach

First, each channel is calibrated for gain and phase using a reference source. The receiver forms

A conventional beam scan provides an initial indication of dominant angular energy. Because the interferer is strong, an MVDR beamformer is then designed for θ₀ = 20°. In practice, diagonal loading is applied:

The value of δ is selected relative to the estimated noise power and validated using held-out data. The constraint w^Ha(20°)=1 preserves the desired direction, while the minimization suppresses energy elsewhere.

A MUSIC analysis is also performed as a diagnostic. If two clear pseudospectrum peaks appear near the expected directions, the result supports the beamformer design. If the sources are coherent because of strong multipath, spatial smoothing is applied before eigendecomposition.

Outcome

The adaptive beamformer increases desired-to-interference ratio relative to conventional summation, but performance degrades if the desired steering vector is wrong by several degrees. A robust implementation therefore uses a small angular uncertainty region, diagonal loading, continuous calibration checks, and protection against placing a null on the desired user.

Learning points

1. Spatial resolution depends on aperture and wavelength, not only on sensor count.
2. Covariance estimates require enough snapshots and representative operating conditions.
3. Adaptive methods improve interference rejection but are more sensitive to mismatch.
4. A DOA estimate is not automatically a reliable communication beamformer; calibration and robustness are essential.

6. Practical Implementation Considerations

Sampling and synchronization

Array channels must be synchronously sampled. Timing offsets create phase errors that appear as direction errors. The sampling rate must satisfy the temporal bandwidth requirement, while spatial aliasing is controlled through element spacing and scan range.

Numerical conditioning

Matrix inversion should generally be implemented using a factorization such as Cholesky or QR rather than explicitly forming an inverse. Ill-conditioned covariance matrices arise with too few snapshots, highly correlated sources, or redundant sensors. Regularization, shrinkage, and diagonal loading improve stability.

Model mismatch

Real arrays depart from ideal models because of mutual coupling, element-pattern differences, cable phase shifts, finite bandwidth, platform scattering, and sensor position errors. Calibration should use measured steering vectors where possible. Wideband signals may require filter-and-sum processing, subband decomposition, or focusing methods because a single narrowband steering vector is inadequate.

Performance evaluation

A complete evaluation should report probability of detection, false-alarm rate, estimation bias, angular root-mean-square error, computational cost, latency, and robustness to interference. Simulation alone is insufficient: controlled laboratory tests and field data are needed to reveal calibration and propagation effects.

7. Ethical, Security, and Governance Issues

Array processing can localize people, devices, vehicles, and conversations. Deployment should therefore follow applicable privacy and telecommunications laws, minimize unnecessary data retention, and restrict access to raw sensor data. Security controls are needed because spoofed signals, replay attacks, malicious interference, and corrupted calibration data can mislead detectors and beamformers.

Performance should be evaluated across operating environments rather than reported only for favorable cases. In safety-critical systems, automated detection should provide confidence measures, audit logs, fail-safe behavior, and human oversight where appropriate. A statistically accurate estimator can still be unacceptable if its use violates privacy, creates discriminatory surveillance, or conceals uncertainty from decision-makers.

8. Contemporary Developments

Modern systems increasingly combine classical statistical signal processing with machine learning. Neural networks can learn denoising, waveform classification, or calibration corrections, but their outputs should be tested against interpretable baselines such as matched filters and MVDR. Learned methods can fail under distribution shift, adversarial interference, or changed propagation conditions.

Sparse and co-prime arrays seek larger effective apertures with fewer sensors, although they introduce ambiguity management and more complicated array processing. Distributed arrays and cooperative sensing provide wider coverage but require synchronization and communication. Joint communication-and-sensing systems use shared waveforms for data transmission and environmental inference. Research remains active in robust covariance estimation, few-snapshot inference, wideband localization, intelligent reflecting surfaces, and uncertainty-aware learning.

Best Practices

• State the observation, noise, and hypothesis models explicitly.
• Separate training, calibration, and evaluation data.
• Use covariance estimates appropriate to stationarity and snapshot availability.
• Check units, angle conventions, phase signs, and sensor ordering.
• Compare adaptive algorithms with transparent baselines.
• Validate false-alarm performance under changing clutter and noise.
• Report uncertainty and sensitivity to mismatch.
• Avoid explicit matrix inversion when stable factorizations are available.
• Treat privacy, security, and human consequences as engineering requirements.

Chapter Summary

Estimation infers continuous unknown quantities from noisy data, while detection decides among competing hypotheses. ML uses the likelihood, MAP combines likelihood with a prior, and MMSE minimizes expected squared error through the posterior mean. Bias, variance, MSE, consistency, Fisher information, and the Cramér–Rao bound provide principled performance measures.

Detection theory uses likelihood ratios, decision costs, and false-alarm constraints. Matched filters are optimal for known signals in Gaussian noise, while GLRTs address unknown parameters and CFAR methods adapt thresholds to changing backgrounds. Array processing adds spatial information through steering vectors and covariance matrices. Conventional beamforming is simple, MVDR adaptively suppresses interference, and subspace methods such as MUSIC exploit eigenstructure for high-resolution direction finding. All methods depend on assumptions concerning noise, calibration, source coherence, snapshots, geometry, and stationarity. Reliable professional systems therefore combine mathematical design with numerical conditioning, validation, uncertainty reporting, security, and responsible governance.

Key Terms

Estimation: Inferring an unknown numerical quantity from observations.

Detection: Deciding which of two or more hypotheses generated the data.

Likelihood: Probability density viewed as a function of unknown parameters.

Prior: Probability model representing knowledge about a parameter before observing current data.

Posterior: Updated probability distribution after combining prior and likelihood.

Maximum likelihood: Parameter estimate that maximizes the likelihood.

MAP estimation: Estimate that maximizes the posterior density.

MMSE estimation: Bayesian estimate minimizing expected squared error.

Bias: Difference between an estimator’s expected value and the true parameter.

Fisher information: Information about a parameter contained in a probability model.

Cramér–Rao lower bound: Lower bound on the covariance of unbiased estimators under regularity conditions.

False alarm: Decision that a signal is present when it is absent.

Probability of detection: Probability of declaring a signal present when it is present.

Matched filter: Filter matched to a known signal waveform for maximum output SNR in Gaussian noise.

CFAR: Detection method designed to maintain a controlled false-alarm rate under varying background power.

Steering vector: Array response vector for a specified source direction.

Beamformer: Weighted spatial combiner that controls array response.

MVDR: Minimum-variance distortionless-response beamformer.

MUSIC: Subspace-based direction-of-arrival estimator using noise-subspace orthogonality.

Spatial aliasing: Ambiguous directional response caused by excessive sensor spacing.

Review Questions

Short Answer Questions

1. Distinguish an estimation problem from a detection problem.
2. Define estimator bias and mean-square error.
3. What is the purpose of whitening in a matched-filter detector?
4. Why can sensor spacing greater than approximately λ/2 cause grating lobes?
5. State one assumption required by MUSIC.

Long Answer Questions

1. Derive the ML estimate of the mean of independent Gaussian observations with known variance and discuss its properties.
2. Explain the Neyman–Pearson criterion and the role of the likelihood-ratio test.
3. Derive the MVDR beamformer using constrained optimization and interpret its constraint.
4. Explain how the eigenstructure of an array covariance matrix supports MUSIC direction finding.
5. Compare conventional beamforming, MVDR, and MUSIC in terms of resolution, robustness, and computational requirements.

Analytical / Application Questions

1. A receiver observes x_k = μ+n_k for N independent samples with known noise variance. Derive the estimator of μ and its variance, then explain how doubling N changes the variance.
2. Design a processing chain for a radar with unknown target amplitude, nonuniform clutter, and a required fixed false-alarm rate. Justify the roles of whitening, GLRT, and CFAR processing.
3. An eight-element ULA experiences two coherent multipath arrivals. Explain why basic MUSIC may fail and propose a covariance-processing remedy, including the limitations of that remedy.

Multiple Choice Questions with Answers

Question 1

Which estimator minimizes the expected squared estimation error in a Bayesian model?

A. Maximum-likelihood estimator

B. Maximum a posteriori estimator

C. Minimum mean-square-error estimator

D. Least absolute deviations estimator

Correct Answer: C. Minimum mean-square-error estimator

Question 2

In a Neyman–Pearson detector, which quantity is constrained directly?

A. Probability of detection

B. Probability of false alarm

C. Estimator bias

D. Array aperture

Correct Answer: B. Probability of false alarm

Question 3

For colored Gaussian noise with covariance R, the matched-filter statistic is proportional to which expression?

A. s^Hx

B. x^HRx

C. s^HR^(−1)x

D. s^HRx

Correct Answer: C. s^HR^(−1)x

Question 4

What is the principal purpose of guard cells in a CFAR detector?

A. To increase ADC resolution

B. To prevent target energy contaminating the background estimate

C. To estimate the carrier frequency

D. To remove spatial aliasing

Correct Answer: B. To prevent target energy contaminating the background estimate

Question 5

For a ULA, increasing the physical aperture generally has what effect?

A. It reduces spatial resolution

B. It improves angular resolution

C. It eliminates calibration requirements

D. It guarantees zero false alarms

Correct Answer: B. It improves angular resolution

Question 6

What constraint does the MVDR beamformer impose in the look direction?

A. Zero gain

B. Unit gain

C. Maximum noise gain

D. Random phase

Correct Answer: B. Unit gain

Question 7

MUSIC identifies source directions primarily through orthogonality between steering vectors and which subspace?

A. Signal subspace

B. Noise subspace

C. Null-data subspace only

D. Time-frequency subspace

Correct Answer: B. Noise subspace

Question 8

If λ=0.1 m, what is the usual half-wavelength spacing?

A. 0.005 m

B. 0.01 m

C. 0.05 m

D. 0.2 m

Correct Answer: C. 0.05 m

Question 9

Which condition commonly makes a sample covariance matrix poorly conditioned?

A. Many independent snapshots and white noise

B. Highly correlated sources and few snapshots

C. Perfect calibration

D. A known steering vector

Correct Answer: B. Highly correlated sources and few snapshots

Question 10

What does the Cramér–Rao lower bound provide under its regularity conditions?

A. An upper bound on detection probability

B. A lower bound on the covariance of unbiased estimators

C. An exact ML estimate for every model

D. A guarantee of zero estimation error

Correct Answer: B. A lower bound on the covariance of unbiased estimators