Random Signals and Statistical DSP

Introduction and context

Many signals encountered in engineering are not exactly repeatable. Thermal noise, interference, speech, biomedical recordings, wireless fading, sensor uncertainty, and packet-arrival processes contain components that cannot be predicted sample by sample. Digital signal processing therefore requires a statistical description in addition to a deterministic waveform description.

A random signal is modelled as a random process: a family of random variables indexed by time. In discrete time, X[n] denotes the random variable at sample index n, while one observed record is called a realization, sample function, or record. Statistical DSP uses ensembles of realizations, or suitable time averages, to estimate quantities such as mean, variance, correlation, and power spectral density (PSD).

The distinction between energy and power is important. For a deterministic sequence, energy is E = ∑_n=-∞^∞|x[n]|², whereas average power is P = lim_N→∞(1/(2N+1))∑_n=-N^N|x[n]|². Random-process definitions replace these quantities by expectations when appropriate.

Learning objectives

After studying this chapter, the reader should be able to:

1. distinguish random variables, random processes, realizations, and ensembles;
2. define stationarity, ergodicity, correlation, covariance, and spectral density;
3. derive the Wiener–Khinchin relationship for discrete-time processes;
4. formulate and use Yule–Walker equations for autoregressive models;
5. construct statistically valid correlation and PSD estimates, including for complex data;
6. analyse linear systems driven by random inputs;
7. evaluate practical limitations, uncertainty, and applications of statistical DSP.

1. Probability models for discrete-time signals

A random variable is a numerical outcome of an experiment. A random process {X[n]} assigns a random variable to every integer time index. A realization x_r[n] is one sequence obtained from the process. The subscript r identifies the experiment or record and is not a time index.

The ensemble mean is

where E{·} is expectation. The autocorrelation is

and the autocovariance is

The star denotes complex conjugation. Consequently,

R_X[n₁,n₂] = C_X[n₁,n₂] + μ_X[n₁]μ_X*[n₂]. For a real-valued process, conjugation has no numerical effect, but retaining it makes the definitions valid for complex signals.

At equal time, the second moment is

The variance is σ_X²[n] = C_X[n,n]. Thus, only for a zero-mean process is R_X[n,n] = σ_X²[n]. For a wide-sense stationary process this becomes R_X[0] = σ_X² + |μ_X|², and it reduces to σ_X² only when μ_X = 0.

Important inequalities follow from covariance positivity. For any finite set of complex coefficients a_k,

and, for a stationary process, |R_X[k]| ≤ R_X[0].

2. Stationarity and ergodicity

A process is strictly stationary if every finite-dimensional distribution is invariant under a common integer shift m:

for every integer m, every order p, and all threshold values x₁,…,x_p. Strict stationarity is a distributional property and can be difficult to verify.

A process is wide-sense stationary (WSS) if:

1. E{X[n]} = μ_X is constant;
2. E{|X[n]|²} < ∞;
3. R_X[n₁,n₂] = R_X[n₁−n₂].

Writing lag k = n₁−n₂, the autocorrelation is R_X[k]. For a complex WSS process,

so the autocorrelation is Hermitian. For real-valued data it is even: R_X[−k] = R_X[k]. The covariance is similarly a function of lag: C_X[k] = R_X[k]−|μ_X|².

Ergodicity concerns whether time averages equal ensemble averages. Mean ergodicity, for example, requires a time-average estimator of the mean to converge to μ_X. WSS does not automatically imply ergodicity. In practice, ergodicity is often assumed when only one long record is available, but the assumption should be tested or justified from the physical mechanism.

3. Correlation and power spectral density

The autocorrelation measures statistical similarity between samples separated by a lag. Cross-correlation between WSS processes X[n] and Y[n] may be defined as

with conventions stated explicitly because texts differ in lag orientation. Cross-correlation supports delay estimation, synchronization, system identification, and matched filtering.

For a WSS process with absolutely summable autocorrelation, the discrete-time PSD is the Fourier transform

and the inverse relationship is

The PSD is periodic with period 2π, nonnegative, and has units of signal-squared per radian/sample when frequency is expressed in radians/sample. The average power is

This includes the mean contribution. If X[n] = μ_X + W[n] with zero-mean WSS W[n], the PSD contains a spectral line 2π|μ_X|²∑_ℓδ(ω−2πℓ) in addition to the continuous PSD of W[n]. The covariance spectrum excludes this deterministic mean line.

Wiener–Khinchin derivation

Consider a finite observation of a zero-mean WSS process:

The expected squared magnitude is

Set k = n−m. For each lag k, there are N−|k| pairs with that difference, giving

If ∑_k|R_X[k]| < ∞, the factor (1−|k|/N) tends to one and the finite sum converges to S_X(e^jω). Therefore,

under the stated regularity conditions. This is the discrete-time Wiener–Khinchin theorem: the PSD is the Fourier transform of autocorrelation and is also the limiting expected periodogram, subject to the usual convergence interpretation. Spectral lines require generalized-function interpretation because their autocorrelation need not be absolutely summable.

4. Estimation from finite records

Suppose one record x[0],…,x[N−1] is available. The biased autocorrelation estimator is

For negative lags, define it explicitly by conjugate reversal:

Equivalently, for k < 0, R̂_b[k] = (1/N)∑_n=0^N−1−|k|x[n]x*[n+|k|]. This ordering is essential for complex records. The unbiased estimator replaces 1/N by 1/(N−|k|) for both signs, while retaining the corresponding conjugate ordering. The biased form is nonnegative-definite and is often preferred when its Fourier transform is used as a PSD estimate; the unbiased form can have undesirable spectral properties for short records.

A practical periodogram is

for a normalized discrete-time sequence. It is asymptotically unbiased under suitable conditions but is not generally consistent: its variance does not vanish as N grows. Windowing reduces leakage but changes resolution and amplitude calibration. Welch’s method divides data into overlapping, windowed segments, averages modified periodograms, and usually reduces variance at the expense of frequency resolution.

For a window w[n], segment length L, and sampling frequency f_s, a power-normalized modified periodogram uses a normalization proportional to U = (1/L)∑_n=0^L−1|w[n]|². Exact scaling depends on whether the spectrum is one-sided or two-sided and on units; these conventions must be documented.

5. Linear systems with random inputs

For a stable LTI system with impulse response h[n] and output Y[n] = ∑_r h[r]X[n−r], a WSS input produces a WSS output under standard summability and finite-power conditions. The output PSD is

where H(e^jω) = ∑_r h[r]e^-jωr. For input-output cross-spectrum under the convention R_XY[k] = E{X[n+k]Y*[n]},

when Y = h*X. The output power is obtained by integrating S_Y over frequency. This result is central to noise filtering, receiver design, vibration analysis, and sensor conditioning.

6. Autoregressive modelling and Yule–Walker equations

An AR(p) model is

where E[n] is zero-mean white innovation noise with variance σ_E². Stability requires all roots of A(z)=1+∑_i=1^pa_i z^-i to lie inside the unit circle in the z-plane when expressed as poles of 1/A(z).

Multiplying the model by X*[n−k] and taking expectations gives the Yule–Walker equations. For k ≥ 1,

For k=0,

For complex models, coefficients and correlation orientation must be treated consistently; a common convention writes the equations using conjugated coefficients in the correlation matrix. For a real AR(2) model, the first two equations are

Given estimated correlations, these equations can be solved for model coefficients. The AR PSD is

provided the model is stable. Levinson–Durbin recursion solves the Toeplitz Yule–Walker system efficiently, with useful reflection-coefficient stability tests.

Worked example: AR(1)

Let X[n] = 0.8X[n−1] + E[n], where E[n] is white with variance σ_E² = 1. In the sign convention above, a₁ = −0.8. The lag-one Yule–Walker equation is R[1]−0.8R[0]=0, hence R[1]=0.8R[0]. At lag zero,

R[0]−0.8R[1]=1. Substitution gives R[0](1−0.64)=1, so R[0]=1/0.36≈2.7778 and R[1]≈2.2222. The stable PSD is

S(e^jω)=1/|1−0.8e^-jω|².

7. Statistical detection and estimation

Statistical DSP supports decisions under uncertainty. In binary detection, hypotheses H₀ and H₁ generate observations with likelihoods p(x|H₀) and p(x|H₁). The likelihood-ratio test compares

with a threshold determined by prior probabilities and costs in a Bayesian detector, or by a prescribed false-alarm probability in a Neyman–Pearson detector. In matched filtering, a known signal in additive white Gaussian noise is correlated with the time-reversed conjugate template. The detector’s performance is described by probability of detection, probability of false alarm, and receiver operating characteristic curves.

The Wiener filter estimates a desired process D[n] from an observed process X[n]. For a noncausal frequency-domain formulation,

when S_X(e^jω) > 0 and the mean-square criterion applies. Causal Wiener filtering requires spectral factorization and is more restrictive. The Kalman filter is a recursive Bayesian minimum-mean-square estimator for linear state-space models with Gaussian noise; its assumptions should not be silently applied to nonlinear or strongly non-Gaussian systems.

8. Illustrative application: vibration-based machine monitoring

Background. Consider a motor monitored by an accelerometer sampled at f_s=12.8 kHz. Normal operation produces broadband vibration plus narrowband components related to shaft rotation and bearing geometry.

Problem. A maintenance team must distinguish normal spectral variation from an emerging bearing fault using a 10-second record.

Analysis. The record contains 128,000 samples. The engineer first removes the sample mean, records sensor calibration and units, and checks for clipping and missing samples. A Welch PSD is computed using 4,096-point Hann-windowed segments with 50% overlap. The frequency-bin spacing before any interpolation is f_s/4096 = 3.125 Hz. Several adjacent segments are averaged to reduce variance. A baseline distribution is obtained from healthy machines under comparable speed and load.

Approach. Narrowband peaks are compared with baseline confidence intervals, while broadband energy is integrated over physically meaningful bands. If rotational speed varies, order tracking is preferred to a fixed-frequency PSD. A classifier may use band powers and spectral kurtosis, but training and test records must be separated by machine and operating session to avoid leakage.

Outcome. A persistent increase near a bearing characteristic order, supported by time-domain impulsiveness and repeated measurements, triggers inspection. One isolated high peak is not treated as proof because leakage, resonance, speed variation, and sensor mounting can produce similar features.

Learning points. Statistical significance is not the same as mechanical causation; calibration, uncertainty, operating-condition control, and independent validation are essential. The same workflow applies to power-electronic converter noise, biomedical monitoring, and communications interference.

9. Limitations, governance, and contemporary practice

Finite records create bias, variance, leakage, and frequency-resolution trade-offs. Nonstationarity invalidates a single global PSD; short-time Fourier analysis, wavelets, cyclostationary analysis, or state-space models may be more appropriate. Heavy-tailed or impulsive noise challenges Gaussian assumptions. Correlation does not establish causation, and preprocessing can create artificial periodicity.

In safety-critical monitoring, retain raw data, processing parameters, calibration records, software versions, and uncertainty estimates. Automated or AI-based anomaly detection should provide traceable features, validation across operating regimes, drift monitoring, and human review for high-consequence decisions. Privacy controls are required for speech, biomedical, and location-linked signals. Models should not be trained with contaminated labels or evaluated on records that also contributed to parameter selection.

Best practice is to state the stationarity assumption, mean-removal policy, correlation convention, window and overlap, PSD scaling, confidence method, sampling rate, units, and validation protocol. Reproducible statistical DSP is as much a measurement discipline as an algorithmic one.

Chapter Summary

Random signals are described through ensembles and probability laws, while realizations are finite records used for estimation. Mean, autocorrelation, covariance, variance, and PSD must be distinguished carefully, especially for nonzero-mean and complex processes. WSS requires constant mean and lag-dependent second-order statistics; strict stationarity is distributional and stronger. The Wiener–Khinchin theorem connects autocorrelation and PSD, and its finite-record derivation explains the periodogram. Correlation estimators must preserve conjugate symmetry for complex data. Linear systems shape random spectra by |H|², while AR models provide compact spectral descriptions through explicitly formulated Yule–Walker equations. Practical analysis requires attention to bias, variance, leakage, nonstationarity, units, calibration, uncertainty, and responsible deployment.

Key Terms

Random process: A time-indexed family of random variables.

Realization: One observed sample sequence from a random process.

Ensemble mean: Expected value at a specified time index.

Autocorrelation: Expected product of a sample and a conjugated lagged sample.

Autocovariance: Correlation after removal of the mean contribution.

WSS: Wide-sense stationarity based on constant mean and lag-dependent correlation.

Strict stationarity: Invariance of all finite-dimensional distributions under time shifts.

Ergodicity: Conditions under which time averages represent ensemble averages.

PSD: Fourier transform of the autocorrelation of a WSS process.

Spectral line: Impulsive spectral component associated with deterministic periodic or mean content.

Periodogram: Squared magnitude of a finite-record Fourier transform with normalization.

Window leakage: Spectral spreading caused by finite observation and windowing.

Cross-correlation: Statistical similarity between two processes at a relative lag.

AR model: Model expressing a sample as a linear combination of past samples plus innovation.

Yule–Walker equations: Correlation equations used to estimate AR parameters.

Innovation: Uncorrelated driving noise in an AR or state-space model.

Wiener filter: Linear minimum-mean-square estimator based on spectra.

Kalman filter: Recursive state estimator for specified stochastic state-space models.

False-alarm probability: Probability of declaring a target when the null hypothesis is true.

Review Questions

Short Answer Questions

1. Distinguish a random process from one of its realizations.
2. State the three conditions for a discrete-time process to be WSS.
3. Why does R_X[0] not always equal the variance?
4. State the conjugate-symmetry property of autocorrelation for a complex WSS process.
5. What is the principal bias–variance trade-off in Welch PSD estimation?

Long Answer Questions

1. Derive the Wiener–Khinchin relationship from the expected squared magnitude of a finite-record Fourier transform.
2. Explain strict stationarity, WSS, and ergodicity, including their logical relationships.
3. Derive the Yule–Walker equations for a general real AR(p) process.
4. Explain how a stable LTI system transforms the PSD of a WSS input.
5. Compare periodogram, Bartlett, and Welch spectral estimators in terms of resolution and variance.

Analytical / Application Questions

1. A zero-mean AR(1) process satisfies X[n]=0.6X[n−1]+E[n], with innovation variance 2. Calculate R[0] and R[1].
2. Design a PSD-estimation workflow for a nonstationary industrial vibration record and justify window, segmentation, and validation choices.
3. For a complex finite record, explain how you would calculate positive and negative lag autocorrelation estimates so that Hermitian symmetry is preserved.

Multiple Choice Questions with Answers

Question 1

For a WSS process with mean μ_X, which expression is generally correct?

A. R_X[0] = σ_X²

B. R_X[0] = σ_X² + |μ_X|²

C. R_X[0] = μ_X

D. R_X[0] = 0

Correct Answer: B. R_X[0] = σ_X² + |μ_X|²

Question 2

For a complex WSS process, the autocorrelation satisfies:

A. R_X[−k] = R_X[k]

B. R_X[−k] = −R_X[k]

C. R_X[−k] = R_X*[k]

D. R_X[−k] = |R_X[k]|

Correct Answer: C. R_X[−k] = R_X*[k]

Question 3

Which property is required for WSS but not necessarily for strict stationarity?

A. Constant mean and lag-dependent second-order statistics

B. Invariance of every finite-dimensional distribution

C. Gaussian marginal distributions

D. White-noise excitation

Correct Answer: A. Constant mean and lag-dependent second-order statistics

Question 4

The PSD of a WSS process is the Fourier transform of its:

A. Mean

B. Autocorrelation

C. Probability density

D. Impulse response

Correct Answer: B. Autocorrelation

Question 5

For the AR(1) process X[n] = 0.8X[n−1] + E[n] with innovation variance 1, R[0] is approximately:

A. 0.36

B. 0.80

C. 1.64

D. 2.78

Correct Answer: D. 2.78

Question 6

For a stable LTI system with frequency response H and input PSD S_X, the output PSD is:

A. H S_X

B. |H| S_X²

C. |H|² S_X

D. S_X/H

Correct Answer: C. |H|² S_X

Question 7

Which estimator commonly reduces periodogram variance by averaging windowed overlapping segments?

A. Welch’s method

B. Zero padding alone

C. A single raw DFT

D. Mean subtraction alone

Correct Answer: A. Welch’s method

Question 8

In the Yule–Walker equations for an AR model, the innovation variance appears in the equation corresponding to:

A. Lag zero

B. Only lag p+1

C. Every positive lag equally

D. No lag

Correct Answer: A. Lag zero

Question 9

If f_s = 12.8 kHz and an FFT segment has 4096 samples, the frequency-bin spacing is:

A. 0.3125 Hz

B. 3.125 Hz

C. 31.25 Hz

D. 4096 Hz

Correct Answer: B. 3.125 Hz

Question 10

Which statement about WSS and ergodicity is correct?

A. Every WSS process is ergodic

B. Ergodicity concerns whether time averages represent ensemble averages

C. Ergodicity requires the process to be deterministic

D. WSS requires all samples to be independent

Correct Answer: B. Ergodicity concerns whether time averages represent ensemble averages