3. Random Signals and Statistical Signal Models

Introduction and Context

Many signals encountered in engineering cannot be described exactly by a deterministic formula. Thermal noise in a receiver, speech disturbances, vibration from rotating machinery, channel fading, biomedical measurements, and packet-arrival fluctuations all contain uncertainty. Nevertheless, uncertainty does not imply complete lack of structure. Repeated observations often reveal regularities that can be described statistically.

A random signal, or stochastic process, is therefore modeled not by predicting every sample exactly, but by specifying probability distributions, averages, correlations, spectral content, and dynamic relationships. In advanced digital signal processing, such models support filter design, prediction, detection, estimation, simulation, system identification, and performance analysis.

This chapter develops the principal statistical tools used to analyze random signals. It then connects them with Gaussian processes, autoregressive and moving-average models, state-space descriptions, and model-validation procedures. The central engineering question is not merely whether a model fits measured data, but whether it captures the properties relevant to the intended decision or system.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Distinguish deterministic signals, random variables, and random processes.

2. Define and interpret mean, autocorrelation, autocovariance, cross-correlation, and cross-covariance.

3. Explain strict-sense stationarity, wide-sense stationarity, ergodicity, and their practical implications.

4. Derive and interpret power spectral density using correlation functions and Fourier transforms.

5. Describe Gaussian random processes and explain why second-order statistics are especially useful for them.

6. Formulate and analyze AR, MA, and ARMA models, including stability and causality conditions.

7. Represent stochastic dynamics in state-space form and relate them to filtering and prediction.

8. Select, estimate, validate, and responsibly apply statistical signal models.

From Uncertainty to a Mathematical Model

A random variable assigns a numerical outcome to an experiment. A random process is a collection of random variables indexed by time. For a discrete-time process, the notation \(X[n]\) denotes a random variable at each integer time index. A particular sequence produced by one experiment is called a realization, sample function, or sample path, denoted by \(x[n]\).

This distinction is important. The process is an ensemble of possible sequences; a recorded waveform is usually only one realization. Statistical analysis attempts to infer ensemble properties from finite observations. This inference is never exact and depends on assumptions about stationarity, measurement quality, and the adequacy of the model.

The joint probability density of \(X[n_1],\ldots,X[n_k]\) describes the process completely in principle. In practice, such a description is usually too demanding. Signal processing therefore often begins with first- and second-order statistics, which are computationally tractable and sufficient for important classes of processes.

Mean and variance

The ensemble mean is

\[

m_X[n]=E\{X[n]\}.

\]

The variance is

\[

\sigma_X^2[n]=E\{(X[n]-m_X[n])^2\}.

\]

The mean represents the average level, while variance measures average fluctuation around that level. For complex signals, variance is defined using the magnitude squared, \(E\{|X[n]-m_X[n]|^2\}\).

Correlation and covariance

The autocorrelation function is

\[

R_X[n_1,n_2]=E\{X[n_1]X^*[n_2]\},

\]

where the conjugate is required for complex-valued signals. It measures statistical similarity between samples at two time indices. The autocovariance is

\[

C_X[n_1,n_2]=E\{[X[n_1]-m_X[n_1]][X[n_2]-m_X[n_2]]^*\}.

\]

Thus,

\[

C_X[n_1,n_2]=R_X[n_1,n_2]-m_X[n_1]m_X^*[n_2].

\]

For a second process \(Y[n]\), the cross-correlation is

\[

R_{XY}[n_1,n_2]=E\{X[n_1]Y^*[n_2]\}.

\]

Cross-statistics are central to adaptive filtering, beamforming, system identification, and sensor fusion because they reveal dependence between signals.

Stationarity and Ergodicity

A process is strictly stationary if every finite-dimensional joint distribution remains unchanged under a common time shift. For example, the joint distribution of \(X[n_1],X[n_2]\) is the same as that of \(X[n_1+k],X[n_2+k]\) for every integer \(k\). Strict stationarity is a powerful but demanding property because it concerns all orders of statistical dependence.

A process is wide-sense stationary (WSS) if its mean is constant and its autocorrelation depends only on the time difference:

\[

m_X[n]=m_X,

\qquad R_X[n_1,n_2]=R_X[n_1-n_2]=R_X[k].

\]

The autocovariance likewise becomes a function of lag. WSS is often sufficient for linear systems and spectral analysis. It is weaker than strict stationarity: a WSS process need not have time-invariant higher-order distributions.

For a WSS process, the autocorrelation has important properties. It is Hermitian, \(R_X[k]=R_X^*[-k]\), and positive semidefinite. In particular, \(|R_X[k]|\leq R_X[0]\), where \(R_X[0]\) is the average signal power. If the mean is nonzero, the correlation may not decay to zero; a constant component contributes \(|m_X|^2\).

Ergodicity concerns whether time averages from one sufficiently long realization equal ensemble averages. Mean ergodicity, for example, implies that

\[

\lim_{N\to\infty}\frac{1}{N}\sum_{n=0}^{N-1}x[n]=E\{X[n]\}

\]

under suitable conditions. Correlation ergodicity extends the idea to correlation estimates. Stationarity does not automatically imply ergodicity. A process can be statistically time-invariant while different realizations retain different fixed characteristics. Engineering data analysis commonly assumes ergodicity because only one record is available, but this assumption should be checked rather than silently accepted.

Power Spectral Density

For a WSS discrete-time process, the power spectral density (PSD) is the discrete-time Fourier transform of the autocorrelation sequence:

\[

S_X(e^{j\omega})=\sum_{k=-\infty}^{\infty}R_X[k]e^{-j\omega k}.

\]

The inverse relation is

\[

R_X[k]=\frac{1}{2\pi}\int_{-\pi}^{\pi}S_X(e^{j\omega})e^{j\omega k}\,d\omega.

\]

The PSD describes how average power is distributed over frequency. It is nonnegative and periodic with period \(2\pi\). The total average power is

\[

R_X[0]=\frac{1}{2\pi}\int_{-\pi}^{\pi}S_X(e^{j\omega})d\omega.

\]

A white-noise process has constant PSD. Ideal discrete-time white noise has zero correlation at all nonzero lags, although physical systems always impose bandwidth and quantization constraints. A low-frequency-concentrated PSD indicates slowly varying behavior, whereas a broad PSD indicates rapid fluctuations.

If a WSS process passes through a stable LTI filter with frequency response \(H(e^{j\omega})\), the output PSD is

\[

S_Y(e^{j\omega})=|H(e^{j\omega})|^2S_X(e^{j\omega}).

\]

For jointly WSS input processes, cross-power spectra provide frequency-domain measures of dependence. These relationships form the basis of noise shaping, Wiener filtering, spectral estimation, and communication-system analysis.

In practice, the PSD must be estimated from finite data. The periodogram is simple but has substantial variance. Windowing reduces leakage but changes resolution. Averaging adjacent or segmented periodograms, as in Welch-type methods, reduces variance at the cost of frequency resolution. Parametric estimates based on AR models can provide high resolution when the model is appropriate, but may be misleading when it is not.

Gaussian Random Processes

A process is Gaussian when every finite collection of samples has a jointly Gaussian distribution. Gaussian models are important because they are mathematically tractable and often arise as approximations to the aggregate effect of many small, weakly dependent disturbances.

For a jointly Gaussian process, the mean vector and covariance matrix completely determine the joint distribution. Consequently, for Gaussian processes, uncorrelated samples are also statistically independent. This equivalence does not hold generally; nonlinear dependence may remain even when correlation is zero.

A zero-mean white Gaussian noise process with variance \(\sigma^2\) has independent samples and PSD \(S_X(e^{j\omega})=\sigma^2\). Gaussian assumptions simplify likelihood calculations, Kalman filtering, detection, and minimum mean-square-error estimation. They can nevertheless be inappropriate for impulsive interference, heavy-tailed sensor noise, clipped measurements, and signals containing pronounced nonlinear structure.

AR, MA, and ARMA Models

Statistical models frequently describe a signal as the output of a linear filter driven by white noise. The autoregressive model of order \(p\), AR(\(p\)), is

\[

X[n]=c+\sum_{i=1}^{p}a_iX[n-i]+W[n],

\]

where \(W[n]\) is white innovation noise. The model explains the present sample using past samples plus an unpredictable innovation. For a zero-mean AR(1) process, \(X[n]=aX[n-1]+W[n]\). It is stable and stationary when \(|a|<1\), and its correlation decays geometrically.

A moving-average model of order \(q\), MA(\(q\)), is

\[

X[n]=\mu+\sum_{j=0}^{q}b_jW[n-j].

\]

An MA process has finite memory in terms of innovations. Its autocorrelation becomes zero beyond lag \(q\), although the samples themselves can remain dependent over more complicated statistical relationships.

The ARMA(\(p,q\)) model combines both mechanisms:

\[

A(z)X[n]=c+B(z)W[n],

\]

where

\[

A(z)=1-a_1z^{-1}-\cdots-a_pz^{-p},

\quad B(z)=b_0+b_1z^{-1}+\cdots+b_qz^{-q}.

\]

The transfer function from innovation to process is \(H(z)=B(z)/A(z)\). Causality and stationary realizability require the poles of the causal model to lie inside the unit circle. An invertible MA representation requires its zeros to lie inside the unit circle. These conditions matter because different parameter sets can produce equivalent or unstable descriptions.

The PSD of a zero-mean ARMA process driven by white noise of variance \(\sigma_W^2\) is

\[

S_X(e^{j\omega})=\sigma_W^2\frac{|B(e^{j\omega})|^2}{|A(e^{j\omega})|^2}.

\]

AR models are attractive because parameters can be estimated efficiently through Yule–Walker equations, least squares, or maximum likelihood. The model order must be selected carefully. Too small an order leaves structure in the residuals; too large an order increases variance, parameter instability, and overfitting. Information criteria such as AIC and BIC provide useful, though not infallible, order-selection guidance.

State-Space Representation

Many stochastic systems are naturally expressed through hidden states:

\[

\mathbf{x}_{n+1}=F\mathbf{x}_n+G\mathbf{w}_n,

\]

\[

\mathbf{y}_n=H\mathbf{x}_n+\mathbf{v}_n.

\]

Here \(\mathbf{x}_n\) is the state, \(\mathbf{y}_n\) is the observation, and \(\mathbf{w}_n\) and \(\mathbf{v}_n\) are process and measurement noise. Their covariance matrices describe uncertainty entering the dynamics and observation channels.

State-space models separate physical evolution from measurement. They accommodate multivariable systems, missing observations, time-varying parameters, and nonstationary behavior more naturally than a single ARMA equation. Under linear Gaussian assumptions, the Kalman filter recursively computes the minimum mean-square estimate of the state. The prediction step propagates the previous estimate through the state equation; the update step combines the prediction with the new measurement according to relative uncertainty.

AR models can be written in state-space form by defining a state vector containing present and past samples. Conversely, state-space models can produce ARMA-like output relationships. This connection allows frequency-domain statistical descriptions and recursive time-domain estimators to be used together.

Estimation and Model Validation

A practical modeling workflow begins with the engineering purpose. Detection, forecasting, compression, control, and fault diagnosis may require different models even for the same data. The analyst should then inspect sampling quality, missing values, outliers, trends, and possible changes in operating conditions.

A useful sequence is:

1. Remove or model deterministic components such as offsets, trends, and known periodic interference.

2. Examine sample mean, variance, autocorrelation, and an initial PSD estimate.

3. Test whether stationarity is plausible over the intended interval.

4. Fit candidate models using estimation methods suited to the data.

5. Select order using physical reasoning, validation error, and information criteria.

6. Analyze residuals and evaluate performance on held-out data.

Residuals should resemble the assumed innovations. For an adequate model, residual autocorrelation should be insignificant except at lag zero, residual variance should be approximately stable, and residual distributions should be compatible with the chosen noise assumption. A whiteness test, such as a portmanteau test, can help identify remaining temporal structure. Normal probability plots or robust distributional checks can reveal departures from Gaussianity.

Validation must be performed on data not used to estimate parameters whenever prediction performance matters. A model with excellent in-sample likelihood may fail out of sample because it has learned noise, leakage, or a temporary operating condition. Cross-validation for dependent data must preserve temporal ordering; random shuffling can create misleading results.

Applied Case: Vibration-Based Monitoring of a Rotating Machine

Background

Consider an illustrative condition-monitoring system for an industrial motor. An accelerometer samples vibration at a fixed rate. The objective is to detect an early bearing defect before a costly failure occurs.

Situation and analysis

The raw acceleration contains a deterministic component at the shaft rotation frequency, broadband sensor noise, structural resonances, and occasional impulsive events. A simple threshold on instantaneous amplitude produces many false alarms because normal operation also generates large transients.

Engineers first remove the estimated rotational component and divide the record into operating-condition intervals. Within a relatively constant-speed interval, the residual appears approximately WSS. Its sample autocorrelation shows decaying oscillations, suggesting a resonant dynamic rather than white noise. The PSD contains a narrow peak near a known bearing-related frequency and a broader resonance band.

Modeling approach

An AR model is fitted to normal-condition records. Its poles summarize the usual resonant structure, while the innovation variance measures unexplained excitation. Candidate orders are compared using BIC and residual whiteness. A low-order model leaves correlated residuals; a very high-order model fits individual noise fluctuations. A moderate order provides a stable compromise.

During operation, the current record is filtered through the normal model and the innovation sequence is monitored. A sustained increase in innovation variance, together with growth of energy in the bearing-related band, becomes an alarm feature. For changing speed, a state-space model with slowly varying parameters is introduced, and the Kalman filter provides a recursively updated estimate.

Outcome and learning points

The statistical model does not claim to predict every vibration sample. Instead, it defines normal dynamic behavior and identifies statistically unusual departures. The case illustrates four principles: preprocessing must respect physical structure; stationarity may hold only locally; residual analysis is essential; and model outputs should be combined with domain knowledge rather than treated as unquestionable diagnoses.

Applications and Professional Use

Random-signal models occur throughout communications and electronics. In wireless systems, fading and interference statistics guide receiver design, channel estimation, diversity combining, and link budgeting. In radar and sonar, covariance models support detection in colored noise and adaptive beamforming. In image and video processing, spatial random fields describe texture, sensor noise, and compression artifacts. In biomedical instrumentation, stochastic models help separate physiological activity from motion and electronic interference.

In organizations, model selection also has operational consequences. A false alarm in an industrial safety system may interrupt production, whereas a missed alarm may threaten personnel or equipment. Consequently, statistical performance must be translated into costs, risk thresholds, maintenance policies, and traceable decision rules. Model documentation should state the data interval, assumptions, parameter-estimation method, validation results, and known failure conditions.

Modern analytics platforms automate PSD estimation, ARIMA-style fitting, Kalman filtering, and anomaly scoring. Machine-learning models can capture nonlinear or non-Gaussian behavior, but they do not eliminate the need for statistical validation. Hybrid systems often combine a physical state-space model with a learned residual model. Such systems can improve flexibility while preserving interpretability and enforcing physical constraints.

Limitations, Risks, and Professional Considerations

The most common error is treating a finite data record as if it were the ensemble itself. Estimates are uncertain, especially for short records. Nonstationarity can cause an average PSD to conceal important transitions. Aliasing, sensor saturation, quantization, synchronization errors, and windowing artifacts can create apparent statistical features that are not present in the underlying process.

Correlation is not causation. A strong cross-correlation may result from a common unobserved source, shared timing, or filtering. Likewise, Gaussianity should not be assumed merely because it makes calculations convenient. Heavy-tailed or impulsive disturbances can make variance-based criteria unreliable.

Ethically and professionally, monitoring systems should account for the consequences of incorrect decisions. Data from people or workplaces may require privacy protection, access control, retention limits, and transparent explanation of automated alerts. Safety-critical models require verification, version control, calibration, and human oversight. Claims about performance should be based on representative test data rather than selected demonstrations.

Current Developments and Best Practices

Emerging work increasingly addresses nonstationary, nonlinear, and high-dimensional processes. Time-frequency methods, switching state-space models, particle filters, sparse Bayesian models, and neural sequence models extend the classical framework. Deep models can learn complex dependencies, but uncertainty quantification, distribution shift, interpretability, and computational cost remain significant concerns.

A sound practice is to begin with the simplest model that satisfies the engineering purpose. Use physical knowledge to define variables and operating regimes, estimate uncertainty rather than reporting only point parameters, validate on future-like data, monitor residuals after deployment, and recalibrate when the data-generating process changes. Classical correlation and spectral tools remain valuable even when advanced learning methods are used, because they provide diagnostics and interpretable baselines.

Chapter Summary

Random signals are modeled statistically because exact sample-by-sample prediction is generally impossible, while ensemble regularities remain measurable. Mean, variance, correlation, covariance, and cross-statistics describe first- and second-order behavior. Strict stationarity concerns all finite-dimensional distributions; wide-sense stationarity requires constant mean and lag-dependent correlation. Ergodicity justifies estimating ensemble properties from a single long record, but it must not be assumed without consideration.

The PSD is the Fourier transform of autocorrelation and describes average power distribution over frequency. Gaussian processes are completely characterized by mean and covariance, making them especially useful in estimation and filtering. AR, MA, and ARMA models represent random signals through relationships between samples and innovations, while state-space models represent hidden dynamics and observations. Model validation requires residual analysis, temporal hold-out testing, order selection, and awareness of nonstationarity and non-Gaussian behavior. Practical systems should combine statistical evidence with physical reasoning, risk assessment, and responsible governance.

Key Terms

Random process: A time-indexed collection of random variables.

Realization: One sample sequence produced by a random process.

Ensemble mean: The expected value of a process at a specified time.

Variance: Mean squared deviation from the process mean.

Autocorrelation: Expected product of samples separated by specified time indices.

Autocovariance: Correlation of deviations from the mean.

Cross-correlation: Statistical similarity between two processes at selected lags.

Strict-sense stationarity: Invariance of all finite-dimensional distributions under time shifts.

Wide-sense stationarity: Constant mean and autocorrelation dependent only on lag.

Ergodicity: Property allowing ensemble statistics to be obtained from time averages of one realization.

Power spectral density: Frequency-domain distribution of average signal power.

White noise: A process with zero correlation at nonzero lags and a flat PSD under the ideal model.

Gaussian process: A process whose every finite sample vector has a joint Gaussian distribution.

Innovation: New unpredictable disturbance entering a predictive model.

AR model: A model expressing the present sample as a combination of past samples and innovation.

MA model: A model expressing the output as a finite weighted sum of current and past innovations.

ARMA model: A model combining autoregressive and moving-average components.

State-space model: A dynamic model using hidden states, process noise, and observations.

Residual: The difference between an observed value and its model-based prediction.

Model validation: Evaluation of assumptions and predictive performance using diagnostic and independent data.

Review Questions

Short Answer Questions

1. Distinguish a random process from one of its realizations.

2. State the conditions for a discrete-time process to be wide-sense stationary.

3. What physical meaning does the value \(R_X[0]\) have for a zero-mean process?

4. Why does a Gaussian process receive special treatment in statistical signal processing?

5. What is the purpose of examining residual autocorrelation after fitting an ARMA model?

Long Answer Questions

1. Explain strict stationarity, wide-sense stationarity, and ergodicity, and discuss why they should not be treated as equivalent.

2. Derive the relationship between autocorrelation and power spectral density and explain the significance of PSD area.

3. Compare AR, MA, and ARMA models in terms of memory, parameters, stability, and applications.

4. Explain the structure and operation of a linear Gaussian state-space model and describe the role of the Kalman filter.

5. Develop a complete workflow for estimating and validating a statistical signal model from a finite record.

Analytical / Application Questions

1. A measured process has a nonzero sample mean and a slowly decaying autocorrelation. Explain how you would determine whether the behavior is caused by a constant component, a low-frequency dynamic, or nonstationarity.

2. An AR model produces low training error, but its residuals show significant correlation at lags 1 through 4. What does this indicate, and what actions should be considered?

3. Design a statistical monitoring strategy for a sensor whose noise changes when the operating temperature changes. Specify how stationarity, model selection, validation, and alarm decisions should be handled.

22. Multiple Choice Questions with Answers

Question 1

Which condition is required for a discrete-time process to be wide-sense stationary?

A. All samples must be statistically independent

B. The mean is constant and autocorrelation depends only on lag

C. The process must have a Gaussian distribution

D. Every realization must be identical

Correct Answer: B. The mean is constant and autocorrelation depends only on lag

Question 2

For a zero-mean WSS process, what does R_X[0] represent?

A. The signal bandwidth

B. The correlation at infinite lag

C. The average signal power

D. The innovation mean

Correct Answer: C. The average signal power

Question 3

Which statement correctly relates PSD and autocorrelation?

A. PSD is the time derivative of autocorrelation

B. PSD is the Fourier transform of the autocorrelation sequence

C. PSD is always equal to the variance

D. PSD exists only for deterministic signals

Correct Answer: B. PSD is the Fourier transform of the autocorrelation sequence

Question 4

Why are second-order statistics sufficient to characterize a jointly Gaussian process?

A. Gaussian processes have no temporal dependence

B. Their mean and covariance determine the joint distribution

C. Their PSD is always flat

D. All Gaussian processes are stationary

Correct Answer: B. Their mean and covariance determine the joint distribution

Question 5

For a causal AR model to represent a stable stationary process, its poles must generally be located:

A. Outside the unit circle

B. On the imaginary axis

C. Inside the unit circle

D. At the origin only

Correct Answer: C. Inside the unit circle

Question 6

A fitted model has strongly correlated residuals. What is the most appropriate interpretation?

A. The model has certainly overestimated the noise variance

B. The residuals may contain dynamics not captured by the model

C. The original signal must be deterministic

D. The data must be Gaussian

Correct Answer: B. The residuals may contain dynamics not captured by the model

Question 7

What is the principal purpose of the innovation sequence in an AR model?

A. To represent the unpredictable component remaining after prediction

B. To remove the sampling clock

C. To define the Fourier-transform convention

D. To guarantee nonstationarity

Correct Answer: A. To represent the unpredictable component remaining after prediction

Question 8

Which advantage is especially associated with a state-space model?

A. It requires no assumptions about noise

B. It can represent hidden dynamics, observations, and time variation

C. It always has fewer parameters than an MA model

D. It eliminates measurement error

Correct Answer: B. It can represent hidden dynamics, observations, and time variation

Question 9

Why can random shuffling be inappropriate when validating a time-series model?

A. It changes the amplitude units

B. It destroys temporal ordering and can leak future information

C. It makes the process Gaussian

D. It forces the PSD to be flat

Correct Answer: B. It destroys temporal ordering and can leak future information

Question 10

An analyst observes zero cross-correlation between two non-Gaussian signals. What conclusion is justified?

A. The signals are necessarily independent

B. The signals are necessarily causally unrelated

C. They are uncorrelated, but nonlinear dependence may still exist

D. Both signals must be white noise

Correct Answer: C. They are uncorrelated, but nonlinear dependence may still exist