Random Processes, Noise and Information Measures

Introduction and context

Communication and electronic systems operate in the presence of uncertainty. A transmitted waveform may be corrupted by thermal agitation, interference, fading, oscillator instability, quantisation, or unpredictable traffic. Engineering mathematics provides a unified language for describing such phenomena through probability, random processes, spectral analysis, and information theory.

A random process is a family of random variables indexed by time or another parameter. Noise is commonly modelled as a random process, while entropy and mutual information quantify uncertainty and the information conveyed by observations. Together, these ideas support receiver design, channel capacity analysis, coding, detection, estimation, radar, wireless communications, and signal processing.

Learning objectives

After studying this chapter, the reader should be able to:

1. distinguish random variables, random processes, stationary processes, and ergodic processes;
2. calculate means, autocorrelation functions, covariance, and power spectral densities;
3. explain white, thermal, shot, flicker, impulse, and quantisation noise;
4. analyse linear systems excited by random inputs;
5. use entropy, joint entropy, mutual information, and channel capacity;
6. evaluate signal-to-noise ratio and the effect of bandwidth and coding;
7. select suitable probabilistic and information-theoretic models for communication systems.

1. Random variables and random processes

A random variable assigns a numerical value to each outcome of an experiment. A random process is written as X(t), where, for every fixed time t, X(t) is a random variable. A single realisation, or sample function, is the waveform obtained when one experiment is observed over time.

The distinction is important:

• an ensemble contains all possible sample functions and their probabilities;
• a sample function is one observed waveform;
• a time average averages one waveform over time;
• an ensemble average averages corresponding values over all possible waveforms.

For a continuous random variable with probability density f_X(x),

and

The variance is

where μ_X = E[X] and σ_X² have units equal to the square of the units of X.

For a process, the mean function is

The autocorrelation function is

where * denotes complex conjugation. For a real process, X* = X. The autocovariance is

Thus,

At equal times, R_X(t,t) is the mean-square value, and C_X(t,t) is the variance.

Important properties of autocorrelation

For a wide-sense stationary process, the autocorrelation depends only on the time difference τ = t₂ − t₁:

For a real stationary process, R_X(τ) = R_X(−τ). More generally, autocorrelation is positive semidefinite. In particular, |R_X(τ)| ≤ R_X(0).

The cross-correlation between X(t) and Y(t) is

Cross-correlation is central to synchronisation, radar ranging, matched filtering, and system identification.

2. Stationarity and ergodicity

A process is strict-sense stationary (SSS) if every joint probability distribution is unchanged by a common time shift. For example, the distribution of (X(t₁),X(t₂)) is the same as that of (X(t₁+a),X(t₂+a)) for every shift a.

A process is wide-sense stationary (WSS) if:

1. m_X(t) = μ_X, a constant;
2. R_X(t₁,t₂) = R_X(t₂ − t₁).

WSS is sufficient for much linear-systems analysis. It requires only first- and second-order statistics, whereas SSS concerns all orders.

A process is ergodic in a statistic if its time average equals the corresponding ensemble average, under suitable limiting conditions. Mean ergodicity, for example, requires the long-time average

to equal E[X(t)] with probability one. Ergodicity permits engineers to estimate ensemble statistics from one sufficiently long record. Stationarity does not automatically imply ergodicity.

3. Power spectral density

For a WSS process, the power spectral density (PSD) is the Fourier transform of the autocorrelation function, known as the Wiener–Khinchin relationship:

The inverse relation is

Here f is frequency in hertz and S_X(f) has units of power per hertz. The average power is

For a real process, the PSD is non-negative and even. A one-sided PSD, defined only for non-negative frequencies, must be used consistently; for a real signal it is usually twice the two-sided PSD for f > 0.

Example 1: Exponentially correlated process

Suppose a zero-mean WSS process has

Using the Fourier transform pair for e^(−a|τ|),

At τ = 0, R_X(0) = σ². Integrating the PSD over all frequencies also gives σ², confirming power consistency. The spectrum is concentrated near zero and becomes smaller as frequency increases.

4. Random processes through linear systems

Let a stable linear time-invariant system have impulse response h(t) and frequency response H(f). If a WSS input X(t) produces output Y(t), then

For two inputs that are jointly WSS,

only when the notation and direction are defined consistently; the safest general method is to derive cross-spectra from the chosen input-output relation. For an input plus independent noise, output powers add because the cross terms have zero expectation.

If white noise has two-sided PSD N₀/2, the output noise power is

The equivalent noise bandwidth of a filter with peak power gain |H(f₀)|² is

For a filter whose peak gain is unity, P_n = (N₀/2)B_eq under the two-sided convention. Care is required because communications texts also use one-sided noise densities and one-sided bandwidths.

5. Noise in electronic and communication systems

Noise is an unwanted random component that limits detection and information transfer. Its statistical model should reflect the physical mechanism and the frequency range of interest.

Thermal noise

For a resistor at absolute temperature T, the open-circuit thermal-noise voltage PSD is

where k = 1.380649×10^(−23) J/K is Boltzmann’s constant and R is resistance in ohms. The corresponding available noise power from a matched resistor over bandwidth B is

This ideal white-noise model applies when quantum effects and frequency-dependent device behaviour can be neglected over the band.

Shot noise

Shot noise arises from discrete charge transport. For average direct current I, the two-sided current PSD is often written

in the one-sided convention, where q is the elementary charge. The factor convention must be stated because two-sided PSD descriptions distribute power over positive and negative frequencies.

Flicker and burst noise

Flicker noise, or 1/f noise, increases as frequency decreases and is important in semiconductor devices, sensors, and oscillators. A simplified model is S(f) ∝ 1/f^α, where α depends on the device. It is not white and cannot be characterised by a single constant noise density over a broad band.

Burst noise consists of sudden transitions between levels and may produce clustered errors. Impulse noise is often generated by switching, ignition, power-line transients, or electromagnetic events. These processes are non-Gaussian and may require impulsive or compound-Poisson models rather than a Gaussian approximation.

Gaussian noise and AWGN

A process is Gaussian if every finite collection of samples has a jointly Gaussian distribution. Additive white Gaussian noise (AWGN) assumes:

• additive disturbance;
• Gaussian amplitude statistics;
• flat PSD over the relevant band;
• independence from the transmitted signal in the basic model.

AWGN is analytically convenient and often a useful baseline, but it does not represent fading, interference, impulsive noise, nonlinear distortion, or coloured device noise.

6. Signal-to-noise ratio and noise figure

The signal-to-noise ratio is

where both powers are measured at the same point and over the same bandwidth. In decibels,

For voltage measurements across the same resistance, the equivalent expression is 20log₁₀(V_s/V_n).

For a receiver, the noise factor is

and noise figure is NF = 10log₁₀F dB. Noise factor is dimensionless and at least unity for a passive or physically realizable noisy system under the usual definition. For cascaded stages, Friis’ formula is

where F_i are linear noise factors and G_i are preceding-stage available power gains, also linear. Gains and noise factors must not be inserted in decibels.

7. Information measures

Information theory treats a message as a random variable. For a discrete variable X with probabilities p(x), Shannon entropy is

The convention 0log₂0 = 0 is used. Entropy is maximum for a uniform distribution over a fixed finite alphabet. For M equally likely symbols, H(X) = log₂M bits.

The joint entropy of X and Y is

Conditional entropy is

and satisfies

The mutual information is

It measures the reduction in uncertainty about one variable produced by observing the other. Mutual information is non-negative and equals zero precisely when the variables are statistically independent.

Example 2: Binary source

Let P(X=1)=0.8 and P(X=0)=0.2. Then

The source carries less than one bit per symbol because its outcomes are not equally likely. Compression can exploit this non-uniformity, although practical codes must account for integer code lengths, delay, and error requirements.

8. Communication channels and capacity

A discrete memoryless channel is described by transition probabilities p(y|x). Its mutual information is

Channel capacity is the maximum mutual information over all permitted input distributions:

For a band-limited AWGN channel with bandwidth B hertz, average signal power S watts, and noise power N watts in that bandwidth,

This is a theoretical upper limit for reliable communication under the stated channel model, coding assumptions, bandwidth, and power constraint. It does not specify a practical code or latency.

If noise is thermal with N = kTB, then

Increasing bandwidth can increase capacity, but the result depends on how total noise power and signal power scale. At low SNR, log₂(1+S/N) is approximately (S/N)/ln2, while at high SNR capacity grows logarithmically with SNR.

9. Worked application: an IoT receiver link

Consider an illustrative low-power sensor transmitting through a channel of bandwidth B = 200 kHz at T = 290 K. The received signal power is S = −90 dBm. Find the thermal noise power, SNR, and Shannon capacity, assuming an ideal matched receiver.

First convert signal power:

Thermal noise power is

In dBm, this is approximately −120.96 dBm. Therefore,

or about 30.96 dB. The ideal capacity is

This result is an upper bound. Receiver noise figure, fading margin, coding overhead, protocol overhead, interference, peak-power limits, and implementation losses reduce usable throughput. If the receiver noise factor is F, the effective thermal noise contribution becomes approximately FN when referred to the receiver input, subject to the adopted reference and bandwidth conventions.

10. Case study: noise-aware wearable telemetry

Background

An illustrative medical-wearable system sends ECG-derived features over a low-power wireless link. The sensor operates near the human body, where motion, electrode contact variation, mains coupling, and radio interference are significant.

Problem

The first prototype used a wide analogue bandwidth and a simple threshold detector. Motion produced large low-frequency disturbances, while radio interference caused occasional bursts. Average SNR appeared acceptable, but packet errors were highly clustered.

Analysis

A power-spectrum estimate showed strong low-frequency coloured noise rather than white noise. Time-domain records showed non-Gaussian bursts. Consequently, an AWGN-only link budget underestimated error probability. The engineering team separated the problem into analogue noise, interference, channel fading, and digital implementation noise.

Approach and solution

The revised design:

1. limited the analogue bandwidth to the medically relevant signal band;
2. added a high-pass stage to reduce baseline wander while preserving diagnostic content;
3. used differential sensing and shielding to reduce common-mode interference;
4. included automatic gain control with saturation detection;
5. applied interleaving and forward-error correction so burst errors were spread across codewords;
6. estimated channel quality from pilot symbols rather than assuming constant SNR;
7. monitored residual noise and alerted the operator when electrode contact degraded.

Outcome

The design reduced false alarms and improved useful packet delivery without simply increasing transmit power. The key improvement came from matching the stochastic model and mitigation method to the observed noise: filtering for coloured noise, shielding for interference, and coding/interleaving for bursts.

Learning points

A single SNR number is insufficient when noise is nonstationary or impulsive. Measurement bandwidth, estimator bias, temporal correlation, and error distribution should be reported. In safety-related systems, a communication-quality indicator must not be interpreted as a clinical diagnosis without appropriate validation and governance.

11. Limitations, research directions, and best practice

Random-process models are abstractions. WSS assumptions may fail during transients; Gaussian assumptions may hide rare but damaging events; ergodic estimates may be unreliable for short records; and PSD estimates involve windowing and finite-sample variance. Researchers should state sampling rate, observation interval, estimator, window, confidence interval, and PSD convention.

Contemporary systems increasingly use learned receivers, intelligent reflecting surfaces, massive multiple-input multiple-output links, terahertz channels, and machine-learning-based anomaly detection. Information theory remains useful for defining limits, while data-driven models can capture non-Gaussian and nonstationary behaviour. However, learned models require representative data, validation under distribution shift, explainable failure analysis, and protection against adversarial or privacy-threatening inference.

Best practice is to:

• define the random quantity and its units;
• distinguish one-sided from two-sided spectra;
• state whether powers are average, peak, or instantaneous;
• test stationarity before applying stationary estimators;
• compare model predictions with measured residuals;
• retain uncertainty estimates and calibration records;
• use conservative margins in safety- or reliability-critical systems;
• protect communication data and disclose monitoring purposes.

Chapter Summary

Random processes model uncertain time-varying signals. Their mean, autocorrelation, covariance, and power spectral density provide complementary descriptions. WSS simplifies analysis by requiring constant mean and time-difference-dependent autocorrelation; ergodicity determines whether time records can represent ensemble statistics. Linear systems transform PSD according to S_Y(f)=|H(f)|²S_X(f). Thermal, shot, flicker, burst, impulse, and quantisation noise have different physical origins and statistical properties, so AWGN is a useful baseline rather than a universal truth.

Entropy quantifies uncertainty, conditional entropy quantifies residual uncertainty, and mutual information measures information shared by source and observation. Channel capacity is the maximum reliable information rate allowed by a channel model. The AWGN capacity formula connects bandwidth and SNR, but practical performance is limited by coding, fading, interference, hardware, latency, and governance requirements. Sound engineering therefore combines mathematical modelling, measurement, uncertainty analysis, and responsible system design.

Key Terms

Random process: A family of random variables indexed by time or another parameter.

Sample function: One waveform realisation of a random process.

Mean function: Expected process value as a function of time.

Autocorrelation: Expected product of process values at two time instants.

Autocovariance: Autocorrelation after removal of mean components.

WSS process: Process with constant mean and time-difference-dependent autocorrelation.

Strict-sense stationarity: Invariance of all joint distributions under time shifts.

Ergodicity: Property allowing suitable time averages to equal ensemble averages.

Power spectral density: Distribution of average signal power with frequency.

White noise: Ideal noise with constant PSD over the modelled band.

AWGN: Additive white Gaussian noise model.

Equivalent noise bandwidth: Noise-passing bandwidth based on integrated squared filter gain.

Signal-to-noise ratio: Ratio of signal power to noise power.

Noise factor: Input SNR divided by output SNR.

Entropy: Average uncertainty of a random variable, measured here in bits.

Conditional entropy: Uncertainty remaining in one variable given another.

Mutual information: Reduction in uncertainty produced by an observation.

Channel capacity: Maximum reliable information rate under stated channel constraints.

Wiener–Khinchin relationship: Fourier-transform relationship between autocorrelation and PSD.

Flicker noise: Low-frequency noise with approximate inverse-frequency spectral behaviour.

Review Questions

Short Answer Questions

1. Define a random process and distinguish a sample function from an ensemble.
2. State the two conditions for wide-sense stationarity.
3. What does the PSD represent, and how is total average power obtained from it?
4. Define entropy and mutual information.
5. Why must one-sided and two-sided noise PSD conventions not be mixed?

Long Answer Questions

1. Derive the relationship between the input and output PSD of a stable LTI system.
2. Compare thermal, shot, flicker, burst, and impulse noise in terms of origin and modelling.
3. Explain stationarity and ergodicity, including why one does not imply the other.
4. Discuss Shannon capacity for the AWGN channel and the assumptions behind the formula.
5. Explain noise figure and derive the significance of Friis’ cascaded noise-factor formula.

Analytical / Application Questions

1. A zero-mean process has R_X(τ)=σ²e^(−a|τ|). Derive its PSD and verify that the integral of the PSD equals σ².
2. A receiver has bandwidth 100 kHz, temperature 300 K, and received signal power −100 dBm. Calculate thermal noise power and ideal AWGN capacity, clearly stating the noise convention.
3. A measured telemetry channel has clustered errors and a strongly frequency-dependent PSD. Explain why an AWGN-only model is inadequate and propose a measurement and mitigation plan.

Multiple Choice Questions with Answers

Question 1

Which condition is required for a wide-sense stationary process?

A. Its probability density must be uniform

B. Its mean must be constant and autocorrelation must depend only on time difference

C. Every sample must be statistically independent

D. Its power spectral density must be zero

Correct Answer: B. Its mean must be constant and autocorrelation must depend only on time difference

Question 2

For a WSS process, the power spectral density is the Fourier transform of the:

A. Mean function

B. Probability density

C. Autocorrelation function

D. Cumulative distribution function

Correct Answer: C. Autocorrelation function

Question 3

The average power of a WSS process is equal to:

A. The derivative of its PSD at zero frequency

B. The integral of its PSD over frequency

C. Its mean divided by bandwidth

D. Its autocorrelation at infinite lag

Correct Answer: B. The integral of its PSD over frequency

Question 4

Thermal noise power in bandwidth B at temperature T is:

A. kT/B

B. kTB

C. B/(kT)

D. qTB

Correct Answer: B. kTB

Question 5

If X and Y are statistically independent, their mutual information is:

A. −1 bit

B. Equal to their joint entropy

C. Zero

D. Infinite

Correct Answer: C. Zero

Question 6

The entropy of a uniformly distributed binary source is:

A. 0 bits

B. 0.5 bits

C. 1 bit

D. 2 bits

Correct Answer: C. 1 bit

Question 7

For an AWGN channel, Shannon capacity is:

A. B(S/N)

B. B log₂(1 + S/N)

C. (S/N) log₂B

D. S + N

Correct Answer: B. B log₂(1 + S/N)

Question 8

A noise figure of 3 dB corresponds to a linear noise factor closest to:

A. 0.5

B. 1

C. 2

D. 3

Correct Answer: C. 2

Question 9

For an LTI system with frequency response H(f), the output PSD is:

A. H(f)S_X(f)

B. |H(f)|²S_X(f)

C. S_X(f)/|H(f)|²

D. H(f)+S_X(f)

Correct Answer: B. |H(f)|²S_X(f)

Question 10

A binary source has probabilities 0.8 and 0.2. Its entropy is approximately:

A. 0.200 bits

B. 0.500 bits

C. 0.722 bits

D. 1.600 bits

Correct Answer: C. 0.722 bits