Probability, Statistics and Data Analytics

Introduction and Context

Artificial-intelligence systems rarely observe complete certainty. A medical model receives noisy measurements, a recommender system infers preferences from incomplete behaviour, and an autonomous system must act despite sensor error. Linear algebra provides the language for representing these uncertainties: vectors represent observations, matrices represent relationships among variables, and quadratic forms describe geometric distance and probability density.

This chapter develops covariance and correlation matrices, multivariate Gaussian distributions, whitening, principal component analysis (PCA), canonical correlation ideas, and dimensionality reduction. The emphasis is not only on computation but also on interpretation: what a matrix says about data, when a transformation is valid, and how uncertainty should influence decisions.

Learning Objectives

By the end of this chapter, you should be able to:

1. Calculate and interpret covariance and correlation matrices.

2. Explain the geometry and assumptions of the multivariate Gaussian model.

3. Use eigenvalues and eigenvectors to derive PCA.

4. Construct and interpret whitening transformations.

5. Distinguish PCA from canonical correlation analysis.

6. Evaluate dimensionality-reduction methods for exploratory analytics and AI.

7. Recognise numerical, ethical, and governance limitations in statistical modelling.

1. Random Vectors and Data Matrices

A random vector is written as

\[

X=(X_1,X_2,\ldots,X_p)^T,

\]

where each component is a random variable. Its mean vector is

\[

\mu=E[X],

\]

and its covariance matrix is

\[

\Sigma=E[(X-\mu)(X-\mu)^T].

\]

The entry \(\Sigma_{ij}=\operatorname{Cov}(X_i,X_j)\). Diagonal entries are variances and therefore cannot be negative. Covariance matrices are symmetric and positive semidefinite: for every vector \(a\),

\[

a^T\Sigma a=\operatorname{Var}(a^TX)\geq 0.

\]

In a data set with \(n\) observations and \(p\) measured variables, let \(Z\) be the \(n\times p\) data matrix. If \(\bar{x}\) is the vector of column means, the centred matrix is \(X_c=Z-\mathbf{1}\bar{x}^T\). The sample covariance matrix is

\[

S=\frac{1}{n-1}X_c^TX_c.

\]

The divisor \(n-1\) gives the usual unbiased estimator of population covariance under independent sampling. Dividing by \(n\) is also used in some machine-learning and maximum-likelihood conventions; the choice must be stated because it changes scale, although not eigenvectors.

Worked Example 1: Covariance and Correlation

Consider three observations of two variables:

\[

(1,2),\quad(2,4),\quad(3,3).

\]

The means are \(\bar{x}=2\) and \(\bar{y}=3\). Deviations are \((-1,-1),(0,1),(1,0)\). Thus

\[

S=\frac12\begin{bmatrix}2&1\\1&2\end{bmatrix}

=\begin{bmatrix}1&0.5\\0.5&1\end{bmatrix}.

\]

The sample standard deviations are both 1, so the correlation matrix is

\[

R=D^{-1}SD^{-1}=\begin{bmatrix}1&0.5\\0.5&1\end{bmatrix},

\]

where \(D=\operatorname{diag}(1,1)\). The positive off-diagonal value indicates that observations above the mean on one variable tend to be above the mean on the other. Correlation is dimensionless and lies between −1 and 1; covariance retains measurement units.

2. Geometry of Covariance

For any direction \(a\), the scalar projection \(a^TX\) has variance \(a^T\Sigma a\). Therefore, covariance defines a directional spread. In two dimensions, equal-density contours of a Gaussian distribution are ellipses. The eigenvectors of \(\Sigma\) give the ellipse axes, while eigenvalues give the variances along those axes.

If \(\Sigma=Q\Lambda Q^T\), with orthonormal eigenvectors in \(Q\) and non-negative eigenvalues in diagonal matrix \(\Lambda\), then the transformed coordinates \(Q^T(X-\mu)\) are uncorrelated and have variances equal to the eigenvalues. This is a rotation (and possibly reflection), not yet a rescaling.

The squared Mahalanobis distance is

\[

d_M^2(x)=(x-\mu)^T\Sigma^{-1}(x-\mu),

\]

when \(\Sigma\) is positive definite. It measures distance relative to the covariance geometry. If \(\Sigma\) is singular, a pseudoinverse or a reduced-dimensional representation is required. Mahalanobis distance is not automatically a probability; under a multivariate Gaussian, its squared value follows a chi-square distribution with \(p\) degrees of freedom.

3. The Multivariate Gaussian Model

A random vector \(X\in\mathbb{R}^p\) has a multivariate normal distribution with mean \(\mu\) and positive-definite covariance \(\Sigma\), written \(X\sim N_p(\mu,\Sigma)\), if its density is

\[

f(x)=\frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}}

\exp\left[-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu)\right].

\]

The determinant \(|\Sigma|\) controls the volume scale of the density, while the quadratic form controls elliptical distance from the mean. Positive definiteness is needed for an ordinary density. A positive-semidefinite covariance may describe a degenerate Gaussian concentrated on a lower-dimensional subspace.

Important properties include:

• Every linear transformation of a Gaussian vector is Gaussian.
• If two jointly Gaussian variables have zero covariance, they are independent. This implication is not true for arbitrary distributions.
• Marginal distributions are Gaussian.
• Conditional distributions are Gaussian when the relevant covariance block is nonsingular.

Partition \(X=(X_A^T,X_B^T)^T\) and partition the mean and covariance accordingly. Then

\[

X_A\mid X_B=b\sim N\left(\mu_A+\Sigma_{AB}\Sigma_{BB}^{-1}(b-\mu_B),

\Sigma_{AA}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{BA}\right).

\]

The conditional mean is a linear prediction, and the conditional covariance is the residual uncertainty. This result underlies Gaussian-process prediction, Kalman filtering, and many Bayesian models.

The Gaussian model is useful because it is mathematically tractable, but it is not universally appropriate. Heavy tails, skewness, multimodality, outliers, and nonlinear dependence can make Gaussian summaries misleading. A covariance matrix captures second-order dependence only; it does not prove causation.

4. Whitening

Whitening transforms centred data so that its covariance becomes the identity matrix. Let \(X_c=X-\mu\), and suppose

\[

\Sigma=Q\Lambda Q^T,

\]

with positive eigenvalues. The PCA or ZCA whitening transformation is

\[

W=\Lambda^{-1/2}Q^T X_c,

\]

for PCA whitening, or

\[

W_{ZCA}=Q\Lambda^{-1/2}Q^T X_c

\]

for ZCA whitening. Since

\[

\operatorname{Cov}(W)=\Lambda^{-1/2}Q^T\Sigma Q\Lambda^{-1/2}=I,

\]

whitened coordinates have unit variance and zero covariance.

PCA whitening rotates into principal coordinates before scaling. ZCA whitening additionally rotates back, producing whitened data that is often closest in squared Euclidean distance to the original centred data. Whitening is not unique: any orthogonal rotation of a whitened vector remains whitened. If an eigenvalue is zero or very small, exact inversion is unstable. A regularised form replaces \(\lambda_i^{-1/2}\) with \((\lambda_i+\varepsilon)^{-1/2}\), where \(\varepsilon>0\) is chosen and reported.

Whitening can improve numerical conditioning and help some learning algorithms, but it may amplify measurement noise and can remove meaningful scale information. It must be fitted on training data only; applying statistics estimated from the full data set can cause leakage.

5. Principal Component Analysis

PCA seeks orthogonal directions that successively maximise variance. Assume centred observations and covariance matrix \(S\). The first principal direction solves

\[

\max_{\|w\|=1}w^TSw.

\]

Using a Lagrange multiplier,

\[

\mathcal{L}(w,\lambda)=w^TSw-\lambda(w^Tw-1).

\]

Differentiating and setting the gradient to zero gives

\[

2Sw-2\lambda w=0\quad\Rightarrow\quad Sw=\lambda w.

\]

Thus stationary directions are eigenvectors. The maximum is the largest eigenvalue, so the first component is \(z_1=w_1^TX_c\), where \(w_1\) is the corresponding unit eigenvector. Subsequent components use orthogonal directions and descending eigenvalues.

If eigenvalues are \(\lambda_1\geq\cdots\geq\lambda_p\), the proportion of sample variance explained by the first \(k\) components is

\[

\frac{\sum_{i=1}^k\lambda_i}{\sum_{i=1}^p\lambda_i}.

\]

For a rank-\(k\) reconstruction, retain \(W_k\), the first \(k\) eigenvectors, and calculate

\[

\hat{X}_c=X_cW_kW_k^T.

\]

PCA gives the best rank-\(k\) linear least-squares reconstruction of centred data. It does not necessarily preserve class labels, rare events, fairness properties, or predictive performance. If variables have very different units, PCA on the covariance matrix may be dominated by large-scale variables. Standardising variables and using the correlation matrix is appropriate when relative scale is more meaningful, but it changes the scientific question.

Worked Example 2: Eigenvalues and Reduction

Suppose a covariance matrix has eigenvalues \(9,4,1,0.5\). Total variance is \(14.5\). The first two components explain

\[

\frac{9+4}{14.5}=\frac{13}{14.5}\approx0.8966,

\]

or approximately 89.66%. A two-dimensional representation therefore retains most, but not all, observed variance. The discarded variance is \(1.5/14.5\approx10.34\%\).

PCA can be computed from an eigendecomposition of \(S\), but singular value decomposition (SVD) is often numerically preferable. If \(X_c=U\Delta V^T\), the columns of \(V\) are principal directions and the covariance eigenvalues are \(\Delta_i^2/(n-1)\).

6. Canonical Correlation and Related Ideas

PCA analyses one variable set and finds directions of maximum internal variance. Canonical correlation analysis (CCA) analyses two sets, \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\), and seeks vectors \(a,b\) maximising

\[

\operatorname{Corr}(a^TX,b^TY).

\]

After centring and, commonly, regularising covariance estimates, the first canonical pair maximises cross-set correlation. Later pairs are constrained to be uncorrelated with earlier canonical variates within each set. CCA is useful for linking image features with text features, physiological measurements with behaviour, or student activity with achievement.

The classical solution assumes relevant covariance blocks are invertible. In high dimensions, sample covariance matrices may be singular; ridge regularisation or sparse CCA is then needed. Canonical correlations can also be inflated by flexible modelling and small samples, so validation and correction for multiple comparisons are important. Association between two projected variables remains association, not evidence that one causes the other.

7. Dimensionality Reduction for Exploratory Analytics

Dimensionality reduction may serve several purposes:

• visualisation of high-dimensional observations;
• compression and storage reduction;
• denoising and reconstruction;
• faster downstream modelling;
• detection of clusters, outliers, or latent structure;
• removal of collinearity in regression.

A sound workflow is:

1. Define the analytical objective and the unit of observation.

2. Inspect missingness, outliers, measurement units, and sampling design.

3. Split data into training, validation, and test sets when prediction is intended.

4. Fit centring, scaling, PCA, or whitening parameters on training data only.

5. Select dimensionality using explained variance, reconstruction error, predictive validation, stability, and domain interpretability—not a single universal threshold.

6. Examine loadings and transformed observations.

7. Validate conclusions under reasonable preprocessing and resampling choices.

PCA is linear and unsupervised. t-SNE and UMAP can produce useful visualisations of nonlinear neighbourhood structure, but distances between separated visual clusters and global geometry should not be interpreted naively. Autoencoders learn nonlinear representations but require architectural choices, regularisation, and careful validation. Reduction can hide minority subgroups or remove low-variance but decision-critical signals.

8. Applied Case: Predictive Maintenance (Illustrative)

Background

Consider an illustrative manufacturing plant monitoring vibration, temperature, pressure, motor current, and acoustic features from rotating equipment. Several sensors are strongly correlated because they respond to load and speed.

Problem

The analytics team wants a compact representation for exploratory dashboards and an early-warning classifier. A model trained directly on all variables is unstable because of multicollinearity, while sensor scales differ substantially.

Analysis

The team first separates machines by time into training and test periods to avoid future information entering the past. Missing values and sensor failures are documented rather than silently replaced. Features are robustly inspected for outliers. Training features are centred and standardised because units and ranges differ. PCA on the training correlation matrix shows that the first three components explain 82% of training variance. Loadings indicate that the first component mainly represents operating load, whereas the second contrasts vibration with temperature.

Approach and Outcome

The team uses the first three components for visual exploration, but also trains a classifier with and without PCA. Test performance, calibration, false-negative rate, and subgroup performance across machine types are compared. PCA improves computational efficiency and reduces coefficient instability, but a low-variance vibration feature is found to be important for a rare bearing failure. It is therefore retained in a separate monitored feature path rather than discarded solely because it contributes little total variance.

The final system reports a failure probability, uncertainty intervals from resampling, and an abstention rule for unfamiliar operating conditions. Engineers can inspect loadings and original sensor traces rather than receiving an opaque component label.

Learning Points

PCA is a tool for representation, not an automatic definition of operational risk. Temporal leakage, class imbalance, rare-event signals, calibration, and interpretability matter as much as variance explained. Decisions should combine statistical evidence with engineering knowledge and documented thresholds.

9. Limitations, Ethics, and Governance

Statistical preprocessing can encode unfairness. If a sensitive attribute is removed, correlated proxy variables may remain. A dimension-reduction model trained on historically unequal outcomes can preserve or obscure those inequalities. Analysts should assess performance and error rates across relevant groups, document features and transformations, and obtain appropriate consent and legal authority for sensitive data.

Privacy is also relevant: covariance structures can reveal group characteristics even when raw records are not displayed. Access controls, retention limits, reproducible versioning, and audit logs should accompany analytical pipelines. For high-impact decisions, explanations should connect model outputs to measurable variables and uncertainty, with human review and an appeal process where appropriate.

Other limitations include sampling bias, nonstationarity, measurement error, missing-not-at-random data, and instability of eigenvectors when eigenvalues are close. A component direction may change substantially under resampling even when explained variance changes little. Bootstrap stability analysis and sensitivity analysis are therefore valuable.

10. Best Practices and Contemporary Developments

Use SVD for large or numerically delicate data matrices, regularise covariance estimates in high dimensions, and report preprocessing choices. Compare covariance-based PCA with correlation-based PCA when units differ. Use robust covariance estimators when outliers are genuine contamination rather than the phenomenon of interest. For streaming data, incremental PCA can update components without retaining all observations, but drift monitoring is required.

Modern representation learning combines linear statistical foundations with neural methods, multimodal alignment, and self-supervised objectives. Nevertheless, covariance analysis remains valuable for diagnostics, whitening, monitoring, uncertainty quantification, and interpreting learned representations. Contemporary practice increasingly combines dimensionality reduction with conformal prediction, distribution-shift detection, and uncertainty-aware deployment rather than treating a low-dimensional embedding as inherently trustworthy.

Chapter Summary

Covariance matrices describe variance and pairwise linear dependence, while correlation matrices remove units through standardisation. Their eigenvectors define principal directions and their eigenvalues quantify directional variance. Multivariate Gaussian distributions provide an analytically useful model whose ellipsoidal geometry is governed by covariance; conditional Gaussian formulas support prediction and uncertainty estimation. Whitening decorrelates and rescales variables but can amplify noise and must be fitted without data leakage. PCA is derived by maximising projected variance and gives the best linear rank-constrained reconstruction, although high variance is not the same as predictive or social importance. CCA finds maximally correlated projections between two data sets. Effective analytics requires validation, stability checks, attention to units and sampling, and ethical governance. Dimensionality reduction should support—not replace—domain reasoning and uncertainty-aware decisions.

Key Terms

Random vector: A vector whose components are random variables.

Mean vector: The vector of expected or average values of random variables.

Covariance matrix: A symmetric matrix containing variances and pairwise covariances.

Correlation matrix: A covariance matrix after each variable is scaled to unit standard deviation.

Positive semidefinite: A matrix satisfying \(a^TMa\geq0\) for every vector \(a\).

Mahalanobis distance: A covariance-adjusted distance from a mean.

Multivariate Gaussian: A distribution defined by a mean vector and covariance matrix with elliptical density contours.

Eigenvector: A nonzero vector whose direction is preserved by a matrix transformation.

Eigenvalue: The scalar describing the scaling associated with an eigenvector.

Principal component: A projected coordinate along an eigenvector of the covariance or correlation matrix.

Explained variance: The proportion of total variance represented by selected components.

Whitening: A transformation producing identity covariance.

PCA whitening: Whitening in principal-coordinate order.

ZCA whitening: Whitening followed by rotation toward the original coordinate system.

Canonical correlation analysis: A method finding maximally correlated linear combinations of two variable sets.

Multicollinearity: Strong dependence among predictor variables.

Dimensionality reduction: Representing data with fewer coordinates.

Data leakage: Use of information unavailable at prediction time or from evaluation data during training.

Regularisation: Adding constraints or penalties to improve numerical stability or generalisation.

Concept drift: Change in the data-generating process over time.

Review Questions

Short Answer Questions

1. Why is a covariance matrix positive semidefinite?

2. Distinguish covariance from correlation.

3. What do PCA eigenvalues represent?

4. Why can whitening be numerically unstable?

5. State one important difference between PCA and CCA.

Long Answer Questions

1. Derive the PCA optimisation problem using a Lagrange multiplier and explain the role of eigenvalues.

2. Explain the geometry of a multivariate Gaussian using covariance ellipses and Mahalanobis distance.

3. Compare covariance-based PCA and correlation-based PCA, including when each is appropriate.

4. Explain the assumptions, uses, and limitations of the multivariate Gaussian conditional-distribution formula.

5. Design a leakage-resistant dimensionality-reduction workflow for a predictive AI application.

Analytical / Application Questions

1. A standardised data set has eigenvalues 6, 2, 1, and 1. What percentage of variance is explained by the first two components? Interpret the result.

2. A data scientist removes a feature because it contributes little to total PCA variance, but that feature detects a rare safety failure. Analyse the decision and propose a better procedure.

3. Two departments provide high-dimensional data sets describing the same customers. Recommend whether PCA, CCA, or another method should be used to study their relationships, and justify the validation and governance steps.

Multiple Choice Questions with Answers

Question 1

For the sample observations (1,2), (2,4), and (3,3), what is the sample covariance between the two variables?

A. 0.25

B. 0.50

C. 1.00

D. 2.00

Correct Answer: B. 0.50

Question 2

Which property must a valid covariance matrix satisfy?

A. It must have determinant exactly one

B. It must be skew-symmetric

C. It must be positive semidefinite

D. All off-diagonal entries must be positive

Correct Answer: C. It must be positive semidefinite

Question 3

For eigenvalues 9, 4, 1, and 0.5, what proportion of variance is explained by the first two principal components?

A. 0.8125

B. 0.8667

C. 0.8966

D. 0.9655

Correct Answer: C. 0.8966

Question 4

Under a multivariate Gaussian distribution, what does a covariance eigenvector represent geometrically?

A. An axis direction of an equal-density ellipsoid

B. A guaranteed causal direction

C. A probability value

D. A missing-data indicator

Correct Answer: A. An axis direction of an equal-density ellipsoid

Question 5

If X is centred and has covariance Sigma, which transformation produces identity covariance when Sigma=Q Lambda Q^T?

A. Q Lambda Q^T X

B. Lambda^(1/2) Q^T X

C. Lambda^(-1/2) Q^T X

D. Q^T Lambda X

Correct Answer: C. Lambda^(-1/2) Q^T X

Question 6

What is the principal objective of the first PCA direction?

A. Maximise projected variance subject to unit length

B. Minimise the number of observations

C. Maximise the number of original variables

D. Guarantee maximum classification accuracy

Correct Answer: A. Maximise projected variance subject to unit length

Question 7

Which method explicitly seeks correlated linear combinations of two variable sets?

A. PCA

B. CCA

C. ZCA whitening

D. Standardisation

Correct Answer: B. CCA

Question 8

Why should PCA preprocessing be fitted only on training data in a predictive workflow?

A. To increase the number of features

B. To prevent test-set information leakage

C. To force all eigenvalues to zero

D. To guarantee causal interpretation

Correct Answer: B. To prevent test-set information leakage

Question 9

If the total variance is 14.5 and retained variance is 13, what percentage is discarded?

A. 5.17%

B. 10.34%

C. 13.00%

D. 89.66%

Correct Answer: B. 10.34%

Question 10

Which statement about PCA is correct?

A. It always preserves class labels

B. It proves causal relationships

C. It gives the best linear rank-k least-squares reconstruction for centred data

D. It requires variables to be measured in identical units

Correct Answer: C. It gives the best linear rank-k least-squares reconstruction for centred data