3. Mathematical Foundations for Data Science

Introduction and Context

Data science is often described as the use of data, computation, and statistical reasoning to support explanation, prediction, and decision-making. Beneath these activities lies a mathematical language. Vectors represent observations and parameters; matrices organise data and transformations; derivatives describe sensitivity; probability expresses uncertainty; optimisation identifies effective parameter values; and information theory quantifies uncertainty and predictive content.

Mathematics is not an ornamental background to data science. It determines what a model can represent, how it learns, why it fails, and how its outputs should be interpreted. A linear regression model, for example, is simultaneously a geometric projection, an optimisation problem, a probabilistic statement about errors, and an information-processing procedure. Understanding these connections enables practitioners to select suitable methods rather than treating algorithms as opaque software commands.

This chapter develops the mathematical foundations most frequently used in postgraduate data science and machine learning. The emphasis is not on abstract proof for its own sake, but on interpretation, assumptions, computational meaning, and professional application.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Represent data using vectors, matrices, subspaces, norms, and inner products.

2. Explain eigenvalues, eigenvectors, rank, orthogonality, and singular value decomposition in data-analysis contexts.

3. Use derivatives, gradients, Hessians, and the chain rule to understand model sensitivity and learning.

4. Distinguish probability, conditional probability, likelihood, and posterior probability.

5. Apply expectation, variance, covariance, and common probability distributions to data problems.

6. Explain optimisation objectives, convexity, gradient descent, regularisation, and constraints.

7. Interpret entropy, cross-entropy, mutual information, and Kullback–Leibler divergence.

8. Identify assumptions, numerical risks, ethical implications, and practical limitations of mathematical models.

Data as Geometry: Vectors and Matrices

An observation with p measured features can be represented as a vector x = (x₁, x₂, …, xₚ)ᵀ in ℝᵖ. A dataset containing n observations is commonly represented by a design matrix X ∈ ℝⁿˣᵖ. Rows represent observations and columns represent variables. A target vector y may contain numerical outcomes or encoded labels.

The distinction between a row and a column is not merely typographical. Matrix dimensions determine whether an operation is defined and often reveal its meaning. For example, Xβ produces one predicted value for each observation when β is a p-dimensional parameter vector. The expression XᵀX summarises relationships among features, whereas Xᵀy summarises relationships between features and the target.

The inner product of two vectors, xᵀz, measures alignment. If vectors are centred and standardised, their inner product is closely related to correlation. The Euclidean norm, ||x||₂ = √(xᵀx), measures length, while the Manhattan or L₁ norm, ||x||₁ = Σ|xᵢ|, measures coordinate-wise magnitude. Norms are used to quantify prediction error, parameter size, and distance between observations.

A matrix can represent a transformation: rotation, scaling, projection, or a combination of these. A diagonal matrix scales coordinates independently. An orthogonal matrix preserves lengths and angles. A symmetric matrix satisfies A = Aᵀ and is especially important because it has real eigenvalues and an orthogonal eigenbasis under standard finite-dimensional conditions.

Rank, Span, and Identifiability

The column space of X is the set of all linear combinations of its columns. Its dimension is the rank of X. If two features are exact linear combinations of one another, the matrix is rank-deficient. In linear regression this creates non-identifiability: multiple parameter vectors can produce the same predictions. Near-linear dependence produces multicollinearity, which can make coefficient estimates unstable even when predictions remain acceptable.

The null space contains vectors v for which Xv = 0. A nonzero null-space vector represents a parameter change that has no effect on fitted values. This distinction is important in interpreting coefficients: a model may predict well while individual coefficients are not reliably interpretable.

Projection and Least Squares

The least-squares estimate chooses β to minimise

||y − Xβ||₂².

Geometrically, Xβ is the projection of y onto the column space of X. The residual r = y − Xβ is orthogonal to every column of X at the optimum, giving the normal equations XᵀXβ = Xᵀy. When XᵀX is invertible, β̂ = (XᵀX)⁻¹Xᵀy. In practice, QR factorisation or singular value decomposition is usually preferred to explicitly computing an inverse because it is more numerically stable.

Eigenvalues and Principal Directions

An eigenvector v of A satisfies Av = λv. The transformation changes the vector’s scale but not its direction; λ is the corresponding eigenvalue. In principal component analysis (PCA), eigenvectors of the covariance matrix identify directions of maximum variance. The first principal component captures the largest possible variance among unit-length directions, the second captures the largest remaining variance subject to orthogonality, and so forth.

PCA can reduce dimensionality, remove correlated structure, and support visualisation. However, maximum variance is not equivalent to maximum predictive usefulness. A low-variance feature may be highly informative about a target, and principal components may be difficult to interpret substantively.

Singular Value Decomposition

Every real matrix can be decomposed as X = UΣVᵀ, where U and V have orthonormal columns and Σ contains non-negative singular values. SVD supports low-rank approximation by retaining the largest singular values. It is used in recommender systems, image compression, latent semantic analysis, denoising, and stable solution of least-squares problems. Small singular values often signal directions that are weakly supported by the data and may amplify noise during inversion.

Calculus and the Behaviour of Models

Calculus provides a language for change. For a scalar function f(x), the derivative f′(x) describes the instantaneous rate of change. In data science, it answers questions such as: how much does loss change when a parameter changes slightly, and which direction reduces the loss?

For a function of several variables, the gradient is

∇f(x) = (∂f/∂x₁, …, ∂f/∂xₚ)ᵀ.

The gradient points in the direction of steepest increase under the Euclidean geometry. Consequently, −∇f is the local direction of steepest decrease. The Hessian, H, is the matrix of second partial derivatives. It describes local curvature and helps distinguish flat, sharply curved, and saddle-like regions.

Chain Rule and Computational Graphs

The chain rule explains how derivatives propagate through compositions. If z = f(g(x)), then dz/dx = f′(g(x))g′(x). Neural networks are nested compositions of affine transformations and nonlinear activation functions. Backpropagation applies the chain rule efficiently from the output layer toward the input, calculating parameter gradients without separately perturbing every parameter.

Poorly chosen activation functions or very deep compositions can produce vanishing or exploding gradients. Normalisation, suitable initialisation, residual connections, and carefully selected optimisers are practical responses to these mathematical effects.

Differentiation and Sensitivity

A derivative is local, not a guarantee of global behaviour. A small gradient may indicate a local optimum, a saddle point, or simply a flat region. Sensitivity analysis examines how predictions or decisions change under changes in inputs, parameters, or assumptions. In regulated or high-impact applications, sensitivity should be considered alongside accuracy because a model that is highly unstable may be operationally unsafe.

Probability: Reasoning under Uncertainty

Probability assigns values between zero and one to events. A random variable maps outcomes to numerical values. Its probability mass function describes discrete outcomes, while its probability density function describes continuous outcomes; a density itself is not a probability, but probabilities are obtained by integrating it over intervals.

For events A and B, conditional probability is

P(A|B) = P(A ∩ B)/P(B), provided P(B) > 0.

Bayes’ theorem follows directly:

P(A|B) = P(B|A)P(A)/P(B).

Here P(A) is the prior, P(B|A) the likelihood, P(B) the evidence, and P(A|B) the posterior. In classification, a posterior probability combines base rates with evidence. Ignoring base rates is a common source of misleading interpretation.

Expectation, Variance, and Covariance

The expectation E[X] is the probability-weighted average of a random variable. Variance, Var(X) = E[(X − E[X])²], measures dispersion. Covariance, Cov(X,Y) = E[(X − E[X])(Y − E[Y])], measures joint linear variation. Correlation is covariance divided by the product of standard deviations and therefore is scale-free.

Covariance does not establish causation. It can arise from confounding, selection effects, reverse causality, or coincidence. Mathematical dependence is not by itself a causal explanation.

The law of total expectation states E[X] = E[E(X|Z)], and the law of total variance states Var(X) = E[Var(X|Z)] + Var(E[X|Z]). These identities help separate within-group variation from between-group variation and are useful in hierarchical data analysis.

Important Distributions

The Bernoulli distribution models one binary trial with success probability p. The binomial distribution counts successes over a fixed number of independent Bernoulli trials. The categorical and multinomial distributions generalise these ideas to multiple categories.

The normal distribution is characterised by mean μ and variance σ² and is central because of its analytical convenience and its connection to aggregation through the central limit theorem. The Poisson distribution models counts over an interval when events occur with a constant average rate under suitable assumptions. The exponential distribution often models waiting times. These models are approximations, not universal laws; their assumptions should be checked against domain evidence.

Optimisation: From Objectives to Decisions

A learning algorithm usually minimises an objective function. For empirical risk minimisation,

J(θ) = (1/n)Σ L(yᵢ, fθ(xᵢ)),

where θ denotes parameters and L is a loss function. Least squares penalises squared errors; logistic regression commonly uses log loss; support vector machines use hinge loss; and neural networks may use cross-entropy or task-specific objectives.

Gradient Descent

The basic gradient-descent update is

θₜ₊₁ = θₜ − η∇J(θₜ),

where η is the learning rate. Batch gradient descent uses all observations, stochastic gradient descent uses one observation, and mini-batch descent uses a subset. Mini-batches offer a practical compromise between computational efficiency and gradient stability.

A learning rate that is too large may cause divergence; one that is too small may produce impractically slow learning. Momentum, adaptive learning rates, and schedules can improve optimisation, but they do not eliminate the need to inspect objective behaviour and validation performance.

Convexity and Constraints

A function is convex if the line segment joining any two points on its graph lies above the graph. In a convex optimisation problem, every local minimum is global. Ordinary least squares with a convex quadratic objective is a prominent example. Deep neural-network objectives are generally non-convex, containing saddle points and multiple parameterisations with similar predictions.

Constraints can express feasibility or policy requirements, such as non-negative weights or a probability vector summing to one. Lagrange multipliers incorporate equality constraints into an objective. Karush–Kuhn–Tucker conditions generalise this reasoning to inequality constraints and provide necessary conditions for many constrained optima.

Regularisation and Generalisation

Regularisation modifies the objective to discourage undesirable solutions. Ridge regression adds λ||β||₂² and tends to shrink correlated coefficients together. Lasso adds λ||β||₁ and can produce exact zeros, supporting sparse representations. Elastic net combines both penalties. Regularisation reduces variance and can improve generalisation, but excessive regularisation creates bias and may remove useful signal.

The penalty coefficient must be selected without using the final test set, typically through cross-validation on the training data. Preprocessing steps, including scaling and feature selection, must also be performed within each training fold to prevent information leakage.

Information Theory and Learning Signals

Entropy measures uncertainty in a probability distribution. For a discrete variable X,

H(X) = −Σ p(x) log p(x).

Entropy is highest for a uniform distribution over a fixed number of outcomes and zero for a certain outcome. The logarithm base determines units: base two gives bits, while the natural logarithm gives nats.

Cross-entropy between a true distribution p and a model distribution q is

H(p,q) = −Σ p(x)log q(x).

For a one-hot class label, cross-entropy reduces to the negative logarithm of the predicted probability assigned to the correct class. It strongly penalises confident incorrect predictions, which explains its common use in classification.

The Kullback–Leibler divergence is

D_KL(p||q) = Σ p(x)log[p(x)/q(x)].

It is non-negative but not symmetric, so it is not a distance in the usual metric sense. Mutual information, I(X;Y) = D_KL(p(x,y)||p(x)p(y)), measures how much knowing one variable reduces uncertainty about another. It can support feature selection, but high mutual information does not establish causal relevance and may favour variables with many possible values unless estimation is handled carefully.

An Applied Case: Predicting Loan Default

Background

Consider an illustrative case involving a lending organisation that wants to estimate the probability that an applicant will default within twelve months. Available variables include income, existing debt, repayment history, employment duration, and a categorical employment sector.

Situation and Mathematical Formulation

The data are encoded as a matrix X and binary outcomes as y. A logistic model estimates

P(Y=1|x) = 1/[1 + exp(−(β₀ + xᵀβ))].

The parameters are learned by minimising cross-entropy, usually with regularisation. Numerical features are scaled within the training process. Categorical variables are encoded consistently, and missing-data procedures are fitted only on training data.

Analysis and Approach

Initial matrix diagnostics reveal strong correlation between income and credit limit and sparse categories in employment sector. SVD or condition-number diagnostics help identify unstable directions. Ridge regularisation is selected through cross-validation because prediction is the primary objective, while coefficient signs are interpreted cautiously. Calibration is assessed in addition to discrimination: a group predicted at 0.20 risk should contain approximately 20 percent defaults if the model is well calibrated.

Bayesian reasoning is useful when default prevalence changes over time. A model trained during a low-risk period may produce poorly calibrated posteriors when the base rate increases. Monitoring must therefore distinguish changes in feature relationships from changes in outcome prevalence.

Outcome and Governance

Suppose validation indicates acceptable ranking performance but poorer calibration for a small applicant group. The organisation should not deploy the model solely on the basis of aggregate accuracy. It should investigate data quality, subgroup sample size, proxy variables, and threshold effects. Recalibration, additional data collection, or human review may be preferable to automatic rejection.

Learning Points

The case demonstrates that linear algebra diagnoses feature dependence, probability supports risk interpretation, optimisation trains the model, and information theory supplies the loss function. It also shows that a mathematically correct model can still be professionally unsuitable if calibration, fairness, explainability, and governance are neglected.

Limitations, Risks, and Professional Judgement

Mathematical models simplify reality. Measurement error, missingness, sampling bias, distribution shift, and incorrect functional assumptions can dominate algorithmic sophistication. Correlated predictors can make coefficients unstable; poorly scaled variables can hinder optimisation; floating-point arithmetic can create numerical error; and ill-conditioned matrices can amplify small perturbations.

Data leakage is one of the most damaging practical mistakes. It occurs when information unavailable at prediction time, or information derived from validation or test observations, influences training. Another mistake is confusing a low training objective with real-world usefulness. Proper evaluation requires a deployment-relevant validation design, suitable metrics, uncertainty analysis, and monitoring after implementation.

Ethically, mathematical transparency includes communicating assumptions and uncertainty, not merely publishing equations. High-impact systems should document data provenance, intended use, limitations, subgroup performance, human oversight, appeal mechanisms, and responsibility for decisions. Privacy-preserving methods, access controls, secure computation, and minimisation of collected data are relevant when mathematical models process personal information.

Current Directions and Best Practice

Modern systems increasingly combine classical mathematical structure with automatic differentiation, probabilistic programming, large-scale optimisation, and representation learning. Differentiable programming allows gradients to pass through complex computational pipelines. Approximate Bayesian methods seek uncertainty estimates in models too complex for exact inference. Foundation models use high-dimensional representations whose geometry is analysed through embeddings, similarity measures, and spectral methods.

Good practice remains grounded in fundamentals:

• define the estimand, prediction target, and decision context before selecting an algorithm;
• inspect scale, rank, sparsity, dependence, and missingness;
• use numerically stable decompositions rather than unnecessary matrix inversion;
• separate training, validation, and testing rigorously;
• match the loss and evaluation metric to the real cost of errors;
• report uncertainty, calibration, subgroup behaviour, and sensitivity;
• monitor data and outcome drift after deployment; and
• treat mathematical output as decision support subject to domain and ethical review.

Chapter Summary

Linear algebra provides the representation and geometry of data through vectors, matrices, projections, eigenvectors, and SVD. Calculus explains sensitivity and enables gradient-based learning through derivatives, gradients, Hessians, and the chain rule. Probability formalises uncertainty using conditional probability, Bayes’ theorem, expectations, variances, covariances, and probability distributions. Optimisation converts modelling aims into objective functions and provides procedures such as gradient descent, constrained optimisation, and regularisation. Information theory measures uncertainty and predictive content through entropy, cross-entropy, KL divergence, and mutual information. Across all these areas, assumptions, numerical stability, leakage prevention, calibration, fairness, and governance determine whether mathematical methods produce trustworthy professional outcomes.

Key Terms

Vector: An ordered numerical quantity used to represent an observation, parameter, or direction.

Matrix: A rectangular arrangement of numbers used to store data or represent transformations.

Inner product: A scalar product that measures alignment between vectors.

Norm: A function measuring the size or length of a vector.

Rank: The dimension of a matrix’s column space.

Null space: The set of vectors mapped to zero by a matrix.

Projection: The closest representation of a vector in a specified subspace under a chosen geometry.

Eigenvector: A nonzero vector whose direction is preserved by a linear transformation.

Singular value decomposition: A factorisation expressing a matrix through orthogonal directions and non-negative singular values.

Gradient: The vector of first partial derivatives of a scalar-valued function.

Hessian: The matrix of second partial derivatives describing local curvature.

Conditional probability: The probability of an event given that another event is known.

Likelihood: The probability of observed data viewed as a function of model parameters.

Expectation: The probability-weighted average value of a random variable.

Covariance: A measure of joint linear variation between two variables.

Convexity: A property ensuring that line segments between feasible points lie within the relevant epigraph or set.

Regularisation: An objective modification that discourages complex or unstable parameter values.

Entropy: A measure of uncertainty in a probability distribution.

Cross-entropy: A measure of the coding or predictive cost of one distribution under another.

Mutual information: The reduction in uncertainty about one variable obtained from another.

Review Questions

Short Answer Questions

1. Why can multicollinearity make regression coefficients unstable even when predictions are accurate?

2. Distinguish a probability density from a probability.

3. What role does the Hessian play in optimisation?

4. Why is KL divergence not generally a distance metric?

5. Explain why preprocessing must be included inside cross-validation folds.

Long Answer Questions

1. Explain least squares geometrically and derive the normal-equation interpretation.

2. Discuss PCA, its relationship to eigenvectors, and two situations in which PCA may be inappropriate.

3. Explain Bayes’ theorem using the concepts of prior, likelihood, evidence, and posterior.

4. Compare ridge, lasso, and elastic-net regularisation, including their effects on model parameters.

5. Describe how gradients and the chain rule support the training of a multilayer neural network.

Analytical / Application Questions

1. A feature matrix has two nearly identical columns and a very small singular value. Explain the likely modelling consequences and recommend a diagnostic and remedy.

2. A classifier ranks applicants well but systematically overestimates risk after the population default rate changes. Analyse the problem and propose an evaluation and recalibration strategy.

3. Design a mathematically and ethically defensible workflow for a high-impact prediction system, including objective selection, validation, uncertainty, subgroup analysis, and post-deployment monitoring.

22. Multiple Choice Questions with Answers

Question 1

In least-squares regression, the fitted vector Xβ̂ is best interpreted geometrically as:

A. The projection of y onto the column space of X

B. The projection of X onto the null space of y

C. An eigenvector of XᵀX

D. A random sample from the residual distribution

Correct Answer: A. The projection of y onto the column space of X

Question 2

A very small singular value of a feature matrix most directly indicates:

A. Perfectly independent features

B. A weakly supported or nearly dependent direction

C. That all features have zero variance

D. That the matrix is necessarily square

Correct Answer: B. A weakly supported or nearly dependent direction

Question 3

Which quantity points in the direction of steepest local increase of a differentiable scalar function?

A. The Hessian

B. The null vector

C. The gradient

D. The residual

Correct Answer: C. The gradient

Question 4

In Bayes’ theorem, P(A|B) represents:

A. The prior probability of A

B. The likelihood of B under A

C. The evidence for B

D. The posterior probability of A given B

Correct Answer: D. The posterior probability of A given B

Question 5

Why can a model with a low training loss still perform poorly after deployment?

A. Training loss always measures causal effects

B. The model may have overfit or encountered distribution shift

C. A lower loss mathematically guarantees worse generalisation

D. Optimisation prevents the use of validation data

Correct Answer: B. The model may have overfit or encountered distribution shift

Question 6

Which regularisation method most directly encourages exact zero coefficients?

A. Lasso using an L₁ penalty

B. Ridge using an L₂ penalty

C. Unregularised least squares

D. Entropy maximisation

Correct Answer: A. Lasso using an L₁ penalty

Question 7

For a fixed number of possible outcomes, entropy is generally greatest when the distribution is:

A. Certain

B. Highly concentrated on one outcome

C. Uniform

D. Impossible to normalise

Correct Answer: C. Uniform

Question 8

A classifier assigns a very high probability to an incorrect class. Cross-entropy responds by:

A. Giving no penalty because the prediction is confident

B. Applying a strong penalty

C. Removing the observation from the dataset

D. Changing the class label automatically

Correct Answer: B. Applying a strong penalty

Question 9

Which statement about mutual information is correct?

A. It proves that one variable causes another

B. It measures dependence or uncertainty reduction

C. It is always negative for dependent variables

D. It can only be used for continuous variables

Correct Answer: B. It measures dependence or uncertainty reduction

Question 10

In the illustrative lending case, why should calibration be assessed in addition to ranking performance?

A. Calibration determines whether predicted probabilities correspond to observed frequencies

B. Ranking performance automatically establishes fairness

C. Calibration eliminates the need for validation data

D. Ranking metrics cannot be computed for binary outcomes

Correct Answer: A. Calibration determines whether predicted probabilities correspond to observed frequencies