Mathematical, Probability and Statistical Foundations

Introduction

Data science is applied mathematics under uncertainty. Linear algebra represents data and transformations; calculus explains change and sensitivity; optimization selects model parameters; probability formalizes uncertainty; and statistics connects finite samples to populations. These foundations are not merely theoretical. They determine whether a model is identifiable, whether an estimate is reliable, whether an optimization procedure converges, and whether a reported prediction is ethically defensible.

This chapter develops the mathematical language needed for postgraduate study of machine learning and data analysis. The emphasis is on interpretation, assumptions, calculation, and professional judgment rather than mechanical formula memorization.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Represent data, transformations, and relationships using vectors, matrices, norms, and eigen concepts.
2. Apply derivatives, gradients, Hessians, and the chain rule to model fitting.
3. Formulate and analyze optimization problems, including constrained and regularized problems.
4. Distinguish probability models, random variables, conditional probability, and independence.
5. Use common probability distributions and interpret their parameters.
6. Explain sampling distributions, estimation, confidence intervals, and hypothesis tests.
7. Apply Bayes’ theorem and evaluate prior assumptions and posterior evidence.
8. Diagnose statistical assumptions and communicate uncertainty responsibly.

1. Linear Algebra for Data Representation

1.1 Vectors, matrices, and data tables

A scalar is a single number. A vector is an ordered collection of numbers. A feature vector for one observation may be written as x ∈ ℝᵖ, where p is the number of features. A dataset with n observations and p features is commonly represented by a matrix X ∈ ℝⁿˣᵖ. The entry xᵢⱼ is the value of feature j for observation i.

A linear combination of vectors is formed by multiplying vectors by scalars and adding them. This idea underlies regression, dimensionality reduction, embeddings, and neural-network layers. Matrix multiplication is defined when the inner dimensions agree. If A ∈ ℝᵐˣⁿ and B ∈ ℝⁿˣᵖ, then AB ∈ ℝᵐˣᵖ, with

The order matters: generally AB ≠ BA. In a machine-learning pipeline, a matrix may represent observations, a transformation, or model parameters; its meaning must be established before interpreting a calculation.

1.2 Inner products, norms, and geometry

For vectors a,b ∈ ℝᵖ, the Euclidean inner product is

The Euclidean norm is ||a||₂ = √(aᵀa). The squared Euclidean distance between observations x and z is ||x − z||₂². Distance-based methods such as k-nearest neighbours and clustering are sensitive to feature scale. A variable measured in thousands can dominate a variable measured in fractions, even if the latter is more informative.

The cosine similarity between nonzero vectors is

It measures orientation rather than magnitude and is often useful for document or embedding representations. It is undefined for a zero vector.

1.3 Rank, linear dependence, and invertibility

The rank of a matrix is the dimension of its column space, equivalently the number of linearly independent columns. If columns are linearly dependent, some features contain redundant information. A square matrix A is invertible only if its columns are linearly independent, equivalently rank(A) equals its dimension, and equivalently det(A) ≠ 0.

The normal-equation solution for ordinary least squares is

when XᵀX is invertible. Here X is the design matrix, y is the response vector, and β̂ is the coefficient vector. In practice, directly computing an inverse is numerically inferior to QR or singular-value decomposition, especially under multicollinearity. The formula is therefore a theoretical characterization, not always the preferred implementation.

1.4 Eigenvalues, eigenvectors, and singular value decomposition

For a square matrix A, a nonzero vector v is an eigenvector if Av = λv, where λ is its eigenvalue. Eigenvectors identify directions preserved by a transformation, while eigenvalues describe scaling in those directions. For a symmetric covariance matrix, eigenvectors can be chosen orthogonal and eigenvalues are real and nonnegative.

Principal component analysis (PCA) finds directions of maximum variance. After centering the columns of X, the sample covariance matrix is

If Svⱼ = λⱼvⱼ and λ₁ ≥ λ₂ ≥ …, the first principal component is the projection onto v₁. PCA is useful for compression and visualization, but its components are not automatically causal factors, and scaling choices can change the result.

The singular value decomposition (SVD) expresses a matrix as X = UΣVᵀ. It is central to least squares, PCA, low-rank approximation, and stable numerical computation. Truncating small singular values can reduce noise, but the appropriate rank depends on validation and the purpose of analysis.

2. Calculus and Sensitivity

2.1 Derivatives and gradients

A derivative measures the local rate of change of a scalar function. For f(x) = x², f′(x) = 2x. For a differentiable scalar function f(w) of a vector w ∈ ℝᵖ, the gradient is

The gradient points in the direction of greatest local increase. A first-order approximation is f(w + Δw) ≈ f(w) + ∇f(w)ᵀΔw for sufficiently small Δw.

The Hessian is the matrix of second partial derivatives:

For twice-differentiable f, a local quadratic approximation is f(w + Δw) ≈ f(w) + ∇f(w)ᵀΔw + ½ΔwᵀH(w)Δw. A positive-definite Hessian at a stationary point indicates a strict local minimum; positive semidefiniteness gives a weaker conclusion.

2.2 Chain rule and computational graphs

If y = f(g(x)), then dy/dx = f′(g(x))g′(x). In multiple dimensions, Jacobian matrices generalize derivatives. Neural networks apply the chain rule repeatedly during backpropagation. Automatic differentiation evaluates exact derivatives of composed computer programs up to floating-point error; it is not the same as finite-difference approximation.

For the sigmoid function σ(z) = 1/(1 + e⁻ᶻ), its derivative is σ′(z) = σ(z)[1 − σ(z)]. This bounded derivative contributes to both numerical stability and the possibility of vanishing gradients in deep compositions.

3. Optimization and Regularization

3.1 Objective functions

An optimization problem has decision variables, an objective, and possibly constraints. In supervised learning, parameters θ are selected by minimizing empirical risk:

where L is a loss function. The empirical objective estimates expected prediction loss but may be optimistic if evaluated on the same data used for fitting.

For differentiable unconstrained optimization, gradient descent updates parameters by

where J is the objective and η > 0 is the learning rate. If η is too large, iterations may diverge; if too small, learning may be impractically slow. Convexity is important: every local minimum of a convex function is global, whereas nonconvex objectives may contain saddle points and multiple local minima.

3.2 Least squares and a worked calculation

For a linear model ŷ = Xβ, the mean squared error is

Assuming fixed observations, a correctly specified linear conditional mean, finite second moments, and an identifiable design, minimizing MSE estimates the conditional mean in the population limit. Normal-equation derivation gives XᵀXβ̂ = Xᵀy.

Consider two observations with one feature and an intercept: X = [[1,1],[1,2]] and y = [2,3]ᵀ. The line ŷ = β₀ + β₁x passes exactly through both points, so β̂₀ = 1 and β̂₁ = 1. Its predictions are 2 and 3, residuals are zero, and training MSE is 0. This does not establish generalization: with only two observations, no conclusion about unseen cases is justified.

3.3 Regularization and constraints

Ridge regression minimizes

where λ ≥ 0 controls shrinkage. For standardized predictors and an unpenalized intercept, the ridge solution is (XᵀX + nλI)⁻¹Xᵀy, provided the indicated inverse exists. Ridge usually reduces variance at the cost of some bias.

Lasso uses λ||β||₁ instead of λ||β||₂². The absolute-value penalty can produce exactly zero coefficients, enabling sparse selection, although correlated predictors may be selected unpredictably. Regularization strength must be selected using validation within the training process; test data should remain untouched until final assessment.

Constrained optimization can be analyzed using Lagrange multipliers. For minimizing f(x) subject to g(x) = 0, define ℒ(x,ν) = f(x) + νg(x). Under differentiability and suitable constraint qualifications, candidate solutions satisfy ∇ₓℒ = 0 and g(x) = 0. Inequality constraints require the Karush–Kuhn–Tucker conditions, including complementary slackness.

4. Probability Foundations

4.1 Probability spaces and conditional probability

A probability model consists of a sample space Ω, events, and a probability measure P satisfying P(Ω) = 1, nonnegativity, and countable additivity. For events A and B with P(B) > 0,

The multiplication rule is P(A ∩ B) = P(A|B)P(B). Events A and B are independent when P(A ∩ B) = P(A)P(B), equivalently P(A|B) = P(A) when the conditional probability is defined. Independence is stronger than zero correlation and is a modeling assumption, not a conclusion from convenience.

Bayes’ theorem is

For hypotheses H₁,…,Hₖ forming a mutually exclusive and exhaustive partition, P(B) = ∑ⱼP(B|Hⱼ)P(Hⱼ). Confusing P(A|B) with P(B|A) is a common and consequential error.

4.2 Random variables and expectation

A random variable maps outcomes to numerical values. Its expected value is a probability-weighted average. For a discrete variable,

and for a continuous variable with density f,

Variance is Var(X) = E[(X − μ)²] = E[X²] − μ², where μ = E[X]. Standard deviation is σ = √Var(X). For constants a and b, E[aX+b] = aE[X]+b and Var(aX+b) = a²Var(X). Covariance is Cov(X,Y) = E[(X−E[X])(Y−E[Y])]. Correlation is ρ = Cov(X,Y)/(σₓσᵧ), when both standard deviations are positive. Correlation measures linear association, not causation.

4.3 Important distributions

The Bernoulli distribution models one trial with success probability p: P(X = 1) = p and P(X = 0) = 1−p. Its mean is p and variance is p(1−p). A Binomial(n,p) variable counts successes in n independent Bernoulli trials and has mean np and variance np(1−p).

A Poisson(λ) variable models counts in a fixed interval under an idealized constant-rate process with independent increments; its mean and variance are both λ. The Normal distribution N(μ,σ²) has density

where σ > 0. The standard normal transformation is Z = (X−μ)/σ. Distributional assumptions should be checked rather than imposed solely because a method is familiar.

5. Sampling, Estimation, and Inference

5.1 Sampling distributions and the central limit theorem

A population is the complete target set; a sample is the observed subset. Sampling bias occurs when the sampling mechanism systematically excludes or overrepresents parts of the population. Random sampling does not eliminate every problem: nonresponse, measurement error, and data leakage can remain.

For independent observations with finite mean μ and variance σ², the sample mean X̄ = (1/n)∑Xᵢ has E[X̄] = μ and Var(X̄) = σ²/n. Its standard error is σ/√n, estimated in practice by s/√n. The central limit theorem states that, under broad conditions, the standardized sample mean approaches a standard normal distribution as n increases. It does not say that raw observations become normal, nor does it guarantee accuracy under dependence or extreme heavy tails.

5.2 Estimators and bias

An estimator is a statistic used to estimate a population quantity. Bias(θ̂) = E[θ̂] − θ, and variance is Var(θ̂). Mean squared error decomposes as

assuming the target θ is fixed. An unbiased estimator is not automatically best: a slightly biased estimator can have much lower variance and MSE.

For independent observations from a Normal(μ,σ²) population, the sample mean is an unbiased estimator of μ. The sample variance

is unbiased for σ² under independent, identically distributed sampling with finite variance. The n−1 correction reflects estimation of the mean.

5.3 Confidence intervals and hypothesis tests

If σ is known and observations are independent with a normal sampling distribution, a 100(1−α)% confidence interval for μ is

where z₁₋α/₂ is the corresponding standard-normal quantile. If σ is unknown and the population is normal, replace σ by s and use a t quantile with n−1 degrees of freedom.

A confidence level describes the long-run coverage of the procedure, not the probability that a fixed interval contains μ after it has been calculated. A p-value is the probability, under the null hypothesis and the analysis assumptions, of results at least as extreme as the observed result. It is not the probability that the null hypothesis is true, nor a measure of practical importance. Multiple testing, optional stopping, and selective reporting can make nominal p-values misleading.

Effect sizes, uncertainty intervals, study design, and domain consequences should accompany significance testing. Statistical significance is not equivalent to business relevance or causality.

6. Bayesian Reasoning

Bayesian inference treats unknown quantities as uncertain and combines a prior distribution with a likelihood. In proportional form,

More explicitly, p(θ|D) = p(D|θ)p(θ)/p(D), where θ is a parameter and D is observed data. The prior encodes information or regularization available before the current data; it should be justified, documented, and subjected to sensitivity analysis.

For a coin with a Beta(α,β) prior and x heads in n independent trials, the posterior is Beta(α+x, β+n−x). This conjugate result is computationally convenient, but convenience does not validate the coin model or the prior. Bayesian predictive inference integrates parameter uncertainty: p(y_new|D) = ∫p(y_new|θ)p(θ|D)dθ. Such predictive distributions can be more informative than a single point estimate.

Bayesian and frequentist methods answer related but distinct questions. Bayesian credible intervals have a posterior probability interpretation conditional on the model and prior; frequentist confidence intervals have repeated-sampling coverage interpretations. Neither framework removes the need for good data, valid measurement, and transparent assumptions.

7. Statistical Assumptions in Data Science

The phrase independent and identically distributed (i.i.d.) means observations follow the same distribution and are mutually independent. Time series, clustered records, repeated patients, social networks, and transactions often violate i.i.d. assumptions. Random train–test splitting can then leak future or related information into training data. Grouped, blocked, or temporal validation may be required.

Common assumptions include linearity, constant error variance (homoscedasticity), weak dependence, appropriate distributional form, absence of extreme influential points, and reliable measurement. Diagnostics include residual plots, calibration curves, sensitivity analysis, influence measures, cross-validation, and posterior predictive checks. No diagnostic proves a model true; diagnostics reveal incompatibilities and guide refinement.

Data preprocessing is part of statistical methodology. Imputation, scaling, feature selection, and dimensionality reduction must be fitted using training data only and then applied to validation or test data. Otherwise, information from evaluation data contaminates performance estimates.

8. Applied Case: Predicting Loan Default

Background

This illustrative case concerns a lender developing a model to estimate the probability that an applicant will default within twelve months. The data include income, debt ratio, repayment history, employment duration, and an outcome indicator.

Problem

The lender wants accurate risk estimates while avoiding disparate impact, leakage, and unjustified certainty. Historical approvals are not a random sample of all applicants, and the outcome is observed only for previously approved applicants.

Analysis

Analysts first define the prediction time and exclude variables unavailable at that time. They inspect missingness, class imbalance, duplicate records, and temporal drift. A random split would be inappropriate if economic conditions change over time, so training uses earlier periods and validation uses later periods. A grouped split is used when applicants have multiple accounts.

A logistic model estimates a default probability pᵢ using

The log-odds transformation requires 0 < pᵢ < 1. Coefficients describe changes in log-odds holding other predictors fixed; exp(βⱼ) is an odds ratio, not a probability increase. Regularization reduces instability from correlated variables. Calibration is assessed alongside discrimination, because a model may rank applicants well while its probabilities are systematically too high or too low.

Approach and governance

The team reports a confusion matrix at policy-relevant thresholds, area under the precision–recall curve for the imbalanced outcome, calibration plots, and subgroup performance. Threshold selection incorporates the costs of false approvals and false denials. Fairness analysis examines legally and ethically relevant groups, but no single fairness metric resolves all trade-offs. Human review, appeal mechanisms, data minimization, access controls, model monitoring, and documented change management are required.

Outcome and learning points

The model is deployed only after temporal validation, leakage review, calibration, and governance approval. Monitoring detects drift and changes in subgroup error rates. The main lesson is that mathematical validity is necessary but insufficient: sampling, measurement, institutional context, and rights of affected people determine whether a statistical system is fit for use.

9. Contemporary Developments and Best Practices

Modern AI increasingly uses high-dimensional representations, stochastic optimization, probabilistic programming, causal inference, and uncertainty quantification. Large models make optimization and linear algebra computationally demanding, while distribution shift challenges classical validation. Conformal prediction offers distribution-free marginal coverage under exchangeability, but coverage can weaken under shift and may not guarantee equal subgroup performance.

Best practice is to:

• define the estimand, population, prediction time, and decision use before modeling;
• standardize units and document transformations;
• use numerically stable decompositions rather than unnecessary matrix inverses;
• separate training, validation, and final test information;
• quantify uncertainty and report effect size, not only p-values;
• test sensitivity to priors, sampling, missing-data mechanisms, and model specification;
• validate on data resembling deployment conditions;
• document assumptions, limitations, subgroup behavior, and monitoring plans;
• treat fairness, privacy, security, and explainability as design requirements.

Challenges and Limitations

Mathematical models simplify reality. Non-identifiability can make distinct explanations fit the same data. Correlation can arise from confounding, selection, or common causes. Small samples produce unstable estimates; large samples can make trivial effects statistically significant. Missing data may be informative, and measurement systems may encode institutional bias. Optimization can find a numerical solution to the wrong objective. Finally, uncertainty intervals conditional on a flawed model may give a false impression of precision.

Chapter Summary

Linear algebra supplies the representation and geometry of data through vectors, matrices, norms, rank, eigenvectors, and SVD. Calculus provides derivatives, gradients, Hessians, and the chain rule needed for sensitivity analysis and learning algorithms. Optimization formalizes parameter selection; gradient descent, convexity, constraints, and regularization determine computational and statistical behavior.

Probability defines uncertainty through conditional probability, independence, random variables, expectations, variance, covariance, and distributions. Sampling theory explains why estimates vary and why the central limit theorem is useful but conditional. Estimation requires attention to bias, variance, MSE, confidence procedures, and the distinction between statistical and practical significance. Bayesian reasoning combines prior information and likelihood while requiring transparent sensitivity analysis. Across all applications, assumptions, validation design, leakage prevention, calibration, fairness, privacy, and monitoring are as important as formula correctness.

Key Terms

Vector: An ordered numerical quantity representing a point, feature set, or direction.

Matrix: A rectangular array used to represent data or transformations.

Norm: A function measuring vector magnitude.

Rank: The dimension of a matrix’s column or row space.

Eigenvector: A nonzero vector whose direction is preserved by a linear transformation.

Gradient: The vector of first partial derivatives.

Hessian: The matrix of second partial derivatives.

Convexity: A property ensuring that line segments between graph points lie above the graph; convex minimization has strong global guarantees.

Regularization: A penalty or constraint used to control complexity and improve generalization.

Random variable: A numerical mapping from outcomes to values.

Expectation: The probability-weighted long-run average of a random variable.

Variance: The expected squared deviation from the mean.

Independence: A condition in which joint probability factors into marginal probabilities.

Likelihood: The probability model for observed data viewed as a function of parameters.

Estimator: A statistic used to infer a population quantity.

Bias: The difference between an estimator’s expected value and its target.

Confidence interval: A repeated-sampling interval procedure with stated coverage.

p-value: A null-model probability of results at least as extreme as the observed result.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data leakage: Use of information unavailable at prediction time or improperly borrowed from evaluation data.

Review Questions

Short Answer Questions

1. Why can AB differ from BA in matrix multiplication?
2. Define the gradient and explain its role in gradient descent.
3. Distinguish independence from zero correlation.
4. What does the central limit theorem approximate?
5. Why must preprocessing be fitted only on training data?

Long Answer Questions

1. Explain how vectors, matrices, norms, rank, and SVD support machine-learning workflows.
2. Derive the normal equations for ordinary least squares and discuss when the inverse formula is unsafe computationally.
3. Compare ridge and lasso regularization, including their statistical and interpretive consequences.
4. Explain confidence intervals, p-values, practical significance, and multiple-testing concerns.
5. Compare Bayesian credible intervals with frequentist confidence intervals and discuss the role of priors.

Analytical / Application Questions

1. A fraud model is trained on randomly split transaction data containing repeated customers. Identify likely leakage and propose a validation design.
2. A classifier has high ranking performance but poor probability calibration. Explain why this matters and propose corrective actions.
3. In the loan-default case, analyze how selection into historical approvals affects estimation and propose a responsible governance strategy.

Multiple Choice Questions with Answers

Question 1

If A ∈ ℝ²ˣ³ and B ∈ ℝ³ˣ⁴, what is the dimension of AB?

A. 2 × 4

B. 3 × 3

C. 4 × 2

D. 2 × 3

Correct Answer: A. 2 × 4

Question 2

Which condition guarantees that a differentiable stationary point is a strict local minimum?

A. The Hessian is negative definite

B. The Hessian is positive definite

C. The gradient is nonzero

D. The function is discontinuous

Correct Answer: B. The Hessian is positive definite

Question 3

For independent Bernoulli trials with success probability p, what is the variance of the number of successes in n trials?

A. np

B. p(1−p)

C. np(1−p)

D. n²p

Correct Answer: C. np(1−p)

Question 4

Bayes’ theorem is used primarily to calculate which quantity from a likelihood and a prior?

A. A posterior probability or distribution

B. A matrix rank

C. A gradient norm

D. A training-set size

Correct Answer: A. A posterior probability or distribution

Question 5

What is the main effect of ridge regularization?

A. It forces every coefficient to zero

B. It shrinks coefficients toward zero using a squared L2 penalty

C. It guarantees causal interpretation

D. It removes the need for validation

Correct Answer: B. It shrinks coefficients toward zero using a squared L2 penalty

Question 6

A 95% frequentist confidence interval is best interpreted as:

A. A 95% probability that this fixed interval contains the parameter

B. A procedure that covers the parameter in 95% of repeated samples under its assumptions

C. A 95% probability that the null hypothesis is false

D. The probability that the sample mean is correct

Correct Answer: B. A procedure that covers the parameter in 95% of repeated samples under its assumptions

Question 7

If X has mean μ and variance σ², what is Var(3X + 5)?

A. 3σ²

B. 9σ²

C. σ² + 5

D. 3σ² + 5

Correct Answer: B. 9σ²

Question 8

Which situation most clearly violates an ordinary random train–test split?

A. Independent laboratory measurements

B. A dataset with time-ordered transactions and future information

C. A balanced dataset with no repeated entities

D. A dataset whose features were standardized using training data

Correct Answer: B. A dataset with time-ordered transactions and future information

Question 9

What does calibration measure for probabilistic predictions?

A. Whether predicted probabilities agree with observed frequencies

B. Whether all coefficients are positive

C. Whether the data matrix is square

D. Whether training loss is exactly zero

Correct Answer: A. Whether predicted probabilities agree with observed frequencies

Question 10

For a Normal population with unknown variance and a small sample, which reference distribution is typically used for a confidence interval for the mean?

A. Uniform distribution

B. Poisson distribution

C. Student’s t distribution with n−1 degrees of freedom

D. Bernoulli distribution

Correct Answer: C. Student’s t distribution with n−1 degrees of freedom