Mathematical, Statistical and Optimization Foundations

Introduction

Machine learning is often described as the construction of systems that learn from data. More precisely, it is the study of algorithms that estimate an unknown relationship between inputs and outputs, or discover useful structure in observations, by optimizing an objective under uncertainty. The mathematics of machine learning provides the language for representing data, defining uncertainty, measuring error, and improving models.

This chapter develops four mutually connected foundations:

1. Linear algebra, for representing observations, parameters, transformations, and geometric relationships.
2. Probability and statistics, for modelling uncertainty, variation, sampling, estimation, and evidence.
3. Calculus and loss functions, for expressing sensitivity and defining what it means for a model to perform well.
4. Optimization, for finding parameter values that minimize loss while controlling complexity and computational cost.

These foundations are not separate topics. A typical supervised-learning algorithm represents data as vectors, defines a probabilistic or deterministic prediction function, measures error through a loss function, and uses derivatives to optimize parameters. Regularization incorporates prior preferences for simpler solutions, while validation data estimate whether the learned relationship generalizes beyond the training sample.

Learning Objectives

After studying this chapter, you should be able to:

• represent data and model parameters using vectors, matrices, and tensors;
• explain norms, inner products, projections, rank, eigenvectors, and positive-definite matrices;
• distinguish probability, likelihood, and posterior probability;
• use common distributions and summary statistics appropriately;
• derive and interpret estimators such as the sample mean and maximum-likelihood estimate;
• apply Bayes’ theorem to revise beliefs using evidence;
• explain derivatives, gradients, Hessians, and the chain rule in model training;
• compare common loss functions and identify their assumptions;
• explain overfitting, bias–variance trade-offs, and regularization;
• analyze gradient descent and its important practical variants;
• connect mathematical assumptions to professional machine-learning decisions.

1. Linear Algebra as the Language of Data

1.1 Scalars, vectors, matrices, and tensors

A scalar is a single numerical value. A vector is an ordered collection of values, commonly written as x ∈ ℝᵈ, where d is the number of features. A dataset containing n observations and d features can be represented by a matrix X ∈ ℝⁿˣᵈ. Its element xᵢⱼ is the value of feature j for observation i. A target vector is often denoted by y ∈ ℝⁿ.

A parameter vector w ∈ ℝᵈ defines the coefficients of a linear model. With an intercept b, prediction for observation x is:

Here, wᵀ is the transpose of w, and the expression is defined when w and x have the same dimension. For all observations, predictions can be written as:

where 1 is an n-dimensional vector of ones. Alternatively, a column of ones may be appended to X, allowing the intercept to be included in an augmented parameter vector.

A tensor generalizes arrays to more than two dimensions. For example, a colour image may be represented by height × width × three colour channels. Matrix operations remain important even when a model processes tensors, because many neural-network layers perform affine transformations.

1.2 Inner products, norms, and geometry

The inner product of two vectors is:

It measures alignment. If xᵀz = 0, the vectors are orthogonal. The Euclidean, or L², norm is:

The L¹ norm is:

Norms measure magnitude and are used in distance calculations and regularization. The Euclidean distance between points x and z is ||x − z||₂. Distances are meaningful only relative to the feature scales: a feature measured in thousands can dominate one measured in fractions. Standardization is therefore often useful:

where μⱼ and σⱼ are the training-set mean and standard deviation of feature j. The transformation is dimensionless, and the same training-set values must be used to transform validation, test, and future data.

The cosine similarity is:

provided neither vector is zero. It measures direction rather than magnitude and is widely used for document and embedding comparison.

1.3 Matrices, rank, and conditioning

Matrix multiplication is defined when the inner dimensions agree. If A ∈ ℝᵐˣᵖ and B ∈ ℝᵖˣⁿ, then AB ∈ ℝᵐˣⁿ. Matrix multiplication is generally not commutative: AB ≠ BA.

The rank of a matrix is the dimension of its column space, equivalently the maximum number of linearly independent columns. Rank deficiency indicates redundant features or insufficient information to identify all parameters uniquely. In linear regression, highly correlated predictors can produce an ill-conditioned design matrix: small changes in data may cause large changes in estimated coefficients.

A square matrix A is symmetric if A = Aᵀ. It is positive semidefinite if zᵀAz ≥ 0 for every vector z, and positive definite if zᵀAz > 0 for every nonzero z. Covariance matrices are positive semidefinite. Positive-definite Hessians characterize locally strictly convex objectives and support stable optimization near a minimum.

An eigenvector v of a square matrix A satisfies:

where λ is the corresponding eigenvalue. Principal component analysis uses eigenvectors of a covariance matrix, or equivalently singular vectors of a suitably centered data matrix, to identify directions of high variance. Singular value decomposition expresses a matrix as X = UΣVᵀ and is useful for dimensionality reduction, least-squares computation, and numerical analysis.

2. Probability and Statistical Reasoning

2.1 Random variables and probability

A random variable maps outcomes of an uncertain process to numerical values. A discrete random variable has a probability mass function p(x), whereas a continuous random variable has a probability density function f(x). For a continuous variable, f(x) itself is not a probability; probabilities are areas under the density:

A valid probability model satisfies non-negativity and total probability one. The expectation of a discrete variable is:

and for a continuous variable:

The variance is:

where μ = E[X]. The standard deviation is σ = √Var(X), in the same units as X. Covariance between X and Y is:

Correlation is covariance divided by σₓσᵧ, provided both standard deviations are nonzero. Correlation is unit-free but does not establish causation.

2.2 Conditional probability, independence, and Bayes’ theorem

Conditional probability is:

when P(B) > 0. Events A and B are independent if P(A ∩ B) = P(A)P(B), equivalently P(A|B) = P(A) when P(B) > 0. Independence is a modelling assumption and should not be inferred merely because variables appear unrelated in a small sample.

Bayes’ theorem follows from the product rule:

In machine learning, A may represent a class and B observed features. P(A) is the prior, P(B|A) the class-conditional likelihood, P(A|B) the posterior, and P(B) the evidence. For multiple mutually exclusive classes:

The denominator normalizes the posterior probabilities.

A crucial distinction is that likelihood is a function of parameters given observed data, whereas a probability distribution describes possible data under fixed parameters. Confusing P(data|model) with P(model|data) leads to incorrect interpretations of evidence.

2.3 Important distributions

The Bernoulli distribution models one binary trial with success probability p:

Its mean is p and variance is p(1 − p). The binomial distribution models the number of successes in n independent Bernoulli trials:

The categorical distribution generalizes a single trial to K classes with probabilities p₁,…,pₖ summing to one. The multinomial distribution counts class outcomes over repeated trials.

The normal distribution with mean μ and variance σ² has density:

It is mathematically convenient and often a reasonable approximation for aggregated measurement noise, but not every feature is normally distributed. The central limit theorem concerns the distribution of suitably normalized sums or sample means under conditions such as independent observations with finite variance; it does not say that raw data must be normal.

The exponential distribution models waiting times under a constant-rate Poisson process. Its rate parameter λ has units of inverse time, and its mean is 1/λ. Distribution choice should follow the data-generating context, support of the variable, and diagnostic evidence.

3. Sampling, Estimation, and Generalization

A population is the complete set of entities of interest; a sample is the observed subset. A statistic is computed from a sample, while a parameter describes the population. Machine learning usually observes a finite training sample and seeks low expected risk on future observations drawn from a target distribution.

For independent observations X₁,…,Xₙ with finite mean, the sample mean is:

If the observations have variance σ², then Var(x̄) = σ²/n. The sample variance is commonly estimated by:

The n − 1 denominator gives an unbiased estimator of population variance under independent identically distributed sampling with finite variance. This correction does not make biased sampling representative.

3.1 Maximum likelihood

Given observations D and a parametric model p(D|θ), maximum likelihood selects:

Because logarithm is monotonic, it is usually equivalent to maximize the log-likelihood:

for independent observations. Minimizing negative log-likelihood is therefore a natural training objective. For a normal-noise regression model with fixed variance, minimizing negative log-likelihood is equivalent, apart from constants and scaling, to minimizing squared error.

3.2 Bias, variance, and predictive risk

For an estimator θ̂ of a parameter θ, bias is E[θ̂] − θ, and variance is E[(θ̂ − E[θ̂])²]. For squared-error prediction at an input x, expected test error can be decomposed conceptually as:

The precise decomposition depends on the data-generating setup and expectation over training samples and noise. A highly flexible model may have low training error but high variance. A rigid model may have high bias. Regularization, feature design, model capacity, and sample size influence this trade-off.

4. Calculus, Loss Functions, and Statistical Learning

4.1 Derivatives and gradients

The derivative measures the local rate of change of a scalar function. For f(x) = x², f′(x) = 2x. For a multivariable function J(w), the gradient is the vector of first partial derivatives:

The gradient points in the direction of steepest local increase under the Euclidean norm; therefore −∇J points toward steepest local decrease. The Hessian is the matrix of second partial derivatives:

When the Hessian is positive semidefinite throughout a convex domain, the function is convex. Every local minimum is then global, although optimization may still be numerically difficult.

The chain rule explains backpropagation. If z = g(u) and u = h(w), then:

For vector-valued functions, the same principle is expressed through Jacobian matrix products.

4.2 Loss functions

A loss function assigns a numerical penalty to a prediction. The empirical risk is the average loss over training data:

The population risk is the expected loss over the target data distribution:

Training minimizes empirical risk as a proxy for population risk.

For regression, mean squared error is:

It strongly penalizes large errors and corresponds to Gaussian noise assumptions when used as a negative log-likelihood objective. Mean absolute error is:

It is less sensitive to outliers and is associated with a Laplace-noise interpretation. Neither loss is universally superior; the choice should reflect the operational cost of errors.

For binary classification, with y ∈ {0,1} and predicted probability p ∈ (0,1), binary cross-entropy is:

For K-class classification, if yₖ is a one-hot target and pₖ predicted class probability:

Probabilities must be valid and nonzero for any class assigned positive target weight. In practice, numerically stable log-softmax implementations should be used rather than separately computing exponentials and logarithms.

5. Regularization and Model Complexity

Regularization modifies the training objective to discourage undesirable parameter values. Ridge regression minimizes:

Lasso regression minimizes:

Here λ ≥ 0 controls penalty strength. The intercept is commonly excluded from the penalty. Ridge tends to shrink correlated coefficients together and usually does not make them exactly zero. Lasso can produce sparse coefficients because the L¹ geometry has corners. The penalty and loss should be scaled consistently; standardizing features is particularly important when coefficient penalties are applied.

From a Bayesian perspective, an L² penalty corresponds to a zero-centred Gaussian prior on coefficients, while an L¹ penalty corresponds to a Laplace prior, under a compatible likelihood and parameterization. This interpretation does not mean that regularization automatically proves a causal prior belief; it provides a useful connection between penalized optimization and probabilistic modelling.

Regularization also includes early stopping, dropout, data augmentation, architectural constraints, and explicit complexity control. Hyperparameters must be selected using validation procedures, not by repeatedly inspecting the untouched test set.

6. Gradient-Based Optimization

6.1 Batch, stochastic, and mini-batch gradient descent

For objective J(w), gradient descent updates parameters as:

where η > 0 is the learning rate. If η is too large, updates may overshoot or diverge; if too small, training may be unacceptably slow.

Batch gradient descent computes the gradient using all training observations. Stochastic gradient descent uses one observation at a time, producing noisy but inexpensive updates. Mini-batch methods use a subset Bₜ:

Under random sampling, the mini-batch gradient is often an approximately unbiased estimate of the full-data gradient, although dependence, weighting, and sampling schemes can alter this property.

Momentum accumulates a moving direction, reducing oscillations in poorly conditioned problems. Adaptive methods such as Adam maintain estimates of first and second moments of gradients. They can be effective in deep learning, but their adaptive dynamics do not guarantee better generalization in every setting. Learning-rate schedules, gradient clipping, normalization, and careful initialization can materially affect results.

6.2 A worked example: one-dimensional regression

Suppose a model predicts ŷ = wx with one training observation x = 2 and y = 6. Use the half-squared loss:

The factor ½ simplifies differentiation. By the chain rule:

Let w = 1. Then ŷ = 2, residual wx − y = −4, and gradient = (−4)(2) = −8. With learning rate η = 0.1:

The updated prediction is 3.6, closer to 6 than the initial prediction. This single update does not establish convergence; repeated updates and a representative dataset are required.

7. Worked Statistical Example: Bayesian Diagnosis

Consider an illustrative screening test. Suppose disease prevalence is 1%, sensitivity is 90%, and the false-positive rate is 5%. Let D denote disease and + a positive result. Bayes’ theorem gives:

Substitution gives:

Thus, approximately 15.4% of positive results correspond to disease under these assumptions. The example illustrates why a high sensitivity does not imply a high posterior probability when the condition is rare. In real healthcare applications, prevalence may vary by population, test characteristics may depend on disease stage, and decisions require clinical validation, consent, privacy protection, and appropriate communication of uncertainty.

8. Applied Case: Predicting Loan Default

Background

This illustrative case concerns a financial institution developing a model to estimate the probability that an applicant will default within twelve months. The model will support, but not replace, credit assessment and governance processes.

Problem

The institution has historical records containing income, debt ratio, repayment history, loan amount, and a binary default outcome. It wants accurate predictions while avoiding discriminatory effects, data leakage, unstable coefficients, and unjustified certainty.

Analysis

The data team first defines the prediction time and outcome window. Features unavailable at application time are excluded to prevent leakage. Numerical variables are examined for missingness, outliers, skewness, and scale. The data are split into training, validation, and test sets using a time-aware design if economic conditions change over time. Class imbalance is assessed using prevalence and appropriate metrics rather than accuracy alone.

A logistic model represents the conditional probability of default as:

The coefficients are estimated by minimizing cross-entropy, possibly with an L¹ or L² penalty. Standardization is fitted only on training data. Validation data are used to select λ, learning rate, and decision threshold. Performance is evaluated using discrimination measures such as area under the ROC curve, calibration, expected cost, and subgroup analyses. Calibration is important because a predicted probability of 0.20 should correspond approximately to a 20% event frequency among comparable cases.

Approach and solution

The institution compares a regularized logistic baseline with a tree-based model. It uses cross-validation within the training period, preserves the final test period for one-time evaluation, and documents features, exclusions, assumptions, and limitations. A threshold is selected according to the relative costs of false approvals and false rejections, rather than automatically using 0.5.

Governance includes access controls, retention limits, audit logs, adverse-action explanations where legally required, monitoring for drift, and a human appeal process. Protected attributes may be restricted from direct use but remain relevant for fairness auditing, subject to lawful and ethical governance.

Outcome

The selected model improves calibrated risk ranking over the existing rule system, but its deployment is conditional on monitoring. If economic conditions or applicant populations shift, training risk may no longer represent operational risk. The organization therefore establishes periodic recalibration, drift alerts, performance review, and a procedure for suspending automated decisions.

Learning points

The case demonstrates that mathematical quality is necessary but insufficient. Data definitions, sampling, loss functions, regularization, threshold choice, calibration, fairness, explainability, security, and accountability all influence whether a model is fit for use.

9. Limitations, Assumptions, and Contemporary Practice

Mathematical models simplify reality. Independent and identically distributed sampling may fail under time dependence, clustered observations, feedback loops, or changing populations. Missing data may be informative rather than random. Measurement error can attenuate or distort relationships. Optimization can find a low training objective without producing reliable predictions under distribution shift.

Best practice includes defining the estimand and decision context before modelling; establishing reproducible preprocessing pipelines; preventing leakage; choosing metrics aligned with consequences; using uncertainty intervals or calibrated probabilities where appropriate; checking sensitivity to modelling assumptions; and reporting subgroup performance. Reproducible experiments should record data versions, code, random seeds where meaningful, hyperparameters, and evaluation protocols.

Contemporary machine learning increasingly uses automatic differentiation, differentiable programming, large-scale stochastic optimization, probabilistic programming, and uncertainty-aware methods. These tools improve expressive power but do not remove the need for statistical assumptions or critical evaluation. A larger model can amplify biased labels, privacy risks, security vulnerabilities, and environmental cost. Responsible practice requires lawful data use, minimization of personal information, protection against unauthorized inference, documentation, human oversight, and mechanisms for contesting harmful outcomes.

Chapter Summary

Machine learning rests on a connected mathematical pipeline. Linear algebra represents observations, parameters, transformations, distances, and geometric structure. Probability describes uncertainty, while statistics provides tools for sampling, estimation, likelihood, and generalization. Bayes’ theorem distinguishes prior information from evidence and posterior belief. Calculus supplies derivatives, gradients, Hessians, and the chain rule needed to understand learning updates. Loss functions formalize predictive error, and maximum likelihood explains several widely used objectives. Regularization controls complexity and can be interpreted through both optimization and Bayesian perspectives. Gradient descent and its variants search parameter spaces, but their success depends on scaling, conditioning, learning rates, data quality, and objective design. Finally, applied machine learning must address leakage, distribution shift, calibration, fairness, privacy, explainability, and governance in addition to numerical performance.

Key Terms

Vector: An ordered collection of numerical values.

Matrix: A rectangular array of values used to represent data or transformations.

Inner product: A scalar product measuring alignment between two vectors.

Norm: A function measuring vector magnitude.

Rank: The dimension of a matrix’s linearly independent column or row space.

Eigenvector: A vector whose direction is unchanged by a linear transformation apart from scaling.

Random variable: A numerical representation of an uncertain outcome.

Expectation: The probability-weighted average value of a random variable.

Variance: The expected squared deviation from the mean.

Likelihood: A function measuring how compatible observed data are with parameter values.

Prior: A probability distribution representing belief before observing current evidence.

Posterior: A probability distribution after updating with evidence.

Estimator: A rule for using sample data to infer an unknown parameter.

Bias: Systematic difference between an estimator’s expected value and the target parameter.

Loss function: A function assigning a penalty to a prediction.

Gradient: The vector of first partial derivatives of a scalar function.

Regularization: An objective modification that controls model complexity.

Learning rate: The step-size parameter in an iterative optimization update.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Distribution shift: A change between the data-generating conditions used for training and those encountered in deployment.

Review Questions

Short Answer Questions

1. Why is feature standardization often important before applying L² regularization?
2. Distinguish probability, likelihood, and posterior probability.
3. What is the practical meaning of the gradient of a loss function?
4. State one important difference between ridge and lasso regularization.
5. Why can accuracy be misleading for a highly imbalanced classification problem?

Long Answer Questions

1. Explain how linear algebra, probability, loss functions, and optimization interact in supervised learning.
2. Derive the gradient-descent update for the half-squared loss of a linear model.
3. Discuss the bias–variance trade-off and explain how regularization affects it.
4. Compare MSE, MAE, and cross-entropy in terms of assumptions and practical use.
5. Explain the assumptions and governance requirements involved in deploying a credit-default model.

Analytical / Application Questions

1. A dataset contains features measured in dollars, years, and percentages. Design a preprocessing and regularization strategy, explaining how scale affects optimization and coefficient penalties.
2. A disease has prevalence 2%, test sensitivity 95%, and false-positive rate 10%. Compute the posterior probability of disease after a positive result and interpret why it differs from sensitivity.
3. A mini-batch gradient has a very large norm and training loss oscillates. Analyze at least three possible causes and propose a technically justified diagnostic and remediation plan.

Multiple Choice Questions with Answers

Question 1

Which expression represents the Euclidean norm of vector x?

A. ∑ⱼ |xⱼ|

B. √(∑ⱼ xⱼ²)

C. ∑ⱼ xⱼ²

D. maxⱼ xⱼ

Correct Answer: B. √(∑ⱼ xⱼ²)

Question 2

If two nonzero vectors have an inner product of zero, what is their relationship?

A. They are parallel.

B. They have equal magnitude.

C. They are orthogonal.

D. They are statistically independent.

Correct Answer: C. They are orthogonal.

Question 3

A disease prevalence is 1%, sensitivity is 90%, and false-positive rate is 5%. Approximately what is the probability of disease given a positive test?

A. 1.0%

B. 5.0%

C. 15.4%

D. 90.0%

Correct Answer: C. 15.4%

Question 4

Which objective is most directly associated with maximum likelihood for independent observations?

A. Maximizing the sum of log-likelihoods

B. Minimizing the feature means

C. Maximizing the L¹ norm only

D. Minimizing the number of observations

Correct Answer: A. Maximizing the sum of log-likelihoods

Question 5

What is the principal effect of an L² penalty in linear regression?

A. It usually shrinks coefficients toward zero.

B. It guarantees every coefficient is exactly zero.

C. It removes the need for a validation set.

D. It converts every feature into a binary variable.

Correct Answer: A. It usually shrinks coefficients toward zero.

Question 6

For gradient descent, the standard update with learning rate η is:

A. wₜ₊₁ = wₜ + η∇J(wₜ)

B. wₜ₊₁ = ηwₜ − ∇J(wₜ)

C. wₜ₊₁ = wₜ − η∇J(wₜ)

D. wₜ₊₁ = wₜ/η + ∇J(wₜ)

Correct Answer: C. wₜ₊₁ = wₜ − η∇J(wₜ)

Question 7

Which loss is generally most sensitive to a single very large regression error?

A. Mean absolute error

B. Mean squared error

C. Zero-one loss

D. Cosine distance

Correct Answer: B. Mean squared error

Question 8

What does calibration mean for a probabilistic classifier?

A. All classes have equal frequency.

B. Predicted probabilities agree with observed frequencies.

C. The model has zero training error.

D. Every feature has unit variance.

Correct Answer: B. Predicted probabilities agree with observed frequencies.

Question 9

Why should the untouched test set not be repeatedly used for hyperparameter selection?

A. It cannot contain numerical values.

B. Repeated inspection can adapt decisions to the test set and bias performance estimates.

C. It always has fewer observations than the training set.

D. It prevents gradient computation.

Correct Answer: B. Repeated inspection can adapt decisions to the test set and bias performance estimates.

Question 10

Which condition is characteristic of a convex optimization problem?

A. Every local minimum is a global minimum.

B. The objective must contain an exponential function.

C. The gradient must be zero everywhere.

D. The objective cannot have constraints.

Correct Answer: A. Every local minimum is a global minimum.