Mathematical Foundations of Machine Learning and AI

Introduction and Context

Machine learning (ML) is the study of computational systems that infer patterns from data and use those patterns to make predictions, decisions, or representations. Artificial intelligence (AI) systems extend this idea to perception, reasoning, planning, language, and action. Although modern ML is implemented with software and specialized hardware, its central structure is mathematical: data are represented as vectors, matrices, tensors, probability distributions, and functions; learning is formulated as optimization; and performance is assessed through statistical generalization.

A useful abstraction is a parameterized model f_θ: X → Y, where θ denotes parameters, X is the input space, and Y is the output space. Training chooses parameters that minimize an objective on observed data while controlling complexity so that the model performs well on unseen observations.

This chapter connects regression, classification, clustering, dimensionality reduction, probabilistic modeling, and neural networks through their objective functions, regularization mechanisms, computational requirements, tensor representations, and differentiable-programming foundations.

Learning Objectives

After studying this chapter, you should be able to:

• formulate common supervised and unsupervised learning problems mathematically;
• distinguish empirical risk, population risk, likelihood, and posterior objectives;
• explain regularization, bias–variance trade-offs, and generalization;
• derive key gradients for linear models and neural networks;
• interpret tensors, computational graphs, automatic differentiation, and backpropagation;
• compare computational complexity and optimization characteristics across model classes;
• evaluate an applied ML pipeline critically, including ethical and governance risks.

1. A Unifying Mathematical View of Learning

Let a dataset be D = {(x_i,y_i)}_i=1ⁿ, where x_i is an input and y_i is a target. A loss function ℓ(f_θ(x_i),y_i) measures disagreement between prediction and target. The empirical risk is

Here, n is the number of observations. The population risk is

where P is the unknown data-generating distribution. Training minimizes an estimate of population risk because P is unavailable. The distinction is fundamental: a model can achieve low empirical risk yet high population risk through overfitting.

A regularized objective commonly has the form

where Ω(θ) penalizes complexity and λ ≥ 0 controls the penalty. The loss and penalty must be interpreted together. For example, squared error has units of the squared target variable, whereas a penalty must be scaled consistently or treated as a dimensionless weighted modeling preference.

Data representation

A scalar is a rank-0 tensor, a vector is rank 1, a matrix is rank 2, and an array with additional axes is a higher-order tensor. A batch of color images may have shape (b,c,h,w), where b is batch size, c channels, and h,w spatial dimensions. Shape conventions must be explicit because a mathematically valid operation can still be semantically wrong if axes are confused.

2. Regression: Geometry, Loss, and Regularization

In linear regression, the prediction is

where X ∈ R^n×p is the design matrix, β ∈ R^p is the coefficient vector, b is an intercept, and 1 is an all-ones vector. Under the assumptions of independent errors with mean zero and constant variance, ordinary least squares minimizes

The gradient with respect to β is

Setting the gradient to zero gives the normal equations. If an intercept is included in an augmented matrix and XᵀX is invertible,

The inverse formula is mainly theoretical or suitable for small, well-conditioned problems. In practice, QR decomposition, singular-value decomposition, or iterative methods are numerically safer. If features are highly collinear, XᵀX may be ill-conditioned or singular.

Ridge and lasso

Ridge regression uses an ℓ₂ penalty:

For a fixed intercept and λ > 0, the solution is

The scaling by n follows the chosen mean-squared-error convention. Other texts place the factor differently; the meaning of λ changes accordingly. Ridge shrinks correlated coefficients and improves conditioning but generally does not set coefficients exactly to zero.

Lasso uses an ℓ₁ penalty:

where ||β||₁ = ∑_j |β_j|. Its non-differentiability at zero can produce sparse solutions. Feature scaling is important for both methods, especially lasso, because penalty strength otherwise depends on measurement units.

Worked example: ridge in one dimension

Suppose centered observations have XᵀX = 10, Xᵀy = 20, n = 5, and λ = 1. Then

The unregularized coefficient would be 20/10 = 2. Regularization has shrunk the coefficient toward zero. This example assumes the intercept has already been removed or separately handled.

3. Classification and Probabilistic Decision Functions

For binary classification, let y ∈ {0,1}. Logistic regression models

The sigmoid maps real scores to probabilities in (0,1). With independent Bernoulli observations, the negative log-likelihood, also called binary cross-entropy, is

The gradient is

where p is the vector of predicted probabilities. A threshold such as 0.5 converts probability into a class label, but the threshold should reflect decision costs, prevalence, and operational requirements rather than being treated as a universal law.

For K classes, softmax produces

and multiclass cross-entropy is −∑_k y_k log p_k for one-hot targets. Numerically stable implementations subtract max_j z_j from all logits before exponentiation; this leaves probabilities unchanged but avoids overflow.

A confusion matrix supports metrics such as precision TP/(TP+FP), recall TP/(TP+FN), and accuracy (TP+TN)/(TP+TN+FP+FN). These metrics answer different questions and can conflict under class imbalance. Proper evaluation requires a prespecified test set or cross-validation and, when probabilities matter, calibration analysis.

4. Clustering and Dimensionality Reduction

k-means

Given points x_i ∈ R^p and K clusters, k-means minimizes within-cluster squared distance:

where c_i ∈ {1,…,K} is an assignment and μ_k is a centroid. The alternating algorithm assigns each point to its nearest centroid and then updates each centroid to the mean of assigned points:

Each iteration decreases or preserves the objective, but the method can converge to a local minimum. Initialization, scaling, outliers, and the selected value of K strongly affect results. Its usual dominant cost is approximately O(nKpT), where T is the number of iterations; this is a practical complexity expression rather than a guarantee for every implementation.

Principal component analysis

For centered data matrix X, PCA finds orthogonal directions of maximum variance. The first direction solves

Equivalently, v₁ is the leading eigenvector of XᵀX or the leading right singular vector of X. The rank-r reconstruction minimizes squared reconstruction error among rank-r linear approximations. PCA is sensitive to feature scale and outliers, and its components are not automatically causal or semantically meaningful.

5. Probabilistic Models and Bayesian Regularization

A generative model specifies a joint distribution p(x,y). The chain rule gives

Bayes’ rule is

where p(y) is the prior, p(x|y) the likelihood, and p(y|x) the posterior. The denominator p(x) normalizes the posterior.

Maximum likelihood estimates parameters by maximizing ∑_i log p_θ(x_i,y_i). Maximum a posteriori estimation maximizes log likelihood plus log prior:

Thus regularization can have a probabilistic interpretation. A Gaussian prior on coefficients produces an ℓ₂-type penalty, while a Laplace prior produces an ℓ₁-type penalty, subject to parameterization and scaling conventions.

Latent-variable models, such as Gaussian mixture models, introduce unobserved variables z. The expectation–maximization algorithm alternates between estimating posterior responsibilities and updating parameters. It can increase likelihood at each iteration but may reach local optima and can encounter degenerate covariance estimates without constraints or regularization.

6. Neural Networks as Compositions of Differentiable Functions

A feed-forward layer is

where W is a weight matrix, b a bias vector, and φ an elementwise or structured activation function. ReLU is φ(a)=max(0,a). A multilayer network is a composition of such maps. The final layer and loss depend on the task: linear output with squared error for some regression problems, or logits with cross-entropy for classification.

Backpropagation

Backpropagation applies the chain rule efficiently to a computational graph. For a scalar loss L and a layer preactivation a^(ℓ),

and

where ⊙ denotes elementwise multiplication. These expressions assume compatible dimensions and differentiability wherever derivatives are required; ReLU is handled with a subgradient convention at zero.

Gradient descent updates parameters by

where η > 0 is the learning rate. Stochastic or mini-batch gradients are estimates of the full gradient. Momentum, adaptive methods, normalization, initialization, and learning-rate schedules influence optimization, but none guarantees a globally optimal solution for a non-convex network.

7. Generalization, Capacity, and Regularization

The bias–variance perspective decomposes expected prediction error, under appropriate squared-error assumptions, into approximation bias, estimation variance, irreducible noise, and sometimes covariance terms depending on the setup. Increasing model flexibility can reduce bias while increasing variance. Regularization, early stopping, data augmentation, dropout, and ensembling can improve generalization, but their effects depend on data and architecture.

Training, validation, and test data must be separated according to the unit of independence. In time series, random splitting can leak future information; in medical or customer data, records from the same person or organization may need to remain in one partition. Hyperparameters must be selected without using the final test set.

Modern overparameterized networks challenge simple textbook intuitions: some highly flexible models interpolate training data yet generalize well. Explanations include implicit regularization from optimization, architecture, data geometry, and parameterization. These phenomena do not eliminate the need for validation and distribution-shift analysis.

8. Computational Complexity and Numerical Reliability

For dense matrix multiplication (m×k)(k×n), the arithmetic cost is typically O(mkn), and storage is O(mk+kn+mn) if all operands and results are retained. A dense neural layer with input width d_in and output width d_out has approximately d_in d_out weights and comparable multiply–add cost per example.

Complexity analysis should distinguish:

Floating-point overflow, underflow, cancellation, and ill-conditioning can invalidate mathematically correct formulas. Standardization, stable softmax, QR/SVD solvers, gradient clipping, and mixed-precision loss scaling are engineering responses, not cosmetic choices.

9. Tensors and Differentiable Programming

A tensor program combines operations on arrays with automatic differentiation. A computational graph records dependencies among variables. Reverse-mode automatic differentiation computes gradients of a scalar output with respect to many inputs efficiently; this is why it is suitable for neural-network training. Forward mode is often preferable when there are few inputs and many outputs.

Differentiable programming extends this framework beyond conventional neural layers. Physics-informed models, differentiable simulators, optimal-control systems, and learned components can be composed when the relevant operations have derivatives or useful approximations. However, differentiability does not imply identifiability, causal validity, convexity, or reliable extrapolation.

A tensor operation must preserve both shape and meaning. For example, attention mechanisms commonly compute a scaled similarity matrix:

where Q, K, and V are query, key, and value matrices, and d_k is the key dimension. The square-root scaling controls the magnitude of dot products; masking may be necessary to prevent access to unavailable positions.

10. Illustrative Case: Credit-Risk Modeling

Background

The following is a fictional illustrative case. A bank wants to estimate the probability that an applicant will default within twelve months. The data include income, debt ratio, repayment history, account age, and demographic variables.

Problem

The bank needs accurate probabilities, transparent governance, resistance to temporal drift, and fair treatment across protected groups. A model that merely maximizes training accuracy is inadequate.

Analysis

Analysts define the target using an observation window and a future performance window, remove post-outcome variables, and split data chronologically. They compare logistic regression, a regularized gradient-boosted model, and a neural network. Logistic regression uses cross-entropy with an ℓ₂ penalty; the boosted model uses additive trees and shrinkage; the neural model uses cross-entropy and early stopping.

Evaluation includes area under the ROC curve, precision–recall behavior under imbalance, calibration, expected cost at operational thresholds, subgroup error rates, and performance on a later time period. Confidence intervals or bootstrap uncertainty should accompany point estimates where decisions are consequential.

Approach and solution

Features are standardized using training data only. Missingness indicators are examined rather than silently imputed. Hyperparameters are selected by time-aware validation. The final test period remains untouched until model selection is complete. Probability calibration is performed on validation data, and a human-review pathway is retained for borderline cases.

Outcome

The regularized logistic model is selected because its predictive performance is close to that of more complex alternatives while its coefficients and monitoring behavior are easier to explain. The bank documents variables, data provenance, thresholds, calibration, subgroup analyses, override policy, and drift triggers. It does not treat statistical parity in one metric as proof of fairness; trade-offs among fairness criteria, predictive accuracy, and legal requirements are reviewed with domain experts.

Learning points

The case demonstrates that objective-function choice is only one part of responsible ML. Leakage control, temporal validation, calibration, complexity, interpretability, fairness, privacy, and post-deployment monitoring are mathematical and governance concerns simultaneously.

11. Limitations, Ethics, and Best Practices

Models inherit sampling bias, measurement error, historical discrimination, and label-definition problems. A highly accurate predictor may still be inappropriate if the target is a proxy for an unjust process. Privacy risks include re-identification, membership inference, and inappropriate secondary use. Security risks include poisoning, evasion, and model extraction.

Best practices include:

1. define the estimand, target, population, and decision use before modeling;
2. establish leakage-resistant splits and a reproducible preprocessing pipeline;
3. scale features when penalties or distance metrics require it;
4. select losses and metrics that match operational costs;
5. use numerically stable implementations and monitor conditioning;
6. compare simple baselines with complex models;
7. quantify uncertainty and examine subgroup and shift performance;
8. document data provenance, model version, assumptions, and limitations;
9. obtain appropriate consent, security controls, and human oversight;
10. monitor after deployment and establish rollback criteria.

Contemporary research increasingly studies foundation models, self-supervised objectives, probabilistic calibration, causal representation learning, efficient fine-tuning, privacy-preserving learning, and reliable uncertainty estimation. These developments expand capability but do not replace the foundational requirements of sound measurement, optimization, validation, and governance.

Chapter Summary

Machine learning can be understood as the optimization of an empirical objective intended to approximate population performance. Regression uses losses such as squared error; classification commonly uses likelihood-based cross-entropy; clustering optimizes within-group structure; PCA optimizes variance or reconstruction; probabilistic models represent uncertainty through distributions; and neural networks compose parameterized differentiable functions. Regularization controls complexity and may be interpreted geometrically or probabilistically. Generalization depends on data splitting, capacity, optimization, distribution shift, and measurement quality. Tensors provide the computational representation, while automatic differentiation and backpropagation provide efficient gradients. Computational complexity, numerical stability, fairness, privacy, interpretability, and monitoring are essential parts of professional ML practice.

Key Terms

Empirical risk: Average loss measured on observed training data.

Population risk: Expected loss under the data-generating distribution.

Regularization: A complexity-controlling term added to an objective.

Ridge regression: Regression with an ℓ₂ coefficient penalty.

Lasso: Regression with an ℓ₁ coefficient penalty.

Logistic regression: A probabilistic binary classifier using the sigmoid function.

Cross-entropy: Negative log-likelihood for categorical or Bernoulli outcomes.

Softmax: A function converting logits into a categorical probability distribution.

k-means: Centroid-based clustering minimizing within-cluster squared distance.

PCA: Linear dimensionality reduction based on variance or reconstruction error.

Likelihood: Probability of observed data conditional on model parameters.

Prior: Distribution representing parameter beliefs before observing data.

Posterior: Updated distribution after combining prior and likelihood.

Tensor: Multidimensional numerical array.

Computational graph: Representation of dependencies among operations and variables.

Automatic differentiation: Systematic computation of exact derivatives through a program.

Backpropagation: Reverse-mode differentiation used to train layered models.

Generalization: Performance on previously unseen data.

Data leakage: Unintended use of information unavailable at prediction time.

Calibration: Agreement between predicted probabilities and observed frequencies.

Review Questions

Short Answer Questions

1. Distinguish empirical risk from population risk.
2. Why does feature scaling matter for lasso and k-means?
3. State the role of the regularization parameter λ.
4. What does the softmax function produce?
5. Why is a chronological split appropriate for many time-series problems?

Long Answer Questions

1. Explain ordinary least squares, ridge regression, and lasso, including their objective functions, assumptions, and practical differences.
2. Derive the gradient of binary cross-entropy for logistic regression and explain its interpretation.
3. Compare k-means, PCA, and probabilistic latent-variable models in terms of objective, assumptions, and limitations.
4. Explain backpropagation using the chain rule and discuss the role of tensors in implementation.
5. Discuss how regularization, optimization, data splitting, and model capacity jointly affect generalization.

Analytical / Application Questions

1. A fraud classifier has excellent accuracy but misses most fraud cases. Which metrics, thresholding strategy, validation design, and cost assumptions should be reconsidered?
2. A PCA pipeline gives different components after changing units from dollars to cents. Diagnose the issue and propose a defensible preprocessing approach.
3. Design a governance and monitoring plan for a deployed credit-risk model facing population drift, subgroup performance differences, and changing economic conditions.

Multiple Choice Questions with Answers

Question 1

Which objective directly estimates expected predictive performance using the observed sample?

A. Population risk

B. Empirical risk

C. Prior probability

D. Computational complexity

Correct Answer: B. Empirical risk

Question 2

For ridge regression using mean squared error plus λ||β||²₂, the main effect of the penalty is to:

A. Force every coefficient to exactly one

B. Shrink coefficients toward zero and improve conditioning

C. Guarantee zero training error

D. Convert regression into clustering

Correct Answer: B. Shrink coefficients toward zero and improve conditioning

Question 3

In binary logistic regression, the sigmoid function maps a real-valued score to:

A. A probability between 0 and 1

B. An integer cluster label

C. A vector whose entries sum to zero

D. A variance estimate

Correct Answer: A. A probability between 0 and 1

Question 4

Which operation is a defining update in k-means?

A. Set each centroid to the mean of its assigned points

B. Set every point to the global mean

C. Maximize the determinant of the covariance matrix

D. Remove all points with nonzero residuals

Correct Answer: A. Set each centroid to the mean of its assigned points

Question 5

PCA’s first direction is commonly characterized as the unit vector that:

A. Minimizes the number of observations

B. Maximizes projected variance

C. Maximizes classification accuracy for every label

D. Makes all features independent

Correct Answer: B. Maximizes projected variance

Question 6

In Bayes’ rule, p(y|x) represents the:

A. Prior

B. Likelihood

C. Posterior

D. Evidence-free parameter penalty

Correct Answer: C. Posterior

Question 7

Reverse-mode automatic differentiation is especially efficient when:

A. There is one scalar output and many inputs

B. There are many unrelated scalar outputs and no shared computation

C. No operation has a computational graph

D. All parameters are discrete

Correct Answer: A. There is one scalar output and many inputs

Question 8

For dense multiplication of an m×k matrix by a k×n matrix, the usual arithmetic complexity is:

A. O(m+k+n)

B. O(mkn)

C. O(m²+n²)

D. O(1)

Correct Answer: B. O(mkn)

Question 9

Why should a final test set remain unused during hyperparameter selection?

A. To increase the number of training examples

B. To prevent optimistic assessment caused by information leakage

C. To guarantee a convex objective

D. To eliminate the need for validation

Correct Answer: B. To prevent optimistic assessment caused by information leakage

Question 10

In the credit-risk case, why was the regularized logistic model selected despite more complex alternatives?

A. It always has the highest accuracy

B. It required no data preprocessing

C. It offered comparable performance with greater transparency and monitoring ease

D. It cannot experience distribution shift

Correct Answer: C. It offered comparable performance with greater transparency and monitoring ease