Mathematical and Statistical Foundations

Introduction

Machine learning is often described as the construction of systems that learn from data. In practice, however, learning is expressed through mathematics. Data are represented as vectors and matrices; uncertainty is described with probability; patterns are summarized through statistics; model parameters are estimated through optimization; and predictions are evaluated using carefully chosen measures. A sound understanding of these foundations enables practitioners to select suitable models, interpret results, diagnose errors, and recognize when an apparently accurate model is unreliable.

This chapter develops the mathematical and statistical language required for introductory machine learning. It emphasizes interpretation and application rather than abstract formalism, while stating the assumptions under which important results are valid.

Learning Objectives

By the end of this chapter, you should be able to:

1. Represent data using scalars, vectors, matrices, and geometric operations.
2. Apply means, variances, covariance, and correlation to summarize data.
3. Explain probability rules, conditional probability, Bayes’ theorem, and common distributions.
4. Distinguish population quantities from sample estimates and explain statistical inference.
5. Use derivatives, gradients, and partial derivatives to understand model optimization.
6. Explain gradient descent, loss functions, regularization, and convergence limitations.
7. Interpret mathematical assumptions and communicate quantitative results responsibly.

1. Mathematical Representation of Data

1.1 Scalars, vectors, and matrices

A scalar is a single numerical value, such as a temperature or a model parameter. A vector is an ordered collection of values. If an observation has p features, it may be represented as

where xⱼ is the value of feature j and ᵀ denotes transposition. A dataset containing n observations and p features is commonly represented by an n × p matrix X. Row i contains one observation, and column j contains one feature.

A target vector y contains the values the model is intended to predict. In supervised regression, y may contain real-valued prices; in classification, it may contain class labels or encoded values.

The dot product of two vectors a and b of equal length is

It produces a scalar and is used in linear models. The vector norm measures size. The Euclidean norm is

The distance between two vectors is the norm of their difference:

These expressions assume numerical features measured on comparable scales. If one feature is measured in rupees and another in millimetres, distance calculations may be dominated by the feature with larger numerical units. Standardization is therefore often required.

1.2 Matrix operations

Matrix multiplication is defined when the inner dimensions agree. If A is m × k and B is k × n, then AB is m × n, with entries

Matrix multiplication is generally not commutative: AB may exist while BA does not, or both may exist but have different values. The transpose reverses dimensions: if A is m × n, then Aᵀ is n × m.

A square matrix may have an inverse A⁻¹ only when it is nonsingular, meaning its determinant is non-zero. In that case, solving Ax = b can be written x = A⁻¹b, although numerical algorithms such as QR or singular-value decomposition are generally preferred to explicitly computing an inverse.

1.3 Geometric interpretation

A vector can represent a point or a direction in feature space. A linear model calculates a weighted combination of features:

Here w is a vector of weights, b is an intercept, and ŷ is the prediction. The equation defines a hyperplane. In two dimensions the boundary is a line; in three dimensions it is a plane. The sign and magnitude of w determine the orientation and influence of features, provided the features have been appropriately scaled.

2. Descriptive Statistics

Descriptive statistics summarize observed data without claiming that the summary necessarily applies to an entire population.

2.1 Measures of central tendency

For observations x₁, x₂, …, xₙ, the arithmetic sample mean is

The mean uses every observation but is sensitive to extreme values. The median is the middle ordered value, or the average of the two middle values when n is even. The mode is the most frequent value and is especially useful for categorical data.

The weighted mean is

where weights wᵢ are non-negative and not all zero. It is appropriate when observations contribute unequally, but the weighting scheme must be justified.

2.2 Measures of spread

The range is the maximum minus the minimum. The interquartile range is

where Q₁ and Q₃ are the first and third quartiles. The IQR is relatively resistant to outliers.

The sample variance is

and the sample standard deviation is s = √s². The denominator n − 1, called Bessel’s correction, gives an unbiased estimator of population variance under independent random sampling with finite variance. If the complete population is available, population variance is instead

where N is population size and μ is the population mean.

Worked Example: Mean and sample variance

Suppose daily prediction errors are 2, 4, 4, and 6 units. The mean is

x̄ = (2 + 4 + 4 + 6)/4 = 4.

The squared deviations are 4, 0, 0, and 4. Therefore,

s² = (4 + 0 + 0 + 4)/(4 − 1) = 8/3 ≈ 2.67,

and s ≈ 1.63 units. The standard deviation is expressed in the original unit, whereas variance is expressed in squared units.

2.3 Standardization and robust scaling

A z-score standardizes an observation using a mean μ and standard deviation σ:

For training data, μ and σ are estimated from the training set and then reused for validation and test data. This prevents information leakage. Standardization is useful for distance-based models, gradient-based optimization, and regularized linear models. It does not make a skewed distribution normal, and it can be distorted by extreme outliers.

A robust alternative uses the median and IQR:

This is useful when the distribution contains substantial outliers, although its interpretation differs from a z-score.

3. Probability and Random Variables

Probability quantifies uncertainty. A sample space Ω contains possible outcomes, and an event A is a subset of Ω. A probability measure P satisfies 0 ≤ P(A) ≤ 1, P(Ω) = 1, and for mutually exclusive events A and B, P(A ∪ B) = P(A) + P(B).

3.1 Conditional probability and independence

Conditional probability is

provided P(B) > 0. The multiplication rule follows:

Events A and B are independent if P(A|B) = P(A), equivalently P(A ∩ B) = P(A)P(B). Independence is a strong assumption and should not be confused with lack of obvious association in a small dataset.

Bayes’ theorem is

where P(A) is the prior probability, P(B|A) is the likelihood, P(B) is the evidence, and P(A|B) is the posterior probability. In classification, the theorem explains how base rates and observed features jointly determine posterior class probabilities.

3.2 Random variables and expectation

A random variable assigns numerical values to outcomes. Its expected value is a probability-weighted average. For a discrete variable X,

For a continuous variable with density f(x),

The variance is

If X and Y are random variables and a and b are constants, expectation is linear:

Independence implies Var(X + Y) = Var(X) + Var(Y), but without independence a covariance term is required.

3.3 Important probability distributions

The normal density is

for σ > 0. The empirical 68–95–99.7 rule applies approximately only to normally distributed data: about 68% lie within one standard deviation of μ, 95% within two, and 99.7% within three.

The central limit theorem states, under commonly used conditions such as independent observations with finite variance, that the standardized sample mean approaches a normal distribution as sample size increases. It concerns the distribution of sample means, not the claim that the raw data themselves are normal.

4. Covariance, Correlation, and Dependence

Sample covariance between variables X and Y is

Positive covariance indicates that values above the mean of X tend to occur with values above the mean of Y. Its magnitude depends on measurement units.

Pearson’s correlation is

where sₓ and sᵧ are sample standard deviations. It lies between −1 and 1 and measures linear association. Correlation is unit-free, but it is sensitive to outliers and does not establish causation. A nonlinear relationship may have correlation near zero, and a confounding variable may create a misleading association.

For ranked or ordinal data, Spearman’s rank correlation may be more suitable. For binary classification, association can also be studied with contingency tables, conditional rates, and measures such as precision, recall, and odds ratios.

5. Statistical Inference

A population parameter, such as μ, is usually unknown. A statistic, such as x̄, is calculated from a sample and used to estimate it. An estimator is unbiased if its expected value equals the parameter it estimates, although unbiasedness alone does not guarantee low error.

5.1 Sampling variability and confidence intervals

For independent observations with population standard deviation σ, the standard error of the sample mean is

When σ is unknown, it is estimated by s. A large-sample approximate 95% confidence interval for μ is

The value 1.96 is the 97.5th percentile of the standard normal distribution. For small samples from an approximately normal population, a t critical value is preferred:

A 95% confidence interval is a procedure that captures the fixed population parameter in approximately 95% of repeated samples under the model assumptions. It is not, strictly speaking, a statement that there is a 95% probability that a fixed parameter lies in the particular interval.

5.2 Hypothesis testing

A hypothesis test begins with a null hypothesis H₀ and an alternative hypothesis H₁. A p-value is the probability, assuming H₀ and the analysis model are correct, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true.

Statistical significance does not imply practical importance. Large datasets can make very small effects statistically significant, while small samples may fail to detect meaningful effects. Confidence intervals and effect sizes should accompany p-values.

5.3 Sampling, bias, and data leakage

Random sampling supports generalization when the sample represents the target population. Convenience samples can produce selection bias. Measurement bias may arise from faulty sensors, inconsistent labels, or systematically missing values. In machine learning, data leakage occurs when information unavailable at prediction time enters training features or preprocessing. Leakage can produce excellent validation results but poor deployment performance.

6. Calculus for Learning Algorithms

Calculus describes how a quantity changes. For a function f(x), the derivative is

when the limit exists. It is the slope of the tangent and indicates local increase or decrease.

For several variables, the gradient is the vector of partial derivatives:

The gradient points in the direction of greatest local increase. Therefore, −∇f points toward greatest local decrease. A stationary point satisfies ∇f(θ) = 0, but it may be a minimum, maximum, or saddle point.

The chain rule explains backpropagation. If z = g(u) and u = h(x), then

For vector-valued computations, the same principle is expressed using Jacobian matrices.

6.1 Common derivatives in machine learning

For f(x) = x², f′(x) = 2x. For the sigmoid function

its derivative is σ′(z) = σ(z)[1 − σ(z)]. The sigmoid maps real values to (0, 1), which is useful for binary probability models, though it can suffer from very small gradients for large positive or negative inputs.

7. Optimization and Model Training

A machine-learning algorithm generally minimizes an objective function. For observations (xᵢ, yᵢ), a common empirical risk is

where θ denotes parameters, fθ is the model, and L is a loss function.

For linear regression, mean squared error is

It penalizes large errors strongly and is differentiable. For binary classification, binary cross-entropy is

where y ∈ {0, 1} and p is the predicted probability of class 1. The logarithm requires p to lie strictly between 0 and 1; implementations use numerical safeguards.

7.1 Gradient descent

The basic update is

where η > 0 is the learning rate. Batch gradient descent uses all training observations; stochastic gradient descent uses one observation; mini-batch gradient descent uses a subset.

A learning rate that is too large may cause divergence or oscillation. One that is too small may make training unnecessarily slow. Scaling features often improves the geometry of the objective and makes optimization more stable.

Worked Example: One gradient update

Consider J(w) = (w − 3)², with current w = 1 and learning rate η = 0.1. The derivative is J′(w) = 2(w − 3), so J′(1) = −4. The update is

wnew = 1 − 0.1(−4) = 1.4.

The parameter moves toward the minimizer w = 3. This is a one-dimensional illustration; real models use a gradient vector and may have non-convex objectives.

7.2 Convexity, regularization, and generalization

A differentiable function is convex if every line segment between two points lies above the graph. In convex optimization, every local minimum is global, although numerical convergence still depends on the algorithm and conditions. Neural-network objectives are generally non-convex.

Regularization adds a complexity penalty. Ridge regression uses

and lasso regression uses

where λ ≥ 0 controls penalty strength. The intercept is commonly excluded from the penalty. Ridge tends to shrink coefficients; lasso can set some coefficients exactly to zero. Regularization may reduce variance and overfitting but can introduce bias.

8. Integrated Applied Case: Predicting Customer Churn

Background

The following is an illustrative case. A subscription company wants to predict whether a customer will cancel within the next month. Available features include tenure, monthly charge, number of support contacts, and recent service interruptions.

Problem

The company has 10,000 historical records, but only 8% represent churn. A model with high overall accuracy could predict “no churn” for every customer and achieve 92% accuracy, while providing no useful intervention strategy.

Analysis

The data team first separates a training set, validation set, and untouched test set using a time-aware split, because future records should not influence past predictions. Missing values are investigated rather than automatically replaced. Numerical features are standardized using training-set means and standard deviations only.

The team examines class proportions, distributions, outliers, and correlations. A high correlation between monthly charge and contract type is not interpreted as causation. It may reflect the company’s pricing policy. Leakage checks identify a feature called “retention offer accepted,” which is recorded after a customer has already been targeted; it is removed because it would not be available at prediction time.

Approach and solution

A logistic model estimates a probability of churn:

The model is trained with cross-entropy and regularization. Evaluation includes precision, recall, F1 score, the confusion matrix, and calibration. Because missing a likely churner is costly, the decision threshold is selected using business costs rather than automatically set to 0.5.

Suppose the intervention cost is ₹200 per contacted customer and the estimated loss from an avoidable churn is ₹2,000. A threshold should be chosen by comparing expected benefits and costs, while considering the accuracy of predicted probabilities. The model is also checked for performance across customer segments to identify disparate error rates.

Outcome

In an illustrative deployment, recall improves substantially compared with the all-negative baseline, while the company limits interventions to a manageable group. The result is not accepted solely because of a high metric: monitoring is established for drift in churn rates, feature distributions, calibration, and segment-level performance.

Learning points

The case demonstrates that mathematical foundations support the entire workflow: vectors encode customers, descriptive statistics reveal data quality, probability produces risk estimates, optimization estimates parameters, regularization controls complexity, and inference and evaluation determine whether results are credible. Business thresholds, fairness, privacy, and operational constraints are part of the model’s real performance.

9. Limitations, Governance, and Best Practices

Mathematical models are abstractions. Their conclusions depend on data quality, sampling, assumptions, and the stability of the environment. A normal approximation may be inappropriate for heavy-tailed data; Pearson correlation may miss nonlinear dependence; gradient descent may converge to a poor solution; and a statistically significant relationship may have little operational value.

Professional practice requires documenting data provenance, feature definitions, units, transformations, missing-value treatment, random seeds, and evaluation procedures. Training and test information must remain separate. Sensitive attributes should be handled according to applicable privacy and anti-discrimination requirements. Models used in consequential decisions should support appropriate human oversight, auditability, explanation, security, and appeal mechanisms.

Contemporary systems increasingly use automatic differentiation, adaptive optimizers such as Adam, probabilistic programming, uncertainty estimation, and distribution-shift monitoring. These tools do not remove the need to understand assumptions. An optimizer can minimize a poorly designed objective, and a sophisticated uncertainty estimate can still be invalid if the data-generating process changes.

Chapter Summary

Linear algebra provides the representation and operations used by machine-learning models. Vectors encode observations, matrices organize datasets, dot products create weighted combinations, and norms express size and distance. Descriptive statistics summarize location and variation, while standardization supports comparability and stable optimization.

Probability formalizes uncertainty through conditional probability, independence, Bayes’ theorem, random variables, expectation, variance, and probability distributions. Covariance and correlation describe association but do not establish causation. Statistical inference distinguishes samples from populations and uses sampling distributions, confidence intervals, effect sizes, and hypothesis tests to quantify uncertainty.

Calculus supplies derivatives and gradients for parameter learning. Optimization minimizes loss functions through procedures such as gradient descent. Learning rates, convexity, regularization, and feature scaling affect training and generalization. Finally, responsible machine learning requires attention to leakage, sampling bias, fairness, privacy, monitoring, and the practical consequences of decisions.

Key Terms

Scalar: A single numerical value.

Vector: An ordered array of numerical values.

Matrix: A rectangular arrangement of values.

Dot product: The sum of pairwise products of two equal-length vectors.

Norm: A function measuring the size of a vector.

Mean: The arithmetic average of observations.

Variance: The average squared deviation from a mean, with denominator depending on whether data are a population or sample.

Standard deviation: The square root of variance.

Z-score: A standardized value expressed in standard-deviation units.

Covariance: A measure of joint variation between two variables.

Correlation: A unit-free measure of linear association.

Conditional probability: The probability of an event given another event.

Bayes’ theorem: A rule for updating prior probability using evidence.

Random variable: A numerical function of uncertain outcomes.

Expected value: The probability-weighted average of a random variable.

Confidence interval: An interval produced by a procedure designed to capture a population parameter at a stated long-run rate.

Gradient: A vector of partial derivatives.

Loss function: A numerical measure of prediction error.

Regularization: A penalty used to control model complexity.

Data leakage: Use of information during training that would not be available at prediction time.

Review Questions

Short Answer Questions

1. Why can feature scaling affect distance-based models and gradient descent?
2. Distinguish population variance from sample variance.
3. State Bayes’ theorem and explain the meaning of its terms.
4. Why does correlation not establish causation?
5. What role does the learning rate play in gradient descent?

Long Answer Questions

1. Explain how vectors, matrices, dot products, and norms are used in a linear prediction model.
2. Compare the mean, median, variance, standard deviation, and IQR, including their sensitivity to outliers.
3. Explain the difference between a confidence interval and a p-value.
4. Describe gradient descent, including its update rule, assumptions, and common failure modes.
5. Compare ridge and lasso regularization and discuss their effects on model generalization.

Analytical / Application Questions

1. A binary classifier is evaluated on a dataset with 95% negative cases. Explain why accuracy alone may be misleading and recommend additional measures.
2. A model’s validation performance is excellent, but its deployment performance is poor. Analyze how sampling shift, leakage, preprocessing, and calibration could explain the difference.
3. Design a mathematically responsible workflow for a churn-prediction system, covering data splitting, descriptive analysis, probability, optimization, evaluation, fairness, and monitoring.

Multiple Choice Questions with Answers

Question 1

For vectors a and b of equal length, which expression represents their dot product?

A. ∑ⱼ aⱼ + bⱼ

B. ∑ⱼ aⱼbⱼ

C. √(∑ⱼ aⱼ²bⱼ²)

D. ∑ⱼ (aⱼ − bⱼ)²

Correct Answer: B. ∑ⱼ aⱼbⱼ

Question 2

For the observations 2, 4, 4, and 6, what is the arithmetic mean?

A. 3

B. 4

C. 4.5

D. 6

Correct Answer: B. 4

Question 3

Which statement about Pearson correlation is correct?

A. It proves that one variable causes another.

B. It can only take positive values.

C. It measures linear association and is unit-free.

D. It is unaffected by outliers.

Correct Answer: C. It measures linear association and is unit-free.

Question 4

If P(A) = 0.2, P(B|A) = 0.5, and P(B) = 0.4, what is P(A|B)?

A. 0.10

B. 0.25

C. 0.40

D. 0.50

Correct Answer: B. 0.25

Question 5

For a Poisson random variable with rate λ = 3, what is its variance?

A. 1

B. 2

C. 3

D. 9

Correct Answer: C. 3

Question 6

What does the gradient of a differentiable function indicate locally?

A. The direction of greatest increase

B. The sample median

C. The probability of a class

D. The number of observations

Correct Answer: A. The direction of greatest increase

Question 7

Using θₜ₊₁ = θₜ − η∇J(θₜ), let θₜ = 2, η = 0.1, and ∇J(θₜ) = 5. What is θₜ₊₁?

A. 1.5

B. 1.95

C. 2.5

D. 7

Correct Answer: A. 1.5

Question 8

Which regularization penalty can encourage some coefficients to become exactly zero?

A. L1 penalty

B. L2 penalty only

C. No penalty

D. A normalization by the sample mean

Correct Answer: A. L1 penalty

Question 9

What is data leakage?

A. Random variation in a sample

B. Using prediction-time-unavailable information during training

C. A model having too few parameters

D. A distribution with finite variance

Correct Answer: B. Using prediction-time-unavailable information during training

Question 10

A 95% confidence interval is best interpreted as which statement?

A. There is exactly a 95% probability that the fixed parameter lies in this interval.

B. The null hypothesis has a 95% probability of being true.

C. The repeated-sampling procedure captures the parameter approximately 95% of the time under its assumptions.

D. Exactly 95% of observations lie inside the interval.

Correct Answer: C. The repeated-sampling procedure captures the parameter approximately 95% of the time under its assumptions.