Mathematical and Statistical Foundations

Introduction

Machine learning is applied mathematics expressed through data. Algorithms estimate patterns, measure similarity, quantify uncertainty, and improve predictions by minimizing error. A data scientist therefore needs more than the ability to call a software library: the practitioner must understand what a numerical result means, which assumptions support it, and when an apparently accurate model may be misleading.

This chapter develops the mathematical and statistical foundations required for introductory machine learning. It covers descriptive statistics, probability, probability distributions, correlation, vectors and matrices, distance measures, calculus, and optimization. The emphasis is on interpretation and practical use rather than formal proof. Examples use small datasets so that calculations can be checked by hand.

Learning Objectives

By the end of this chapter, you should be able to:

• summarize data using appropriate descriptive statistics;
• distinguish population quantities from sample estimates;
• apply basic probability rules and interpret conditional probability;
• identify common probability distributions and their machine-learning uses;
• calculate and interpret covariance and correlation;
• represent data with vectors and matrices;
• select suitable distance measures for different data types;
• explain derivatives, gradients, and partial derivatives;
• describe optimization through objective functions and gradient descent;
• recognize assumptions, limitations, scaling issues, and ethical risks in quantitative analysis.

1. Data, Variables, and Notation

An observation is one record, such as one customer or one transaction. A variable is a measurable attribute, such as age, income, or purchase status. A dataset with n observations and p features can be represented by a matrix X with n rows and p columns. The value of feature j for observation i is commonly written xᵢⱼ. A target variable, written y, is the quantity to be predicted.

Variables may be numerical or categorical. Numerical variables can be discrete, such as the number of purchases, or continuous, such as temperature. Categorical variables represent groups, such as payment method. Nominal categories have no natural order; ordinal categories do. Encoding a category as 1, 2, and 3 does not automatically make it a numerical quantity.

A population is the complete set of units of interest, whereas a sample is the subset observed. A population mean is often written μ and a population standard deviation σ. Sample estimates are commonly written x̄ and s. Confusing these two levels can lead to incorrect claims about uncertainty.

2. Descriptive Statistics

Descriptive statistics organize and summarize observed data. They do not, by themselves, establish causation or prove that a sample represents a population.

2.1 Measures of location

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here n is the number of observations. The mean is appropriate for numerical data when values are meaningfully additive. It is sensitive to extreme values.

The median is the middle value after sorting. If n is odd, it is the value at position (n + 1)/2. If n is even, it is usually the average of the two middle values. The median is more resistant to outliers. The mode is the most frequent value and is particularly useful for categorical data.

Worked example: mean and median

Suppose daily orders are 4, 5, 5, 6, and 20. The mean is (4 + 5 + 5 + 6 + 20)/5 = 40/5 = 8. The median is 5. The unusually large value 20 raises the mean substantially, so the median better represents a typical day in this small dataset.

2.2 Measures of spread

The range is maximum minus minimum. It is simple but depends only on two observations. The interquartile range is:

where Q₁ is the first quartile and Q₃ is the third quartile. The IQR describes the middle 50% of observations and is relatively resistant to extreme values. Quartile conventions can differ slightly for small samples, so a report should document the convention used.

Population variance is:

Sample variance is:

N is the population size, μ is the population mean, n is the sample size, and x̄ is the sample mean. The n − 1 denominator is Bessel’s correction; it provides an unbiased estimator of population variance under standard random-sampling assumptions. Variance has squared units, while standard deviation returns to the original units:

For the sample 2, 4, and 6, x̄ = 4. Deviations are −2, 0, and 2; squared deviations sum to 8. Thus s² = 8/(3 − 1) = 4 and s = 2.

2.3 Standardization and z-scores

A z-score expresses a value relative to a mean in standard-deviation units:

When a sample is used, x̄ and s replace μ and σ. A z-score has no units. Standardization is useful when features have different scales, although it does not make a skewed distribution normal. It is calculated using training-set statistics only; using the full dataset before splitting causes data leakage.

2.4 Shape and outliers

Skewness describes asymmetry. Right-skewed data have a long upper tail and often have mean greater than median. Left-skewed data have a long lower tail. A histogram, box plot, or empirical quantile plot should accompany numerical summaries when distribution shape matters.

A common exploratory rule identifies potential outliers below Q₁ − 1.5 IQR or above Q₃ + 1.5 IQR. This is a screening rule, not proof that an observation is erroneous. An extreme value may be a valid rare event, a measurement error, or evidence of a separate subgroup. Removing it without investigation can create bias.

3. Probability and Uncertainty

Probability quantifies uncertainty. A sample space Ω contains possible outcomes, and an event A is a subset of Ω. The probability P(A) satisfies 0 ≤ P(A) ≤ 1, P(Ω) = 1, and for mutually exclusive events A and B, P(A ∪ B) = P(A) + P(B).

The complement rule is:

The general addition rule is:

The subtraction prevents the overlap from being counted twice. Conditional probability is:

Events are independent when P(A | B) = P(A), equivalently P(A ∩ B) = P(A)P(B). Independence must not be assumed merely because two variables appear unrelated.

Bayes’ theorem updates a prior probability using evidence:

In classification, A may be a class and B an observed feature pattern. The prior P(A) is the class prevalence, P(B | A) is the likelihood, and P(A | B) is the posterior probability. A rare disease can produce many false positives even when a medical test has high sensitivity and specificity because the base rate is low.

Expected value and variance

For a discrete random variable X with possible values xᵢ and probabilities pᵢ:

where ∑ᵢpᵢ = 1. Its variance is:

Expected value is a long-run average, not necessarily an outcome that must occur. For random variables X and Y, linearity gives E[aX + bY] = aE[X] + bE[Y] for constants a and b, whether or not X and Y are independent.

4. Probability Distributions

A probability mass function assigns probabilities to discrete outcomes. A probability density function describes continuous variables; the probability of an exact single continuous value is generally zero, and probabilities are found over intervals by area under the density. A cumulative distribution function gives P(X ≤ x).

4.1 Bernoulli and binomial distributions

A Bernoulli variable records one trial with success probability p: X = 1 for success and X = 0 for failure. Its mean is p and variance is p(1 − p). If n independent trials share the same p, the number of successes K follows a binomial distribution:

The binomial model requires a fixed number of trials, two outcomes, constant success probability, and independence. These conditions may fail when customer actions influence one another or probabilities change over time.

4.2 Normal distribution

A normal distribution is characterized by mean μ and standard deviation σ. Its density is:

It is symmetric and bell-shaped. The empirical 68–95–99.7 rule states that approximately 68%, 95%, and 99.7% of observations lie within 1, 2, and 3 standard deviations of the mean when the distribution is normal. This is a model-based approximation, not a universal law.

The central limit theorem explains why sample means can be approximately normal under broad conditions when sample size is sufficiently large and observations are independent with finite variance. It concerns the distribution of a statistic across repeated samples, not necessarily the shape of the raw data.

4.3 Other useful distributions

The Poisson distribution models counts in a fixed interval when events occur independently at a constant average rate λ:

Its mean and variance are both λ. Real event data may show overdispersion, dependence, or changing rates, making a basic Poisson model unsuitable. The exponential distribution is often used for waiting times between Poisson events, but its memoryless assumption is restrictive.

5. Covariance, Correlation, and Dependence

Covariance measures the direction in which two variables vary together. Sample covariance is:

Positive covariance indicates that high values of one variable tend to accompany high values of the other. Its magnitude depends on measurement units, so it is difficult to compare across pairs.

Pearson correlation standardizes covariance:

provided sₓ and sᵧ are nonzero. It lies between −1 and 1 and measures linear association. A value near zero does not rule out a nonlinear relationship. Correlation is also sensitive to outliers and does not establish causation. Confounding variables, selection bias, and reverse causality can all produce misleading relationships.

Spearman’s rank correlation applies Pearson correlation to ranks and is often more suitable for monotonic relationships or ordinal data. Correlation matrices are useful for exploratory analysis, but highly correlated predictors can create multicollinearity in regression, making individual coefficient interpretation unstable.

6. Linear Algebra for Machine Learning

A scalar is a single number. A vector is an ordered list, such as x = (x₁, x₂, …, xₚ). A matrix is a rectangular array. In a design matrix X, rows usually represent observations and columns represent features. Convention must be stated because some mathematical treatments use the opposite orientation.

Vector addition requires equal dimensions. Scalar multiplication multiplies every component. The dot product is:

For x = (2, 3) and y = (4, 1), x · y = 2×4 + 3×1 = 11. The Euclidean norm is:

The dot product can also be written ||x||||y||cos θ, where θ is the angle between nonzero vectors. A zero vector has no defined direction.

Matrix multiplication is defined for A of size m×p and B of size p×q, producing an m×q matrix:

The inner dimensions must match, and generally AB ≠ BA. The transpose Xᵀ exchanges rows and columns. An identity matrix I leaves compatible vectors unchanged: Ix = x.

A linear model can be written:

where X is the feature matrix, β is a coefficient vector, and ŷ is the vector of predictions. An intercept is included by adding a column of ones to X. In ordinary least squares, a common objective is:

The objective is minimized over β. The normal-equation solution β̂ = (XᵀX)⁻¹Xᵀy applies only when XᵀX is invertible and the usual least-squares conditions are appropriate. In practice, numerical solvers are often preferred because explicitly computing an inverse can be unstable or inefficient.

7. Distance and Similarity Measures

Distance measures quantify dissimilarity. Euclidean distance between vectors x and y is:

It is suitable for continuous features when scales are comparable. Manhattan distance is:

It can be useful on grid-like spaces and is less dominated by one large coordinate than squared Euclidean distance. Minkowski distance generalizes both:

Cosine similarity measures orientation rather than magnitude:

It is frequently used for text vectors. It is undefined when either vector is zero. Hamming distance counts positions at which two equal-length categorical or binary strings differ. For mixed data, a single unexamined distance formula may be inappropriate; feature-specific distances or carefully designed preprocessing are needed.

Distance-based algorithms such as k-nearest neighbours, k-means, and some anomaly detectors are highly sensitive to feature scale. Standardization, robust scaling, or domain-based weighting should be fitted on training data and documented.

8. Calculus and Optimization

Calculus describes change. For a function f(x), the derivative at x is:

when the limit exists. It is the slope of the tangent and indicates how rapidly the output changes with the input. For f(x) = x², f′(x) = 2x. At x = 3, the slope is 6.

A partial derivative measures change in a multivariable function while holding other variables constant. The gradient collects partial derivatives:

The gradient points in the direction of steepest local increase. Therefore, gradient descent updates parameters in the opposite direction:

θ is the parameter vector, η > 0 is the learning rate, and J is the objective function. The update assumes a differentiable objective, or a suitable subgradient method for some nondifferentiable objectives.

For J(θ) = θ², the derivative is 2θ. Starting at θ₀ = 4 with η = 0.1 gives θ₁ = 4 − 0.1(8) = 3.2. The parameter moves toward the minimum at θ = 0. If η is too small, learning is slow; if too large, updates may oscillate or diverge. A stopping rule may use a maximum number of iterations, a small improvement in objective value, or a small gradient norm.

Batch gradient descent uses all training observations per update. Stochastic gradient descent uses one observation, and mini-batch gradient descent uses a small subset. Stochastic methods can be faster for large datasets but introduce noisy updates. Convex objectives have no suboptimal local minima, while non-convex objectives may contain local minima, saddle points, and flat regions. A trained model is not automatically globally optimal merely because an algorithm stopped.

Regularization modifies the objective to control complexity. Ridge regression adds an L2 penalty:

Lasso adds an L1 penalty:

λ ≥ 0 controls penalty strength. The intercept is commonly excluded from penalization. L2 tends to shrink coefficients toward zero; L1 can produce exact zeros and therefore a sparse model. Penalties do not remove the need for validation, sensible features, and leakage prevention.

9. Integrated Case: Predicting Delivery Delays

Background

The following is a fictional illustrative case. A logistics company wants to predict whether an order will be delivered late so that customers can be notified proactively.

Problem

The data team has delivery distance, package weight, weather category, dispatch hour, and a binary late indicator. Distances range from 1 to 900 kilometres, while weather is categorical. Management initially proposes using raw Euclidean distance in a nearest-neighbour model and judging success only by accuracy.

Analysis

Exploration shows that distance is strongly right-skewed, and a few extreme routes are legitimate rather than erroneous. The late class represents only 8% of orders. Accuracy can therefore appear high if a model predicts “on time” for nearly every order. Correlation between distance and lateness is positive but does not prove that distance causes delay; weather, depot workload, and route type may confound the relationship.

Approach and solution

The team first separates training, validation, and test data before fitting transformations. Numerical features are standardized using training means and standard deviations. Weather is one-hot encoded without assigning artificial numerical order. A baseline always predicts the majority class. A logistic classifier and a k-nearest-neighbour model are compared using precision, recall, F1 score, and a precision–recall curve. The decision threshold is selected with operations staff because missing a likely late delivery may be more costly than sending an unnecessary warning.

The k-nearest-neighbour model uses standardized numerical features and an explicitly documented mixed-data treatment. Hyperparameters are selected on validation data, and the final test set is used once for an unbiased estimate. Probability calibration is checked because a warning system needs meaningful risk estimates, not only rankings.

Outcome

The final model improves recall over the baseline while maintaining an acceptable number of false alerts. The company uses the predictions as decision support rather than automatically penalizing delivery staff. Monitoring is established for changes in route patterns, weather frequencies, missingness, and class prevalence.

Learning points

The case demonstrates that mathematical foundations affect the entire workflow: descriptive statistics reveal skew and imbalance; probability clarifies base rates; encoding and scaling affect distances; correlation supports exploration but not causal claims; optimization fits model parameters; and evaluation must reflect the operational cost of errors. Governance is also essential: customers and workers should not be subjected to opaque or inaccurate decisions without review and an opportunity to challenge consequential outcomes.

10. Best Practices, Limitations, and Governance

1. Inspect distributions, missingness, units, and outliers before modelling.
2. Use robust summaries when data are skewed, and report the chosen definitions of quartiles and variance.
3. Fit imputation, scaling, feature selection, and encoding procedures on training data only.
4. Match probability models to their assumptions; independence and constant rates are empirical questions.
5. Treat correlation as association, not causation.
6. Check numerical conditioning and avoid unnecessary matrix inversion.
7. Select distance measures according to data type, scale, and domain meaning.
8. Tune optimization settings on validation data and monitor convergence.
9. Report uncertainty, class imbalance, subgroup performance, and important failure modes.
10. Protect privacy and avoid using sensitive attributes or proxies in ways that create discrimination.

Mathematical accuracy does not guarantee a fair or useful system. Sampling bias, measurement error, historical discrimination, and unequal access to data can be embedded in a model. Documentation should identify data provenance, intended use, assumptions, limitations, and responsible owners. Reproducible code, versioned datasets, and auditable evaluation are practical controls.

Chapter Summary

Descriptive statistics summarize location, spread, and shape, while careful interpretation distinguishes resistant summaries from outlier-sensitive ones. Probability provides rules for uncertainty, including conditional probability and Bayes’ theorem. Bernoulli, binomial, normal, and Poisson distributions are useful models, but each has assumptions. Covariance and correlation describe association and cannot establish causation.

Vectors and matrices provide the language of features, predictions, and linear models. Dot products, norms, and matrix multiplication support many algorithms. Euclidean, Manhattan, Minkowski, cosine, and Hamming measures serve different data structures and require attention to scale. Derivatives and gradients describe change, and gradient descent minimizes objectives through iterative updates. Regularization controls complexity but does not replace sound validation. Across all applications, assumptions, leakage prevention, uncertainty, fairness, and operational consequences determine whether mathematical results become responsible machine-learning practice.

Key Terms

Descriptive statistics: Numerical and graphical methods for summarizing observed data.

Mean: Arithmetic average of numerical observations.

Median: Middle ordered value, or average of the two middle values.

Variance: Average squared deviation from a mean, with a population or sample form.

Standard deviation: Square root of variance, expressed in the original units.

Z-score: Standardized value measured in standard-deviation units from a mean.

Probability: Numerical measure of uncertainty ranging from 0 to 1.

Conditional probability: Probability of one event given that another event has occurred.

Bayes’ theorem: Rule for updating a probability using prior information and evidence.

Random variable: Variable whose value is determined by a random outcome.

Probability distribution: Specification of probabilities or density across possible values.

Covariance: Measure of joint directional variation between two variables.

Correlation: Unit-free measure of linear association.

Vector: Ordered numerical list representing a point, feature set, or parameter set.

Matrix: Rectangular arrangement of numbers used to represent structured data and transformations.

Norm: Measure of vector magnitude.

Distance measure: Function quantifying dissimilarity between observations.

Gradient: Vector of partial derivatives of a multivariable function.

Objective function: Quantity that an algorithm seeks to minimize or maximize.

Regularization: Penalty added to an objective to control model complexity.

Review Questions

Short Answer Questions

1. Why can the median be more informative than the mean for a right-skewed dataset?
2. State the difference between a population variance and a sample variance.
3. What does conditional probability measure?
4. Why must feature scaling be considered before using Euclidean distance?
5. What is the role of the learning rate in gradient descent?

Long Answer Questions

1. Explain Bayes’ theorem and describe how base-rate imbalance can affect classification decisions.
2. Compare Pearson correlation and Spearman correlation, including situations in which each may be appropriate.
3. Explain how vectors, matrices, dot products, and norms are used in a linear prediction model.
4. Describe gradient descent, including its update rule, learning-rate issues, and stopping criteria.
5. Compare L1 and L2 regularization and discuss their effects on model coefficients.

Analytical / Application Questions

1. A dataset contains customer income values of 20, 22, 23, 25, and 110 thousand units. Calculate the mean and median, identify the likely effect of the extreme value, and recommend a suitable summary for a typical customer.
2. A fraud detector is applied to a population in which 1% of transactions are fraudulent. Explain why a high-accuracy model could still be operationally poor, and identify at least two evaluation measures that should be considered.
3. A nearest-neighbour model uses age in years and annual spending in currency units. Explain why raw Euclidean distance may be misleading and propose a leakage-safe preprocessing strategy.

Multiple Choice Questions with Answers

Question 1

Which statistic is generally most resistant to a single extreme outlier?

A. Mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

For a sample with n observations, the usual unbiased estimator of population variance uses which denominator?

A. n

B. n + 1

C. n − 1

D. 2n

Correct Answer: C. n − 1

Question 3

If P(A) = 0.3, what is P(Aᶜ)?

A. 0.3

B. 0.7

C. 1.3

D. −0.3

Correct Answer: B. 0.7

Question 4

Which statement about Pearson correlation is correct?

A. It proves that one variable causes another.

B. It measures linear association and is sensitive to outliers.

C. It is always positive.

D. It is defined when one variable has zero variance.

Correct Answer: B. It measures linear association and is sensitive to outliers.

Question 5

For vectors x = (2, 3) and y = (4, 1), what is x · y?

A. 5

B. 8

C. 11

D. 14

Correct Answer: C. 11

Question 6

Which distance is based on the sum of absolute coordinate differences?

A. Euclidean distance

B. Manhattan distance

C. Cosine distance

D. Hamming distance

Correct Answer: B. Manhattan distance

Question 7

What does the gradient of an objective function indicate locally?

A. The direction of steepest increase

B. The number of observations

C. The class prevalence

D. The median feature value

Correct Answer: A. The direction of steepest increase

Question 8

In gradient descent, the parameter update generally moves in which direction?

A. Along the positive gradient

B. Opposite to the gradient

C. Toward the largest parameter value

D. Randomly without reference to the objective

Correct Answer: B. Opposite to the gradient

Question 9

Which regularization penalty can encourage exact zero coefficients?

A. L1 penalty

B. L2 penalty

C. No penalty

D. A mean-centering operation

Correct Answer: A. L1 penalty

Question 10

Why should scaling parameters be fitted using training data only?

A. To increase the number of features

B. To prevent information from the validation or test data leaking into training

C. To guarantee a normal distribution

D. To make all categorical variables ordinal

Correct Answer: B. To prevent information from the validation or test data leaking into training