Mathematics and Statistics for ML

Introduction

Machine learning models learn relationships from data, but the language used to describe those relationships is mathematics. Vectors represent observations and features, matrices represent datasets and transformations, functions describe models, probability represents uncertainty, statistics summarizes evidence, and calculus supports learning through optimization. A practical understanding of these ideas enables an AI practitioner to interpret model behaviour, diagnose errors, select suitable algorithms, and communicate results responsibly.

This chapter develops the mathematical and statistical foundations required for introductory machine learning. The emphasis is not on abstract manipulation alone, but on understanding what quantities mean, when formulas apply, and how they influence practical model design.

Learning Objectives

By the end of this chapter, you should be able to:

• represent data using scalars, vectors and matrices;
• interpret functions, parameters and transformations in ML models;
• calculate and interpret probability, common distributions and descriptive statistics;
• distinguish covariance from correlation;
• calculate common distance and similarity measures;
• explain derivatives, loss functions and optimization;
• perform a simple gradient-descent update;
• identify assumptions, limitations and ethical risks in quantitative modelling.

1. Mathematical Representation of Data

A scalar is a single value, such as a person’s age or a model’s accuracy. A vector is an ordered list of values. If an observation has height, weight and age, it may be represented as

x = [h, w, a]^T

where T denotes transpose. A vector is an element of a vector space and has both magnitude and direction, although the interpretation of direction depends on the application.

A dataset containing n observations and p features is commonly represented by a matrix X ∈ R^n×p. The entry x_ij is the value of feature j for observation i. The target values may be stored in a vector y.

Vector operations

For vectors of equal length, addition is performed component by component:

Multiplication by a scalar c changes each component:

The dot product is

It produces a scalar and is central to linear models. The Euclidean norm or length of a vector is

The dot product can also be written as u · v = ‖u‖‖v‖cos θ, where θ is the angle between the vectors. Thus, it measures both size and directional alignment.

Matrices and linear transformations

A matrix can transform a vector, combine features, or store model parameters. Matrix multiplication is defined only when the inner dimensions agree. If A has dimensions m×n and B has dimensions n×q, then AB has dimensions m×q.

For a linear regression model with p features,

where w is a weight vector, b is an intercept, and ŷ is the prediction. For many observations this becomes

where 1 is an all-ones vector. The model is linear in its parameters, even if features have been transformed.

A matrix may have an inverse only when it is square and nonsingular. If A^-1 exists, AA^-1=I, where I is the identity matrix. In practice, direct matrix inversion is often avoided because it may be numerically unstable; algorithms commonly use matrix factorization or iterative methods instead.

Scaling and representation

Features measured in different units can distort distance calculations and optimization. Two common transformations are:

where μ is the training-set mean and σ is the training-set standard deviation, and min–max scaling:

Standardization is suitable when a feature is approximately symmetric and extreme values are not dominant. Scaling parameters must be calculated using training data only; otherwise information from the test set leaks into the model.

2. Functions and Model Behaviour

A function maps inputs to outputs. In ML, a model is often a parameterized function:

where x is the input, θ contains parameters, and ŷ is the prediction. A function may be linear, polynomial, piecewise, or nonlinear.

For binary classification, the logistic function converts a real-valued score into a probability:

Its output lies strictly between 0 and 1. A classifier may predict class 1 when σ(z) ≥ 0.5, although another threshold may be appropriate when the costs of errors differ.

The composition of functions allows layered models. A neural network can be described generally as

f(x) = f_L(…f₂(f₁(x))).

Changing a parameter changes the function’s output. Sensitivity to such changes is studied using derivatives.

3. Probability and Random Variables

Probability quantifies uncertainty. For an event A, 0 ≤ P(A) ≤ 1. If A and B are mutually exclusive, then P(A ∪ B)=P(A)+P(B). The complement rule is

The conditional probability of A given B, provided P(B)>0, is

The multiplication rule is P(A ∩ B)=P(A|B)P(B). Two events are independent if P(A|B)=P(A), equivalently P(A ∩ B)=P(A)P(B). Independence must not be assumed merely because two variables appear unrelated.

Bayes’ theorem is

In classification, it explains why a positive test or predicted label must be interpreted alongside the base rate of the class.

A random variable assigns numerical values to outcomes. A discrete random variable has countable outcomes; a continuous random variable can take values over an interval. Its expected value is the probability-weighted long-run average:

for a discrete variable. Its variance is

where μ=E[X]. Standard deviation is σ=√Var(X). Variance is measured in squared units, while standard deviation has the original units.

4. Important Distributions

A Bernoulli distribution models one trial with success probability p. If X ∈ {0,1}, then P(X=1)=p and P(X=0)=1−p. It is useful for a single binary outcome.

A binomial distribution counts successes in n independent Bernoulli trials with constant success probability p:

where k is an integer from 0 to n.

The normal distribution is continuous, symmetric and characterized by mean μ and standard deviation σ. Its density is

The empirical rule states that approximately 68%, 95% and 99.7% of observations lie within one, two and three standard deviations of the mean, respectively, when the distribution is normal. This is an approximation, not a universal law.

The central limit theorem states, under suitable conditions such as independent observations with finite variance, that the sampling distribution of the sample mean approaches a normal distribution as sample size increases. It does not state that the original data must be normally distributed.

5. Descriptive Statistics

Descriptive statistics summarize observed data. For values x₁,…,x_n, the sample mean is

The median is the middle ordered value, or the average of the two middle values when n is even. The mode is the most frequent value. The mean is sensitive to outliers; the median is generally more robust.

The sample variance is

The denominator n−1 gives an unbiased estimator of population variance under standard random-sampling assumptions. The population variance instead divides by n.

The range is maximum minus minimum. The interquartile range is IQR=Q₃−Q₁, where Q₁ and Q₃ are the first and third quartiles. A common outlier rule identifies values below Q₁−1.5IQR or above Q₃+1.5IQR, though this is a diagnostic convention rather than proof of an error.

Worked example: summary statistics

For the values 2, 4, 4 and 10, the mean is 20/4=5, while the median is 4. Deviations from the mean are −3, −1, −1 and 5; their squared sum is 36. Therefore the sample variance is 36/(4−1)=12, and the sample standard deviation is approximately 3.46. The difference between mean and median indicates right-skew caused by the relatively large value 10.

6. Covariance, Correlation and Dependence

Covariance indicates whether two variables tend to increase or decrease together:

Its units are the product of the two variables’ units, making magnitude difficult to compare across datasets.

Pearson correlation standardizes covariance:

where s_x and s_y are sample standard deviations. It lies between −1 and 1. Values near 1 indicate strong positive linear association; values near −1 indicate strong negative linear association; values near 0 indicate little linear association.

Correlation does not establish causation. It can be distorted by outliers, nonlinear relationships, restricted ranges and confounding variables. A correlation near zero does not rule out a strong nonlinear relationship. For ordinal or non-normal data, rank-based measures such as Spearman correlation may be more suitable.

7. Distance and Similarity Measures

Distance measures are used in clustering, nearest-neighbour algorithms, anomaly detection and recommendation systems.

The Euclidean distance between vectors x and y is

It is appropriate when straight-line geometry and comparable feature scales are meaningful. Manhattan distance is

and can be less affected by a single large coordinate difference. The general Minkowski distance is

Cosine similarity compares direction rather than magnitude:

It requires nonzero vectors. It is widely used for text representations, where document length may be less important than word-pattern direction. Scaling and feature encoding strongly influence every distance-based method.

8. Derivatives and Optimization

A derivative measures the instantaneous rate of change of a function. For f(x)=x²,

The derivative is the slope of the tangent. A positive derivative means the function increases locally; a negative derivative means it decreases locally. The chain rule for a composite function is

For multiple parameters, the gradient is the vector of partial derivatives:

where J(θ) is a loss or objective function. The gradient points in the direction of steepest local increase. Therefore, minimizing the objective uses the negative gradient.

Least-squares loss

For predictions ŷ_i and observed values y_i, mean squared error is

The factor of 1/2 is sometimes included for algebraic convenience:

It does not change the minimizing parameters, only the scale of the gradient.

Gradient descent

Gradient descent updates parameters according to

where η>0 is the learning rate and t is the iteration number. A very small learning rate makes learning slow; a very large one can cause oscillation or divergence. Batch gradient descent uses all training examples, stochastic gradient descent uses one example at a time, and mini-batch gradient descent uses a subset.

#### Worked example: one update

Suppose J(θ)=(θ−3)², the current parameter is θ₀=0, and η=0.1. Since

dJ/dθ=2(θ−3),

at θ₀=0 the gradient is −6. Thus

θ₁=0−0.1(−6)=0.6.

The update moves toward the minimum at θ=3. Repeated updates approach the minimum when the learning rate is suitably chosen.

Optimization may stop when the change in loss, parameter change, or gradient norm becomes sufficiently small. A local minimum is not necessarily a global minimum for a non-convex objective. Regularization adds a penalty, such as λ‖w‖₂², to discourage overly complex parameter values; λ≥0 controls its strength.

9. Integrated Applied Case: Predicting Customer Churn

This is an illustrative case. A telecommunications company wants to identify customers likely to cancel service.

Background and problem

The dataset contains monthly charge, contract length, support contacts, data usage and a binary churn label. The business wants useful early warnings, but an inaccurate system could lead to unfair retention offers or unnecessary contact.

Analysis

The data team first inspects missing values, distributions and class balance. Monthly charge is right-skewed, so the median is compared with the mean. Features are standardized using training-set means and standard deviations. Contract type is encoded carefully; arbitrary numerical codes must not imply a false ordering.

Correlation is examined as a screening tool, not as evidence of causation. Distance-based methods would be inappropriate without scaling because monthly charge and support contacts use different units. A logistic model provides an interpretable baseline:

P(Y=1|x)=σ(wᵀx+b).

The data is divided into training, validation and test sets. The test set is used only for final evaluation. If false negatives are costly, the decision threshold may be lowered, but precision, recall and the business costs of errors must be considered rather than relying only on accuracy.

Approach and outcome

The model is trained by minimizing a suitable classification loss with an iterative optimizer. Performance is evaluated across demographic and contract groups. Suppose the model performs well overall but has substantially lower recall for customers in a small group. The team investigates sample size, missingness, proxy variables and threshold effects before deployment. The appropriate outcome is not simply a high score; it is a validated, monitored system with documented limitations and a human review process for consequential actions.

Learning points

The case demonstrates that mathematical preparation affects the entire workflow: descriptive statistics reveal data quality, scaling affects optimization, probability supports calibrated interpretation, derivatives enable learning, and statistical evaluation reveals whether performance is reliable across groups.

10. Limitations, Governance and Best Practice

Mathematical models are simplified representations. Their conclusions depend on data quality, sampling, measurement choices and assumptions. A precise calculation can still be misleading if the data is biased or the target is poorly defined.

Best practice includes:

• define variables, units and targets before modelling;
• separate training, validation and test data;
• fit preprocessing transformations only on training data;
• inspect distributions and outliers rather than removing them automatically;
• use plots and summary statistics together;
• report uncertainty and class-specific performance;
• test robustness under distribution shift;
• document assumptions, transformations and known limitations;
• protect personal data and restrict access to sensitive attributes;
• assess fairness, explainability and contestability when predictions affect people.

Contemporary ML increasingly uses automatic differentiation, probabilistic modelling, uncertainty estimation and fairness-aware evaluation. These tools do not eliminate the need for foundational mathematics; they make understanding assumptions even more important.

Chapter Summary

Vectors and matrices provide the basic language for representing observations, features and model parameters. Dot products and matrix multiplication support linear prediction and transformations. Functions describe how inputs become predictions, while probability provides a framework for uncertainty and conditional reasoning. Distributions such as Bernoulli, binomial and normal models help describe data and sampling behaviour. Descriptive statistics summarize centre, spread and shape, but must be interpreted with attention to outliers and sampling assumptions. Covariance and correlation describe association without proving causation. Distance measures support similarity-based algorithms but require appropriate scaling and representations. Derivatives and gradients describe how objectives change. Gradient descent uses the negative gradient to iteratively reduce loss. Sound ML practice combines these mathematical tools with validation, ethical review, privacy protection and monitoring.

Key Terms

Scalar: A single numerical value.

Vector: An ordered list of values.

Matrix: A rectangular array of values.

Dot product: Sum of pairwise products of two equal-length vectors.

Norm: A measure of vector magnitude.

Function: A mapping from inputs to outputs.

Parameter: A value learned or selected by a model.

Probability: A numerical measure of uncertainty.

Conditional probability: Probability of an event given another event.

Random variable: A numerical representation of uncertain outcomes.

Expected value: Probability-weighted average of a random variable.

Variance: Average squared deviation from the mean.

Standard deviation: Square root of variance.

Correlation: Standardized measure of linear association.

Euclidean distance: Straight-line distance between vectors.

Derivative: Instantaneous rate of change.

Gradient: Vector of partial derivatives.

Loss function: Quantity measuring prediction error.

Gradient descent: Iterative optimization using the negative gradient.

Data leakage: Use of information during training that would not be available at prediction time.

Review Questions

5 Short Answer Questions

1. What is the difference between a scalar, vector and matrix?
2. Why should features often be scaled before using a distance-based algorithm?
3. Distinguish covariance from correlation.
4. What does a gradient represent in optimization?
5. Why does correlation not prove causation?

5 Long Answer Questions

1. Explain how vectors, matrices and dot products are used in linear regression.
2. Compare the mean, median, variance, standard deviation and interquartile range, including their sensitivity to outliers.
3. Explain conditional probability, independence and Bayes’ theorem in an ML context.
4. Compare Euclidean, Manhattan and cosine-based measures and describe suitable applications for each.
5. Explain gradient descent, including the roles of the gradient, learning rate, initialization and stopping criteria.

3 Analytical / Application Questions

1. A dataset contains income in currency units and age in years. Explain how you would preprocess these variables before applying k-nearest neighbours, and justify your decisions.
2. A classifier has high accuracy but very low recall for a minority group. Identify possible statistical and governance causes and propose an evaluation plan.
3. For J(θ)=(θ−5)², start at θ₀=1 with learning rate η=0.2. Calculate the first two gradient-descent updates and explain whether the parameter moves toward the minimum.

Multiple Choice Questions with Answers

Question 1

Which operation produces a scalar from two equal-length vectors?

A. Vector addition

B. Dot product

C. Matrix inversion

D. Feature scaling

Correct Answer: B. Dot product

Question 2

For a sample of values, which measure is generally most robust to a single extreme outlier?

A. Median

B. Mean

C. Variance

D. Range

Correct Answer: A. Median

Question 3

If P(B) > 0, which expression defines conditional probability?

A. P(A) + P(B)

B. P(A)P(B)

C. P(A ∩ B)/P(B)

D. 1 − P(B)

Correct Answer: C. P(A ∩ B)/P(B)

Question 4

What is the main purpose of standardization?

A. To convert labels into classes

B. To place features on a comparable scale

C. To guarantee normality

D. To remove all outliers

Correct Answer: B. To place features on a comparable scale

Question 5

A Pearson correlation close to zero indicates:

A. No possible relationship of any kind

B. Perfect negative association

C. Little linear association

D. That one variable causes the other

Correct Answer: C. Little linear association

Question 6

Which distance is based on the sum of absolute coordinate differences?

A. Euclidean distance

B. Manhattan distance

C. Cosine distance

D. Minkowski distance with r = 2 only

Correct Answer: B. Manhattan distance

Question 7

In gradient descent, parameters are updated in which direction for minimization?

A. The positive gradient direction

B. The negative gradient direction

C. A random direction only

D. The direction of maximum parameter value

Correct Answer: B. The negative gradient direction

Question 8

For J(θ)=(θ−3)², what is the gradient at θ=0?

A. −6

B. −3

C. 3

D. 6

Correct Answer: A. −6

Question 9

Which distribution models the number of successes in a fixed number of independent trials with constant success probability?

A. Normal

B. Binomial

C. Uniform

D. Exponential

Correct Answer: B. Binomial

Question 10

Why must test-set information generally be excluded when fitting a feature scaler?

A. It makes matrix multiplication impossible

B. It causes data leakage

C. It guarantees underfitting

D. It changes labels into probabilities

Correct Answer: B. It causes data leakage