Mathematical and Statistical Foundations

Introduction

Data science converts observations into descriptions, predictions, and decisions. Although software packages perform many calculations automatically, reliable analysis requires an understanding of the mathematical and statistical ideas behind those calculations. Linear algebra provides a language for representing data and transformations; functions describe relationships between variables; statistics summarizes data and quantifies variation; probability provides a framework for uncertainty; sampling connects observed data to wider populations; correlation helps identify association; and optimization supports model fitting.

These foundations are not merely computational tools. They help an analyst decide which method is appropriate, recognize misleading results, communicate assumptions, and evaluate whether a model is useful in practice.

Learning Objectives

By the end of this chapter, you should be able to:

• represent data using scalars, vectors, matrices, and tensors;
• interpret functions, linear transformations, and common model forms;
• calculate and interpret descriptive statistics;
• apply basic probability rules and probability distributions;
• distinguish populations, samples, parameters, and statistics;
• interpret covariance, correlation, and uncertainty responsibly;
• explain the purpose of optimization and gradient-based methods;
• identify assumptions, limitations, and ethical risks in quantitative analysis.

1. Mathematical Representation of Data

Scalars, vectors, matrices, and tensors

A scalar is a single numerical value, such as a temperature or a customer age. A vector is an ordered collection of values. If a customer is described by age, annual income, and number of purchases, the feature vector may be written as

The superscript T denotes transposition from a row representation to a column representation. A vector with p components belongs to the p-dimensional space ℝᵖ. A dataset containing n observations and p features is commonly represented by an n × p matrix X. Row i represents observation i, and column j represents feature j.

A matrix is a rectangular array of numbers. Matrix notation is useful because many operations on an entire dataset can be expressed compactly. A tensor generalizes vectors and matrices to three or more dimensions; for example, a collection of colour images can be represented using height × width × colour-channel dimensions, with an additional dimension for the number of images.

Vector operations

For vectors of equal length, addition is component-wise:

Scalar multiplication changes the magnitude of a vector:

The dot product of x and y is

It is defined for vectors with the same number of components and produces a scalar. The Euclidean length, or norm, of x is

The norm measures the distance from the origin. The distance between two vectors is ‖x − y‖₂. In machine learning, distances are meaningful only when features are measured on comparable scales or have been appropriately standardized.

The angle between non-zero vectors can be examined through

This cosine similarity lies between −1 and 1. It is often useful for text and high-dimensional sparse data, where direction may matter more than magnitude.

Matrices and transformations

If A is an m × p matrix and x is a p-dimensional vector, Ax is an m-dimensional vector. The number of columns of A must equal the number of components of x. Matrix multiplication is not generally commutative: AB may exist while BA does not, or the two products may differ.

A square matrix may have an inverse A⁻¹ satisfying

where I is the identity matrix. An inverse exists only when A is square and nonsingular. In numerical practice, solving Ax = b is generally preferable to explicitly calculating A⁻¹b because it is usually more stable and efficient.

A linear transformation preserves vector addition and scalar multiplication. Translation, such as adding an intercept in regression, is technically an affine rather than a purely linear transformation. This distinction is important when interpreting geometric models.

Eigenvalues and eigenvectors

For a square matrix A, a non-zero vector v is an eigenvector if

where λ is the corresponding eigenvalue. Eigenvectors identify directions that a transformation does not rotate, while eigenvalues indicate scaling in those directions. Principal component analysis uses eigenvectors of a covariance matrix to identify directions of greatest variance. In computation, eigenvalues may be estimated numerically, and their interpretation depends on the matrix being analyzed.

2. Functions and Model Relationships

A function maps each permitted input to exactly one output. In data science, a function may represent a prediction rule, a transformation, a probability density, or a loss function. A simple affine model is

where β₀ is the intercept, β₁ is the slope, x is the predictor, and y is the predicted response. The slope measures the predicted change in y for a one-unit increase in x, provided the relationship is appropriate and other relevant conditions are held constant.

For multiple predictors, linear regression is written as

Here ε represents unexplained variation. The model is linear in its coefficients, even if transformed predictors such as x² or log(x) are included. A regression equation is not automatically causal: association may arise from confounding, selection bias, or coincidence.

Common functions

• The exponential function f(x) = ae^bx models multiplicative growth or decay when a constant proportional rate is plausible.
• The logarithm is the inverse of exponentiation and converts multiplicative relationships into additive ones. It is defined for positive inputs.
• The logistic function is

It maps any real-valued z to the interval (0, 1), making it useful for modelling probabilities, although the output is a probability only when the model and calibration are appropriate.

A function may be continuous, discontinuous, monotonic, bounded, or unbounded. These properties affect optimization, interpolation, and interpretation. Extrapolation beyond the observed range is especially risky because the functional relationship may change outside the data.

3. Descriptive Statistics

Descriptive statistics summarize observed data; they do not, by themselves, establish general conclusions about a population.

For observations x₁, x₂, …, xₙ, the arithmetic mean is

The mean uses the same units as the observations and is sensitive to extreme values. The median is the middle ordered value, or the average of the two middle values when n is even. It is more resistant to outliers. The mode is the most frequently occurring value and is especially useful for categorical data.

The range is max(xᵢ) − min(xᵢ). The population variance is

where N is population size and μ is the population mean. For a sample, the conventional unbiased estimator of population variance is

The denominator n − 1 reflects the loss of one degree of freedom because x̄ is estimated from the same data. The standard deviation is the positive square root of variance: s = √s². Variance has squared units, whereas standard deviation has the original units.

The coefficient of variation is

usually reported as a percentage. It is meaningful for ratio-scale data with a meaningful zero and a non-zero positive mean; it is not suitable for variables such as temperature measured in Celsius, whose zero is arbitrary.

Quantiles and standardization

The pth quantile divides ordered data so that approximately p of the observations lie below it. The interquartile range is IQR = Q₃ − Q₁, where Q₁ and Q₃ are the first and third quartiles. It is resistant to extreme values and is commonly used in box plots.

A z-score standardizes an observation:

For a sample, x̄ and s are sample estimates. Standardization produces a dimensionless value and is useful when features have different units. It does not make a skewed variable normally distributed, nor does it remove outliers.

Worked example: summary statistics

Consider the daily numbers of support requests: 2, 3, 3, 4, and 8. The mean is x̄ = (2 + 3 + 3 + 4 + 8)/5 = 4. The median is 3, and the range is 6. Deviations from the mean are −2, −1, −1, 0, and 4; their squared sum is 22. Thus the sample variance is s² = 22/4 = 5.5, and the sample standard deviation is approximately 2.35 requests. The mean exceeds the median because the value 8 pulls the mean upward. Reporting both centre and spread gives a more informative description than reporting the mean alone.

4. Probability and Uncertainty

Probability quantifies uncertainty. A sample space Ω contains possible outcomes, and an event A is a subset of Ω. Probability obeys three basic principles: 0 ≤ P(A) ≤ 1, P(Ω) = 1, and for mutually exclusive events A and B, P(A ∪ B) = P(A) + P(B).

For any events A and B,

The conditional probability of A given B is

provided P(B) > 0. Events are independent when P(A ∩ B) = P(A)P(B), equivalently when P(A|B) = P(A) if the probabilities are defined.

Bayes’ theorem reverses conditional probabilities:

It is central to diagnosis, classification, risk assessment, and spam filtering. A common error is to confuse P(A|B) with P(B|A).

Random variables and expectation

A random variable assigns a numerical value to each outcome. A discrete random variable has countable possible values; a continuous random variable can take values over intervals. For a discrete variable X with probability mass function p(x), the expected value is

For a continuous variable with density f(x), the corresponding expression is E[X] = ∫ x f(x) dx, over the support of X. Expected value is a long-run average, not necessarily an outcome that can actually occur.

Variance can be expressed generally as

If a and b are constants, E[aX + b] = aE[X] + b and Var(aX + b) = a²Var(X).

Important distributions

The Bernoulli distribution models one trial with success probability p, where X is 1 for success and 0 otherwise. Its mean is p and variance is p(1 − p). The binomial distribution counts successes in n independent Bernoulli trials with constant success probability p:

where k = 0, …, n and C(n,k) = n!/[k!(n−k)!]. Its mean is np and variance is np(1 − p). Independence, a fixed number of trials, and constant p are required.

The Poisson distribution models counts in a fixed interval when events occur independently at a constant average rate λ:

Its mean and variance are both λ. The assumptions may fail when events cluster, inhibit one another, or have changing rates.

The normal distribution has density

where μ is the mean and σ > 0 is the standard deviation. It is symmetric and continuous. The empirical rule is approximate: about 68%, 95%, and 99.7% of observations lie within one, two, and three standard deviations of μ when the normal model is appropriate. The central limit theorem explains why standardized sample means often become approximately normal as sample size increases under suitable independence and finite-variance conditions; it does not state that every dataset is normally distributed.

5. Populations, Samples, and Sampling

A population is the complete set of units of interest. A sample is the subset observed. A population parameter, such as μ, describes the population; a sample statistic, such as x̄, is calculated from observed data and used to estimate the parameter.

Probability sampling methods include simple random sampling, stratified sampling, systematic sampling, and cluster sampling. Stratification can improve representation of important subgroups, while cluster sampling can reduce cost but may increase sampling variability when units within clusters are similar. Convenience sampling is easy but can produce serious selection bias.

The standard error of the sample mean, under independent observations with finite variance, is

When σ is unknown, it is estimated by s/√n. Increasing n reduces standard error at a square-root rate, not linearly. A 4-fold increase in sample size approximately halves the standard error under the same assumptions.

A confidence interval is an interval-producing procedure with a long-run coverage property. A commonly used approximate interval for a mean when σ is unknown is

where t is a critical value from the t distribution, α is the error level, and n − 1 is the degrees of freedom. A 95% confidence procedure has 95% long-run coverage under its assumptions; it is not technically correct to say that there is a 95% probability that a fixed parameter lies in this particular interval in the frequentist interpretation.

Sampling bias, nonresponse, measurement error, survivorship bias, and data leakage can invalidate an analysis even when the sample is large. Precision cannot compensate for systematic bias.

6. Covariance, Correlation, and Association

For paired observations (xᵢ, yᵢ), sample covariance is

Positive covariance indicates that values tend to be above or below their means together; negative covariance indicates opposite movement. Its magnitude depends on measurement units, so it is difficult to compare across datasets.

Pearson’s correlation coefficient is

where sₓ and sᵧ are sample standard deviations. It lies between −1 and 1 and measures the strength and direction of linear association. Correlation is unchanged by positive rescaling of either variable but changes sign if one variable is multiplied by a negative number.

A correlation near zero does not rule out a strong nonlinear relationship. Outliers can substantially alter r, and restriction of range can weaken it. Spearman’s rank correlation is based on ranks and is useful for monotonic relationships or ordinal data, but it also does not establish causation.

The statement “correlation does not imply causation” is not a claim that causal relationships never correlate. It means that correlation alone does not identify the causal mechanism. Randomized experiments, natural experiments, longitudinal designs, causal diagrams, and careful adjustment for confounders provide stronger evidence.

7. Basic Optimization

Optimization seeks values of parameters that minimize or maximize an objective function. In supervised learning, a loss function measures disagreement between predictions and observed outcomes. For example, mean squared error is

where xᵢ is a feature vector, yᵢ is the target, β is a parameter vector, and an intercept may be included as an additional feature. The units of J are squared target units.

For a differentiable function J(θ), the gradient is the vector of partial derivatives. Gradient descent updates parameters by

where t is the iteration index, η > 0 is the learning rate, and ∇J is evaluated at θₜ. The minus sign moves in the direction of decreasing objective value. If η is too large, the method may overshoot or diverge; if too small, learning may be unnecessarily slow.

A convex function has no non-global local minima. For differentiable convex optimization, a point with zero gradient is a global minimum under suitable conditions. Non-convex objectives, common in neural networks, may contain saddle points and multiple local minima. Stochastic gradient descent estimates the gradient from a minibatch rather than the full dataset, trading noisier updates for lower computational cost.

Regularization modifies the objective to discourage excessive complexity. Ridge regression uses an L2 penalty:

where λ ≥ 0 controls the penalty and the intercept is commonly excluded from the penalty. Larger λ generally shrinks coefficients and may reduce variance, but excessive regularization can increase bias. L1 regularization uses λ∑ⱼ|βⱼ| and can produce exact zero coefficients, supporting sparse representations.

Applied Case: Predicting Delivery Delays

Background

A logistics company wants to predict whether a delivery will arrive late. Its historical data include distance, traffic level, weather category, dispatch hour, vehicle type, and a late-delivery indicator.

Problem

The operations team initially reports that distance and lateness have a correlation of 0.62 and proposes assigning more drivers to all long-distance deliveries. However, long-distance routes also tend to occur in congested regions and at peak hours. The company needs an analysis that distinguishes description from prediction and avoids unfair allocation.

Analysis and approach

The analyst first defines the unit of analysis as one delivery and separates training, validation, and test data chronologically. A chronological split is appropriate because future deliveries should not influence past predictions. Features are inspected for missing values, inconsistent units, and information that would only become available after delivery; such post-outcome variables would create leakage.

Descriptive statistics show that delivery distance is right-skewed, so the median and IQR are reported alongside the mean and standard deviation. Categorical variables are encoded, and numeric variables are standardized using training-set statistics only. A logistic model estimates the probability of lateness:

The model is evaluated using a confusion matrix, precision, recall, calibration, and performance across relevant regions and vehicle groups. A high accuracy score may be misleading if late deliveries are rare. The threshold is therefore selected according to the operational cost of missed late deliveries versus unnecessary interventions.

Outcome

The model shows that distance is associated with lateness, but the association is weaker after traffic and dispatch hour are included. The company uses predicted risk to prioritize proactive customer communication rather than automatically penalizing drivers. Monitoring is scheduled because traffic patterns, routes, and weather distributions can change.

Learning points

1. Correlation is useful for exploration but is not evidence of causation.
2. Data leakage can make a model appear more accurate than it will be in production.
3. Class imbalance requires metrics beyond accuracy.
4. Standardization and preprocessing must be learned from training data only.
5. Operational and fairness consequences should influence threshold selection.

Limitations, Governance, and Best Practice

Mathematical models simplify reality. Their outputs depend on data quality, variable definitions, sampling design, and assumptions. Analysts should document the population, measurement process, missing-data treatment, transformations, model objective, and evaluation protocol.

Uncertainty should be communicated rather than hidden behind excessive decimal places. Estimates should include appropriate intervals or predictive distributions when decisions depend on risk. A confidence interval for an average does not automatically describe the uncertainty of a new individual observation; prediction intervals are generally wider because they include individual-level variation.

Data science also raises governance issues. Personal data should be collected and used for legitimate, communicated purposes, with access controls and retention limits. Sensitive attributes may require special protection, but excluding them does not guarantee fairness because proxy variables may remain. Models should be tested for disparate error rates, documented with versioned data and code, and monitored after deployment. Human review is important when predictions affect employment, credit, healthcare, education, or access to essential services.

Contemporary systems increasingly combine probabilistic modelling, automatic differentiation, distributed computation, and AI-assisted analysis. These technologies do not remove the need to understand assumptions. Automated feature selection can exploit spurious patterns; generative systems can produce plausible but unsupported numerical claims; and large datasets can amplify historical bias. Reproducibility, independent validation, sensitivity analysis, and clear communication remain essential.

Chapter Summary

Linear algebra represents observations and transformations using vectors and matrices. Dot products, norms, matrix multiplication, and eigenvectors support regression, similarity measurement, dimensionality reduction, and machine learning. Functions describe relationships, but extrapolation and causal interpretation require care.

Descriptive statistics summarize centre, spread, and shape. The mean and standard deviation are useful but sensitive to outliers, while medians and IQRs are more robust. Probability defines events, conditional relationships, independence, expectation, and variance. Bernoulli, binomial, Poisson, and normal distributions are useful models only when their assumptions are reasonably appropriate.

Sampling connects data to populations, but bias and measurement problems cannot be solved merely by increasing sample size. Covariance and correlation describe association, not causation. Optimization chooses parameters by minimizing an objective; gradient descent and regularization are fundamental tools for model training and controlling complexity. Across all applications, sound analysis requires explicit assumptions, suitable evaluation, uncertainty communication, ethical governance, and monitoring.

Key Terms

Scalar: A single numerical value.

Vector: An ordered collection of numerical components.

Matrix: A rectangular array of numbers.

Norm: A measure of vector magnitude or length.

Dot product: The sum of pairwise products of two equal-length vectors.

Eigenvector: A non-zero vector whose direction is unchanged by a matrix transformation.

Function: A mapping that assigns one output to each permitted input.

Mean: The arithmetic average of observations.

Median: The middle ordered observation.

Variance: The average squared deviation from a mean, with the denominator depending on whether the data describe a population or sample.

Standard deviation: The positive square root of variance.

Probability: A numerical measure of uncertainty ranging from 0 to 1.

Conditional probability: The probability of one event given that another event has occurred.

Random variable: A numerical representation of outcomes from a random process.

Sampling bias: Systematic deviation caused by how observations are selected or retained.

Standard error: The standard deviation of a sampling distribution or its estimate.

Covariance: A unit-dependent measure of joint variation between two variables.

Correlation: A standardized measure of linear association.

Loss function: An objective measuring disagreement between predictions and observations.

Regularization: A penalty or constraint used to control model complexity.

Review Questions

Short Answer Questions

1. Distinguish a vector from a matrix and give one data-science example of each.
2. Why can the median be more informative than the mean for skewed data?
3. State the difference between a population parameter and a sample statistic.
4. What does a correlation coefficient measure, and what does it not establish?
5. What is the role of the learning rate in gradient descent?

Long Answer Questions

1. Explain how vectors, matrices, dot products, and norms are used in a predictive modelling workflow.
2. Compare the mean, median, variance, standard deviation, and IQR, including their sensitivity to outliers.
3. Explain conditional probability, independence, and Bayes’ theorem using a classification example.
4. Discuss sampling bias, standard error, confidence intervals, and the effect of sample size.
5. Explain regularization, comparing L1 and L2 penalties and their effects on model complexity.

Analytical / Application Questions

1. A dataset contains monthly incomes of 1,800, 2,000, 2,100, 2,200, and 12,000 units. Which measures of centre and spread would you report, and why?
2. A fraud detector identifies 95% of fraudulent transactions but incorrectly flags 8% of legitimate transactions. Design an evaluation plan that considers class imbalance, costs, calibration, and fairness.
3. A delivery model has strong correlation between distance and lateness. Propose an analysis to determine whether traffic and dispatch time confound this relationship, and identify risks that would remain even after adjustment.

Multiple Choice Questions with Answers

Question 1

Which operation produces a scalar when applied to two equal-length vectors?

A. Vector addition

B. Dot product

C. Scalar multiplication

D. Matrix stacking

Correct Answer: B. Dot product

Question 2

For the data 2, 3, 3, 4, 8, what is the arithmetic mean?

A. 3

B. 3.5

C. 4

D. 4.5

Correct Answer: C. 4

Question 3

Which statistic is generally most resistant to an extreme outlier?

A. Mean

B. Variance

C. Standard deviation

D. Median

Correct Answer: D. Median

Question 4

If events A and B are independent, which relationship holds?

A. P(A|B) = P(B|A)

B. P(A ∩ B) = P(A)P(B)

C. P(A ∪ B) = 0

D. P(A) = P(B)

Correct Answer: B. P(A ∩ B) = P(A)P(B)

Question 5

A binomial model requires which condition?

A. A changing success probability on every trial

B. Dependent trials with varying sample sizes

C. A fixed number of independent trials with constant success probability

D. Continuous measurements only

Correct Answer: C. A fixed number of independent trials with constant success probability

Question 6

If the sample size is increased by a factor of four, the standard error of a sample mean approximately:

A. Doubles

B. Quadruples

C. Does not change

D. Halves

Correct Answer: D. Halves

Question 7

A correlation close to zero may still occur when two variables have:

A. No relationship of any kind

B. A strong nonlinear relationship

C. Identical units only

D. Equal means

Correct Answer: B. A strong nonlinear relationship

Question 8

In gradient descent, the update θₜ₊₁ = θₜ − η∇J(θₜ) uses η to represent the:

A. Learning rate

B. Sample variance

C. Intercept

D. Classification threshold only

Correct Answer: A. Learning rate

Question 9

Which regularization penalty can produce exact zero coefficients?

A. L1 penalty

B. L2 penalty only

C. No penalty

D. A standard deviation penalty

Correct Answer: A. L1 penalty

Question 10

Why is chronological splitting useful for evaluating a model intended for future predictions?

A. It guarantees perfect accuracy

B. It prevents future observations from influencing past training

C. It removes all sampling bias

D. It makes every variable normally distributed

Correct Answer: B. It prevents future observations from influencing past training