3. Mathematical, Statistical and Data Foundations

Introduction and Context

Artificial intelligence and machine learning are often presented through algorithms, software libraries and impressive applications. Yet every model ultimately operates on mathematical representations of data. A recommendation system compares vectors, a neural network transforms matrices, a classifier estimates probabilities, and a training procedure minimizes an objective function. Statistical reasoning determines whether observed patterns are meaningful, while information theory helps quantify uncertainty and information. Consequently, mathematical and data foundations are not optional technical details; they shape what a system can learn, how reliably it learns, and whether its conclusions deserve trust.

This chapter develops the foundations required to read, formulate and evaluate machine-learning problems. It connects linear algebra with data representation, probability with uncertainty, descriptive statistics with exploratory analysis, inferential statistics with generalization, and optimization with model training. It also examines datasets, features, labels, distributions, sampling, correlation, causation, bias and data quality. The objective is not to turn the reader into a specialist mathematician, but to establish sufficient conceptual and operational fluency for advanced study and professional practice.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Represent datasets using scalars, vectors, matrices and tensors.

2. Explain the statistical meaning of probability, distributions, expectation, variance and conditional dependence.

3. Calculate and interpret central-tendency and dispersion measures.

4. Distinguish population parameters from sample statistics and explain sampling uncertainty.

5. Interpret covariance, correlation and causation without confusing their roles.

6. Formulate optimization problems using objective functions, gradients and constraints.

7. Explain entropy, cross-entropy and mutual information in machine-learning contexts.

8. Diagnose common forms of data quality problems, sampling bias and data leakage.

9. Apply these foundations to model development and responsible decision-making.

From Observations to Learning Problems

A dataset is a structured collection of observations. An observation may represent a customer, image, medical encounter, transaction or time point. A feature is a measurable attribute used to describe an observation, such as age, temperature or word frequency. In supervised learning, a label or target is the outcome the model is expected to predict. In unsupervised learning, labels may be absent and the aim may be to discover structure.

Suppose a credit dataset contains income, debt ratio and repayment history for each applicant. If the task is to predict default, these variables are features and default status is the label. The same data can support different tasks: predicting a numerical loss is regression, predicting a category is classification, and grouping applicants without a target is clustering. The mathematical representation must reflect the task, measurement scale and temporal context.

A dataset is commonly represented by a design matrix X, whose rows are observations and columns are features. If there are n observations and p features, X belongs to R^(n×p). A label vector y may contain n numerical or categorical outcomes. This representation is useful, but it does not guarantee that the data are suitable. Missing values, inconsistent units, duplicate records, selection effects and measurement error can make a mathematically well-formed matrix scientifically misleading.

Measurement and representation

Features may be numerical, categorical, ordinal, binary, textual, spatial or temporal. Nominal categories such as payment method have no natural order; ordinal categories such as satisfaction levels do. Treating nominal codes as numerical magnitudes can create false relationships. Numerical variables may be discrete, such as number of purchases, or continuous, such as blood pressure. Scaling is often important because a feature measured in thousands can dominate a distance or optimization calculation even when it is not more informative.

A scalar is a single number, a vector is an ordered list of numbers, a matrix is a rectangular array, and a tensor generalizes arrays to more dimensions. An image can be represented as a height-by-width matrix for grayscale or a three-dimensional tensor for color channels. A document may be represented by a vector of word counts or learned embeddings. These representations allow algorithms to perform algebraic operations on objects that originally have semantic meaning.

Linear Algebra as the Language of Data

For vectors x and y of equal dimension, the dot product is xᵀy = Σᵢ xᵢyᵢ. It measures alignment and forms the basis of linear prediction, similarity search and neural-network layers. The Euclidean norm ||x||₂ measures vector length. A distance such as ||x − y||₂ can be used to compare observations, although its usefulness depends on feature scaling and the geometry of the representation.

A linear model can be written as ŷ = wᵀx + b, where w is a vector of weights and b is an intercept. Each weight indicates the model’s contribution associated with a feature, subject to the units, coding and presence of other variables. In a matrix form, predictions for many observations are Xw + b. Matrix multiplication therefore expresses the simultaneous application of a model to a dataset.

The transpose exchanges rows and columns. The identity matrix acts as a multiplicative neutral element, and an inverse, when it exists, can solve systems of linear equations. In practice, numerical algorithms often use decompositions such as QR or singular value decomposition rather than directly computing an inverse, because direct inversion can be unstable or inefficient. Singular value decomposition is especially useful for dimensionality reduction, denoising and understanding correlated features.

The geometric interpretation is important. A linear model defines a hyperplane; training selects a hyperplane that best fits the data according to a chosen loss. High-dimensional data cannot always be visualized, but geometric concepts remain useful for understanding distances, projections, margins and separating boundaries. Principal component analysis, for example, identifies directions of high variance and projects observations onto a lower-dimensional subspace. High variance does not necessarily mean high predictive value, so dimensionality reduction must be evaluated in relation to the task.

Probability, Distributions and Uncertainty

Probability provides a formal language for uncertain events. A random variable maps possible outcomes to numerical values. A probability distribution specifies how probability is assigned to those values. For a discrete variable, probabilities sum to one; for a continuous variable, a density describes relative concentration and probabilities are obtained over intervals.

The expected value E[X] is the probability-weighted average of a random variable. Variance, Var(X) = E[(X − E[X])²], measures dispersion around the mean, and standard deviation is its square root in the original units. These quantities summarize distributions but do not fully describe them. Two distributions can have the same mean and variance yet differ in skewness, tails or multimodality.

Conditional probability, P(A|B), describes the probability of A given that B has occurred. Bayes’ theorem connects inverse conditional relationships:

P(A|B) = P(B|A)P(A) / P(B).

This distinction is essential in diagnosis and classification. The probability of observing a positive test given disease is not the same as the probability of disease given a positive test. The latter also depends on prevalence and false-positive behavior.

Independence means that knowledge of one event does not change the probability of another. In symbols, P(A,B) = P(A)P(B). Statistical learning often relies on simplifying assumptions, such as independent and identically distributed observations, but real data may contain dependence across people, locations or time. Ignoring dependence can produce overly confident estimates and invalid evaluation results.

A distribution may be summarized through its probability mass function, density, cumulative distribution function or quantiles. Common distributions include Bernoulli for a single binary outcome, binomial for counts of successes, categorical for multiple classes and Gaussian for continuous measurements approximated by a bell-shaped form. The normal distribution is mathematically convenient, but assuming normality without checking can distort inference, especially with small samples or heavy-tailed data.

Descriptive Statistics and Exploratory Reasoning

Descriptive statistics organize observed data before formal modeling. The arithmetic mean is sensitive to extreme values, whereas the median is more robust for skewed distributions. The mode identifies the most frequent value and is particularly meaningful for categorical data. The range gives the distance between the largest and smallest values; the interquartile range, Q3 − Q1, describes the middle half and is less affected by outliers.

A useful exploratory workflow combines numerical summaries with visual inspection. Histograms reveal shape, box plots expose possible outliers, scatter plots show pairwise structure, and line plots reveal temporal patterns. Missingness should itself be summarized: a variable with many missing values may indicate a measurement or process problem rather than a minor inconvenience.

An outlier is an observation unusually distant from the rest under a chosen definition. It may reflect an error, a rare but valid case, fraud or a change in the data-generating process. Automatic removal is dangerous. The appropriate response depends on domain knowledge, provenance and the intended use of the model. Robust methods, transformations or explicit anomaly modeling may be preferable.

Standardization converts a value into a z-score, z = (x − mean)/standard deviation. It places variables on comparable scales but does not make data normal and can be affected by outliers. Min–max scaling maps values into a specified interval. Any transformation parameters must be estimated using the training set only; using the complete dataset can leak information from the test set.

Samples, Inference and Generalization

A population is the complete set about which a study seeks knowledge, while a sample is the observed subset. A population parameter, such as the true mean, is usually unknown; a sample statistic estimates it. Sampling variability means that different samples produce different estimates even when the procedure is unbiased.

A confidence interval communicates an estimation procedure and its uncertainty. In a frequentist interpretation, a 95% confidence procedure would capture the fixed parameter in approximately 95% of repeated samples under its assumptions. It does not mean that there is a 95% probability that a fixed parameter lies in one already computed interval. A p-value measures compatibility between data and a null model under specified assumptions; it is not the probability that the null hypothesis is true and does not measure practical importance.

Sampling methods influence validity. Simple random sampling gives units a known chance of selection. Stratified sampling ensures representation of defined subgroups, while cluster sampling selects groups rather than individuals. Convenience samples are easy to obtain but may systematically differ from the target population. In machine learning, random train-test splitting can also be inappropriate when observations are time-dependent, grouped by person, or spatially related. The split must mimic the conditions under which the model will be used.

Generalization is performance on new data drawn from the relevant deployment distribution. A model may fit its training sample extremely well yet fail on new cases because of overfitting, distribution shift, leakage or a mismatch between the sample and the population. Cross-validation estimates predictive performance by repeatedly separating fitting and evaluation roles, but it cannot repair a fundamentally biased sampling design.

Association, Correlation and Causation

Covariance indicates whether two variables tend to move together, but its magnitude depends on units. Pearson correlation standardizes covariance by the two standard deviations and lies between −1 and 1. A value near one indicates strong linear association, while a value near zero indicates little linear association; it does not rule out nonlinear dependence.

Correlation does not establish causation. A third variable may influence both observed variables, selection may create an association, or the direction of influence may be reversed. For example, umbrella sales and traffic accidents may rise together because rainfall affects both. A causal claim requires stronger design or assumptions, such as randomization, valid natural experiments, longitudinal evidence or carefully justified causal models.

This distinction matters in organizational decision-making. A predictive feature can be useful without being causal, but interventions based on it may fail if the relationship changes. Conversely, a causally important variable may have weak predictive power in a particular dataset. Analysts should state whether their objective is prediction, explanation, estimation of an intervention effect or discovery of a mechanism.

Optimization: Turning Learning into an Objective

Machine learning commonly defines a loss function that measures disagreement between predictions and observed targets. Training seeks parameters θ that minimize empirical risk:

J(θ) = (1/n) Σᵢ L(fθ(xᵢ), yᵢ).

For squared-error regression, large errors receive disproportionately large penalties. For classification, cross-entropy penalizes confident incorrect probabilities. Regularization adds a penalty for undesirable parameter behavior, such as large weights, producing an objective such as empirical loss plus λR(θ). The parameter λ controls the trade-off between fit and complexity.

The gradient is the vector of partial derivatives of an objective with respect to its parameters. Gradient descent updates parameters in the negative-gradient direction: θ ← θ − η∇J(θ), where η is the learning rate. A rate that is too large can cause divergence; one that is too small can make training inefficient. Stochastic and mini-batch gradient descent estimate gradients using subsets of observations, allowing scalability and sometimes helping optimization move away from poor regions.

Convex problems have a useful global structure: every local minimum is global. Many modern models produce non-convex objectives, so initialization, architecture, data scale and optimization settings matter. Regularization, early stopping, feature selection and validation-based model comparison can reduce overfitting. Optimization success, however, does not imply that the objective is appropriate or that the data encode a fair decision rule.

Information Theory and Learning Signals

Information theory quantifies uncertainty and dependence. The entropy of a discrete variable X is H(X) = −Σ p(x) log p(x). Entropy is higher when outcomes are more uncertain and lower when one outcome dominates. The logarithm base determines the unit: base two gives bits.

Conditional entropy measures remaining uncertainty about one variable after observing another. Mutual information, I(X;Y) = H(X) − H(X|Y), measures the reduction in uncertainty about X provided by Y. It can detect nonlinear dependence more broadly than Pearson correlation, although estimating it reliably requires sufficient data and careful handling of continuous variables.

Cross-entropy compares a true distribution with a predicted distribution and is widely used for probabilistic classification. If a model assigns very low probability to the observed class, the penalty is large. This encourages calibrated probabilistic predictions, but performance should still be assessed with metrics appropriate to the application, such as calibration error, recall or expected cost.

Data Quality, Bias and Responsible Use

Data quality has several dimensions: accuracy, completeness, consistency, validity, uniqueness, timeliness and provenance. A duplicated customer record affects counts; inconsistent currency units distort numerical relationships; stale data may be inappropriate for current decisions. Imputation can be useful, but it introduces assumptions and should preserve a missingness indicator when the fact of missingness carries information.

Bias can enter through problem formulation, measurement, sampling, labeling, historical decisions and deployment feedback. A dataset may be representative overall yet underrepresent a vulnerable subgroup. A model can also amplify existing disparities when predictions determine access to future data or resources. Fairness analysis should examine relevant subgroup performance, error rates, calibration and the consequences of decisions rather than rely on a single universal metric.

Data leakage occurs when information unavailable at prediction time enters training. Examples include using a post-outcome field, allowing the same patient to appear in both training and test sets, or selecting features using the complete dataset before splitting. Leakage produces deceptively strong evaluation results and is among the most consequential practical mistakes in machine learning.

Privacy and governance require purpose limitation, access controls, retention policies, documentation and lawful processing. Sensitive attributes may be needed for auditing fairness even when they are excluded from prediction, but their collection and use must be justified and protected. Reproducible pipelines, data dictionaries, versioning and model cards support accountability.

Applied Case: Predicting Hospital Readmission

Background

Consider an illustrative case in which a hospital develops a model to estimate whether a patient will be readmitted within thirty days of discharge. The proposed use is to prioritize follow-up support, not to deny treatment.

Situation and problem

The hospital has three years of electronic health records. Features include age, diagnoses, previous admissions, laboratory results, discharge department and medication information. Initial modeling reports high accuracy, but clinicians observe that performance falls for patients from a recently opened community clinic.

Analysis

Several foundational questions must be addressed. First, the label must be defined precisely: does readmission to another hospital appear in the records? Second, features must be restricted to information available before discharge. A field documenting a follow-up appointment could be leakage if it is entered after discharge and influenced by clinicians’ expectations. Third, the random split may place multiple encounters from the same patient in both training and test sets, inflating performance. A patient-level split is more appropriate. If deployment concerns future patients, a temporal validation set is also necessary.

Descriptive analysis may reveal that the community clinic serves a different population and records laboratory tests less frequently. Missingness is therefore not random. A model that treats absent tests as ordinary zeros may encode access differences rather than clinical risk. Evaluation should report sensitivity, precision, calibration and error rates by clinic and relevant demographic groups, while considering the cost of missed high-risk patients and unnecessary interventions.

Approach and solution

The team creates a clear prediction-time data specification, removes post-discharge variables, deduplicates patient histories and uses patient-level temporal validation. Missing values are handled through a documented training-only imputation process, with indicators where clinically meaningful. The model is compared with a simple baseline and calibrated before use. Thresholds are selected according to the capacity of the follow-up team and the relative consequences of errors.

The hospital establishes monitoring for data drift, subgroup performance and changes in readmission patterns. Clinicians receive explanations framed as contributing evidence rather than causal claims. Patients are not automatically denied services, and a human review process is retained for high-impact decisions.

Outcome and learning points

The revised evaluation produces lower but more credible performance. More importantly, it reveals that some apparent predictive power came from leakage and repeated-patient overlap. The model becomes a decision-support tool rather than an unquestioned authority. The case demonstrates that data definitions, sampling, uncertainty, optimization, fairness and governance are inseparable from model accuracy.

Common Errors and Professional Best Practices

Frequent errors include interpreting a correlation as a cause, reporting only accuracy on an imbalanced problem, removing outliers without investigation, scaling before splitting data, assuming missing values are harmless, and treating a statistically significant effect as practically important. Another error is optimizing a proxy that is convenient to measure but poorly aligned with the real objective.

Strong practice begins with a precise unit of analysis, prediction time and target definition. Analysts should inspect distributions and missingness, establish simple baselines, separate training from evaluation, document assumptions and quantify uncertainty. They should test robustness across subgroups and plausible distribution shifts, use domain expertise to interpret anomalies, and distinguish predictive utility from causal evidence. Reproducible code, versioned data, clear feature definitions and post-deployment monitoring are technical requirements, not administrative extras.

Current Directions

Modern machine learning increasingly uses high-dimensional representations, foundation models and automated data pipelines. These developments make linear algebra and optimization more computationally demanding, while large-scale pretraining raises questions about sampling, provenance, memorization and distribution shift. Probabilistic and uncertainty-aware methods are receiving renewed attention in safety-critical applications. Causal machine learning seeks to combine predictive modeling with explicit questions about interventions, although causal conclusions still depend on design and assumptions. Responsible AI practice is also moving toward continuous evaluation, dataset documentation, model monitoring and governance throughout the system life cycle.

Chapter Summary

Mathematical and data foundations determine how machine-learning problems are represented, analyzed and evaluated. Scalars, vectors, matrices and tensors provide the language for data and models. Probability describes uncertainty through distributions, conditional relationships and expectations. Descriptive statistics reveal center, spread, shape and anomalies, while inferential reasoning distinguishes sample evidence from population claims. Sampling design and data splitting determine whether performance generalizes.

Covariance and correlation describe association but do not by themselves establish causation. Optimization expresses learning as the minimization of an objective, with gradients guiding parameter updates and regularization controlling complexity. Entropy, mutual information and cross-entropy quantify uncertainty and predictive information. Finally, data quality, leakage, bias, privacy and governance determine whether a technically accurate model is trustworthy and appropriate for use.

Key Terms

Feature: A measurable input attribute used by a model.

Label: The observed target outcome in supervised learning.

Design matrix: A matrix whose rows represent observations and columns represent features.

Vector: An ordered collection of numerical values.

Tensor: A multidimensional numerical array.

Dot product: The sum of pairwise products of two equal-length vectors.

Random variable: A mapping from uncertain outcomes to numerical values.

Expected value: The probability-weighted average of a random variable.

Variance: The expected squared deviation from the mean.

Conditional probability: The probability of an event given another event or condition.

Distribution: A specification of how probability is allocated across possible values.

Population: The complete set about which a study seeks knowledge.

Sample: The subset of a population that is observed.

Correlation: A standardized measure of linear association between variables.

Causation: A relationship in which changing one factor produces a change in another under defined conditions.

Objective function: A mathematical quantity that a learning procedure seeks to minimize or maximize.

Gradient: The vector of partial derivatives of a function with respect to its parameters.

Entropy: A measure of uncertainty in a probability distribution.

Mutual information: The reduction in uncertainty about one variable obtained from another.

Data leakage: The unintended use of information unavailable at prediction time during model development.

Review Questions

Short Answer Questions

1. What are the roles of features and labels in supervised learning?

2. Why can a mean be less representative than a median for a skewed dataset?

3. Distinguish a population parameter from a sample statistic.

4. What does a gradient indicate during optimization?

5. Why does correlation not necessarily imply causation?

Long Answer Questions

1. Explain how vectors and matrices represent data and support linear prediction.

2. Discuss conditional probability, Bayes’ theorem and their importance in classification.

3. Explain sampling variability, confidence intervals and generalization in machine learning.

4. Describe empirical risk minimization, regularization and gradient descent.

5. Discuss the main dimensions of data quality and their effects on model reliability.

Analytical / Application Questions

1. A fraud model is trained using a feature created after a transaction has been investigated. Identify the methodological problem and propose a valid redesign.

2. Two variables have a Pearson correlation close to zero, but a scatter plot forms a clear U-shape. Explain the apparent contradiction and suggest an alternative analysis.

3. A hospital model performs well overall but has substantially lower recall for one clinic. Analyze possible statistical and data-quality causes and recommend an evaluation and governance plan.

22. Multiple Choice Questions with Answers

Question 1

A dataset contains 10,000 transactions and 12 measured attributes for each transaction. Which representation is most appropriate for the feature data?

A. A 10,000 × 12 design matrix

B. A 12 × 12 covariance matrix only

C. A single scalar containing the mean

D. A probability distribution with no observations

Correct Answer: A. A 10,000 × 12 design matrix

Question 2

Why is a patient-level split preferable to a purely random row-level split when records include multiple visits per patient?

A. It guarantees that every feature is normally distributed

B. It prevents information from the same patient appearing in both training and evaluation sets

C. It always increases the measured accuracy

D. It removes the need for a test set

Correct Answer: B. It prevents information from the same patient appearing in both training and evaluation sets

Question 3

A positive medical test is more likely among diseased patients, but the disease is rare. Which concept explains why the probability of disease given a positive result may still be modest?

A. Vector normalization

B. Bayes’ theorem and the base rate

C. Gradient descent

D. The interquartile range

Correct Answer: B. Bayes’ theorem and the base rate

Question 4

Which statistic is generally more robust to a small number of extreme observations?

A. Mean

B. Range

C. Median

D. Variance

Correct Answer: C. Median

Question 5

A model minimizes an objective using the update θ ← θ − η∇J(θ). What does η represent?

A. The label encoding

B. The learning rate

C. The sample size

D. The entropy of the data

Correct Answer: B. The learning rate

Question 6

A feature has a strong association with loan default because it records whether a collections action occurred after default. What is the principal problem?

A. Underfitting

B. Data leakage

C. Stratified sampling

D. Low entropy only

Correct Answer: B. Data leakage

Question 7

Which statement best describes mutual information?

A. It measures only the difference between two sample means

B. It measures the reduction in uncertainty about one variable obtained from another

C. It proves that one variable causes another

D. It is always identical to Pearson correlation

Correct Answer: B. It measures the reduction in uncertainty about one variable obtained from another

Question 8

A model has excellent training performance but poor performance on new data. Which explanation is most directly consistent with this pattern?

A. Overfitting

B. A perfectly representative sample

C. Zero variance in the labels

D. Guaranteed causal validity

Correct Answer: A. Overfitting

Question 9

Why can scaling features be important for distance-based algorithms?

A. It makes labels unnecessary

B. It prevents a feature with large units from dominating distances

C. It guarantees independence between observations

D. It converts categorical variables into causal variables

Correct Answer: B. It prevents a feature with large units from dominating distances

Question 10

Which conclusion is justified by a high correlation between umbrella sales and traffic accidents during rainy periods?

A. Umbrella sales necessarily cause accidents

B. Accidents necessarily cause umbrella purchases

C. The variables are associated, possibly because rainfall affects both

D. The relationship must remain unchanged in every population

Correct Answer: C. The variables are associated, possibly because rainfall affects both