Machine Learning Foundations and Mathematical Essentials

Introduction and Context

Machine learning (ML) is the study of computational methods that improve their performance on a task through experience or data. Instead of specifying every rule explicitly, a practitioner provides data, defines an objective, selects a model class, and uses an optimization procedure to estimate model parameters. The resulting system is expected to generalize: it should perform well not only on observations used during training but also on suitable unseen observations.

Machine learning is therefore both a mathematical discipline and an engineering practice. Mathematics provides the language for representing data, uncertainty, functions, and optimization. Engineering provides methods for collecting data, validating assumptions, deploying models, and monitoring performance. A technically accurate model can still fail if its data are biased, its evaluation is invalid, or its deployment context changes.

This chapter develops the foundations required for later study of regression, classification, neural networks, ensemble methods, and advanced algorithms.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish supervised, unsupervised, semi-supervised, and reinforcement learning.
2. Explain essential terminology, including features, labels, parameters, hyperparameters, training data, and generalization.
3. Use vectors, matrices, norms, dot products, and matrix multiplication in ML contexts.
4. Interpret probability distributions, conditional probability, expectation, variance, covariance, and correlation.
5. Explain derivatives, gradients, loss functions, and gradient-based optimization.
6. Distinguish empirical risk from expected risk and explain the role of regularization.
7. Interpret bias, variance, underfitting, overfitting, and the bias–variance trade-off.
8. Apply a basic learning workflow responsibly in a practical case.

1. The Machine Learning Problem

A dataset consists of observations. In supervised learning, an observation is commonly written as (x_i, y_i), where x_i is an input and y_i is a target or label. The input may be a scalar, vector, image, text document, audio signal, or structured record. A model is a parameterized function f_θ(x), where θ denotes parameters learned from data.

The learning objective is usually to find parameters that minimize a loss. For a dataset of n observations, the empirical risk is

Here, L is a loss function, f_θ(x_i) is the prediction, and y_i is the observed target. The average is appropriate when observations are weighted equally. If observations have different importance, a weighted average may be used, with non-negative weights that sum to one.

The deeper objective is expected risk:

where P is the unknown data-generating distribution. Since P is generally unavailable, training uses empirical data as an approximation. Generalization concerns the difference between performance on the observed sample and performance on new data drawn from the same or a suitably related distribution.

Important Terminology

2. Learning Paradigms

Supervised Learning

Supervised learning uses paired inputs and targets. Regression predicts a quantitative value, such as energy consumption. Classification predicts a category, such as fraudulent or legitimate. Binary classification has two classes; multiclass classification has more than two; multilabel classification permits several labels for one instance.

A regression model may predict house price from area, location, and age. A classification model may estimate whether a transaction is suspicious. The quality of the target labels is central: systematically incorrect labels create systematic learning errors.

Unsupervised Learning

Unsupervised learning uses inputs without target labels. Clustering seeks groups of similar observations. Dimensionality reduction represents data using fewer variables while preserving selected structure. Density estimation models how observations are distributed. Because there is no universally correct label, evaluation often combines quantitative criteria with domain interpretation.

Semi-Supervised and Self-Supervised Learning

Semi-supervised learning combines a small labelled dataset with a larger unlabelled dataset. It is useful when labelling requires expert time. Self-supervised learning creates supervisory signals from the data itself; for example, a system may hide part of an input and learn to predict it. The learned representation can later support supervised tasks.

Reinforcement Learning

In reinforcement learning, an agent interacts with an environment. At time t, it observes a state s_t, selects an action a_t, receives a reward r_t, and moves to a new state. A policy maps states to actions. A discounted return is often defined as

where γ satisfies 0 ≤ γ < 1 and expresses the relative importance of future rewards. The infinite sum assumes rewards are bounded or otherwise produces a finite expected return. Reinforcement learning differs from ordinary supervised learning because the correct action is often not supplied directly and actions can affect future observations.

3. Linear Algebra for Machine Learning

Scalars, Vectors, and Matrices

A scalar is a single number. A vector is an ordered list, commonly written x ∈ ℝᵈ, where d is the number of features. A dataset containing n observations and d features can be represented by a matrix X ∈ ℝⁿˣᵈ. Row i contains the features of observation i.

The dot product of two vectors is

It produces a scalar and requires vectors of equal dimension. In a linear model,

where w is a weight vector and b is an intercept. If x and w have matching feature units, individual products may have different units unless the features have been appropriately scaled or the weights carry reciprocal units.

Matrix multiplication is defined when the inner dimensions agree. If A is m×p and B is p×q, then AB is m×q, with elements

This operation is central to batch prediction and neural-network layers.

Norms and Distance

The Euclidean norm is

The squared Euclidean norm is often used because it is differentiable and avoids a square root. The Manhattan norm is ‖x‖₁ = ∑ⱼ|xⱼ|. Norms quantify magnitude; distances quantify separation, for example ‖x-z‖₂. Feature scaling is important because a feature measured in large numerical units can dominate distance-based algorithms.

Geometric Interpretation

The dot product can be written as xᵀz = ‖x‖₂‖z‖₂ cos θ, where θ is the angle between nonzero vectors. Thus, it reflects both magnitude and directional alignment. Cosine similarity removes magnitude effects:

It is defined only for nonzero vectors.

4. Probability and Statistics

Random Variables and Distributions

A random variable assigns numerical values to outcomes. A discrete variable has a probability mass function; a continuous variable has a density function. A probability distribution must assign non-negative probability and total probability one. For continuous variables, probabilities are obtained by integrating density over intervals, and the probability at one exact point is normally zero.

Conditional probability is

provided P(B)>0. Bayes’ theorem follows as

It is useful when evidence updates prior beliefs. In ML, P(y|x) represents a conditional predictive distribution, while P(x) describes the input distribution.

Expectation, Variance, and Covariance

For a discrete random variable X,

For a continuous variable, the corresponding expression is an integral. Variance is

where μ=𝔼[X]. Standard deviation is σ=√Var(X), expressed in the same units as X. For two variables,

and correlation is

provided both standard deviations are positive. Correlation is unitless and lies between −1 and 1. It measures linear association, not causation and not necessarily nonlinear dependence.

Sampling and Standardization

A sample mean is x̄=(1/n)∑ᵢx_i. A commonly used sample variance is

The denominator n−1 provides an unbiased estimator of population variance under independent, identically distributed sampling with finite variance. It does not correct biased sampling or dependence.

Standardization transforms a value using

where s>0. In a predictive workflow, x̄ and s must be calculated on the training data only and then applied to validation, test, and production data. Otherwise information leaks from evaluation data into training.

5. Calculus, Loss Functions, and Optimization

Derivatives and Gradients

A derivative measures the local rate of change. For a differentiable scalar function J(θ), the gradient is

The gradient points in the direction of greatest local increase. Therefore, gradient descent updates parameters in the opposite direction:

where η>0 is the learning rate. This update is a local optimization method. It does not guarantee a global minimum for non-convex objectives, and convergence depends on the function, learning rate, initialization, and numerical precision.

Common Loss Functions

For regression, mean squared error (MSE) is

It heavily penalizes large errors and has squared target units. Mean absolute error is

It has the target’s units and is less sensitive to extreme errors, though it is not differentiable at zero. A differentiable alternative is the Huber loss, which is quadratic for small errors and linear for large errors.

For binary classification, if y∈{0,1} and p is the predicted probability of class 1, binary cross-entropy is

with 0<p<1. Numerical implementations clip probabilities away from exactly zero and one. The loss penalizes confident incorrect predictions strongly.

Regularization

Regularization adds a penalty to discourage undesirable parameter values. Ridge regression minimizes

where λ≥0. The intercept is commonly excluded from the penalty. Larger λ generally shrinks weights more strongly, but the best value must be selected using validation procedures. Lasso regression replaces the squared norm with ‖w‖₁, encouraging sparsity under suitable conditions. Regularization changes the optimization objective and can improve generalization; it does not guarantee fairness or eliminate data bias.

Worked Example: One Gradient Step

Suppose a one-parameter model has objective J(w)=(w−3)² and current value w=1. Its derivative is dJ/dw=2(w−3), so at w=1, the gradient is −4. With learning rate η=0.1,

The parameter moves toward 3, the minimizer. This is a simple convex example; real models can have many parameters and more complex surfaces.

6. Empirical Risk, Evaluation, and Data Splitting

A reliable evaluation estimates performance on data not used to fit the model or select its settings. A typical workflow divides data into training, validation, and test sets. The training set estimates parameters; the validation set supports model and hyperparameter choices; the test set is used once, or sparingly, for an approximately unbiased final report.

Random splitting is not always valid. Time-dependent data should usually be split chronologically. Records from the same patient, customer, household, or device may need group-based splitting so related records do not appear in both training and test sets. In classification, stratification can preserve class proportions when appropriate.

Cross-validation repeatedly divides training data into folds. In k-fold cross-validation, each fold serves as validation once, and the scores are averaged. Cross-validation estimates comparative performance but does not protect against every form of leakage, distribution shift, or repeated experimentation on the final test set.

Relevant metrics depend on the task. Accuracy is the fraction of correct classifications, but it can be misleading under class imbalance. Precision is the proportion of predicted positives that are truly positive; recall is the proportion of actual positives detected. For regression, MAE is often interpretable in target units, while RMSE is √MSE and gives greater emphasis to large errors.

7. Bias, Variance, and Generalization

Bias is systematic error caused by restrictive assumptions or an unsuitable model class. Variance is sensitivity to the particular training sample. A highly flexible model may fit training noise, producing low training error but high test error. This is overfitting. A model that cannot represent the underlying pattern has high training and test error and is underfitting.

For squared-error prediction, a standard conceptual decomposition at a fixed input is

The precise decomposition requires assumptions such as repeated random training samples, a fixed input, squared loss, and an additive noise model. Noise variance represents irreducible randomness or measurement uncertainty. It cannot generally be removed by choosing a more complex model.

Model complexity often reduces bias but increases variance. More data, regularization, early stopping, data augmentation, and suitable model selection can reduce variance. The trade-off is not a rule that every problem has one simple optimum; it is a framework for understanding why training performance and generalization can diverge.

8. Assumptions and Failure Modes

Machine learning methods make assumptions, whether explicit or implicit. Common assumptions include:

• Training and future data are sufficiently related.
• Observations are independent, or dependence is correctly modelled.
• Features are measured with acceptable reliability.
• Labels represent the operational definition of the target.
• The sample is adequate for the intended population.
• The chosen loss reflects the real cost of errors.

Violations can produce misleading results. Distribution shift occurs when the relationship between training and deployment data changes. Covariate shift changes the input distribution; concept drift changes the relationship between inputs and targets. Missing data may be informative rather than random. Outliers may be errors, rare valid cases, or evidence of a different population.

Data leakage is any use of information during training that would not be available at the intended prediction time. Examples include using a post-outcome variable, normalizing before splitting, or allowing duplicate entities across splits. Leakage can produce excellent validation scores and poor deployment performance.

9. Illustrative Case: Predicting Equipment Failure

Background

A manufacturing company operates motors equipped with temperature, vibration, load, and operating-time sensors. The company wants to predict whether a motor will fail within the next seven days so that maintenance can be scheduled.

Problem

A naive model might label each sensor record according to whether a failure eventually occurred. This can leak future information and may not match the seven-day prediction horizon. The operational target must instead be defined as failure within seven days after the timestamp, using only information available at that timestamp.

Analysis

Engineers inspect missing readings, sensor calibration, failure frequencies, and maintenance records. Since readings from one motor are correlated over time, random row-level splitting would allow nearly identical records from the same motor to appear in both training and testing. A chronological split, possibly combined with a motor-level holdout, is more credible.

Features may include recent temperature averages, vibration trends, load variability, and time since maintenance. Rolling features must be calculated using past values only. A baseline model, such as logistic regression with regularization, provides an interpretable reference. A tree-based model may capture nonlinear interactions, but it should be compared using a validation design that reflects deployment.

Approach and Solution

The team establishes a training period, a validation period, and a later test period. Class imbalance is addressed through suitable metrics and decision thresholds rather than accuracy alone. The model outputs a probability of failure. A threshold is selected using the relative costs of missed failures, unnecessary inspections, and maintenance capacity.

Calibration is assessed because a probability of 0.8 should have a meaningful operational interpretation. Engineers also examine false positives and false negatives by motor type and operating condition. Explanations are reviewed as decision support, not treated as proof of causation.

Outcome

The model supports maintenance scheduling and is monitored after deployment. Performance is tracked over time, sensor changes are documented, and retraining is triggered by predefined evidence of drift. The model does not replace safety procedures or engineering inspection.

Learning Points

The case demonstrates that target definition, temporal validity, leakage prevention, evaluation design, threshold selection, calibration, and monitoring are as important as algorithm choice. A more complex algorithm cannot compensate for an invalid prediction setup.

10. Responsible and Professional Use

ML systems can affect employment, credit, healthcare, education, insurance, and access to services. Practitioners should document data provenance, intended use, limitations, performance by relevant subgroups, and human oversight. Fairness cannot be inferred from overall accuracy alone. Different fairness criteria may conflict, particularly when groups have different base rates, so the appropriate criterion must be justified in context.

Privacy requires data minimization, access controls, retention limits, and compliance with applicable law. Sensitive attributes may need protection, but excluding them does not automatically remove discrimination because correlated variables can act as proxies. Security concerns include data poisoning, model extraction, adversarial inputs, and unauthorized access.

Professional practice therefore includes reproducibility, version control, audit logs, clear ownership, model cards or equivalent documentation, and a process for appeal or correction when decisions affect people.

11. Contemporary Developments

Modern ML increasingly combines representation learning, large-scale pretraining, transfer learning, and multimodal data. These approaches can reduce the amount of task-specific labelled data, but they introduce concerns about computational cost, provenance, copyright, privacy, hallucinated outputs, and hidden bias. Foundation models should be evaluated for the specific deployment task rather than assumed to be reliable because they are large.

Automated machine learning can assist with preprocessing, model selection, and hyperparameter search. It does not automate problem definition, causal interpretation, ethical judgment, or monitoring. Explainability methods can help investigate model behaviour, but many are approximate and should not be interpreted as guaranteed causal explanations.

Best-Practice Workflow

A disciplined workflow is:

1. Define the decision, prediction time, target, and cost of errors.
2. Inspect data provenance, quality, missingness, and representativeness.
3. Create leakage-resistant training, validation, and test splits.
4. Establish a simple baseline.
5. Preprocess using training data only.
6. Select a loss and metrics aligned with the application.
7. Train, tune, and compare models using reproducible procedures.
8. Assess calibration, subgroup performance, robustness, and uncertainty.
9. Document assumptions and limitations.
10. Deploy with monitoring, access controls, human oversight, and a rollback plan.

Chapter Summary

Machine learning estimates functions or decision policies from data. Supervised, unsupervised, semi-supervised, self-supervised, and reinforcement learning differ in the information available and the form of feedback. Vectors and matrices represent data and transformations; norms and dot products support geometry and similarity. Probability and statistics provide tools for modelling uncertainty, estimating variation, and reasoning from samples.

Loss functions quantify error, while gradients support parameter optimization. Empirical risk is a data-based approximation to expected risk. Regularization constrains model complexity and can improve generalization. Bias and variance explain underfitting and overfitting, but their formal decomposition depends on assumptions. Sound evaluation requires leakage prevention, appropriate splitting, relevant metrics, and awareness of distribution shift. Responsible ML additionally requires privacy, fairness analysis, documentation, security, human oversight, and continuous monitoring.

Key Terms

Feature: An input variable supplied to a model.

Label: A target value used in supervised learning.

Parameter: A value estimated from training data.

Hyperparameter: A configuration chosen outside ordinary parameter fitting.

Model: A function that maps inputs to predictions or decisions.

Supervised learning: Learning from input–target pairs.

Unsupervised learning: Learning structure from unlabelled inputs.

Reinforcement learning: Learning actions through interaction and rewards.

Generalization: Performance on appropriate unseen data.

Empirical risk: Average loss on a finite dataset.

Expected risk: Average loss under the data-generating distribution.

Gradient: Vector of partial derivatives of an objective.

Regularization: A penalty or constraint used to control model complexity.

Overfitting: Fitting training-specific variation that does not generalize.

Underfitting: Failing to represent important structure in the data.

Bias: Systematic prediction error from restrictive assumptions.

Variance: Sensitivity of a fitted model to the training sample.

Data leakage: Unavailable or future information entering model development.

Distribution shift: Change between development and deployment data or relationships.

Calibration: Agreement between predicted probabilities and observed frequencies.

Review Questions

Short Answer Questions

1. What is the difference between a parameter and a hyperparameter?
2. Why must preprocessing statistics normally be learned from the training set only?
3. Define empirical risk and expected risk.
4. What does a gradient indicate about an objective function?
5. Give one reason accuracy may be unsuitable for an imbalanced classification problem.

Long Answer Questions

1. Compare supervised, unsupervised, semi-supervised, and reinforcement learning, giving one application of each.
2. Explain how vectors, dot products, norms, and matrix multiplication support machine learning models.
3. Derive the gradient-descent update conceptually and discuss the role of the learning rate.
4. Explain underfitting, overfitting, bias, variance, and regularization as related concepts.
5. Design a leakage-resistant evaluation workflow for a time-dependent prediction problem.

Analytical / Application Questions

1. A hospital predicts readmission using repeated records from the same patient. Explain why a random row-level split may be invalid and propose a better evaluation design.
2. A fraud detector has high accuracy but misses most fraudulent transactions. Which metrics and operational choices should be examined, and why?
3. In the equipment-failure case, analyse how target definition, prediction horizon, probability calibration, and decision thresholds affect deployment value.

Multiple Choice Questions with Answers

Question 1

Which quantity is learned directly from training data in a linear model?

A. The learning-rate schedule

B. The model weights

C. The test-set definition

D. The business cost of failure

Correct Answer: B. The model weights

Question 2

For x∈ℝᵈ and z∈ℝᵈ, what is the result of xᵀz?

A. A scalar

B. A d×d matrix

C. A vector of length 2d

D. Undefined in all cases

Correct Answer: A. A scalar

Question 3

Which expression defines mean squared error?

A. (1/n)∑ᵢ|ŷᵢ−yᵢ|

B. (1/n)∑ᵢ(ŷᵢ−yᵢ)²

C. ∑ᵢŷᵢ/yᵢ

D. (1/n)∑ᵢlog(ŷᵢ−yᵢ)

Correct Answer: B. (1/n)∑ᵢ(ŷᵢ−yᵢ)²

Question 4

In gradient descent, assuming a differentiable objective, parameters are generally updated in which direction?

A. The direction of the positive gradient

B. The direction opposite the gradient

C. A randomly selected direction only

D. The direction of maximum parameter magnitude

Correct Answer: B. The direction opposite the gradient

Question 5

What is the primary purpose of a validation set?

A. To estimate parameters while preventing any training error

B. To select models or hyperparameters

C. To replace the training set after deployment

D. To guarantee zero distribution shift

Correct Answer: B. To select models or hyperparameters

Question 6

Which situation is an example of data leakage?

A. Using a training-only scaler on later data

B. Using a future outcome variable as a predictor at prediction time

C. Evaluating a model on a held-out test set

D. Removing duplicate records before splitting

Correct Answer: B. Using a future outcome variable as a predictor at prediction time

Question 7

If the current parameter is w=1, the gradient is −4, and η=0.1, what is the gradient-descent update?

A. 0.6

B. 1.4

C. 4.9

D. −0.4

Correct Answer: B. 1.4

Question 8

Which statement about correlation is correct?

A. It proves that one variable causes another

B. It measures only perfect nonlinear dependence

C. It measures linear association and is unitless

D. It can exceed 100 when variables are strongly related

Correct Answer: C. It measures linear association and is unitless

Question 9

What is a typical effect of increasing ridge-regularization strength?

A. Weights are generally shrunk toward zero

B. The test set becomes part of training automatically

C. All bias is eliminated

D. The number of observations increases

Correct Answer: A. Weights are generally shrunk toward zero

Question 10

Why can a chronological split be preferable for equipment-failure prediction?

A. It guarantees the model is causal

B. It reflects the fact that deployment predicts future events from past data

C. It always produces balanced classes

D. It removes the need for a validation set

Correct Answer: B. It reflects the fact that deployment predicts future events from past data