Supervised Machine Learning

Introduction and Context

Supervised machine learning learns a relationship between observed inputs and known outputs. A training dataset consists of pairs (x_i, y_i), where x_i ∈ R^p is a vector of p features and y_i is the response. The objective is to estimate a function f that performs well on previously unseen observations, not merely to reproduce the training data.

Supervised learning supports credit-risk assessment, medical diagnosis, demand forecasting, fraud detection, recommendation, predictive maintenance and language classification. Its central difficulty is generalization: a model must capture stable structure while ignoring noise and accidental patterns.

Learning Objectives

After completing this chapter, students should be able to:

• distinguish regression from classification and select suitable loss functions;
• formulate linear, generalized linear, tree-based, ensemble, support-vector and nearest-neighbor models;
• explain regularization, optimization and the bias–variance trade-off;
• evaluate models using validation procedures and task-appropriate metrics;
• interpret model outputs, diagnose failure and address ethical and governance risks.

1. Problem Formulation

Let the data be D = {(x_i,y_i)}_i=1ⁿ, drawn from an unknown joint distribution P(X,Y). A learning algorithm uses a training set to produce f_hat. For a loss function L, the empirical risk is

Here, n is the number of observations. The expected or population risk is

The expectation refers to future observations from the same target population. Empirical risk minimization is useful only when combined with controls against overfitting, appropriate sampling, and honest evaluation.

In regression, y is usually numeric and continuous. In binary classification, y ∈ {0,1}; in multiclass classification, y belongs to one of K categories. Features may be numeric, categorical, textual or derived from other data. Preprocessing must be fitted using training data only; otherwise information from the test set leaks into the model.

2. Regression Models

2.1 Linear Regression

The multiple linear regression model is

where β_0 is the intercept, β_j are slopes, x_ij are features, and ε_i is an error term. In matrix form, y = Xβ + ε. Ordinary least squares estimates coefficients by minimizing the residual sum of squares:

When X^TX is invertible, the closed-form solution is

The formula assumes a column for the intercept if one is required. It is not appropriate when X^TX is singular; numerical solvers or regularization are then used. The coefficient β_j represents the expected change in y for a one-unit increase in feature j, holding other included features constant. This interpretation can be misleading under strong collinearity, omitted confounding, extrapolation or nonlinear relationships.

A standard regression estimate is the mean squared error:

MSE has squared response units and heavily penalizes large errors. Root mean squared error is RMSE = √MSE, in the original response units. Mean absolute error is

MAE is less sensitive to outliers than MSE. R² is commonly defined as

where y_bar is the training or evaluation-set mean, as appropriate. R² can be negative on a test set and does not establish causation or practical usefulness.

2.2 Regularized Linear Models

Regularization adds a penalty to discourage excessive coefficient magnitude. Ridge regression solves

and lasso solves

The intercept is generally excluded from the penalty. λ ≥ 0 controls the penalty: λ = 0 gives the unregularized objective, while larger λ increases shrinkage. Ridge usually retains correlated predictors and stabilizes estimates; lasso can set coefficients exactly to zero and therefore perform embedded feature selection. Features should normally be standardized before penalizing them, because penalty magnitude depends on scale.

Elastic net combines both penalties:

where 0 ≤ α ≤ 1. The exact scaling convention varies across software, so reported hyperparameters must be documented.

2.3 Generalized Linear Models

Linear regression is unsuitable when the response distribution or mean–feature relationship is constrained. A generalized linear model has: (1) a random component from an exponential-family distribution, (2) a linear predictor η = β₀ + x^Tβ, and (3) a link function g satisfying g(μ)=η, where μ = E[Y|X=x].

For binary classification, logistic regression uses the logit link:

Thus

The model estimates a probability, which becomes a class prediction only after selecting a threshold. β_j is the change in log-odds for a one-unit increase in x_j, holding other variables fixed; exp(β_j) is the corresponding odds ratio. Logistic regression is linear in log-odds, not necessarily in probability.

For count data, Poisson regression commonly uses log(μ)=β₀+x^Tβ, so exp(β_j) is a multiplicative rate ratio under the model. Overdispersion, excess zeros and exposure time may require alternatives such as negative-binomial models or an offset.

3. Optimization and Generalization

Many models minimize an objective consisting of data loss plus regularization. Gradient descent updates parameters by

where θ is the parameter vector, η > 0 is the learning rate, J is the objective, and ∇J is its gradient. If η is too large, optimization may diverge; if too small, training is inefficient. Stochastic and mini-batch gradient descent use subsets of observations and introduce useful computational noise. Convex objectives such as ordinary least squares and logistic loss have favorable global optimization properties, whereas tree construction and some ensemble procedures use different search strategies.

The bias–variance decomposition, for squared prediction error, is conceptually

where σ² is irreducible noise, bias reflects systematic error from an overly restricted model, and variance reflects sensitivity to the training sample. Increasing complexity often decreases bias but increases variance. Regularization, more data, validation-based tuning, bagging and feature control can reduce harmful variance. There is no universally optimal complexity: it depends on the data-generating process and decision costs.

Cross-validation estimates generalization performance. In k-fold cross-validation, data are split into k folds; each fold is held out once while the other k−1 folds train the model. Hyperparameters must be selected inside the training portion, preferably with nested cross-validation when an unbiased performance estimate is required. A final test set should be used once for final assessment.

4. Decision Trees and Ensembles

A decision tree recursively partitions feature space using rules such as x_j ≤ c. A regression leaf predicts the mean response of its training observations; a classification leaf predicts a class probability or majority class.

For classification, a node impurity measure may be the Gini impurity:

where p_k is the proportion of class k in the node. Entropy is -∑p_k log₂(p_k). A split is selected by its impurity reduction, weighted by child sizes. Trees can model interactions and nonlinearities without feature scaling, but deep trees have high variance and are vulnerable to noisy features and small sample changes. Maximum depth, minimum leaf size, pruning and minimum split size are important controls.

Random forests average many trees trained on bootstrap samples and random feature subsets. For regression, averaging reduces variance; for classification, class voting or averaged probabilities is used. Out-of-bag observations can provide an internal performance estimate, although they do not replace careful external validation in high-stakes applications.

Boosting builds an additive model sequentially, with later learners emphasizing prior errors. In gradient boosting, a new learner approximately follows the negative gradient of the loss. Learning rate, number of trees, tree depth and subsampling govern the bias–variance balance. Modern implementations, including histogram-based and regularized gradient boosting, are powerful for tabular data but can overfit, encode leakage and produce poorly calibrated probabilities unless assessed carefully.

5. Support Vector Machines

A linear support vector classifier seeks a separating hyperplane w^Tx+b=0. For labels y_i ∈ {−1,+1}, the soft-margin objective is commonly written

subject to y_i(w^Tx_i+b) ≥ 1-ξ_i and ξ_i ≥ 0. The slack variables ξ_i permit violations, and C controls the trade-off between margin width and training violations. A large C penalizes violations strongly and may increase variance; a smaller C allows a wider margin and more regularization.

The kernel method replaces inner products with a kernel function, such as the radial basis function kernel

where γ > 0 controls locality. Kernel parameters and features must be tuned using validation. SVMs are sensitive to feature scale, may be computationally expensive for very large datasets, and produce scores that are not automatically calibrated probabilities.

6. Nearest-Neighbor Methods

The k-nearest-neighbor algorithm predicts from the k closest training cases under a distance measure. Euclidean distance is

For classification, the prediction is often the majority class; for regression, it is the mean of neighbor responses. Small k gives flexible, low-bias but high-variance predictions; large k smooths the function and increases bias. Distance-based methods require scaling, thoughtful treatment of categorical variables and protection against irrelevant or high-dimensional features. Approximate neighbor indexes may be needed for large datasets.

7. Evaluation, Calibration and Model Selection

A classification confusion matrix contains true positives, true negatives, false positives and false negatives. Important metrics include

and

Specificity is TN/(TN+FP), and accuracy is (TP+TN)/(TP+TN+FP+FN). Accuracy is misleading under severe class imbalance. F1 is the harmonic mean of precision and recall:

ROC-AUC measures ranking across classification thresholds, whereas precision–recall curves are often more informative when the positive class is rare. The metric must reflect operational costs. If false negatives are substantially more costly than false positives, the threshold should not automatically be 0.5.

Calibration asks whether predicted probabilities correspond to observed frequencies. A group of cases predicted at 0.8 should contain approximately 80% positives if the model is well calibrated for that population. Calibration curves, reliability diagrams, Brier score and recalibration methods are useful. Discrimination and calibration are distinct: a model can rank cases well while producing systematically inaccurate probabilities.

A robust workflow includes problem definition, data audit, stratified or time-aware splitting, leakage-resistant preprocessing, baseline modeling, cross-validated tuning, held-out testing, subgroup evaluation, calibration, documentation and monitoring. For time-dependent prediction, random splitting may be invalid because it allows future information to influence training.

8. Illustrative Case: Credit-Card Default Prediction

Background

A financial institution wants to identify accounts at high risk of default within 90 days. The data include transaction summaries, account age, repayment history and current balance. This is an illustrative case, not a report of a particular institution.

Problem

The target is binary and the positive class is uncommon. Missingness, changing economic conditions and unequal error costs are expected. A model that maximizes accuracy could classify nearly every account as non-default and still appear successful.

Analysis and Approach

The team first defines the prediction time and excludes variables recorded after that time, preventing temporal leakage. A time-based split reserves the most recent period for testing. Numeric variables are imputed and standardized within each training fold; categorical variables are encoded within the same pipeline. Logistic regression with regularization provides an interpretable baseline. A gradient-boosted tree is then tuned using nested time-aware validation.

Evaluation emphasizes recall at an acceptable precision, expected financial loss, calibration and performance across customer subgroups. Threshold selection is performed on validation data, not on the final test set. Probability calibration is checked because risk scores determine intervention intensity. The model card records data provenance, exclusions, intended use, limitations and monitoring plans.

Outcome and Learning Points

Suppose the boosted model ranks risk better than the baseline but is poorly calibrated for newer customers. The institution should not deploy it unchanged. Recalibration, additional data or a restricted-use policy may be preferable to optimizing a single aggregate score. Human review, adverse-action explanations, appeal procedures and drift monitoring are part of the system, not optional additions.

The case illustrates four principles: define the prediction moment, align metrics with costs, separate ranking from probability accuracy, and evaluate fairness and stability before operational use.

9. Limitations, Ethics and Governance

Supervised models inherit defects in labels and sampling. Historical decisions may encode discrimination; missing data may be systematically related to protected status; and a model can perform differently across demographic or geographic groups. Fairness cannot be reduced to one universal metric: demographic parity, equal opportunity and calibration may conflict under differing base rates.

Practical safeguards include data minimization, lawful and transparent purposes, access control, lineage, documentation, subgroup testing, explainability appropriate to the audience, human oversight and appeal mechanisms. Privacy-preserving practices should limit unnecessary collection and retention. In regulated domains, auditability, reproducibility and evidence of model validation are essential. Explanations should distinguish statistical association from causal reasoning.

Contemporary practice increasingly combines predictive models with uncertainty estimation, conformal prediction, interpretable generalized additive models, automated monitoring and responsible MLOps. These tools improve reliability but do not eliminate assumptions or governance obligations.

Best-Practice Checklist

1. Define the target, prediction horizon and decision cost.
2. Establish a simple baseline before selecting complex models.
3. Split data according to deployment reality, especially time.
4. Fit preprocessing only on training data.
5. Tune hyperparameters with cross-validation and preserve a final test set.
6. Compare discrimination, calibration, subgroup performance and operational cost.
7. Regularize and control complexity rather than relying on training accuracy.
8. Document assumptions, data provenance, limitations and intended use.
9. Monitor drift, missingness, performance and fairness after deployment.
10. Retire or revise the model when evidence shows that it is no longer reliable.

Chapter Summary

Supervised machine learning estimates predictive relationships from labeled examples. Linear and generalized linear models offer transparent structures and useful statistical interpretations; regularization controls instability and can support feature selection. Trees represent nonlinear rules and interactions, while random forests reduce variance through averaging and boosting builds sequentially improved predictors. Support vector machines optimize margins and can represent nonlinear boundaries through kernels. Nearest-neighbor methods are simple and locally adaptive but depend strongly on scaling and dimensionality.

Generalization depends on the bias–variance trade-off, appropriate optimization, leakage-resistant validation and complexity control. Regression and classification require different losses and metrics, and classification performance must consider threshold choice, imbalance and calibration. A professional model is not defined only by predictive score: ethics, fairness, privacy, documentation, monitoring and decision consequences are equally important.

Key Terms

Supervised learning: Learning from examples with observed target values.

Regression: Prediction of a numeric response.

Classification: Prediction of a discrete class or class probability.

Loss function: Quantitative penalty for an incorrect prediction.

Empirical risk: Average loss on observed training examples.

Generalization: Performance on unseen data from the target population.

Regularization: Penalization used to control model complexity.

Ridge regression: L2-penalized linear regression.

Lasso: L1-penalized regression that can produce zero coefficients.

Generalized linear model: A model using a distribution, linear predictor and link function.

Logit link: The logarithm of odds used in logistic regression.

Decision tree: A recursively partitioned prediction structure.

Random forest: An ensemble of randomized bootstrap trees.

Boosting: Sequential construction of learners to improve a loss objective.

Support vector machine: A margin-based predictive method.

Kernel: A function representing an implicit feature-space inner product.

Nearest neighbor: Prediction based on similar training observations.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data leakage: Use of information unavailable at prediction time.

Bias–variance trade-off: Balance between systematic error and sensitivity to sample variation.

Review Questions

Short Answer Questions

1. What is the difference between empirical risk and population risk?
2. Why should features usually be standardized before ridge, lasso, SVM or nearest-neighbor modeling?
3. What does the coefficient of a logistic regression model represent?
4. State one advantage and one limitation of decision trees.
5. Why can accuracy be misleading for an imbalanced classification problem?

Long Answer Questions

1. Explain the bias–variance trade-off and discuss how regularization, bagging and boosting affect it.
2. Compare ordinary least squares, ridge regression and lasso regression, including their objectives and practical uses.
3. Explain how generalized linear models extend linear regression and use logistic regression as an example.
4. Compare random forests, gradient boosting, SVMs and nearest-neighbor methods in terms of assumptions, preprocessing and scalability.
5. Design a leakage-resistant model-development workflow for a regulated prediction problem.

Analytical / Application Questions

1. A fraud dataset has 1% positive cases. Recommend evaluation metrics, a validation strategy and a threshold-selection procedure, explaining your choices.
2. A model has excellent ROC-AUC but poor calibration. Explain what this means and propose a deployment remedy.
3. A credit model performs well overall but has substantially lower recall for one subgroup. Analyze possible causes and propose technical and governance responses.

Multiple Choice Questions with Answers

Question 1

Which method can set some coefficients exactly to zero?

A. Ridge regression

B. Lasso regression

C. Ordinary least squares

D. Nearest-neighbor regression

Correct Answer: B. Lasso regression

Question 2

In logistic regression, exp(β_j) represents the multiplicative change in which quantity for a one-unit increase in feature j, holding other features fixed?

A. Mean squared error

B. Probability difference in all cases

C. Odds

D. Residual variance

Correct Answer: C. Odds

Question 3

Which metric is generally most sensitive to large regression errors?

A. MAE

B. MSE

C. Median absolute error

D. Accuracy

Correct Answer: B. MSE

Question 4

What is the principal purpose of a validation set?

A. To increase the number of labels

B. To select models or hyperparameters without using the final test set

C. To guarantee causal interpretation

D. To remove all measurement error

Correct Answer: B. To select models or hyperparameters without using the final test set

Question 5

A classifier predicts 90 true positives and 30 false positives. Its precision is:

A. 0.25

B. 0.67

C. 0.75

D. 0.90

Correct Answer: C. 0.75

Question 6

Which ensemble primarily reduces variance by averaging many randomized trees?

A. Random forest

B. Single decision stump

C. Logistic regression

D. Linear SVM

Correct Answer: A. Random forest

Question 7

In a soft-margin SVM, parameter C controls the trade-off between:

A. Feature scaling and imputation

B. Margin size and penalty for violations

C. Precision and recall only

D. Number of classes and observations

Correct Answer: B. Margin size and penalty for violations

Question 8

Why is scaling especially important for k-nearest neighbors?

A. It changes the target labels

B. It ensures all classes have equal frequency

C. Distance calculations can otherwise be dominated by large-scale features

D. It removes the need for validation

Correct Answer: C. Distance calculations can otherwise be dominated by large-scale features

Question 9

What is data leakage?

A. Random variation in a test score

B. Using information unavailable at prediction time during training

C. A model with too few parameters

D. A calibrated probability estimate

Correct Answer: B. Using information unavailable at prediction time during training

Question 10

A model with strong ranking performance but poor calibration most directly needs assessment or improvement of its:

A. Probability estimates

B. Feature count only

C. Confusion-matrix labels

D. Training-set size units

Correct Answer: A. Probability estimates