Supervised Learning

Introduction and Context

Supervised learning is the branch of machine learning in which an algorithm learns a mapping from observed inputs to known outputs. Each training example contains a feature vector and a target value. The target may be a continuous quantity, such as house price, or a discrete class, such as fraudulent versus legitimate transaction. After learning from labelled examples, the model predicts targets for previously unseen inputs.

The central problem is to estimate a function f(x) that generalises beyond the training data. A dataset is commonly represented as D={(x_i,y_i)}_i=1ⁿ, where x_i is the feature vector for observation i, y_i is its target, and n is the number of observations. Supervised learning is fundamental to credit scoring, medical decision support, predictive maintenance, demand forecasting, speech recognition and computer vision.

A successful model must balance two risks: underfitting, where it is too simple to capture important structure, and overfitting, where it memorises training-specific noise. The goal is not merely high training accuracy, but reliable performance on future data.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish regression from classification and select suitable evaluation metrics.
2. Explain the assumptions, objective functions and limitations of linear and logistic regression.
3. Construct and interpret decision trees and ensemble models.
4. Explain support vector machines, nearest-neighbour methods and probabilistic classifiers.
5. Apply a principled workflow for preprocessing, training, validation and model selection.
6. Evaluate supervised models critically, including fairness, calibration, interpretability and operational risk.

1. Problem Formulation and Generalisation

Let X denote the input space and Y the output space. A supervised algorithm uses a labelled training set to select a model f_θ parameterised by θ. The model is fitted by minimising an empirical loss:

Here, L is a loss function, Ω(θ) is an optional regularisation penalty, and λ≥0 controls the penalty strength. The formulation assumes that examples are sufficiently representative of future cases and that the loss reflects the practical cost of errors. Regularisation is useful when model flexibility or correlated features creates overfitting.

The expected prediction risk is:

where P is the unknown population distribution. Since P is not available, training uses empirical risk and estimates generalisation through validation or test data.

A sound experimental design separates data into training, validation and test sets. The training set estimates parameters; the validation set selects hyperparameters such as tree depth or regularisation strength; the test set is used once, at the end, to estimate final performance. Data leakage occurs when information unavailable at prediction time enters training, preprocessing or model selection.

Regression and classification

Regression predicts a numerical output. Typical losses include mean squared error (MSE) and mean absolute error (MAE):

MSE penalises large errors more strongly and is differentiable, whereas MAE is more resistant to outliers. Both have the same units as the target after taking the square root of MSE; RMSE is RMSE=√MSE.

Classification predicts a class label. In binary classification, the target is often represented as y∈{0,1}. The confusion matrix records true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). Important measures include:

Accuracy is informative when class costs and frequencies are reasonably balanced. Precision matters when false alarms are expensive; recall matters when missed positives are dangerous. The F1 score is appropriate when a balance between precision and recall is required, but it does not include true negatives and should not be treated as universally superior.

A classifier often produces a score or probability rather than a final class. A threshold converts that output into a decision. Changing the threshold changes the trade-off between false positives and false negatives. The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate over thresholds; the area under the curve (ROC–AUC) measures ranking ability, not probability calibration. For strongly imbalanced data, precision–recall curves may be more informative.

2. Linear Regression

Linear regression models a continuous target as a linear combination of features:

The intercept is β₀, β_j is the expected change in the target for a one-unit increase in x_j, holding other included features constant, and ŷ is the prediction. The equation is linear in its parameters; transformed features such as x² can still be included.

Ordinary least squares (OLS) estimates coefficients by minimising the residual sum of squares:

In matrix notation, when X^TX is invertible:

The matrix formula assumes a correctly specified linear conditional mean, independent observations, finite variance, and no perfect multicollinearity. Normality of errors is not required for unbiased coefficient estimation, but is useful for small-sample hypothesis tests and confidence intervals.

Important practical assumptions are linearity of the conditional mean, independent errors, approximately constant error variance (homoscedasticity), and absence of severe multicollinearity. Residual plots can reveal nonlinearity or changing variance. High correlation among predictors can make individual coefficients unstable even when predictions remain accurate.

Regularised linear models

Ridge regression adds an L2 penalty:

Lasso regression uses an L1 penalty:

λ controls shrinkage and is normally selected by cross-validation. Ridge tends to distribute weight across correlated features; lasso can set some coefficients exactly to zero and therefore perform feature selection. The intercept is generally not penalised, and predictors should usually be standardised when penalty magnitudes are to be comparable.

Worked example: linear prediction

Suppose a fitted model predicts monthly electricity use from floor area:

where A is area in square metres and ŷ is kilowatt-hours per month. For a 100 m² building, ŷ=120+0.85(100)=205 kWh/month. The coefficient 0.85 means that, within the model’s data range and holding omitted influences implicit, an additional square metre is associated with 0.85 kWh/month higher predicted use. It is not proof that increasing area causally produces that increase.

3. Logistic Regression

Logistic regression is a linear model for binary classification. It models the log-odds of the positive class:

The probability is obtained with the sigmoid function:

The output p lies between 0 and 1. A default threshold of 0.5 predicts class 1 when p≥0.5, although the operationally appropriate threshold depends on error costs and prevalence.

Parameters are estimated by maximum likelihood, equivalently by minimising binary cross-entropy:

The observations are assumed to be conditionally independent, and the log-odds are assumed to be linear in the predictors. Logistic regression does not require normally distributed predictors. Coefficient β_j represents the change in log-odds for a one-unit increase in x_j, holding other predictors constant; e^β_j is the corresponding odds ratio.

For example, if β_j=0.7, the odds ratio is approximately e^0.7≈2.01. Thus a one-unit increase multiplies the odds by about 2.01, subject to the model assumptions. Odds are not the same as probability, so the coefficient should not be described as a direct percentage-point increase in probability.

Complete or quasi-complete separation occurs when predictors perfectly divide classes, causing very large coefficient estimates. L2 or L1 regularisation, feature revision and more data can help. Class weighting can address unequal error costs or imbalance, but it does not automatically solve biased sampling.

4. Decision Trees

A decision tree recursively partitions the feature space into regions and assigns a prediction to each terminal leaf. A regression leaf commonly predicts the mean target in that region. A classification leaf commonly predicts the majority class or class proportions.

For classification, a node impurity measure can be the Gini impurity:

where p_k|t is the proportion of class k at node t. Entropy is another measure:

A split is selected when it produces a sufficiently large reduction in weighted impurity. If a parent node has proportion w_L and w_R of observations in its left and right children, the weighted child impurity is w_LI(L)+w_RI(R). The impurity decrease is parent impurity minus this quantity.

Trees handle nonlinear relationships, interactions and mixed feature scales with little preprocessing. However, fully grown trees can memorise training data and are unstable: a small data change may produce a different structure. Control complexity through maximum depth, minimum leaf size, minimum split size, pruning and cross-validation. A tree’s apparent interpretability can be misleading when it is very deep or when correlated variables permit many interchangeable splits.

5. Ensemble Methods

Ensembles combine multiple models to improve predictive performance.

Bagging trains models on bootstrap samples and averages regression predictions or votes in classification. Averaging reduces variance when component models make partly independent errors. Random forests add random feature selection at each split, decorrelating trees and often improving generalisation.

Boosting builds models sequentially, with later models focusing on errors made by earlier ones. In gradient boosting, each new learner approximately follows the negative gradient of the loss with respect to current predictions. A learning rate, number of iterations and base-learner complexity control the bias–variance trade-off. Modern gradient-boosted trees are powerful for tabular data, but require careful validation and can be sensitive to leakage and hyperparameters.

A random forest is often robust with minimal scaling; boosted trees can achieve excellent accuracy but may be less transparent. Feature importance must be interpreted carefully. Impurity-based importance can favour high-cardinality variables, while permutation importance can be distorted by correlated predictors. Partial-dependence and local explanation methods describe model behaviour but are not causal explanations.

6. Support Vector Machines

For linearly separable binary data, a support vector machine seeks a separating hyperplane w^Tx+b=0 with maximum margin. The margin is the distance between the boundary and the nearest training points. With soft margins, the optimisation is:

subject to

Here ξ_i are slack variables and C>0 controls the penalty for violations. A large C prioritises fitting training points; a smaller value permits more violations and a wider-margin solution. Features should generally be scaled because distances and inner products depend on feature magnitude.

Nonlinear boundaries can be obtained through kernels, such as the radial basis function kernel:

where γ controls the locality of influence. Large γ can create highly local, complex boundaries. Kernel SVMs can be computationally expensive for very large datasets and their probability outputs may require calibration.

7. Nearest-Neighbor Methods

The k-nearest-neighbours (k-NN) method predicts from the labels of the k training observations closest to a query point. For numerical features, Euclidean distance is commonly used:

Classification may use majority voting, while regression may average neighbours. Small k gives flexible, low-bias predictions but high variance; larger k smooths predictions but may underfit. Select k by validation.

Distance-based methods require meaningful feature scaling. A common standardisation is:

where μ_j and σ_j are calculated from the training set only. Categorical variables require an appropriate representation and distance function. k-NN has low training cost but potentially high prediction cost and suffers in high dimensions, where distances become less informative.

8. Probabilistic Classifiers

Naïve Bayes

Bayes’ rule gives:

Naïve Bayes assumes that features are conditionally independent given the class:

This assumption is often unrealistic, but the classifier can perform well, particularly with text data. Gaussian naïve Bayes models continuous features with class-specific normal distributions; multinomial naïve Bayes is common for count-based text features. Smoothing prevents zero likelihoods for unseen categorical or word events.

Calibration and decision quality

A probability of 0.8 should mean that approximately 80% of comparable cases are positive if the model is calibrated. A model can rank cases well but be poorly calibrated. Reliability diagrams, calibration curves and the Brier score are useful diagnostics. The Brier score for binary outcomes is:

Lower values are better, although the score is affected by prevalence and separates calibration from discrimination only imperfectly.

9. A Professional Modelling Workflow

A defensible workflow is iterative rather than a one-time algorithm choice.

1. Define the decision. Specify the prediction time, target, population, acceptable errors and intervention capacity.
2. Audit data. Examine missingness, duplicates, outliers, label quality, sampling bias and temporal structure.
3. Split appropriately. Use a time-based split for forecasting or random stratification when observations are exchangeable. Keep a final test set untouched.
4. Build a baseline. Examples include predicting the training mean for regression or the majority class for classification.
5. Preprocess within a pipeline. Fit imputation, scaling, encoding and feature selection only on training folds.
6. Train candidates. Compare interpretable models with nonlinear and ensemble alternatives.
7. Tune with cross-validation. Select hyperparameters using validation folds, preferably with nested cross-validation when estimating performance after extensive tuning.
8. Evaluate relevant metrics. Report uncertainty, subgroup performance, calibration and a confusion matrix at the proposed threshold.
9. Check robustness. Test temporal drift, missing inputs, distribution shift and sensitivity to plausible changes.
10. Deploy and monitor. Track data drift, performance, fairness, latency, failures and changes in the decision environment.

Applied Case: Predicting Loan Default

The following is an illustrative case, not a report of a specific institution.

Background and problem

A lender wants to identify applications requiring additional review. The target is whether an account becomes seriously delinquent within 12 months. False negatives may cause financial loss, while false positives may unfairly delay or exclude reliable applicants.

Analysis

Available variables include income, debt-to-income ratio, repayment history, account age and application channel. The team discovers that a randomly selected split would place applications from the same customer in both training and test sets. This would inflate performance, so customer-level grouping is required. Because lending decisions change over time, a later-period test set is also retained.

Approach and solution

A regularised logistic regression is developed as an interpretable baseline. A random forest and gradient-boosted tree model are then trained in leakage-safe pipelines. Missing income values are imputed within each training fold; numerical features are scaled for logistic regression but not required for trees. Class weights are considered because defaults are less frequent than non-defaults.

The models are compared using recall, precision, PR–AUC, calibration and subgroup metrics rather than accuracy alone. A threshold is chosen with business and compliance staff. The output is used to prioritise human review, not to make an irreversible automatic rejection. Calibration is checked on a recent validation period, and documentation records training data, features, exclusions, threshold rationale and known limitations.

Outcome and learning points

The case demonstrates that the strongest model by a single score may not be the best operational model. A slightly less accurate but well-calibrated and explainable model may be preferable when decisions affect access to credit. Performance must be assessed on future-like data, and the organisation must distinguish prediction from causal claims: a risk score identifies association with default, not the effect of granting or denying credit.

Limitations, Ethics and Governance

Supervised models inherit limitations in their labels and data. Historical decisions may encode discrimination; missing groups may be underrepresented; and a proxy such as postcode may reproduce protected characteristics. Fairness is not a single universally agreed metric. Demographic parity, equal opportunity and calibration can conflict, especially when group base rates differ. The relevant legal and institutional requirements must therefore guide metric selection.

Data minimisation, access control, retention limits and secure handling are essential. Individuals may require notice, explanation, appeal or human review, depending on jurisdiction and application. Sensitive attributes may be excluded from the model but still needed for fairness auditing under controlled governance. Documentation should state intended use, out-of-scope use, data provenance, evaluation population and monitoring plan.

Models can degrade because of concept drift, measurement changes, policy changes or adversarial behaviour. Monitoring should compare current feature distributions and outcomes with reference data, while avoiding automatic retraining without validation. Human oversight must be meaningful: reviewers need authority, information and time to challenge predictions.

Contemporary Developments and Best Practices

Current practice increasingly combines strong tabular learners with explainability, calibration and responsible-AI controls. Automated machine-learning systems can search preprocessing and model configurations, but automation does not remove the need for problem definition, leakage control or governance. Cost-sensitive learning and conformal prediction can provide useful decision information, such as prediction sets or intervals under stated assumptions. Deep learning is particularly important for images, audio and language, whereas linear models and tree ensembles remain highly competitive for many structured business datasets.

Best practice is to begin with a transparent baseline, establish a leakage-resistant evaluation design, compare models using decision-relevant metrics, document assumptions, and monitor the deployed system. Technical performance should be considered alongside interpretability, fairness, cost, security and maintainability.

Chapter Summary

Supervised learning estimates relationships between labelled inputs and targets for prediction. Regression predicts continuous values and commonly uses MSE, MAE and RMSE; classification predicts categories and requires careful use of confusion-matrix metrics, thresholds and calibration. Linear regression provides an interpretable additive model, while regularisation controls variance and multicollinearity. Logistic regression models log-odds and produces probabilities through the sigmoid function.

Decision trees represent nonlinear rules but require complexity control. Bagging and random forests reduce variance, while boosting builds an accurate sequence of weak learners. Support vector machines maximise a margin and can use kernels; k-nearest neighbours relies on scaled distances; and naïve Bayes applies Bayes’ rule with a conditional-independence assumption. No algorithm is universally best. Reliable practice depends on representative data, leakage-safe validation, relevant metrics, fairness assessment, documentation and ongoing monitoring.

Key Terms

Supervised learning: Learning from examples with known target values.

Feature: An input variable used for prediction.

Target: The outcome a model is trained to predict.

Regression: Prediction of a continuous or numerical outcome.

Classification: Prediction of a discrete class or category.

Loss function: A numerical measure of prediction error.

Regularisation: A penalty used to control model complexity.

Overfitting: Learning noise or training-specific detail that harms generalisation.

Underfitting: Failure to represent important structure in the data.

Log-odds: The logarithm of the odds of an event.

Confusion matrix: A table of true and false prediction outcomes.

Precision: The proportion of predicted positives that are positive.

Recall: The proportion of actual positives that are detected.

Decision tree: A recursive sequence of feature-based partitions.

Random forest: An ensemble of decorrelated decision trees.

Boosting: Sequentially combining learners that focus on prior errors.

Support vector machine: A margin-based classifier or regressor.

Nearest neighbour: A method that predicts from nearby training examples.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data leakage: Use of information unavailable at prediction time.

Review Questions

Short Answer Questions

1. Distinguish regression from classification and give one example of each.
2. Why should preprocessing be fitted within training folds rather than on the complete dataset?
3. What does a logistic-regression coefficient represent on the log-odds scale?
4. State one advantage and one limitation of decision trees.
5. Why is feature scaling important for k-nearest neighbours and support vector machines?

Long Answer Questions

1. Explain the bias–variance trade-off and relate it to regularised regression, tree depth and ensemble methods.
2. Compare linear regression and logistic regression in terms of target type, objective function, assumptions and interpretation.
3. Explain how a confusion matrix supports threshold selection in a high-stakes classification problem.
4. Compare random forests and gradient boosting with respect to training strategy, variance, interpretability and tuning.
5. Describe a leakage-resistant workflow for developing and deploying a supervised-learning model.

Analytical / Application Questions

1. A medical screening model has high accuracy but misses many positive cases. Which metrics and threshold changes should be investigated, and why?
2. A k-NN model performs well before deployment but poorly after a new measurement device is introduced. Diagnose likely causes and propose a validation and monitoring plan.
3. A credit model has excellent ROC–AUC but poorly calibrated probabilities and unequal recall across groups. Recommend a technically and ethically defensible deployment strategy.

Multiple Choice Questions with Answers

Question 1

Which task is a regression problem?

A. Predicting whether an email is spam

B. Predicting the monthly energy consumption of a building

C. Assigning an image to one of five classes

D. Detecting whether a transaction is fraudulent

Correct Answer: B. Predicting the monthly energy consumption of a building

Question 2

What does the coefficient β_j in logistic regression directly describe?

A. A change in probability measured in percentage points for every case

B. A change in the log-odds for a one-unit increase in x_j, holding other variables constant

C. The variance of feature x_j

D. The classification accuracy contributed by feature x_j

Correct Answer: B. A change in the log-odds for a one-unit increase in x_j, holding other variables constant

Question 3

Which metric is most directly focused on the proportion of predicted positives that are truly positive?

A. Recall

B. Accuracy

C. Precision

D. Specificity

Correct Answer: C. Precision

Question 4

What is the main purpose of regularisation?

A. To guarantee causal interpretation

B. To increase the number of features automatically

C. To control model complexity and reduce overfitting

D. To remove the need for validation data

Correct Answer: C. To control model complexity and reduce overfitting

Question 5

Which method combines predictions from many bootstrap-trained trees and introduces random feature selection?

A. Random forest

B. Logistic regression

C. Naïve Bayes

D. Single-layer perceptron

Correct Answer: A. Random forest

Question 6

In a support vector machine, parameter C primarily controls:

A. The number of classes

B. The penalty assigned to margin violations

C. The number of nearest neighbours

D. The probability threshold used after deployment

Correct Answer: B. The penalty assigned to margin violations

Question 7

Why should numerical features usually be scaled before using k-nearest neighbours?

A. Scaling converts regression into classification

B. Scaling prevents all missing values

C. Distances can otherwise be dominated by features with large numerical units

D. Scaling guarantees perfect accuracy

Correct Answer: C. Distances can otherwise be dominated by features with large numerical units

Question 8

The naïve Bayes classifier assumes that features are:

A. Conditionally independent given the class

B. Always normally distributed in the complete population

C. Causally independent of the target

D. Measured on identical numerical scales

Correct Answer: A. Conditionally independent given the class

Question 9

A model has TP=40, FP=10 and FN=20. What is its recall?

A. 0.50

B. 0.67

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 10

Which practice is an example of data leakage?

A. Using a later-period dataset as a final test set

B. Selecting a classification threshold using a validation set

C. Fitting an imputer on the full dataset before cross-validation

D. Comparing a regularised model with a baseline

Correct Answer: C. Fitting an imputer on the full dataset before cross-validation