Supervised Machine Learning

Introduction and Context

Supervised machine learning develops a predictive relationship between input variables and a known target variable. A learning algorithm receives examples of the form (xᵢ, yᵢ), where xᵢ contains measured characteristics and yᵢ is the observed outcome. It then estimates a function that can predict y for new, previously unseen observations.

Supervised learning is central to data science because many organisational questions have a labelled outcome: What will a customer spend? Will a transaction be fraudulent? Is a medical image likely to contain a tumour? Will a student complete a course? The quality of a model depends not only on the algorithm but also on how the problem is defined, how data are collected, how features are engineered, and how predictions are evaluated and governed.

Regression predicts a numerical quantity, whereas classification predicts a category. This chapter introduces linear models, decision trees, ensemble methods, and introductory neural networks. It also explains feature engineering, model evaluation, interpretation, leakage, fairness, and deployment considerations.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish regression from classification and formulate each as a supervised-learning problem.
2. Prepare features and targets while avoiding data leakage.
3. Explain and apply linear regression, logistic regression, decision trees, random forests, gradient boosting, and basic neural networks.
4. Select suitable evaluation metrics and interpret their limitations.
5. Compare models using validation data and cross-validation.
6. Interpret model outputs using coefficients, feature importance, permutation importance, and partial-dependence ideas.
7. Identify practical, ethical, and governance risks in predictive modelling.

1. The Supervised-Learning Framework

A dataset contains n observations and p input features. The feature vector for observation i is xᵢ = (xᵢ₁, …, xᵢₚ), and its target is yᵢ. A model is a parameterised function f̂(x) that estimates the unknown relationship between inputs and outcomes.

Here, ŷᵢ is the prediction for observation i. Training chooses model parameters to minimise a loss function, often with a complexity penalty:

θ denotes model parameters, L measures prediction error, Ω penalises complexity, and λ ≥ 0 controls the trade-off between fit and simplicity. The expression is applicable to many models, although the specific loss and penalty vary.

The available data are commonly divided into training, validation, and test sets. The training set estimates parameters; the validation set supports model selection and hyperparameter tuning; and the test set provides a final, relatively unbiased estimate of generalisation performance. When data are limited, k-fold cross-validation divides the training data into k folds, trains on k−1 folds, and validates on the remaining fold, repeating until each fold has been used for validation.

A model that memorises training observations but performs poorly on new data is overfitted. A model that is too simple to capture important structure is underfitted. The aim is not merely low training error but good performance on the target population.

Regression and classification

In regression, y is numerical, such as income, temperature, or delivery time. In classification, y belongs to a finite set of classes, such as {approved, rejected}. Binary classification has two classes; multiclass classification has more than two. A classifier often produces a probability before converting it into a class using a threshold.

A useful distinction is between parameters and hyperparameters. Parameters are learned from data, such as regression coefficients. Hyperparameters are selected by the analyst, such as tree depth, the number of trees, or the regularisation strength.

2. Data Preparation and Feature Engineering

The representation of the data often matters as much as the algorithm.

Data quality

Before modelling, inspect missing values, duplicates, inconsistent units, outliers, class imbalance, and unusual target values. Numerical variables may require imputation, transformation, or scaling. Categorical variables can be represented using one-hot encoding, in which each category becomes a binary indicator. For ordered categories, an ordinal representation may be appropriate, but assigning arbitrary numerical values to unordered categories can create a false relationship.

Standardisation transforms a feature x with mean μ and standard deviation σ into:

The mean and standard deviation must be calculated on the training data only and then applied to validation, test, and production data. Standardisation is particularly useful for linear models with regularisation and neural networks; tree-based models generally do not require it.

Feature engineering

Feature engineering uses domain knowledge to construct useful predictors. Examples include:

• extracting day of week, month, or hour from a timestamp;
• converting total expenditure into average expenditure per order;
• creating an interaction such as advertising spend × season;
• using logarithms for heavily right-skewed positive variables;
• aggregating transaction-level data into customer-level measures;
• encoding text through counts, n-grams, or learned representations.

Feature engineering must respect the prediction time. A variable recorded after the outcome occurs cannot legitimately be used to predict that outcome. This error is called target leakage and can produce impressive but invalid test performance.

A complete preprocessing pipeline should learn transformations only from the training partition. The same sequence must be applied consistently during validation and deployment.

3. Linear Models for Regression

Linear regression assumes that the conditional mean of y can be approximated by a weighted sum of features:

β₀ is the intercept, βⱼ is the coefficient for feature j, and εᵢ is the unexplained error. The ordinary least-squares estimator minimises the sum of squared residuals:

The model is most interpretable when the relationship is approximately additive and linear, errors are reasonably independent, and extreme outliers do not dominate the fit. A coefficient represents the expected change in y for a one-unit increase in xⱼ, holding the other included features constant. This is an associational interpretation, not automatically a causal one.

The coefficient of determination is:

SST = ∑ᵢ(yᵢ − ȳ)² is the total sum of squares and ȳ is the training or evaluation mean, as appropriate. R² measures improvement over a constant-mean baseline. It can be negative on test data and should not be treated as a universal measure of usefulness.

Regularisation can reduce variance. Ridge regression minimises:

Lasso regression minimises:

The intercept is usually not penalised. Ridge shrinks correlated coefficients together, while lasso can set some coefficients exactly to zero. λ is selected using validation or cross-validation. Features should generally be scaled before applying these penalties.

4. Logistic Regression for Classification

Logistic regression models the probability of a binary outcome. It first models the log-odds as a linear function:

The probability is obtained with the sigmoid function:

where z is the linear predictor. A common decision rule assigns class 1 when p ≥ 0.5, but the threshold should reflect the costs of false positives and false negatives. For example, screening for a serious disease may use a lower threshold to increase sensitivity.

The loss commonly used is binary cross-entropy:

For a positive coefficient βⱼ, increasing xⱼ increases the log-odds, holding other variables constant. The quantity e^βⱼ is an odds ratio for a one-unit increase, subject to the model specification. Logistic regression is often a strong baseline because it is fast, transparent, and probabilistic, but it may miss nonlinear boundaries and interactions unless features are added explicitly.

5. Decision Trees

A decision tree recursively divides observations using rules such as “income < 40,000?” A regression tree predicts a value in each terminal leaf, often the mean target in that leaf. A classification tree predicts a class or class probabilities based on the class distribution in the leaf.

For classification, a common impurity measure is Gini impurity:

pₖ is the proportion of class k in a node. Gini impurity is zero when a node contains only one class. Entropy is another measure:

A split is attractive when it produces child nodes with lower weighted impurity than the parent. Tree growth can be restricted by maximum depth, minimum samples per leaf, or minimum impurity improvement. Without restriction, a tree can fit noise.

Trees capture nonlinear relationships, thresholds, and interactions without requiring feature scaling. They are easy to explain as rules, but a single tree can be unstable: a small change in data may produce a different structure. Deep trees also tend to overfit.

6. Ensemble Methods

An ensemble combines multiple models to improve predictive performance.

Bagging and random forests

Bagging trains models on bootstrap samples—samples drawn with replacement—and averages regression predictions or votes for classification. A random forest adds random feature selection at each split. This decorrelates trees and generally reduces variance compared with a single tree.

Random forests are effective for mixed nonlinear relationships and require relatively little preprocessing. Their limitations include larger computational cost, less direct interpretability, and probability estimates that may require calibration.

Boosting

Boosting builds models sequentially. Each new weak learner focuses on errors or residual structure left by earlier learners. In gradient boosting, a differentiable loss guides the direction of improvement. Important hyperparameters include the number of estimators, learning rate, tree depth, and subsampling rate.

A smaller learning rate often requires more trees but can improve generalisation. Excessive depth or too many trees can overfit, so validation and early stopping are important. Modern gradient-boosting implementations are highly competitive on tabular data, but they can be sensitive to leakage, noisy features, and unexamined distribution shifts.

7. Introductory Neural Networks

A basic artificial neuron computes a weighted sum followed by an activation function:

x is the input vector, w is a vector of weights, b is a bias, and φ is an activation function. A multilayer perceptron places layers of neurons between the input and output. Common hidden-layer activations include ReLU, φ(z) = max(0, z), and sigmoid. A regression output may be linear; a binary classifier commonly uses a sigmoid output.

Training minimises a loss using gradient-based optimisation. Backpropagation applies the chain rule to calculate how each weight contributes to the loss. Learning rate, batch size, number of layers, number of units, regularisation, and number of epochs are hyperparameters.

Neural networks can learn complex nonlinear functions but usually require careful preprocessing, sufficient data, and tuning. They are less interpretable than a small tree or a sparse linear model. For ordinary structured tabular data, tree ensembles may be a stronger first choice; neural networks become especially attractive for images, audio, language, and very high-dimensional representations.

8. Evaluation and Model Selection

Evaluation must match the decision problem.

Regression metrics

Mean absolute error is:

It is expressed in the target’s units and is less sensitive to large errors than squared-error metrics. Mean squared error is:

Root mean squared error is RMSE = √MSE and therefore has the target’s units. MSE and RMSE penalise large errors more strongly. Mean absolute percentage error can be misleading when actual values are zero or close to zero.

Classification metrics

A confusion matrix records true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Accuracy is:

Precision = TP/(TP + FP) measures the reliability of positive predictions. Recall, also called sensitivity, is TP/(TP + FN) and measures the proportion of actual positives detected. Specificity is TN/(TN + FP). The F1 score is the harmonic mean of precision and recall:

Accuracy is unsuitable as the sole metric for severe class imbalance. ROC-AUC evaluates ranking across thresholds, while precision–recall curves are often more informative when the positive class is rare. If predicted probabilities will support decisions, calibration should be assessed: among cases assigned probability about 0.7, approximately 70% should be positive over a sufficiently large group.

A sound comparison procedure

Establish a simple baseline, define the primary metric before testing models, use stratified splits for classification where appropriate, tune only with training and validation data, and evaluate the chosen model once on the untouched test set. Report uncertainty where possible, investigate subgroup performance, and assess operational costs rather than relying on one aggregate score.

9. Interpreting Model Outputs

Interpretability has several meanings. A model may be intrinsically simple, its predictions may be explained locally for individual cases, or its average behaviour may be examined globally.

For a linear model, coefficients are directly interpretable under the stated assumptions. Tree feature importance based on impurity can favour variables with many possible split points. Permutation importance measures the performance decrease after randomly shuffling one feature, but correlated features can make importance appear artificially low or distribute importance among substitutes.

Partial dependence describes the average predicted response as a feature changes while other features are averaged over the data. It can be misleading when the examined feature combinations are unrealistic or when strong interactions exist. Local explanation methods can help investigate an individual prediction, but they are approximations and should not be treated as proof of causal reasoning.

Always distinguish explanation from causation. A predictive feature may be a proxy for protected status, a consequence rather than a cause, or merely correlated because of sampling practices.

10. Applied Case: Predicting Customer Churn

This is an illustrative case, not a report of a specific organisation.

Background and problem

A subscription company wants to identify customers at risk of cancelling within the next 30 days so that retention teams can offer appropriate support. The target is binary: churn or no churn.

Analysis

Available data include tenure, plan type, monthly charge, service incidents, recent usage, payment history, and customer-service contacts. The data scientist first defines the prediction timestamp and excludes information recorded after that time, such as a cancellation request. Churn prevalence is 12%, so accuracy alone would be misleading: predicting “no churn” for every customer would achieve 88% accuracy but detect no at-risk customers.

Approach and solution

The team creates a time-based training and test split to imitate deployment. Numerical variables are imputed and standardised for logistic regression; categorical variables are one-hot encoded. A regularised logistic model provides a transparent baseline. A random forest and a gradient-boosting model are then tuned using validation data.

The primary metric is recall subject to a minimum precision, because the retention team has limited capacity and unnecessary offers are costly. Threshold selection is performed on validation predictions rather than on the test set. Calibration is checked because managers need meaningful risk probabilities.

Feature review finds that “days since last login” is predictive, but “cancellation workflow opened” is excluded because it occurs too late to support an intervention. The final model is compared across customer segments. If performance differs substantially by language, region, or disability status, the company must investigate data quality, unequal service processes, and possible discriminatory effects.

Outcome and learning points

The selected model produces a ranked list rather than an automatic cancellation decision. Human staff review high-risk cases and record outcomes. Performance is monitored after deployment because pricing, competitor activity, and customer behaviour can change. The principal lessons are that target definition, time-aware validation, threshold selection, calibration, and governance are as important as algorithm choice.

11. Challenges, Ethics, and Governance

Important risks include:

• Leakage: future or target-derived information enters the features.
• Sampling bias: training data do not represent the population receiving predictions.
• Historical bias: labels reflect unequal past decisions or access to services.
• Fairness concerns: error rates or opportunities may differ across groups.
• Privacy: features may reveal sensitive behaviour or enable re-identification.
• Automation bias: users may accept predictions without scrutiny.
• Distribution shift: relationships change after deployment.
• Feedback loops: predictions influence outcomes, which then become future training data.

Governance should document the purpose, data provenance, target definition, exclusions, model version, metrics, limitations, and responsible owner. Access controls, retention limits, audit logs, incident procedures, and periodic review are practical controls. High-impact applications may require human oversight, contestability, transparency, and compliance with applicable data-protection and sector-specific regulations.

12. Best Practices and Contemporary Developments

A reliable workflow is:

1. Define the decision, prediction horizon, target, and cost of errors.
2. Audit data quality, representativeness, permissions, and privacy.
3. Split data appropriately before learning transformations.
4. Establish an interpretable baseline.
5. Engineer features using domain knowledge and prediction-time availability.
6. Compare models with an agreed metric and cross-validation strategy.
7. Tune hyperparameters without touching the final test set.
8. Check calibration, subgroup performance, robustness, and interpretability.
9. Deploy with monitoring for drift, latency, errors, and changes in prevalence.
10. Retrain or retire the model when evidence shows that it no longer serves its purpose.

Contemporary practice increasingly includes automated machine-learning pipelines, model cards, data sheets, explainability tooling, probabilistic forecasting, and fairness assessment. These tools support—not replace—professional judgement. A more complex model is justified only when its additional performance is reliable, relevant, and acceptable to stakeholders.

Chapter Summary

Supervised machine learning learns predictive relationships from labelled examples. Regression predicts numerical outcomes, while classification predicts categories or probabilities. Linear and logistic regression provide strong, interpretable baselines; trees model thresholds and interactions; random forests reduce variance through bagging; boosting sequentially improves weak learners; and neural networks learn layered nonlinear representations.

Successful modelling depends on sound feature engineering, leakage prevention, appropriate data splitting, and metrics aligned with decisions. MAE, RMSE, precision, recall, F1, ROC-AUC, and calibration answer different questions. Interpretability tools can illuminate model behaviour but do not establish causation. In professional settings, fairness, privacy, documentation, monitoring, human oversight, and resistance to distribution shift are essential components of a responsible machine-learning system.

Key Terms

Supervised learning: Learning from examples with known target values.

Feature: An input variable used by a model.

Target: The outcome a model is trained to predict.

Regression: Prediction of a numerical outcome.

Classification: Prediction of a categorical outcome.

Parameter: A value learned during model training.

Hyperparameter: A setting selected outside the fitting process.

Overfitting: Excessive adaptation to training data that harms generalisation.

Underfitting: Failure to capture important structure in the data.

Regularisation: A penalty or constraint used to control model complexity.

Feature engineering: Construction or transformation of useful input variables.

Data leakage: Use of information unavailable at prediction time.

Decision tree: A model that applies recursive feature-based rules.

Random forest: An ensemble of decorrelated decision trees.

Boosting: Sequential combination of weak learners.

Precision: The proportion of positive predictions that are correct.

Recall: The proportion of actual positives that are detected.

Calibration: Agreement between predicted probabilities and observed frequencies.

Distribution shift: Change in the data-generating process after training.

Permutation importance: Performance change caused by shuffling a feature.

Review Questions

Short Answer Questions

1. Distinguish regression from classification and give one example of each.
2. Why must preprocessing transformations be fitted using training data only?
3. What is the purpose of regularisation in a predictive model?
4. Define precision and recall.
5. Why can a deep decision tree overfit?

Long Answer Questions

1. Explain the complete supervised-learning workflow from problem definition to deployment monitoring.
2. Compare linear regression, logistic regression, decision trees, random forests, and gradient boosting.
3. Explain how feature engineering can improve a model and how it can create leakage.
4. Discuss why accuracy may be unsuitable for an imbalanced classification problem.
5. Explain the difference between model interpretation and causal explanation.

Analytical / Application Questions

1. A fraud classifier is used when fraudulent transactions represent 1% of all transactions. Which metrics should be reported, and why might accuracy be misleading?
2. Design a validation strategy for predicting next-month customer churn from monthly historical records. Explain how you would prevent temporal leakage.
3. A model has excellent test AUC but performs poorly for a minority subgroup. Analyse possible causes and recommend technical and governance actions.

Multiple Choice Questions with Answers

Question 1

Which task is a regression problem?

A. Predicting whether an email is spam

B. Predicting a house selling price

C. Assigning an image to one of three categories

D. Determining whether a transaction is fraudulent

Correct Answer: B. Predicting a house selling price

Question 2

What does standardisation using z = (x − μ)/σ accomplish?

A. It converts a numerical feature to a category

B. It makes the feature have approximately mean 0 and standard deviation 1

C. It removes all outliers

D. It guarantees a normal distribution

Correct Answer: B. It makes the feature have approximately mean 0 and standard deviation 1

Question 3

In logistic regression, the sigmoid function converts the linear predictor into a value between:

A. −∞ and ∞

B. −1 and 1

C. 0 and 1

D. 0 and 100

Correct Answer: C. 0 and 1

Question 4

Which metric is especially concerned with the proportion of predicted positives that are correct?

A. Recall

B. Precision

C. Specificity

D. Mean squared error

Correct Answer: B. Precision

Question 5

A model has TP = 80 and FN = 20. What is its recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 6

Which technique combines many bootstrapped decision trees and random feature selection?

A. Ridge regression

B. Random forest

C. Logistic regression

D. Single-layer perceptron

Correct Answer: B. Random forest

Question 7

What is target leakage?

A. A model having too few features

B. A feature containing information unavailable at prediction time

C. Random variation in the target

D. A method for balancing classes

Correct Answer: B. A feature containing information unavailable at prediction time

Question 8

Which statement about lasso regularisation is correct?

A. It uses the sum of squared coefficients and never produces zero coefficients

B. It uses the sum of absolute coefficient values and can set coefficients to zero

C. It can be used only with decision trees

D. It guarantees the highest test accuracy

Correct Answer: B. It uses the sum of absolute coefficient values and can set coefficients to zero

Question 9

Why can accuracy be misleading in a highly imbalanced classification problem?

A. Accuracy cannot be calculated from a confusion matrix

B. A model can achieve high accuracy by always predicting the majority class

C. Accuracy is measured in target units

D. Accuracy applies only to regression

Correct Answer: B. A model can achieve high accuracy by always predicting the majority class

Question 10

Which practice best estimates performance after deployment when the data have a strong time order?

A. Randomly mixing all observations before splitting

B. Using future observations to impute past values

C. Using a time-based holdout that reflects the prediction setting

D. Evaluating only on the training data

Correct Answer: C. Using a time-based holdout that reflects the prediction setting