Supervised Machine Learning

Introduction and Context

Supervised machine learning is the branch of data science concerned with learning a relationship between input variables and a known outcome from labelled examples. The learned relationship is then used to predict outcomes for new observations. The word supervised indicates that the training data contain a target, or response, variable against which predictions can be compared.

Supervised learning supports applications such as predicting house prices, detecting fraudulent transactions, classifying medical images, estimating customer churn and forecasting equipment failure. Its value lies not merely in producing predictions, but in producing predictions that are accurate, appropriately evaluated, interpretable where necessary, and suitable for the decision context.

Two broad supervised-learning tasks are central:

• Regression, where the target is numerical, such as income, temperature or demand.
• Classification, where the target is categorical, such as fraud/not fraud or high-risk/low-risk.

This chapter develops the principles and practical use of linear regression, logistic regression, decision trees, ensemble methods and nearest-neighbor approaches. It also examines assumptions, model evaluation, overfitting, ethical risks and professional implementation practices.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the supervised-learning workflow and distinguish regression from classification.
2. Fit and interpret linear and logistic regression models.
3. Explain how decision trees, ensembles and nearest-neighbor methods make predictions.
4. Select suitable evaluation metrics for different business and technical contexts.
5. Diagnose overfitting, data leakage and violations of modelling assumptions.
6. Apply supervised-learning methods to a realistic organisational problem.
7. Discuss interpretability, fairness, privacy and governance requirements.

1. The Supervised-Learning Framework

Let the dataset contain n observations. For observation i, the feature vector is x_i = (x_i1, x_i2, ..., x_ip), and the observed target is y_i. A supervised algorithm estimates a function f such that:

Here, ŷ_i is the predicted target and f is learned from the training data. In a regression problem, y_i is numerical. In classification, it is a class label or, commonly, a probability of class membership.

A typical workflow is:

1. Define the decision problem and prediction target.
2. Collect, clean and document labelled data.
3. Split data into training, validation and test sets.
4. Pre-process features using operations learned only from training data.
5. Train candidate models.
6. Tune hyperparameters using validation or cross-validation.
7. Evaluate the final model once on untouched test data.
8. Deploy, monitor and periodically reassess performance.

The separation of data is essential. A training set estimates model parameters; a validation set supports model selection; and a test set provides an approximately unbiased estimate of performance on future data. When data are limited, k-fold cross-validation divides the training data into k folds, trains on k − 1 folds and validates on the remaining fold, repeating the process so each fold is used for validation.

Features, parameters and hyperparameters

A feature is an input variable supplied to a model. A parameter is estimated during training, such as a regression coefficient. A hyperparameter is selected before or around training, such as a tree's maximum depth or the value of k in nearest neighbors. Confusing these categories can lead to incorrect tuning procedures.

Generalisation and the bias–variance problem

A model should perform well on unseen data, not merely reproduce its training examples. Underfitting occurs when a model is too simple and has high bias. Overfitting occurs when a model learns noise or idiosyncrasies and has high variance. Increasing complexity often reduces training error while eventually increasing test error. Cross-validation, regularisation, pruning and careful feature design help manage this trade-off.

2. Linear Regression

Linear regression models a numerical target as a linear combination of predictors. With p predictors, the multiple linear regression model is:

where β₀ is the intercept, β_j is the expected change in the target for a one-unit increase in predictor j while other predictors are held constant, and ε_i is the error for observation i. The fitted prediction is:

The ordinary least-squares estimates minimise the residual sum of squares:

The method assumes that the conditional mean is appropriately linear, observations are independent or dependence is modelled, predictors are not perfectly collinear, and—when inferential tests and confidence intervals are required—error variance and distributional assumptions are reasonably satisfied. Normal errors are not required merely to calculate predictions, but they matter for small-sample classical inference.

Interpretation

Suppose a fitted model predicts monthly sales in thousands of units from advertising expenditure in thousands of currency units:

The intercept 12 is predicted sales when expenditure is zero, although this may not be meaningful if zero lies outside the observed range. The slope 2.4 means that an additional one thousand currency units of expenditure is associated with an estimated increase of 2.4 thousand units of sales, under the model and within the observed range. Association is not automatically causation; an experimental or strong causal design is needed for causal claims.

Evaluation metrics

For actual values y_i and predictions ŷ_i, common regression measures include:

MAE is in the target's units and is comparatively robust to outliers. MSE penalises large errors more strongly. RMSE is also in the target's units but remains sensitive to large errors. Mean absolute percentage error can be misleading when actual values are zero or close to zero, so it should not be used without checking the data.

Regularisation

When predictors are numerous or correlated, regularisation can reduce variance. Ridge regression minimises:

Lasso regression minimises:

Here λ ≥ 0 controls the penalty. Larger values shrink coefficients more strongly. Ridge usually retains all predictors, whereas lasso can shrink some coefficients exactly to zero. Predictors should generally be standardised before penalisation when they are measured on different scales. The intercept is normally not penalised.

3. Logistic Regression

Logistic regression is used for classification, especially binary classification. It models the probability that an observation belongs to class 1. The logistic function is:

The output p(x) lies between 0 and 1. The model is linear in the log-odds, not necessarily in probability:

The odds ratio associated with predictor j is e^β_j. Holding other variables constant, a one-unit increase in x_j multiplies the odds by this value. For example, if β_j = 0.693, then e^0.693 ≈ 2, meaning approximately doubled odds, not doubled probability.

Parameters are commonly estimated by maximum likelihood rather than ordinary least squares. A classification decision requires a threshold. At the conventional threshold 0.5, predict class 1 when p(x) ≥ 0.5, but the threshold should reflect the costs of false positives and false negatives.

Confusion matrix and metrics

For binary classification, predictions can be true positives (TP), true negatives (TN), false positives (FP) or false negatives (FN).

Accuracy can be inappropriate for imbalanced data. If only 1% of transactions are fraudulent, a model predicting “not fraud” for every case achieves 99% accuracy but detects no fraud. Precision–recall analysis is often useful when the positive class is rare. The receiver operating characteristic curve examines true-positive rate against false-positive rate across thresholds; the area under this curve summarises ranking ability but does not by itself guarantee useful probabilities or acceptable performance at the operating threshold.

4. Decision Trees

A decision tree recursively partitions the feature space. Internal nodes contain tests such as “income < 40,000,” branches represent outcomes of tests, and leaves provide a prediction. A regression tree predicts a numerical value, often the mean target in a leaf. A classification tree predicts a class or class probability based on the class distribution in the leaf.

For classification, a common impurity measure is Gini impurity:

where p_c is the proportion of observations in a node belonging to class c. Entropy is another measure:

A split is preferred when it produces purer child nodes, usually measured by the impurity reduction weighted by child size. A pure node has impurity zero. Trees do not require feature scaling and can represent nonlinear interactions, but deep unpruned trees may overfit and can be unstable: a small change in data may produce a substantially different tree.

Controls include maximum depth, minimum samples per leaf, minimum impurity decrease and cost-complexity pruning. A tree is attractive when transparent rules are important, but interpretability declines when the tree becomes very large.

5. Ensemble Methods

An ensemble combines multiple models to obtain better predictive performance or stability.

Bagging and random forests

Bagging trains models on bootstrap samples—samples drawn with replacement—and averages regression predictions or votes for classification. Random forests add randomness by considering a random subset of features at each split. This reduces correlation among trees and often improves generalisation compared with a single tree. Random forests are robust, handle nonlinear relationships and provide useful variable-importance measures, although importance measures can be biased by feature scale, cardinality or correlation and should not be treated as causal evidence.

Boosting

Boosting builds models sequentially. Later models concentrate on errors made by earlier models. Gradient boosting fits new learners to a direction related to the loss function's gradient. Learning rate, number of estimators and tree depth are important hyperparameters. Boosting can achieve excellent accuracy on structured tabular data, but it requires careful tuning and may overfit noisy data. Modern implementations include gradient-boosted decision-tree systems designed for efficient handling of large datasets.

Stacking

Stacking trains several base models and uses a meta-model to combine their predictions. To avoid leakage, the meta-model must generally be trained on out-of-fold predictions rather than predictions generated from the same data used to fit each base model.

6. Nearest-Neighbor Methods

The k-nearest-neighbor (k-NN) method predicts an observation using the labels or values of the k most similar training observations. For numeric features, Euclidean distance is commonly defined as:

For classification, the majority class among the neighbors is commonly selected; for regression, their mean may be used. A distance-weighted version gives nearer observations greater influence.

Feature scaling is essential when variables have different units. Otherwise, a feature measured in large numerical units can dominate the distance. The choice of k controls the bias–variance trade-off: small k gives flexible but noisy predictions; large k gives smoother but potentially biased predictions. k-NN can be computationally expensive at prediction time and performs poorly in very high-dimensional spaces, a difficulty associated with the curse of dimensionality.

7. Comparing and Selecting Models

No algorithm is universally best. Selection should reflect the target, data structure, error costs, interpretability requirements, latency, maintenance resources and regulatory context.

A sound comparison uses the same data partitions and an evaluation metric aligned with the decision. It should include a simple baseline. Probability calibration should be assessed when predicted probabilities drive decisions. A model with slightly lower accuracy may be preferable if it is more interpretable, fairer, cheaper or easier to monitor.

Diagnostics and common errors

Residual plots can reveal nonlinearity, unequal variance and outliers in regression. Classification models should be assessed by subgroup, threshold and calibration, not only by a single aggregate score. Common failures include:

• Data leakage: information unavailable at prediction time enters the features or preprocessing.
• Target leakage: a feature is a direct consequence of the target or future outcome.
• Random splitting of time-dependent data: future information indirectly influences training.
• Improper preprocessing: scaling or imputation is fitted using the complete dataset before splitting.
• Multiple testing and repeated tuning: repeated experimentation on the test set produces optimistic estimates.
• Class imbalance: the model ignores a rare but important class.

For time-dependent applications, use chronological splits or rolling-origin evaluation. For grouped observations, such as several transactions from one customer, split by group when the deployment task concerns unseen customers.

8. Illustrative Case: Predicting Customer Churn

Background

This fictional case concerns a telecommunications company that wants to identify customers likely to cancel service within the next 30 days. The purpose is to offer retention support, not to deny service.

Problem

The data include tenure, monthly charge, contract type, service complaints, payment method and recent usage. The target is churn within 30 days. Only 8% of customers churn, so accuracy alone would be misleading. The business estimates that contacting a likely churner is inexpensive, while missing a genuine churner is costly.

Analysis and approach

The data science team first defines the prediction timestamp and removes variables created after cancellation, such as final account-closure codes. It uses a chronological training period and a later validation period to imitate deployment. Missing values and categorical encoding are fitted within the training pipeline. A regularised logistic regression provides an interpretable baseline; a random forest and gradient-boosted trees are compared with it.

Because recall is important, the team evaluates recall, precision, F1 score, precision–recall curves and calibration. The classification threshold is selected using the expected cost of contact and missed churn rather than automatically using 0.5. Performance is also examined across customer segments to identify unequal error rates.

Outcome

The boosted model ranks potential churners better than the baseline, but the calibrated logistic model produces sufficiently competitive results and is easier for retention managers to explain. The company therefore deploys the logistic model initially, with a threshold that supports a manageable contact volume. A controlled experiment compares targeted offers with ordinary retention activity. The model is monitored for drift in contract mix, customer behaviour and subgroup performance.

Learning points

The best model is not necessarily the most complex. Target definition, temporal validity, cost-sensitive evaluation, calibration, fairness analysis and an intervention experiment are as important as algorithm choice. A predictive model identifies risk; it does not establish why an individual will churn or justify an unfair intervention.

9. Ethics, Governance and Professional Practice

Supervised models can reproduce historical discrimination when labels reflect unequal treatment or when features act as proxies for protected characteristics. Removing a protected attribute does not guarantee fairness because correlated variables may remain. Practitioners should document data provenance, consent and lawful purpose; minimise personal data; control access; and define retention periods.

Important governance practices include model cards or equivalent documentation, reproducible pipelines, version control, audit logs, human oversight and an appeal process for consequential decisions. Fairness should be assessed in context because statistical criteria can conflict, especially when base rates differ between groups. In high-impact domains, explainability, impact assessment and regulatory requirements may be mandatory. Explanations should distinguish correlation from causation and should not overstate what a model knows.

10. Contemporary Developments

Current practice increasingly combines classical models with automated machine-learning pipelines, explainable-AI tools, model monitoring and responsible-AI controls. Gradient-boosted trees remain strong for many tabular problems, while representation-learning methods are important for text, images and audio. However, more complex models do not remove the need for high-quality labels, valid evaluation and governance. Distribution shift, adversarial behaviour, privacy-preserving learning and energy use are active concerns. The durable professional skill is not memorising algorithms, but matching a method to a well-defined problem and maintaining its validity after deployment.

Chapter Summary

Supervised machine learning learns predictive relationships from labelled data. Regression predicts numerical outcomes, while classification predicts classes or class probabilities. Linear regression estimates an additive conditional mean by minimising squared errors; logistic regression models the log-odds of a binary outcome and requires a decision threshold. Decision trees create recursive rules, random forests reduce variance through many decorrelated trees, boosting builds models sequentially, and k-nearest neighbors use local similarity.

Model quality must be judged on unseen data using metrics appropriate to the decision. MAE, RMSE, precision, recall, F1 score, ROC analysis and calibration each answer different questions. Cross-validation, regularisation, pruning, scaling and temporal or grouped splitting help control overfitting and leakage. Deployment additionally requires monitoring, interpretability, privacy, fairness, documentation and human governance. A technically accurate model is useful only when its data, assumptions, evaluation and operational purpose are valid.

Key Terms

Supervised learning: Learning from examples containing input features and known targets.

Feature: An input variable used to make a prediction.

Target: The outcome a model is trained to predict.

Regression: Prediction of a numerical outcome.

Classification: Prediction of a categorical outcome.

Parameter: A quantity estimated during model training.

Hyperparameter: A setting selected outside the model's parameter-estimation process.

Overfitting: Learning training-specific noise, causing poor generalisation.

Underfitting: Using a model too simple to capture relevant structure.

Regularisation: Penalising model complexity to improve generalisation.

Log-odds: The logarithm of the odds, log(p/(1−p)).

Confusion matrix: A table of binary prediction outcomes.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positives that are detected.

Decision tree: A recursive sequence of feature-based decision rules.

Ensemble: A combination of several predictive models.

Random forest: An ensemble of randomised decision trees.

Boosting: Sequentially building models that address previous errors.

Nearest neighbor: A training observation selected because it is similar to a new case.

Data leakage: Use of information that would not be available at prediction time.

Review Questions

Short Answer Questions

1. What distinguishes regression from classification?
2. Why must preprocessing be fitted using training data only?
3. Interpret a positive logistic-regression coefficient.
4. Why can accuracy be misleading for an imbalanced classification problem?
5. Why is feature scaling important for k-nearest neighbors?

Long Answer Questions

1. Explain the supervised-learning workflow from problem definition to deployment and monitoring.
2. Compare linear regression and logistic regression, including their outputs, assumptions and evaluation measures.
3. Explain how decision trees, random forests and boosting differ in the way they control error.
4. Discuss overfitting, the bias–variance trade-off and the role of cross-validation and regularisation.
5. Evaluate the ethical and governance issues involved in deploying a churn-prediction model.

Analytical / Application Questions

1. A fraud model achieves 99% accuracy but detects only 5% of fraudulent transactions. Explain why this may be unacceptable and recommend alternative metrics and threshold considerations.
2. A k-NN model uses annual income and number of complaints as features. Explain how unequal measurement scales could distort predictions and propose a preprocessing strategy.
3. Design a model-evaluation plan for predicting machine failure from sensor readings collected over time. Specify data splitting, leakage controls, suitable metrics and monitoring requirements.

Multiple Choice Questions with Answers

Question 1

Which task is a regression problem?

A. Predicting whether an email is spam

B. Predicting the selling price of a house

C. Assigning a customer to a risk class

D. Recognising a handwritten digit category

Correct Answer: B. Predicting the selling price of a house

Question 2

In ordinary least-squares linear regression, the fitted coefficients minimise which quantity?

A. The sum of absolute residuals

B. The sum of squared residuals

C. The classification error rate

D. The number of predictors

Correct Answer: B. The sum of squared residuals

Question 3

What does a logistic-regression output usually represent before a classification threshold is applied?

A. A probability between 0 and 1

B. A residual measured in target units

C. A tree depth

D. A Euclidean distance

Correct Answer: A. A probability between 0 and 1

Question 4

If false negatives are especially costly, which measure should generally receive particular attention?

A. Recall

B. Specificity only

C. Mean squared error

D. Ridge penalty

Correct Answer: A. Recall

Question 5

What is a major purpose of regularisation?

A. To guarantee causal interpretation

B. To reduce overfitting by penalising complexity

C. To remove the need for test data

D. To convert classification into regression

Correct Answer: B. To reduce overfitting by penalising complexity

Question 6

Why should numerical features commonly be scaled before applying k-nearest neighbors?

A. To make the target categorical

B. To prevent large-unit features from dominating distance

C. To increase the number of classes

D. To eliminate all missing values automatically

Correct Answer: B. To prevent large-unit features from dominating distance

Question 7

Which method builds many trees using randomisation and combines their predictions?

A. Random forest

B. Ordinary logistic regression

C. Single nearest neighbor

D. Principal-component analysis

Correct Answer: A. Random forest

Question 8

A feature recorded after a customer has cancelled a service is most likely an example of what problem if used for prediction?

A. Underfitting

B. Data leakage

C. Feature scaling

D. Bootstrap sampling

Correct Answer: B. Data leakage

Question 9

For a binary classifier with TP = 80 and FN = 20, what is recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 10

Why can a chronological split be preferable to a random split for forecasting equipment failure?

A. It guarantees perfect accuracy

B. It imitates the direction of time in deployment and reduces future-information leakage

C. It removes the need for labels

D. It ensures every class has equal size

Correct Answer: B. It imitates the direction of time in deployment and reduces future-information leakage