7. Supervised Learning Models

Introduction and Context

Supervised learning is the branch of machine learning in which an algorithm learns a relationship between input variables and a known target variable. Each training example consists of features, represented by a vector x, and an observed label or value, represented by y. The learning task is to estimate a function f such that f(x) produces useful predictions for previously unseen cases.

The word supervised reflects the presence of an answer during training. In regression, the answer is numerical—for example, house price, energy consumption, or delivery time. In classification, the answer is categorical—for example, whether a transaction is fraudulent, whether a patient belongs to a risk group, or whether an email is spam. The model does not simply memorize the training records; it should generalize from them.

Supervised learning models differ in their assumptions, representation of relationships, treatment of variables, computational demands, interpretability, and sensitivity to noise. A linear model may be highly transparent but unable to represent complex interactions. A decision tree may capture nonlinear rules but overfit without appropriate control. A support vector machine can produce a powerful boundary in high-dimensional spaces, while a nearest-neighbour model can make flexible predictions with almost no explicit training. Model selection is therefore a problem of matching the characteristics of an algorithm to the data, decision context, and cost of errors.

This chapter examines the principal supervised learning models used in applied artificial intelligence: linear and logistic regression, decision trees, random forests, nearest-neighbour methods, naïve Bayes, support vector machines, boosting, and ensemble methods. It also considers evaluation, interpretation, deployment, governance, and the practical trade-offs involved in selecting a model.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Distinguish regression from classification and explain the supervised learning workflow.

2. Describe the assumptions, mechanisms, strengths, and weaknesses of major supervised models.

3. Interpret coefficients, probabilities, decision rules, margins, and feature importance appropriately.

4. Select models according to data structure, predictive objective, computational constraints, and interpretability requirements.

5. Explain overfitting, regularization, validation, class imbalance, and data leakage in supervised learning.

6. Apply supervised learning reasoning to professional and organizational decision problems.

7. Critically evaluate ethical, governance, and operational risks in predictive systems.

From labelled data to a predictive function

A supervised learning project normally begins with a labelled dataset. The feature matrix X contains observations and predictors, while y contains the target. The data are divided into training and testing portions; a validation set or cross-validation procedure is commonly used to tune model settings. The model minimizes a loss function during training. For regression, squared error or absolute error may be used. For classification, cross-entropy loss, hinge loss, or a related objective is common.

The central distinction is between fitting and generalization. A model that performs extremely well on training data may have learned accidental properties of those observations rather than the underlying pattern. This is overfitting. Underfitting occurs when the model is too simple, or insufficiently trained, to capture important structure. The bias–variance perspective describes this tension: high bias reflects systematic simplicity, whereas high variance reflects excessive sensitivity to the sample.

A typical workflow includes problem definition, data preparation, exploratory analysis, feature engineering, model training, validation, evaluation, threshold selection, deployment, and monitoring. Preprocessing operations such as scaling, imputation, and encoding must be learned from the training data only. Applying information from the full dataset before splitting can cause data leakage and produce deceptively optimistic results.

The appropriate metric depends on the decision. Regression may be assessed using mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE), or R-squared. Classification may use accuracy, precision, recall, specificity, F1 score, log loss, or area under the receiver operating characteristic curve. Accuracy is often inappropriate when one class is rare. In a fraud-screening system, recall may be prioritized, while in an expensive manual-review process, precision may be more important.

Linear regression: a transparent baseline

Linear regression models a numerical response as a weighted combination of predictors:

y-hat = beta-zero + beta-one x-one + ... + beta-p x-p.

The coefficients estimate the expected change in the response associated with a one-unit change in a predictor, holding the other included predictors constant. Ordinary least squares selects coefficients that minimize the sum of squared residuals. The residual is the difference between an observed and predicted value.

Linear regression is particularly valuable as a baseline because it is fast, relatively easy to interpret, and often effective when relationships are approximately additive and linear. It is also useful for inference when the data-generating assumptions and sampling design support such interpretation.

Important assumptions include linearity of the conditional mean, independent observations, a suitable treatment of influential observations, and—when conventional statistical inference is required—constant error variance and appropriately behaved residuals. Multicollinearity occurs when predictors are strongly related to one another. It can make coefficient estimates unstable even when predictions remain reasonable. Feature transformations and interaction terms can extend the model, but they increase modelling complexity.

Regularization controls coefficient size. Ridge regression adds a penalty proportional to the squared coefficients and tends to stabilize correlated predictors. Lasso regression uses an absolute-value penalty and can shrink some coefficients to zero, producing a sparse model. Elastic net combines both penalties. Regularization generally improves generalization when predictors are numerous or correlated, but the penalty strength must be selected through validation.

A coefficient should not automatically be interpreted as a causal effect. For example, a model predicting salary from education and experience describes conditional association in the available data; it does not by itself establish that changing education will produce a specific salary increase.

Logistic regression: probability and classification

Logistic regression is used mainly for binary classification. It models the log-odds of an event as a linear function of the predictors:

log(p/(1-p)) = beta-zero + beta-one x-one + ... + beta-p x-p.

The logistic function converts the linear score into a probability between zero and one. A classification threshold then converts the probability into a class decision. The familiar threshold of 0.5 is not universally appropriate; it should reflect class prevalence, error costs, and operational capacity.

A coefficient represents a change in log-odds, holding other variables constant. Exponentiating a coefficient gives an odds ratio. This interpretation is often useful in credit, health, marketing, and policy contexts, although it remains dependent on model specification and data quality. Multiclass outcomes can be addressed through extensions such as one-versus-rest or multinomial logistic regression.

Logistic regression assumes a linear relationship between predictors and log-odds, not necessarily between predictors and probability. It may perform poorly when the true boundary is strongly nonlinear unless transformations, interactions, splines, or other features are introduced. It remains an important benchmark because its probability outputs can be calibrated and its reasoning can be communicated comparatively clearly.

Decision trees: prediction through recursive questions

A decision tree partitions the feature space through a sequence of questions. A node may ask whether income is below a threshold, whether a category equals a particular value, or whether a measurement exceeds a boundary. Leaves contain predictions: a mean value for regression or a class distribution for classification.

During tree construction, the algorithm evaluates candidate splits using an impurity or error measure. Gini impurity and entropy are common classification criteria; variance reduction is common in regression. The process is recursive and greedy: at each stage it selects a locally useful split, although the resulting tree is not necessarily globally optimal.

Trees can represent nonlinear relationships and interactions without requiring feature scaling. Their rule-like structure is accessible to many stakeholders. However, an unrestricted tree can memorize training data. Maximum depth, minimum leaf size, pruning, and related parameters regulate complexity. Small changes in data can sometimes produce substantially different trees, indicating high variance.

Tree explanations require care. A split near the root may be useful for partitioning the sample but should not automatically be treated as a causal or universally important variable. Standard impurity-based feature importance can also be biased toward variables with many possible split points. Permutation importance and local explanations can provide complementary evidence.

Random forests and the logic of bagging

A random forest is an ensemble of decision trees trained on bootstrap samples, with random subsets of features considered at each split. For regression, predictions are commonly averaged; for classification, votes or averaged class probabilities are used. Random feature selection reduces correlation between trees, while aggregation reduces variance.

Random forests usually require less delicate tuning than a single tree and work well with mixed nonlinear relationships. They provide an out-of-bag estimate because each bootstrap sample leaves out some observations from each tree. They can also estimate feature importance, although such estimates should not be confused with causal importance.

The trade-off is reduced transparency. A forest containing many trees cannot normally be explained through one short set of rules. It may also require more memory and computation than a linear model or small tree. Extrapolation in regression can be weak because tree predictions are based on observed regions rather than a smooth functional form.

Nearest-neighbour methods: local similarity as a model

The k-nearest-neighbour (k-NN) method predicts an observation using the labels of the k most similar training observations. Classification commonly uses a majority vote or distance-weighted vote; regression uses an average or weighted average of nearby responses.

The method has little explicit training cost but substantial prediction cost because new observations must be compared with stored training examples. It is therefore called a lazy learner. Distance measurement is central. Euclidean distance is common for numeric features, but it becomes less informative in high-dimensional spaces, a problem known as the curse of dimensionality. Standardization is usually essential when variables use different units. Categorical variables require appropriate encodings and distance definitions.

A small k produces flexible, low-bias predictions but may be noisy. A large k produces smoother, higher-bias predictions. Cross-validation can select k and other design choices. Nearest-neighbour methods are useful when local similarity is meaningful and the dataset is moderate in size, but they can be difficult to deploy at scale and can inherit biases or irrelevant patterns from the stored data.

Naïve Bayes: probabilistic classification with a simplifying assumption

Bayes’ theorem relates a posterior probability to a prior probability and a likelihood:

P(class | features) is proportional to P(features | class) P(class).

Naïve Bayes assumes that features are conditionally independent given the class. This assumption is rarely literally true, particularly in text, but the model can nevertheless classify effectively because it estimates relatively few parameters. Variants include Gaussian naïve Bayes for continuous variables, multinomial naïve Bayes for count data such as word frequencies, and Bernoulli naïve Bayes for binary features.

The model is fast, works well with sparse high-dimensional data, and can perform surprisingly well with limited training data. Smoothing prevents zero probabilities when a feature value is absent from a class. Its probability estimates may require calibration, and correlated features can make the independence assumption problematic. Nevertheless, its speed and simplicity make it a useful baseline for document classification, filtering, and routing tasks.

Support vector machines: margins and kernels

A support vector classifier seeks a decision boundary that separates classes with the largest possible margin. The observations closest to the boundary are the support vectors; they determine the fitted boundary. A soft-margin formulation permits some misclassification in exchange for a wider and more generalizable margin. The parameter C controls the trade-off between margin width and training errors.

The kernel approach enables an SVM to represent nonlinear boundaries without explicitly transforming every feature. Common kernels include linear, polynomial, and radial basis function kernels. The radial basis function model has a parameter gamma controlling the influence of individual observations. Feature scaling is generally important, and hyperparameter tuning can be computationally demanding.

SVMs are effective in high-dimensional settings and can perform well when the classes are separated by a meaningful margin. They are less convenient when very large datasets make training expensive, when probability estimates are required directly, or when explanations must be extremely simple. A linear SVM and a kernel SVM should be treated as distinct practical choices.

Boosting and modern ensemble thinking

Boosting constructs a strong predictor by combining a sequence of weak or modest learners. Each new learner concentrates on observations or residuals that previous learners handled poorly. In gradient boosting, the next learner approximates the direction that reduces the chosen loss function. Decision trees are common base learners because shallow trees provide flexible incremental corrections.

AdaBoost changes the emphasis placed on training observations, while gradient boosting fits learners to residual-like errors. Implementations such as gradient-boosted tree libraries can include regularization through learning rate, tree depth, subsampling, early stopping, and penalties. A small learning rate combined with more stages often produces a useful bias–variance trade-off, but tuning remains important.

Boosted trees frequently deliver excellent predictive performance on structured tabular data. Their disadvantages include greater tuning complexity, higher risk of leakage-related overconfidence, and reduced direct interpretability. Feature importance, partial-dependence plots, and local explanation methods can assist analysis, but explanations should be validated against the model and the data-generating process.

Ensembling need not mean using one algorithm repeatedly. Voting combines predictions from different classifiers; averaging combines regression predictions; stacking trains a meta-model to combine base-model outputs. Diversity among errors is important: combining models that make identical mistakes provides little benefit. Ensemble performance must be evaluated using out-of-sample predictions to avoid information leakage.

Choosing among models

No model is best for every problem. A useful selection process begins with the decision requirement rather than algorithmic fashion. If coefficients and auditability are central, regularized linear or logistic regression may be preferred. If stakeholders need readable rules, a constrained tree may be suitable. If strong performance on heterogeneous tabular data is the priority, random forests or boosting are strong candidates. For sparse text, naïve Bayes or a linear classifier may be computationally attractive. For moderate-sized, scaled data with complex boundaries, an SVM may be appropriate. k-NN is compelling when local similarity and rapid model updating matter.

Model comparison should use the same data splits, preprocessing pipeline, metric, and decision threshold. Cross-validation estimates expected performance but does not eliminate sampling uncertainty. A final holdout test set should be used sparingly, ideally after model and threshold decisions are complete. Calibration is important when predicted probabilities drive resource allocation or risk ranking. A model can rank cases well while producing probabilities that are systematically too high or too low.

The operational cost of an error must be made explicit. A false negative in a safety-monitoring system may be more serious than a false positive; in a customer-contact system, unnecessary intervention may be costly or harmful. Decision curves, cost-sensitive learning, class weights, resampling, and threshold adjustment can address unequal consequences, but resampling should occur within the training folds rather than before cross-validation.

Illustrative case study: predicting delayed medical appointments

Background

This illustrative case concerns a hospital seeking to reduce missed or substantially delayed outpatient appointments. The hospital has historical records containing appointment details, lead time, prior attendance patterns, communication history, and broad demographic and access-related variables. The target is whether an appointment will be missed.

Situation and problem

The hospital initially reports 92 percent accuracy using a simple model. However, only 8 percent of appointments are missed. A model that predicts “not missed” for everyone would already achieve 92 percent accuracy, so the reported result has little operational value. The hospital needs a system that identifies a manageable number of high-risk appointments for reminders or transport support without unfairly burdening particular groups.

Analysis

The team defines recall for missed appointments as an important measure, while also monitoring precision because staff capacity is limited. It separates historical records by time so that future appointments are not used to predict past ones. Any variable recorded after the appointment time is removed. Missing values and categorical encodings are handled within a training pipeline. Logistic regression provides an interpretable baseline, a random forest captures nonlinear interactions, and gradient boosting is evaluated as a higher-capacity alternative.

The team checks subgroup performance, calibration, and the relationship between threshold selection and reminder capacity. It discovers that one communication variable is partly a proxy for unequal access to digital services. A high-performing model using that variable may intensify existing disadvantage if it automatically labels certain patients as unreliable.

Approach and solution

The hospital chooses a regularized logistic model for the first deployment because its coefficients, calibration, and threshold can be explained to clinical and administrative reviewers. A model score is used to prioritize supportive interventions rather than deny appointments or penalize patients. The threshold is selected according to available outreach capacity, and different intervention options are considered. The model is monitored by month and across relevant patient groups.

A more accurate boosted model is retained for experimentation, but it is not deployed until its calibration, stability, explanation quality, and fairness implications are acceptable. Staff can override recommendations and record reasons, creating a feedback process for later evaluation.

Outcome

The illustrative outcome is not a claim about a real hospital. Under an agreed threshold, the system concentrates outreach on a smaller group with substantially higher observed risk than the general appointment population. The organization evaluates the intervention by missed appointments, successful attendance, staff workload, patient experience, and subgroup effects—not by accuracy alone.

Learning points

The case demonstrates that model selection is inseparable from the intervention. A technically accurate predictor can be inappropriate if its features encode structural disadvantage, its probability estimates are poor, or its output is used for a punitive purpose. Time-aware validation, leakage prevention, calibration, threshold design, human oversight, and ongoing monitoring are all part of supervised learning practice.

Limitations, risks, and common mistakes

Several recurring mistakes undermine supervised learning projects. Training and testing on the same records produces invalid performance claims. Leakage may occur through future information, target-derived features, duplicated individuals, or preprocessing performed before splitting. Optimizing accuracy on an imbalanced target can conceal poor minority-class detection.

Another mistake is treating feature importance as explanation or association as causation. Models can exploit proxies for protected attributes, historical discrimination, or administrative artefacts. Distribution shift occurs when the population, measurement process, or operating environment changes. A model that was valid when trained may degrade after a policy change or new data-collection system.

High-capacity models may be computationally expensive, difficult to reproduce, and hard to audit. Conversely, simple models can be misleading if linearity, independence, or calibration assumptions are ignored. Human review does not automatically remove bias; reviewers may over-trust automated scores, a phenomenon often called automation bias.

Ethical, governance, and professional considerations

A supervised model should have a documented purpose, data provenance, feature rationale, evaluation protocol, intended users, and known limitations. Privacy protection requires data minimization, access control, retention limits, and appropriate handling of sensitive attributes. Fairness assessment should examine relevant group-level error rates, calibration, selection effects, and the context in which predictions are used. No single fairness criterion resolves every policy conflict.

High-impact applications may require consent, explanation, appeal, human review, record keeping, and compliance with sector-specific regulation. Governance should assign responsibility for model approval, monitoring, incident response, retraining, and retirement. Security controls should address poisoning, unauthorized model access, and manipulation of inputs. Professional practice requires communicating uncertainty rather than presenting a prediction as a fact.

Current developments and best practices

Recent practice increasingly emphasizes automated machine-learning pipelines, gradient-boosted tabular models, calibrated probability estimation, explainable AI, fairness-aware evaluation, and monitoring for drift. Foundation and generative models have expanded the range of feature representations, but conventional supervised models remain important for structured organizational data because they are efficient, testable, and often easier to govern.

Best practice is to establish a simple baseline, use leakage-resistant pipelines, select metrics linked to decisions, tune models through nested or carefully controlled validation, assess calibration and subgroup performance, and preserve a genuinely untouched test set. Deployment should include versioning, reproducibility, data-quality checks, monitoring, periodic revalidation, and a defined rollback process. The most sophisticated model is not necessarily the most responsible or useful model.

Chapter Summary

Supervised learning estimates relationships between labelled inputs and known outputs. Regression predicts numerical values, while classification predicts categories or probabilities. Linear regression provides an interpretable additive model; regularization helps manage multicollinearity and high-dimensional predictors. Logistic regression models log-odds and produces probabilities for classification.

Decision trees represent nonlinear rules but can overfit. Random forests reduce tree variance through bootstrap aggregation and feature randomization. Nearest-neighbour methods rely on local similarity and require careful scaling. Naïve Bayes uses Bayes’ theorem with a conditional-independence assumption and is especially efficient for sparse data. Support vector machines seek large-margin boundaries and can use kernels for nonlinear separation. Boosting builds an additive predictor sequentially, while broader ensembles combine diverse models.

Effective model choice depends on data structure, computational resources, interpretability, probability calibration, error costs, and operational context. Validation, leakage prevention, class-imbalance treatment, fairness analysis, monitoring, and human governance are as important as the algorithm itself.

Key Terms

Supervised learning: Learning a predictive relationship from examples containing inputs and known targets.

Regression: Supervised learning in which the target is numerical or continuous.

Classification: Supervised learning in which the target represents one or more categories.

Feature: An input variable used by a predictive model.

Target: The observed outcome that a supervised model learns to predict.

Overfitting: Fitting training-specific noise so that generalization performance declines.

Underfitting: Using a model too simple to capture important structure in the data.

Regularization: A method of controlling model complexity by penalizing parameters or structures.

Multicollinearity: Strong dependence among predictors that can destabilize linear-model coefficients.

Log-odds: The logarithm of the odds of an event, modelled linearly in logistic regression.

Decision tree: A recursive sequence of feature-based splits producing predictions at terminal leaves.

Random forest: An ensemble of randomized, bootstrap-trained decision trees.

Nearest neighbour: A prediction method based on the outcomes of similar stored observations.

Naïve Bayes: A probabilistic classifier based on Bayes’ theorem and conditional feature independence.

Support vector machine: A model that seeks a separating boundary with a large margin.

Boosting: Sequentially combining weak learners so later learners correct earlier errors.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Data leakage: Use of information during training that would not be available at prediction time.

Concept drift: Change over time in the relationship between predictors and the target.

Ensemble: A predictive system that combines outputs from multiple models.

Review Questions

Short Answer Questions

1. What distinguishes regression from classification in supervised learning?

2. Why is a validation procedure needed in addition to a training set?

3. What does a coefficient represent in logistic regression?

4. Why is feature scaling important for k-nearest-neighbour methods?

5. What is the purpose of regularization?

Long Answer Questions

1. Explain the assumptions, interpretation, and limitations of linear regression.

2. Compare a decision tree, random forest, and boosted-tree model.

3. Describe how a support vector machine uses margins and kernels.

4. Explain the naïve Bayes assumption and discuss why the method can still work when the assumption is imperfect.

5. Discuss how model selection should incorporate interpretability, computational requirements, error costs, and governance.

Analytical / Application Questions

1. A fraud dataset contains 1 percent fraudulent transactions. Which evaluation measures and modelling decisions would you recommend, and why would accuracy be inadequate?

2. A company wants to predict employee turnover but plans to use a feature recorded only after resignation. Identify the methodological problem and propose a valid workflow.

3. A hospital must choose between a highly accurate boosted model and a slightly less accurate logistic model for prioritizing supportive outreach. What additional evidence should guide the decision?

22. Multiple Choice Questions with Answers

Question 1

Which situation most directly indicates data leakage?

A. Scaling features using statistics learned from the training fold

B. Using a variable recorded after the predicted event as an input

C. Selecting the classification threshold using a validation set

D. Comparing models with the same test set

Correct Answer: B. Using a variable recorded after the predicted event as an input

Question 2

In logistic regression, exponentiating a coefficient gives the associated change in:

A. Mean squared error

B. Feature variance

C. Odds, holding other included variables constant

D. Decision-tree depth

Correct Answer: C. Odds, holding other included variables constant

Question 3

Why does a random forest generally reduce the variance of a single decision tree?

A. It removes all nonlinear relationships

B. It averages many less-correlated trees

C. It forces every tree to use the same features

D. It fits one tree to the entire dataset without sampling

Correct Answer: B. It averages many less-correlated trees

Question 4

Which model is especially suited to sparse high-dimensional count data such as word frequencies?

A. Multinomial naïve Bayes

B. Unpruned regression tree only

C. Unscaled k-nearest neighbours only

D. Ordinary least squares with no target definition

Correct Answer: A. Multinomial naïve Bayes

Question 5

What is the main effect of choosing a very small k in k-nearest-neighbour classification?

A. A smoother and more biased boundary

B. A flexible boundary that may be sensitive to noise

C. Removal of the need for feature scaling

D. A guaranteed reduction in test error

Correct Answer: B. A flexible boundary that may be sensitive to noise

Question 6

In a support vector machine, the support vectors are important because they:

A. Are the observations closest to the decision boundary

B. Are always the largest observations

C. Determine the number of target classes

D. Are used only to calculate feature means

Correct Answer: A. Are the observations closest to the decision boundary

Question 7

A model has 92 percent accuracy on a dataset where only 8 percent of cases are positive. What is the most appropriate initial concern?

A. The model must be perfectly calibrated

B. Accuracy may be achieved by ignoring the minority class

C. The target is necessarily continuous

D. The model cannot be evaluated with recall

Correct Answer: B. Accuracy may be achieved by ignoring the minority class

Question 8

What distinguishes boosting from ordinary bagging most directly?

A. Boosting trains learners sequentially to address earlier errors

B. Boosting never uses decision trees

C. Bagging always produces a linear model

D. Boosting eliminates the need for validation

Correct Answer: A. Boosting trains learners sequentially to address earlier errors

Question 9

Why might a hospital prefer a regularized logistic model over a more accurate black-box model for outreach prioritization?

A. Logistic regression can never overfit

B. The simpler model may offer clearer interpretation and governance

C. Black-box models cannot process numerical variables

D. Logistic regression always has higher recall

Correct Answer: B. The simpler model may offer clearer interpretation and governance

Question 10

Which practice best prevents preprocessing leakage during cross-validation?

A. Fit imputation and scaling on the full dataset before splitting

B. Fit preprocessing steps separately within each training fold

C. Use the test labels to select the threshold

D. Duplicate minority examples before creating folds

Correct Answer: B. Fit preprocessing steps separately within each training fold