Supervised Machine Learning

Introduction and Context

Supervised machine learning learns a mapping from observed input variables to a known target outcome. A dataset consists of pairs (x_i, y_i), where x_i is a vector of predictor variables and y_i is the response. The objective is to learn a function f such that predictions f(x) generalize to previously unseen cases.

Supervised learning is central to data science because many organizational decisions have observable outcomes: whether a transaction is fraudulent, the expected price of a property, whether a customer will default, or the number of support requests expected next week. The two principal task types are:

• Classification: predicting a categorical response, such as approved/rejected or disease/no disease.
• Regression: predicting a numerical response, such as revenue, temperature, or delivery time.

A model is useful only when it performs reliably on future data and satisfies operational requirements. Consequently, model selection must consider predictive accuracy, interpretability, computational cost, fairness, data availability, latency, maintainability, and the consequences of errors.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain the supervised learning workflow and distinguish classification from regression.

2. Apply logistic regression, k-nearest neighbors, decision trees, random forests, gradient boosting, support vector machines, and neural networks in R.

3. Select appropriate evaluation metrics and interpret confusion matrices and regression errors.

4. Use resampling, preprocessing, regularization, and tuning to reduce overfitting.

5. Compare models using accuracy, interpretability, computational requirements, and operational constraints.

6. Identify ethical, governance, and deployment risks in predictive systems.

The Supervised Learning Workflow

A defensible workflow normally includes the following stages.

Problem formulation

Define the decision to be supported, the prediction time, the unit of analysis, the target variable, and the cost of different errors. A model predicting loan default must use information available when the loan decision is made; using later information creates leakage.

Data preparation

Inspect missing values, duplicates, invalid values, outliers, class imbalance, and variable types. Categorical predictors may require encoding. Numerical variables may require scaling, especially for distance-based or margin-based algorithms.

Data partitioning

Separate data into training and test sets before substantial model development. The training set is used for fitting and tuning; the test set is reserved for final assessment. A validation set or cross-validation is used during model selection. For time-dependent data, use chronological splits rather than random splitting.

Model fitting and tuning

A model estimates parameters from training data. Hyperparameters, such as k in k-nearest neighbors or tree depth, are selected using cross-validation. All preprocessing operations that learn from data—such as imputation, scaling, and feature selection—must be performed within each resampling fold to prevent leakage.

Evaluation and deployment

Evaluate on untouched test data using metrics aligned with the business objective. A deployment plan should specify monitoring, retraining, access controls, explanation procedures, and a process for handling model failure or drift.

Statistical Foundations

Let the training data be D = {(x_i, y_i): i = 1, ..., n}. A learning algorithm chooses an estimated function f-hat from a model class. The expected prediction error on future observations can be understood as a combination of noise, bias, and variance. A highly rigid model may underfit and have high bias; an excessively flexible model may overfit and have high variance.

Regularization controls complexity by adding a penalty to the fitting objective. For example, ridge regression minimizes:

sum_i (y_i - beta_0 - x_i^T beta)^2 + lambda sum_j beta_j^2

where beta_0 is the intercept, beta is the coefficient vector, and lambda >= 0 controls shrinkage. This expression applies to a numerical response under squared-error loss. Larger lambda generally reduces coefficient magnitude and variance, but may increase bias. Scaling predictors is important because the penalty otherwise depends on measurement units.

Cross-validation estimates generalization performance by repeatedly fitting on one portion of the training data and evaluating on another. In k-fold cross-validation, the training data are divided into k folds; each fold is used once as validation data while the other k - 1 folds are used for fitting. Stratified folds should generally be used for classification so that class proportions are preserved.

Classification Evaluation

For binary classification, a confusion matrix contains:

• True positives (TP): positive cases correctly classified.
• True negatives (TN): negative cases correctly classified.
• False positives (FP): negative cases incorrectly classified as positive.
• False negatives (FN): positive cases incorrectly classified as negative.

Accuracy is (TP + TN)/(TP + TN + FP + FN). It can be misleading when classes are imbalanced.

Precision, also called positive predictive value, is TP/(TP + FP). It answers: among predicted positives, how many are actually positive?

Recall, sensitivity, or true-positive rate is TP/(TP + FN). It answers: among actual positives, how many were detected?

Specificity, or true-negative rate, is TN/(TN + FP). The F1 score is the harmonic mean of precision and recall:

F1 = 2(precision × recall)/(precision + recall)

The F1 score is useful when both false positives and false negatives matter and a single balance measure is desired. It should not replace a cost analysis. A receiver operating characteristic curve examines sensitivity against false-positive rate across thresholds. Its area under the curve (ROC-AUC) measures ranking ability, but may appear optimistic under severe class imbalance. Precision-recall curves are often more informative when the positive class is rare.

A classification model commonly produces a probability p-hat rather than a fixed class. A threshold converts the probability into a decision. Changing the threshold changes the confusion matrix; therefore, the default threshold of 0.5 is not universally appropriate.

Regression Evaluation

For numerical prediction, the mean squared error is:

MSE = (1/n) sum_i (y_i - y-hat_i)^2

The root mean squared error is RMSE = sqrt(MSE), expressed in the same units as the response. Mean absolute error is:

MAE = (1/n) sum_i |y_i - y-hat_i|

MAE is less sensitive to extreme errors than RMSE. R-squared is 1 - SSE/SST, where SSE is the residual sum of squares and SST is the total sum of squares around the training-response mean. R-squared describes relative explained variation and does not by itself establish that a model is useful operationally. Always compare errors with a meaningful baseline, such as predicting the training mean or the previous period's value.

Core Supervised Learning Methods

Logistic Regression

Logistic regression models the probability of a binary outcome. For predictors x_1, ..., x_p:

log(p/(1-p)) = beta_0 + beta_1x_1 + ... + beta_px_p

where p is the probability of the event and log(p/(1-p)) is the log-odds. Equivalently, p = 1/[1 + exp(-(beta_0 + beta^T x))]. The coefficient beta_j represents the change in log-odds for a one-unit increase in x_j, holding other predictors constant. exp(beta_j) is the corresponding odds ratio.

Logistic regression assumes an appropriate linear relationship between predictors and log-odds, independent observations, limited problematic multicollinearity, and an adequately specified outcome process. It is highly interpretable and produces probabilities, but may underperform when decision boundaries are strongly nonlinear unless interactions or transformations are added. Penalized logistic regression can improve performance with many or correlated predictors.

In R, a basic model is:

```r

fit <- glm(default ~ income + age + balance,

data = train, family = binomial())

prob <- predict(fit, newdata = test, type = "response")

pred <- ifelse(prob >= 0.5, "yes", "no")

```

The threshold should be selected using validation data and the relative costs of errors.

k-Nearest Neighbors

k-nearest neighbors (kNN) predicts an observation using the labels or values of its k closest training observations. Classification commonly uses majority voting; regression uses the mean of neighboring responses. Distance is often Euclidean:

d(x, z) = sqrt(sum_j (x_j - z_j)^2)

This distance is appropriate only when variables are comparably scaled and the geometry is meaningful. A small k gives flexible, low-bias predictions but can be noisy; a large k produces smoother, higher-bias predictions. kNN has almost no explicit training phase, but prediction can be expensive because it compares new cases with stored training data. It is sensitive to irrelevant variables, missing values, high dimensionality, and uneven feature scales.

Decision Trees

A decision tree recursively partitions the feature space using questions such as “is income less than 50,000?” A classification split may be selected using Gini impurity:

Gini = 1 - sum_c p_c^2

where p_c is the proportion of observations in a node belonging to class c. Entropy, -sum_c p_c log_2(p_c), is another impurity measure. For regression, splits commonly minimize within-node squared error.

Trees are easy to visualize and can represent nonlinear relationships and interactions without requiring scaling. They handle mixed predictor types well. However, fully grown trees can overfit and are unstable: small data changes may produce different structures. Pruning, maximum depth, minimum node size, and minimum split size are important complexity controls.

Random Forests

A random forest constructs many decision trees using bootstrap samples and random subsets of predictors at each split. The final classification prediction is generally a vote, while regression averages tree predictions. Randomization reduces correlation among trees, and aggregation reduces variance.

Important hyperparameters include the number of trees, the number of candidate variables at each split, and minimum node size. Random forests usually provide strong performance with limited preprocessing and can estimate variable importance. However, importance measures can be biased toward variables with many possible split points or categories, and the ensemble is less interpretable than a single tree. Partial dependence or accumulated local effects can assist interpretation but should not be treated as causal evidence.

Gradient Boosting

Gradient boosting builds trees sequentially. Each new tree focuses on reducing the loss of the existing ensemble, often by approximating the negative gradient of the loss function. A simplified additive form is:

F_m(x) = F_(m-1)(x) + nu h_m(x)

where h_m is the new tree and nu is the learning rate. Smaller learning rates generally require more trees but can improve generalization when tuned appropriately.

Key hyperparameters include learning rate, number of boosting iterations, tree depth, subsampling fraction, and regularization. Boosting often produces excellent tabular-data performance, but it can overfit, be computationally intensive, and require careful tuning. Modern implementations such as XGBoost, LightGBM, and CatBoost add efficient tree construction and regularization; their use remains subject to validation, governance, and explainability requirements.

Support Vector Machines

A support vector classifier seeks a separating boundary with a large margin. With a soft margin, some observations may violate the margin, controlled by cost parameter C. The optimization balances margin width against classification errors. A kernel, such as the radial basis function (RBF), allows nonlinear boundaries through implicit feature mapping.

The RBF kernel is commonly written as K(x, z) = exp(-gamma ||x - z||^2), where gamma controls the locality of influence. Large gamma creates highly local boundaries; small gamma creates smoother ones. SVMs require careful scaling and tuning of C and gamma. They can perform well in high-dimensional settings, but probability calibration, large datasets, and explanation may be challenging.

Neural Networks

A feed-forward neural network transforms inputs through layers of weighted sums and nonlinear activation functions. A hidden unit may compute:

h = g(w_0 + sum_j w_jx_j)

where g is an activation function such as ReLU or sigmoid. The network parameters are learned by minimizing a loss function using backpropagation and gradient-based optimization.

Neural networks can represent highly nonlinear functions and are valuable for images, language, signals, and large-scale structured data. For ordinary tabular datasets, tree ensembles may be competitive or superior. Neural networks require scaling, architecture and optimization choices, regularization, and sufficient data. They are often difficult to interpret, and reproducibility requires control of random seeds, software versions, and training procedures.

Model Selection by Accuracy, Interpretability, and Operations

No algorithm is universally best. A practical comparison is:

| Method | Typical strengths | Important limitations |

|---|---|---|

| Logistic regression | Transparent coefficients, fast, calibrated probabilities when specified well | Linear log-odds assumption; limited nonlinear structure |

| kNN | Simple, flexible local patterns | Scaling, memory, dimensionality, prediction latency |

| Decision tree | Visual and rule-based interpretation | Instability and overfitting |

| Random forest | Strong baseline, robust nonlinear modeling | Less transparent; larger models |

| Gradient boosting | Often excellent tabular accuracy | Tuning, computational cost, explanation complexity |

| SVM | Effective margins and high-dimensional boundaries | Scaling and parameter tuning; less interpretable |

| Neural network | Flexible representation learning | Data, tuning, compute, and explainability demands |

For regulated lending or clinical triage, a slightly less accurate but explainable and auditable model may be preferable. For real-time fraud screening, prediction latency and class imbalance may dominate. For batch demand forecasting, computational cost may be less restrictive, allowing extensive ensembles. Model selection should therefore use a documented utility function or decision framework rather than accuracy alone.

Implementation in R

The tidymodels ecosystem supports a consistent workflow. A conceptual example is:

```r

library(tidymodels)

set.seed(42)

split <- initial_split(data, prop = 0.80, strata = outcome)

train <- training(split)

test <- testing(split)

rec <- recipe(outcome ~ ., data = train) |>

step_impute_median(all_numeric_predictors()) |>

step_normalize(all_numeric_predictors())

spec <- logistic_reg() |>

set_engine("glm") |>

set_mode("classification")

wf <- workflow() |> add_recipe(rec) |> add_model(spec)

fit_obj <- fit(wf, data = train)

```

For kNN, SVM, and neural networks, normalization is usually essential. For tree-based models it is generally unnecessary, although consistent preprocessing may still simplify a production pipeline. Use `vfold_cv()` for resampling, `tune_grid()` or an appropriate search strategy for hyperparameters, and `yardstick` metrics such as `accuracy`, `roc_auc`, `sens`, `spec`, `rmse`, and `mae`.

Reproducible R projects should record package versions, random seeds, data definitions, preprocessing steps, model objects, and evaluation results. The test set must not be repeatedly consulted during tuning.

Worked Example: Classification Metrics

Suppose a fraud classifier produces the following results on 1,000 transactions: TP = 72, TN = 850, FP = 48, and FN = 30. Then:

• Accuracy = (72 + 850)/1,000 = 0.922, or 92.2%.
• Precision = 72/(72 + 48) = 0.600, or 60.0%.
• Recall = 72/(72 + 30) = 0.706, approximately 70.6%.
• Specificity = 850/(850 + 48) = 0.947, approximately 94.7%.
• F1 = 2(0.600 × 0.706)/(0.600 + 0.706) approximately 0.648.

Although accuracy is high, 40% of flagged transactions are not fraudulent. If manual investigation is expensive, precision matters. If missed fraud is much more costly, the threshold may be lowered to increase recall. The final threshold should be chosen using explicit costs and validation data, not the test set.

Applied Case: Illustrative Retail Credit Screening

Background

A fictional retail bank wants to identify applicants likely to default within twelve months. It has income, debt-to-income ratio, credit history length, prior delinquency, loan amount, employment category, and the observed default outcome for historical applicants.

Problem

The bank initially reports 94% accuracy from a random forest. However, only 6% of applicants default. A trivial classifier that predicts “no default” for everyone achieves 94% accuracy, so the reported result is not useful.

Analysis

The data science team first defines the prediction point and removes variables recorded after approval. It uses a chronological split because economic conditions and lending policies change over time. Missing-value imputation, encoding, and scaling are placed inside resampling. Stratified cross-validation is used on the training period.

The team compares penalized logistic regression, a pruned tree, random forest, gradient boosting, and an RBF SVM. It reports recall, precision, PR-AUC, calibration, expected financial cost, and subgroup performance rather than accuracy alone. The cost of a missed default is estimated to be substantially greater than the cost of reviewing a false alarm.

Approach and solution

Logistic regression provides a transparent benchmark and a coefficient-based explanation. Gradient boosting gives the strongest validation ranking performance, but its probabilities require calibration. The bank chooses the calibrated boosting model subject to a recall constraint and a review-capacity limit. Logistic regression is retained as a challenger and fallback model.

The threshold is selected so that the predicted number of manual reviews is operationally feasible. A model card documents intended use, training data, limitations, performance by subgroup, and known exclusions. Human reviewers receive the model score and important contributing factors, but the system does not automatically reject every flagged applicant.

Outcome

On the locked chronological test period, the selected model improves detection at the same review capacity as the previous rules. Monitoring is established for missingness, feature drift, approval-rate changes, calibration, subgroup recall, and realized default rates. Retraining is triggered only after documented review rather than automatically on every fluctuation.

Learning points

The case illustrates that class imbalance makes accuracy inadequate, temporal validation is more realistic than random splitting for changing processes, leakage can create unrealistic performance, threshold selection is a business decision, and governance must continue after deployment.

Challenges, Ethics, and Governance

Data leakage and shift

Leakage occurs when information unavailable at prediction time enters training. Distribution shift occurs when future data differ from training data. Concept drift occurs when the relationship between predictors and outcome changes. Monitoring should distinguish changes in inputs, predictions, calibration, and actual outcomes.

Fairness

A model may reproduce historical discrimination even when sensitive attributes are excluded. Fairness assessment should examine performance and decision rates across relevant groups, while recognizing that fairness criteria can conflict. Sensitive attributes may be needed for auditing, subject to law and privacy safeguards. Group-level metrics do not prove absence of individual discrimination.

Privacy and security

Apply data minimization, purpose limitation, access control, retention limits, encryption, and appropriate anonymization or pseudonymization. Models can leak sensitive information, and adversaries may manipulate inputs. High-impact decisions require documented human oversight and appeal mechanisms where applicable.

Interpretability and accountability

Feature importance is not causality. Local explanations can be unstable, and explanations should be validated with domain experts. Maintain versioned data, code, model parameters, evaluation reports, approvals, and incident records. Establish responsibility for monitoring and override decisions.

Reproducibility and maintenance

A production model includes preprocessing, feature definitions, software dependencies, model artifacts, and serving logic. Changes to any component can alter predictions. Use automated tests, deployment approval, rollback capability, drift alerts, and periodic independent review.

Contemporary Developments and Best Practices

AutoML can accelerate baseline construction, but it does not replace problem formulation, leakage control, fairness analysis, or operational validation. Explainable AI methods, conformal prediction, probabilistic calibration, and uncertainty estimation are increasingly used to communicate confidence and support human decisions. Federated learning and privacy-preserving methods may reduce centralized data exposure but introduce statistical, communication, and governance challenges.

Recommended practice is to:

1. Define the decision, prediction horizon, and error costs before selecting an algorithm.

2. Establish a simple, interpretable baseline.

3. Split data according to the data-generating process.

4. Put learned preprocessing inside cross-validation.

5. Tune only on training data and reserve a locked test set.

6. Report several metrics, confidence intervals where appropriate, and subgroup results.

7. Calibrate probabilities when decisions depend on risk estimates.

8. Prefer the simplest model meeting performance and operational requirements.

9. Document assumptions, data provenance, limitations, and intended use.

10. Monitor deployed performance and provide retraining and retirement criteria.

Chapter Summary

Supervised machine learning learns predictive relationships from labeled examples and supports classification and regression decisions. Logistic regression offers interpretable probability models; kNN uses local similarity; trees provide rule-based partitions; random forests reduce variance through aggregation; gradient boosting builds accurate sequential ensembles; SVMs use margins and kernels; and neural networks learn layered nonlinear representations. Each method has assumptions, computational implications, and interpretability trade-offs.

Reliable practice requires careful problem formulation, leakage-resistant preprocessing, appropriate train-validation-test design, hyperparameter tuning, and metrics aligned with error costs. Accuracy alone is inadequate for imbalanced or high-stakes problems. Thresholds, calibration, subgroup performance, privacy, fairness, monitoring, and human oversight are essential components of deployment. The best model is not necessarily the most complex or accurate in isolation; it is the model that achieves acceptable predictive performance while remaining interpretable, maintainable, equitable, and operationally feasible.

Key Terms

Supervised learning: Learning a predictive function from labeled examples.

Classification: Predicting a categorical outcome.

Regression: Predicting a numerical outcome.

Feature/predictor: An input variable used for prediction.

Target/response: The outcome to be predicted.

Overfitting: Modeling training-specific patterns that fail to generalize.

Regularization: Penalizing model complexity to improve generalization.

Cross-validation: Resampling procedure for estimating performance and tuning models.

Data leakage: Use of information unavailable at prediction time or improperly shared across splits.

Logistic regression: A model that relates predictors to the log-odds of a binary outcome.

Odds ratio: The multiplicative change in odds associated with a one-unit predictor change.

k-nearest neighbors: A method predicting from nearby training observations.

Decision tree: A recursively partitioned model represented by decision rules.

Random forest: An ensemble of randomized bootstrap decision trees.

Gradient boosting: Sequential construction of models that reduce a chosen loss.

Support vector machine: A margin-based learning method that may use kernels.

Neural network: A layered model composed of weighted nonlinear transformations.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are detected.

Calibration: Agreement between predicted probabilities and observed frequencies.

Review Questions

Short Answer Questions

1. Distinguish classification from regression and give one example of each.

2. Why can accuracy be misleading in an imbalanced classification problem?

3. What does the coefficient in logistic regression represent before exponentiation?

4. Why should numerical variables generally be scaled for kNN and SVM?

5. What is the purpose of a locked test set?

Long Answer Questions

1. Explain the complete supervised learning workflow, including data splitting, preprocessing, tuning, evaluation, and deployment.

2. Compare decision trees, random forests, and gradient boosting in terms of bias, variance, interpretability, and computational requirements.

3. Explain precision, recall, specificity, F1 score, ROC-AUC, and PR-AUC, and discuss when each is useful.

4. Describe the assumptions, advantages, and limitations of logistic regression for credit-risk prediction.

5. Discuss ethical, privacy, fairness, and accountability requirements for a high-impact predictive system.

Analytical / Application Questions

1. A medical screening model has TP = 180, TN = 760, FP = 40, and FN = 20. Calculate accuracy, precision, recall, and specificity. Which metric should receive priority if missed cases are substantially more harmful than unnecessary referrals?

2. A company has 2,000 records, with 5% positive cases. Compare the meaning of 95% accuracy from an all-negative baseline with a model reporting 90% recall and 35% precision. Which additional information should management request before deployment?

3. Design an evaluation strategy for a demand-prediction model trained on five years of monthly data. Explain the split strategy, validation method, metrics, leakage risks, and criteria for choosing between an interpretable model and a boosted ensemble.

Multiple Choice Questions with Answers

Question 1

Which metric is the proportion of predicted positive cases that are actually positive?

A. Recall

B. Precision

C. Specificity

D. Accuracy

Correct Answer: B. Precision

Question 2

In logistic regression, exp(beta_j) is interpreted as the:

A. Change in probability for one unit of x_j

B. Change in squared error

C. Odds ratio for a one-unit increase in x_j, holding other variables constant

D. Correlation between x_j and the response

Correct Answer: C. Odds ratio for a one-unit increase in x_j, holding other variables constant

Question 3

Which method is most directly affected by differences in numerical measurement scales?

A. k-nearest neighbors

B. A single decision tree

C. Random forest

D. Gradient boosting with trees

Correct Answer: A. k-nearest neighbors

Question 4

What is the principal variance-reduction mechanism in a random forest?

A. Using one maximally deep tree

B. Averaging or voting across randomized trees

C. Removing all correlated predictors

D. Using logistic loss only

Correct Answer: B. Averaging or voting across randomized trees

Question 5

A false negative is a case that is:

A. Actually negative and predicted negative

B. Actually positive and predicted positive

C. Actually negative and predicted positive

D. Actually positive and predicted negative

Correct Answer: D. Actually positive and predicted negative

Question 6

Which practice prevents preprocessing leakage during cross-validation?

A. Scaling the entire dataset before splitting

B. Fitting preprocessing separately within each resampling analysis

C. Using the test set to choose the threshold

D. Removing the validation folds

Correct Answer: B. Fitting preprocessing separately within each resampling analysis

Question 7

Gradient boosting primarily builds its model by:

A. Training sequential learners to reduce the current loss

B. Selecting the nearest observations only

C. Maximizing the distance between all observations

D. Averaging independent neural networks without fitting

Correct Answer: A. Training sequential learners to reduce the current loss

Question 8

If TP = 80 and FN = 20, what is recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 9

Which model is generally easiest to explain using a small set of explicit decision rules?

A. A single decision tree

B. A deep neural network

C. An RBF support vector machine

D. A large gradient-boosting ensemble

Correct Answer: A. A single decision tree

Question 10

For a time-dependent prediction problem, which evaluation design is usually most appropriate?

A. Randomly mixing all observations before splitting

B. Using future observations to impute past values

C. Chronological training and test periods

D. Evaluating only on the training data

Correct Answer: C. Chronological training and test periods