7. Supervised Data Mining: Classification and Prediction

Introduction and Context

Supervised data mining learns a relationship between observed input variables and a known target outcome. The available data contain examples for which the correct answer is already recorded: a customer may be labelled as having churned or not churned, a transaction may be marked fraudulent or legitimate, or a house may have a known selling price. The purpose of supervised learning is to use these examples to construct a model that can estimate outcomes for new cases.

The two central forms of supervised data mining are classification and prediction. Classification estimates a categorical outcome, such as “default” versus “no default” or “approved” versus “rejected”. Prediction, often called regression, estimates a numerical quantity, such as demand, revenue, delivery time, or credit loss. In practice, the distinction is not merely technical. It determines the type of model, evaluation measure, feature preparation, decision threshold, and business cost analysis required.

A model is not valuable simply because it produces accurate estimates on historical data. It must generalize to future observations, operate within practical constraints, provide an appropriate level of explanation, and support decisions whose consequences are understood. A highly accurate fraud model that generates too many false alarms may be less useful than a slightly less accurate model that investigators can efficiently act upon. Similarly, a regression model with a low average error may still perform poorly for the high-value customers or extreme events that matter most to management.

This chapter examines major supervised learning approaches: linear and logistic models, decision trees, rule learners, nearest-neighbour methods, support vector machines, and ensemble models. The emphasis is on connecting algorithmic assumptions with feature engineering, validation, interpretation, and business costs.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Distinguish classification from numerical prediction and formulate each as a supervised learning problem.

2. Explain the assumptions, strengths, and weaknesses of linear, logistic, tree-based, rule-based, nearest-neighbour, support vector, and ensemble methods.

3. Design features and preprocessing procedures appropriate to different supervised algorithms.

4. Select evaluation measures that reflect both statistical performance and organizational costs.

5. Compare models using valid resampling and explainability practices.

6. Apply supervised learning methods to a realistic business problem and identify risks involving bias, leakage, governance, and deployment.

From Labelled Examples to Decision Functions

Let the input for observation i be represented by a feature vector xᵢ and the observed target by yᵢ. A supervised algorithm uses a training set D = {(xᵢ, yᵢ)} to estimate a function f such that f(x) produces either a predicted value or a class score for a new observation. The function is learned by minimizing a loss function, usually with some form of regularization to discourage unnecessarily complex solutions.

For regression, y is numerical and a common objective is squared error:

L(y, ŷ) = (y − ŷ)².

For binary classification, y may take values 0 and 1. A model often produces a probability p̂ = P(y = 1 | x), which is converted into a class using a threshold. The familiar threshold of 0.5 is not universally appropriate. If missing a positive case is very costly, a lower threshold may be justified; if investigating a positive prediction is expensive, a higher threshold may be preferable.

The modelling process therefore has at least three layers. First, the statistical model estimates an outcome. Second, an operating rule converts that estimate into an action. Third, the organization evaluates whether the action creates sufficient value. Confusing these layers is a frequent source of poor deployment decisions.

Classification and prediction terminology

A class label is the categorical outcome assigned to an observation. A feature or predictor is an input variable used by the model. The response, target, or dependent variable is the outcome being estimated. A prediction score may be a continuous value, while a predicted class is the final categorical decision.

For binary classification, a confusion matrix contains true positives, true negatives, false positives, and false negatives. Accuracy is the proportion classified correctly, but it can be misleading when classes are imbalanced. Precision is the proportion of predicted positives that are truly positive; recall, or sensitivity, is the proportion of actual positives detected. Specificity is the proportion of actual negatives correctly rejected. The F1 score is the harmonic mean of precision and recall. Receiver operating characteristic analysis examines the trade-off between sensitivity and false-positive rate across thresholds, while precision–recall analysis is often more informative when positive cases are rare.

For regression, mean absolute error (MAE) reports the average absolute difference between actual and predicted values. Root mean squared error (RMSE) gives greater influence to large errors. Mean absolute percentage error can be difficult when actual values are zero or close to zero. R-squared describes the proportion of variation explained relative to a baseline, but it is not a direct measure of business usefulness.

Linear Models: Transparent Baselines and Strong Predictors

Multiple linear regression assumes that the expected target can be represented as a weighted combination of features:

ŷ = β₀ + β₁x₁ + β₂x₂ + … + βₚxₚ.

The coefficients express the expected change in the target associated with a one-unit change in a feature, holding other included features constant. This makes linear regression an important baseline and, in many regulated or managerial settings, a valuable explanatory model.

The basic model relies on assumptions concerning the form of the relationship, independence of observations, and the behaviour of residuals. Constant error variance and normally distributed errors are particularly relevant for classical inference, although prediction can remain useful when some assumptions are imperfect. Linearity does not mean that every raw variable must have a linear effect. Logarithmic transformations, polynomial terms, interaction terms, splines, and carefully constructed domain features can represent nonlinear relationships while retaining a relatively interpretable model.

Multicollinearity makes coefficient interpretation unstable when predictors carry highly overlapping information. Regularization addresses this problem. Ridge regression shrinks coefficients toward zero and is useful when many correlated variables contribute small effects. Lasso regression can shrink some coefficients exactly to zero, providing a form of feature selection. Elastic net combines both penalties and is often useful with correlated high-dimensional features.

Feature engineering is central to linear modelling. A timestamp may be decomposed into hour, weekday, month, and holiday status; a transaction table may produce customer-level totals, recency, frequency, and monetary value; and a text collection may be represented using term frequencies or embeddings. Scaling is especially important when regularization or distance-based comparisons are used.

Logistic Models for Probabilistic Classification

Logistic regression applies a linear predictor to the log-odds of a binary outcome:

log(p / (1 − p)) = β₀ + β₁x₁ + … + βₚxₚ.

The model estimates a probability between zero and one. A coefficient indicates the change in log-odds associated with a feature, while exp(βⱼ) is interpreted as an odds ratio under the model specification. Interpretation requires care when features are transformed, standardized, or correlated.

Logistic regression is often effective when class boundaries are approximately linear in the engineered feature space. It handles sparse representations well and provides a useful benchmark against more complex methods. Regularization can improve generalization and manage large numbers of features. Class weights or resampling may be used when the positive class is rare, but these procedures do not eliminate the need for appropriate threshold selection and probability calibration.

A model can discriminate well while producing poorly calibrated probabilities. Calibration asks whether cases assigned a probability of 0.7 are positive approximately 70 percent of the time within a suitable group. Calibration matters when probabilities are used for pricing, staffing, medical triage, or financial provisioning.

Decision Trees and Rule Learners

A decision tree recursively partitions the feature space. At each internal node, the algorithm selects a feature and split that improves a criterion such as impurity reduction. A terminal node supplies a class probability, class label, or numerical prediction. Trees naturally represent nonlinear effects, thresholds, and interactions. They can work with minimal scaling and produce a decision path that many users find accessible.

An unrestricted tree can memorize noise. Maximum depth, minimum observations per leaf, minimum split size, pruning, and validation-based tuning control complexity. Small changes in the data can sometimes produce a different tree, so an apparently simple structure should not automatically be treated as stable or causal.

A rule learner expresses knowledge as IF–THEN statements. For example: “IF recent complaint count is high AND account age is low, THEN classify as high risk.” Rule systems can be extracted from trees or learned directly. Their main attraction is operational interpretability: a policy team can inspect, challenge, and sometimes implement a rule without a black-box scoring service. However, a large rule list may overlap, conflict, or create uncovered cases. Rule quality should therefore be assessed by coverage, accuracy, simplicity, and stability.

Categorical encoding deserves attention. Some tree implementations can process categories directly, while others require one-hot or carefully controlled target encoding. Target encoding must be fitted inside the training folds; otherwise, information from validation observations leaks into the features.

Nearest-Neighbour Methods: Learning by Similarity

The k-nearest-neighbour (k-NN) method predicts an observation using the outcomes of the k most similar training observations. For classification, the majority class or a distance-weighted vote is used; for regression, neighbouring target values are averaged or weighted.

The method is conceptually simple and makes few distributional assumptions. Its quality depends heavily on the definition of similarity. Variables measured on large numerical scales can dominate distance calculations, so standardization is usually necessary. Irrelevant features, highly correlated variables, missing values, and sparse high-dimensional representations can make neighbourhoods unreliable. This is one reason feature selection or dimensionality reduction may be more important for k-NN than for a regularized linear model.

The value of k controls variance and bias. A small k can capture local structure but is sensitive to noise; a large k produces smoother predictions but may overlook meaningful local patterns. Prediction can also be computationally expensive because new observations must be compared with stored training cases, although indexing and approximate-neighbour methods can reduce the burden.

Support Vector Machines and the Geometry of Separation

A support vector machine (SVM) seeks a separating hyperplane with a large margin between classes. The observations closest to the boundary are the support vectors. A soft-margin formulation permits some errors and uses a penalty parameter C to balance margin width against training violations.

When a linear boundary is inadequate, kernel functions implicitly map observations into a richer feature space. The radial basis function kernel, for example, can represent curved boundaries controlled by a scale parameter. Kernel methods can be powerful for moderate-sized, complex data, but their hyperparameters require careful tuning and their predictions are generally less transparent than those of a small tree or linear model.

SVMs are sensitive to feature scale, particularly with distance-based kernels. Standardization should be learned from the training data only. Class-weighted SVMs can respond to imbalance, but as with other classifiers, the decision threshold and operational consequences must be examined separately from the fitted boundary.

Ensembles: Combining Imperfect Learners

Ensemble learning combines several models to improve predictive performance or stability. Bagging trains models on bootstrap samples and averages their predictions. Random forests add random feature selection to decorrelate trees, reducing variance while retaining the ability to model nonlinear interactions. They are usually robust and require less preprocessing than linear or distance-based methods.

Boosting builds models sequentially, with later learners concentrating on errors made by earlier learners. Gradient boosting minimizes a specified loss through a sequence of weak learners, commonly shallow trees. Modern implementations such as gradient-boosted tree systems can achieve strong performance on structured tabular data, especially when missingness, nonlinearities, and interactions are present. Important controls include learning rate, number of iterations, tree depth, row or feature subsampling, and regularization.

Stacking combines predictions from different model families through a meta-model. It can improve generalization when component models make different errors, but the meta-model must be trained using out-of-fold predictions to avoid leakage. Ensembles often improve accuracy at the cost of interpretability, computational resources, monitoring complexity, and explanation effort. Feature importance measures from ensembles should not be interpreted as causal effects. Permutation importance, partial dependence, accumulated local effects, and local explanation methods can assist interpretation, but each has assumptions and possible failure modes.

Connecting Algorithm Assumptions with Feature Design

Feature engineering should be driven by the interaction between the data-generating process, the algorithm, and the decision objective. A linear model may require log transformations for skewed monetary variables and explicit interaction terms for combined effects. A tree ensemble can discover many thresholds and interactions automatically, but it may still benefit from meaningful aggregates, leakage controls, and domain-specific time windows. k-NN and SVMs require particular attention to scale and distance. Logistic regression benefits from sparse, regularized representations in text classification.

Time is a frequent source of leakage. A feature such as “number of complaints in the next 30 days” may be highly predictive in historical data but unavailable at the moment a decision is made. The training design must reproduce the information available at scoring time. For temporal problems, chronological validation is generally more realistic than randomly mixing past and future cases.

Categorical variables may be one-hot encoded, grouped by meaningful frequency, or represented using learned embeddings. Missingness may itself carry information, but replacing missing values without an indicator can conceal operational processes. Imputation statistics, encoders, scalers, and feature selectors must be fitted within the training data or cross-validation pipeline.

The modelling team should also distinguish between prediction features and policy variables. A feature may be predictive because a previous decision already incorporated the target or because it reflects unequal treatment. Removing it may reduce direct discrimination but leave proxy variables that reproduce the same pattern. Feature review therefore requires domain, legal, and ethical analysis, not only statistical testing.

Model Development, Evaluation, and Selection

A sound workflow begins with a precise decision statement: who will be scored, at what time, for which action, and with what consequences? The target definition should specify the observation window, outcome window, exclusions, and treatment of ambiguous cases. Data are then divided into training and test sets, with validation performed within the training portion for model and hyperparameter selection. The test set should be used sparingly as an estimate of final generalization.

Random splits are unsuitable when repeated observations from the same customer appear in both training and validation, because the model may recognize the customer rather than learn a transferable pattern. Grouped splitting prevents this. Time-based splitting protects against future information and reveals performance drift.

Model comparison should include a simple baseline. For regression, this may be a mean or seasonal forecast; for classification, it may be a majority-class or prior-probability model. A complex model should justify its additional operational cost through improved performance, better targeting, or greater value.

Threshold selection should use a cost matrix. Suppose a fraud system assigns a cost C_FN to a missed fraud and C_FP to an incorrectly blocked legitimate transaction. The preferred threshold depends on these costs, the prevalence of fraud, investigation capacity, and the value of prevented loss. Optimizing accuracy alone implicitly assumes equal costs and balanced classes, assumptions rarely valid in practice.

Performance should be examined across important subgroups and operating conditions. Confidence intervals or repeated cross-validation can reveal whether apparent differences are stable. Calibration, error analysis, drift monitoring, and review of rejected or abstained cases are essential parts of deployment readiness.

Applied Case Study: Predicting Subscription Churn

Background

The following is an illustrative case involving a fictional digital learning provider, LearnPlus. The organization offers monthly subscriptions and wants to identify customers who are likely to cancel in the next 30 days so that retention resources can be allocated effectively.

Situation and problem

The existing campaign sends the same discount to all customers. This is expensive, and some customers who would have remained subscribed receive unnecessary incentives. Management wants a model that ranks customers by churn risk and supports three actions: no intervention, a low-cost reminder, or personal contact by a retention adviser.

Analysis and feature design

The modelling team defines the prediction date as the end of each month and uses only information available at that date. Features include sessions during the previous 7, 30, and 90 days; days since last login; number of completed lessons; support contacts; payment failures; subscription age; and prior discount use. A feature describing cancellation requests after the prediction date is excluded because it would leak the outcome.

Churn is relatively uncommon, so accuracy is rejected as the primary measure. The team uses recall and precision at the number of customers that advisers can contact, together with expected financial value. A chronological split is used: earlier months for training and validation, and the most recent month for testing.

Approach and comparison

A regularized logistic model provides a transparent baseline. It shows that recent inactivity, repeated payment failure, and unresolved support issues are associated with higher estimated risk. A random forest captures nonlinear effects, such as a sharp increase in risk after several weeks of inactivity. A gradient-boosted tree model performs best on the validation data but requires calibration and explanation tools.

The team does not automatically select the model with the highest area under the ROC curve. Instead, it evaluates precision among the top 10 percent of scored customers, expected retained margin after incentive cost, subgroup performance, and stability across months. A threshold is selected so that the adviser team receives a manageable list, while low-cost reminders are sent to a broader group.

Outcome

The selected system is a calibrated boosted-tree model accompanied by a logistic benchmark and a documented decision policy. Customers are ranked daily, but the intervention rules are reviewed monthly. The organization monitors churn outcomes, response to interventions, changes in customer behaviour, and whether discounts are being allocated disproportionately to particular groups.

Learning points

The case illustrates several principles. First, the prediction time defines which features are legitimate. Second, model selection must reflect intervention capacity and financial costs. Third, a high-performing algorithm does not replace a transparent baseline or governance process. Finally, the value of a churn model should be evaluated through the outcomes of the intervention, not merely through offline classification metrics.

Challenges, Risks, and Professional Governance

Overfitting is the central technical risk: a model learns accidental patterns in the training data and fails on new cases. Leakage, inappropriate splitting, unstable labels, and excessive hyperparameter search can create falsely optimistic results. Class imbalance may cause the model to ignore minority cases, while distribution shift may make historical relationships unreliable after a policy or market change.

Interpretability is not identical to simplicity. A small model can be easy to describe but inaccurate or misleading if its features are poorly defined. A complex ensemble can be accompanied by global and local explanations, yet explanations may not fully reveal interactions or causal mechanisms. Decisions with significant consequences require human accountability, appeal mechanisms, documentation, and monitoring.

Fairness assessment should consider selection rates, error rates, calibration, and the social meaning of the target. These criteria can conflict, particularly when groups differ in base rates. Sensitive attributes may be needed for auditing even when they are excluded from prediction. Privacy principles require data minimization, purpose limitation, access control, secure retention, and clear communication about automated decision-making. In regulated settings, model documentation should record the target definition, training data, feature lineage, validation design, known limitations, approval authority, and change history.

Automation also creates organizational risks. A score may become a substitute for professional judgement, or staff may treat a probability as a certainty. Systems should support abstention, escalation, and human review when data are missing, the case is unusual, or the cost of error is high.

Current Developments and Best Practice

Recent supervised-learning practice increasingly emphasizes end-to-end pipelines, automated experiment tracking, model registries, and continuous monitoring. Automated machine learning can compare preprocessing and model configurations, but it does not define the business target, resolve leakage, establish fairness, or decide acceptable costs. These remain professional responsibilities.

For structured business data, gradient-boosted trees remain important, while linear models continue to be valuable for sparse data, calibration, and explanation. Representation learning and pretrained embeddings expand supervised modelling for text, images, audio, and multimodal data. Nevertheless, complex representations require careful governance, reproducibility, and assessment of whether the training data reflect the intended population.

Effective practice generally follows these principles:

• define the decision, prediction time, target window, and cost of errors before model fitting;
• establish a simple and interpretable baseline;
• place preprocessing and feature selection inside a reproducible validation pipeline;
• use temporal, grouped, or stratified evaluation as appropriate;
• report several metrics, including calibration and subgroup performance;
• select thresholds according to capacity and consequences rather than convention;
• document assumptions, data lineage, limitations, and model ownership;
• monitor drift, feedback effects, intervention outcomes, and fairness after deployment;
• retain human oversight for consequential decisions.

Chapter Summary

Supervised data mining learns from labelled examples to classify observations or predict numerical outcomes. Linear and logistic models offer transparent, regularized baselines; trees and rule learners represent thresholds and interactions; nearest-neighbour methods depend on meaningful, scaled similarity; SVMs construct margin-based boundaries; and ensembles combine learners to improve stability or accuracy. No algorithm is universally best. Feature design, validation, calibration, threshold selection, and business costs determine whether a model is useful.

Reliable practice prevents leakage, respects time and group structure, compares models against simple baselines, and evaluates performance with measures appropriate to the target and decision. Deployment requires more than predictive accuracy: it requires monitoring, documentation, privacy protection, fairness analysis, explainability, and human accountability.

Key Terms

Supervised learning: Learning a mapping from input features to a known target using labelled examples.

Classification: Estimation of a categorical outcome.

Regression: Estimation of a numerical outcome.

Feature: An input variable used by a predictive model.

Target: The response or outcome the model is trained to estimate.

Regularization: A penalty or constraint used to reduce model complexity and overfitting.

Logistic regression: A model that estimates class probability through a linear function of log-odds.

Decision tree: A recursive sequence of feature-based partitions used for prediction.

Rule learner: A method that represents predictions as IF–THEN conditions.

Nearest-neighbour method: A method that predicts from the outcomes of similar stored observations.

Support vector machine: A margin-based method that constructs a separating boundary between classes.

Ensemble: A combination of multiple predictive models.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positive cases correctly identified.

Data leakage: Use of information during training that would not be available at prediction time.

Overfitting: Learning noise or idiosyncrasies that reduce performance on new data.

Concept drift: Change over time in the relationship between features and the target.

Threshold: A decision boundary used to convert a score or probability into an action or class.

Review Questions

Short Answer Questions

1. Distinguish classification from regression and give one example of each.

2. Why can accuracy be misleading for an imbalanced classification problem?

3. What is the purpose of regularization in linear or logistic models?

4. Why should features be scaled for k-nearest-neighbour methods?

5. Define data leakage and provide one example.

Long Answer Questions

1. Compare logistic regression, decision trees, and gradient-boosted trees in terms of assumptions, interpretability, feature preparation, and likely applications.

2. Explain how a confusion matrix supports threshold selection when false positives and false negatives have different costs.

3. Discuss the role of chronological, grouped, and stratified validation in supervised model development.

4. Explain how feature engineering differs for linear models, tree ensembles, nearest-neighbour methods, and SVMs.

5. Discuss the governance requirements for deploying a predictive model in a high-consequence organizational setting.

Analytical / Application Questions

1. A bank wants to identify loan applications likely to default, but rejecting a reliable applicant is costly and missing a default is even more costly. Propose an evaluation framework, threshold strategy, and validation design.

2. A retailer has a churn dataset containing “number of customer complaints during the 30 days after the scoring date.” Explain why this feature is problematic and propose a valid alternative.

3. A boosted-tree model has better AUC than a logistic model but produces poorly calibrated probabilities and cannot be used by staff without explanations. Recommend a model-selection and deployment strategy, giving reasons.

22. Multiple Choice Questions with Answers

Question 1

A fraud classifier has very few fraudulent transactions. Which measure is generally more informative than accuracy when evaluating the ability to identify fraud cases?

A. Precision and recall

B. Training-set size only

C. The mean of all feature values

D. The number of model coefficients

Correct Answer: A. Precision and recall

Question 2

What is the main purpose of regularization in a logistic regression model with many correlated features?

A. To guarantee that every prediction is correct

B. To reduce excessive model complexity and improve generalization

C. To convert a classification problem into clustering

D. To remove the need for validation data

Correct Answer: B. To reduce excessive model complexity and improve generalization

Question 3

Why is a feature describing complaints made after the prediction date an example of data leakage?

A. It has a categorical value

B. It is too expensive to calculate

C. It uses information unavailable when the prediction must be made

D. It necessarily has a missing value

Correct Answer: C. It uses information unavailable when the prediction must be made

Question 4

Which characteristic is most directly associated with k-nearest-neighbour methods?

A. Predictions depend on a meaningful distance or similarity measure

B. They require every relationship to be linear

C. They always produce calibrated probabilities

D. They do not store or use training observations at prediction time

Correct Answer: A. Predictions depend on a meaningful distance or similarity measure

Question 5

A business wants to minimize missed high-risk cases, even if this creates more false alarms. Which threshold change will usually increase recall?

A. Lowering the classification threshold

B. Increasing the threshold to 1.0

C. Removing the target variable

D. Using a smaller test set

Correct Answer: A. Lowering the classification threshold

Question 6

What is a defining feature of a decision tree?

A. It recursively partitions observations using feature-based decisions

B. It can only model normally distributed targets

C. It requires all variables to have identical units

D. It cannot represent interactions

Correct Answer: A. It recursively partitions observations using feature-based decisions

Question 7

Why can a chronological split be preferable to a random split for a demand-prediction problem?

A. It guarantees zero prediction error

B. It better represents the task of predicting future observations from past data

C. It eliminates the need for feature engineering

D. It makes the target categorical

Correct Answer: B. It better represents the task of predicting future observations from past data

Question 8

What does calibration assess in a probabilistic classifier?

A. Whether predicted probabilities correspond to observed event frequencies

B. Whether the model has the largest possible number of features

C. Whether all classes have equal sample sizes

D. Whether the training data contain no numerical variables

Correct Answer: A. Whether predicted probabilities correspond to observed event frequencies

Question 9

Which statement best describes boosting?

A. It trains sequential learners that focus on errors made by earlier learners

B. It randomly deletes all difficult observations

C. It uses only one fixed linear equation

D. It prevents any model from overfitting without tuning

Correct Answer: A. It trains sequential learners that focus on errors made by earlier learners

Question 10

In the churn case, why should model selection consider adviser capacity and intervention cost?

A. Because statistical metrics have no meaning

B. Because a model's practical value depends on the actions enabled and their consequences

C. Because high-capacity teams always prefer simpler models

D. Because cost information changes the definition of a feature

Correct Answer: B. Because a model's practical value depends on the actions enabled and their consequences