Supervised Learning for Prediction and Classification

Introduction

Supervised learning is the branch of machine learning in which an algorithm learns a relationship between input variables and a known target variable. The target may be numerical, such as monthly sales, or categorical, such as whether a customer will default. After learning from historical labelled data, the model predicts outcomes for previously unseen cases.

Within data warehousing and data mining, supervised learning converts integrated historical data into operational and strategic decisions. A data warehouse may contain customer profiles, transactions, service interactions, campaign responses and financial outcomes. Data mining methods use these data to estimate demand, identify credit risk, predict churn, detect fraud and support resource planning.

Prediction quality alone is insufficient in business settings. A useful model must also be appropriate to the decision, robust to changing data, computationally feasible, interpretable when required, fair to affected groups and governed throughout its lifecycle.

Learning Objectives

After completing this chapter, the reader should be able to:

• distinguish regression from classification;
• formulate supervised-learning problems using training, validation and test data;
• explain and apply regression, logistic regression, decision trees, Bayesian methods, k-nearest neighbours, support vector machines, ensembles and neural networks;
• select evaluation metrics according to business costs and class distributions;
• diagnose overfitting, data leakage, bias and calibration problems;
• compare models using both statistical evidence and managerial requirements;
• design a governed predictive solution for a business application.

1. The supervised-learning problem

Let a dataset contain n observations. Each observation consists of a feature vector xᵢ and an observed target yᵢ. A supervised-learning algorithm estimates a function f such that:

Here, ŷᵢ is the prediction, θ denotes learned parameters, and f is selected from a model class. The model is trained by minimizing a loss function, often with a complexity penalty:

L measures prediction error, Ω controls model complexity, and λ ≥ 0 determines the strength of regularization. This formulation is general: the loss and penalty depend on the task and model. Training data estimate parameters; validation data support model selection and hyperparameter tuning; the test set provides an approximately unbiased estimate of final generalization performance. In time-dependent business data, chronological splitting is usually preferable to random splitting.

Regression and classification

Regression predicts a continuous or approximately continuous quantity. Examples include revenue, delivery time and energy consumption. Classification predicts a class label. Binary classification has two classes, while multiclass classification has more than two. A classifier may output a probability before applying a decision threshold.

The expected business loss is not necessarily the same as average statistical error. If missing a fraudulent transaction costs substantially more than investigating a legitimate transaction, the decision threshold should reflect those asymmetric costs.

Data preparation

A typical warehouse-to-model pipeline includes:

1. define the prediction target and prediction time;
2. create a modelling table at the correct grain, such as one row per customer;
3. construct features using only information available at prediction time;
4. handle missing values, inconsistent categories and extreme values;
5. split data without allowing future information to enter the past;
6. train, validate and test the model;
7. deploy predictions with monitoring and governance.

Feature engineering may include rolling transaction totals, recency, frequency, monetary value, ratios and interaction indicators. Every transformation learned from data—such as a mean used for imputation or a scaling parameter—must be fitted on training data and then applied unchanged to validation and test data. Otherwise, information leakage produces overoptimistic results.

2. Linear regression for numerical prediction

The multiple linear regression model is:

where yᵢ is the response, xᵢⱼ is feature j for observation i, β₀ is the intercept, βⱼ is the partial effect of feature j, and εᵢ is an error term. The ordinary least-squares estimator minimizes the residual sum of squares:

In matrix form, when X has full column rank:

This closed-form expression assumes a linear model and is computationally less convenient than numerical optimization for very large or highly regularized problems. The classical inferential assumptions include linearity in parameters, independent errors, constant error variance and, for exact small-sample tests, normally distributed errors. Prediction can still be useful when some inferential assumptions are imperfect, but diagnostics remain essential.

A common fit measure is the coefficient of determination:

R² is the proportion of sample variation explained relative to a mean-only baseline. It is not a percentage accuracy measure and can be misleading when comparing models on different datasets. For prediction, mean absolute error and root mean squared error are often more interpretable:

MAE has the same units as y and treats errors linearly. RMSE gives greater weight to large errors. Linear regression is transparent and efficient, but it may underfit nonlinear relationships, react strongly to outliers and become unstable under multicollinearity. Ridge regression adds λ∑βⱼ², while lasso adds λ∑|βⱼ|; the intercept is normally excluded from these penalties.

3. Logistic regression for classification

Logistic regression models the probability of a binary outcome. If p(x) = P(Y = 1 | x), then:

where:

Equivalently, the log-odds are linear:

The coefficient e^βⱼ is the multiplicative change in the odds for a one-unit increase in xⱼ, holding other variables constant. This interpretation requires careful attention to coding, scaling and interactions. Logistic regression parameters are generally estimated by maximum likelihood rather than ordinary least squares.

For a threshold t, the predicted class is 1 if p(x) ≥ t and 0 otherwise. The default t = 0.5 is not universally appropriate. Threshold selection should reflect prevalence, capacity constraints and the relative costs of false positives and false negatives.

The confusion matrix contains true positives, true negatives, false positives and false negatives. Important metrics include:

The F1 score is the harmonic mean of precision and recall:

Accuracy is unsuitable as the principal metric for severely imbalanced classes. ROC-AUC evaluates ranking across thresholds, whereas precision–recall curves are often more informative when the positive class is rare. Probability calibration should be assessed separately from discrimination: a model can rank cases well while producing poorly calibrated probabilities.

4. Decision trees

A decision tree recursively partitions the feature space. Internal nodes apply rules such as “income < £40,000,” branches represent outcomes, and leaves provide a prediction. For regression, a node commonly predicts its mean response. For classification, it predicts the majority class or class probabilities.

For a classification node with class proportions p₁, …, pₖ, Gini impurity is:

Entropy is:

A split is selected by reducing weighted child impurity. If a parent has N observations and child j has Nⱼ, the impurity decrease is:

The formula applies to either Gini impurity or entropy, provided the same criterion is used consistently. A pure node has impurity zero. Trees require little preprocessing, naturally capture interactions and are easy to explain. However, unrestricted trees overfit, are unstable under small data changes and may favour variables with many possible split points.

Pruning, maximum depth, minimum leaf size and minimum impurity decrease control complexity. A single tree is often improved by ensembles, although the resulting model may be less transparent.

5. Bayesian methods

Bayesian classification uses Bayes’ theorem:

Cₖ is class k, P(Cₖ) is its prior probability, P(x | Cₖ) is the class-conditional likelihood, and P(Cₖ | x) is the posterior probability. Since P(x) is common across classes, classification can compare P(x | Cₖ)P(Cₖ).

Naïve Bayes assumes conditional independence of features given the class:

The assumption is often unrealistic, but the method can perform well with small datasets, high-dimensional sparse text and rapidly changing streams. Gaussian naïve Bayes models continuous features with class-specific normal distributions; multinomial naïve Bayes is common for word counts. Zero-frequency problems can be addressed by smoothing, such as adding a small pseudo-count. Bayesian models are fast and provide probabilistic outputs, but correlated predictors can distort probability estimates and the independence assumption limits representation of complex interactions.

6. k-nearest neighbours

The k-nearest neighbours (k-NN) method stores training cases and predicts a new case from its k closest observations. For classification, the majority class is selected; for regression, the mean or distance-weighted mean is used.

For two vectors x and z, Euclidean distance is:

This distance is appropriate when numeric features are comparable and the geometry is meaningful. Standardization is usually necessary because a feature measured in large units can dominate the distance. In classification, odd k values may reduce ties, but k must be selected through validation. Small k gives flexible, high-variance predictions; large k gives smoother, potentially biased predictions. k-NN can be expensive at prediction time and performs poorly in high-dimensional spaces because distances become less informative—the curse of dimensionality.

7. Support vector machines

A support vector machine seeks a separating hyperplane with a large margin. For a linear classifier, the decision function is:

For separable data, the margin is 2/||w||. Soft-margin SVM permits violations and solves an optimization of the form:

subject to yᵢ(wᵀxᵢ + b) ≥ 1 − ξᵢ and ξᵢ ≥ 0, where yᵢ ∈ {−1,+1}, ξᵢ are slack variables and C controls the trade-off between margin width and training violations.

Kernel methods implicitly map observations into a richer feature space. A radial basis function kernel is commonly written:

where γ controls the locality of influence. Larger γ can create highly flexible boundaries. SVMs require careful scaling and tuning of C and kernel parameters. They are powerful for medium-sized, high-dimensional datasets but can be computationally demanding and less interpretable than linear or tree models. Standard SVM scores are not automatically calibrated probabilities.

8. Ensemble learning

Ensembles combine multiple models to improve generalization. Bagging trains models on bootstrap samples and averages their predictions. Random forests add random feature selection to decorrelate trees. For classification, majority voting or averaged class probabilities is used; for regression, predictions are averaged.

Boosting constructs models sequentially, with later models focusing on prior errors. Gradient boosting minimizes a loss by adding weak learners in the direction that reduces current loss. A generic additive form is:

where hₘ is the new learner and η is the learning rate. Small learning rates combined with more trees often improve generalization, but depth, number of estimators, subsampling and regularization require validation. Modern implementations such as gradient-boosted decision-tree systems are highly effective for structured tabular data.

Stacking trains a meta-model on predictions from several base models. To avoid leakage, meta-model training must use out-of-fold predictions rather than predictions generated from the same data used to fit each base learner. Ensembles commonly improve accuracy but increase computational cost, maintenance complexity and explanation requirements.

9. Neural networks

A feed-forward neural network transforms inputs through layers. A neuron computes:

where w is a weight vector, b is a bias, and g is an activation function. ReLU is g(z) = max(0,z). A sigmoid output is suitable for binary probability prediction, while a softmax output for K classes is:

Training uses gradient-based optimization and backpropagation to minimize a loss such as binary cross-entropy:

Neural networks can learn nonlinear interactions and are particularly valuable for images, language, audio and very large datasets. For ordinary structured business tables, boosted trees frequently provide a stronger accuracy–effort trade-off unless the dataset is large or includes unstructured inputs. Networks require scaling, architecture choices, regularization, validation and substantial computing resources. Their opacity also increases the need for explanation, monitoring and human oversight.

10. Model evaluation and selection

Evaluation must mirror deployment. A random split is inappropriate when observations are temporally dependent, when multiple records belong to the same customer, or when repeated measurements leak identity across partitions. Grouped or time-based validation is then required.

Cross-validation divides training data into folds, fits on some folds and evaluates on the remaining fold. The process estimates expected performance, but it does not eliminate dataset shift. Hyperparameter tuning must occur inside the training portion of nested validation when an unbiased research estimate is required.

Model comparison should consider:

The bias–variance perspective explains a central trade-off. High bias reflects an overly restrictive model; high variance reflects excessive sensitivity to training data. Regularization, more data, better features and appropriate validation can improve the trade-off. A model should be compared with a simple baseline, such as the mean for regression or the majority class for classification.

11. Illustrative case: predicting retail customer churn

Background and problem

A subscription retailer wants to identify customers at risk of cancelling within the next 60 days. The warehouse contains customer tenure, monthly spend, support contacts, payment history, product usage and campaign interactions. The business can contact only 20% of customers each month, and an unnecessary contact costs less than a missed cancellation.

Analysis

The modelling grain is one row per customer at the monthly scoring date. The target is cancellation during the following 60 days. Features are calculated from data available before that date. Cancellation records and future campaign responses are excluded from predictors. Because customer behaviour changes over time, the last two months are reserved as a chronological test period.

A logistic regression provides a transparent benchmark. A pruned decision tree explains simple segments. A random forest and gradient-boosted tree model capture nonlinear effects, such as high risk among customers with both declining usage and repeated service complaints. Class weights or threshold selection may be used if churn is relatively uncommon.

Suppose the business evaluates the top 20% of customers ranked by predicted risk. The relevant question is not merely overall accuracy, but how many genuine churners are captured in that contact capacity, the cost of intervention, the retention value and whether probabilities remain calibrated across customer groups. A lift chart, confusion matrix at the selected capacity and expected-profit analysis are more useful than an unqualified accuracy figure.

Approach and solution

The selected model is the simplest model meeting the operational target. Its score is calibrated on validation data, and a threshold is chosen using expected value:

Here V is the value of successfully retained customers, C is the cost of unnecessary interventions, and A is the fixed or variable operating cost of the campaign. The formula is illustrative and must be adapted to the organization’s actual costs, including customer inconvenience and long-term effects.

The deployment system writes a scored customer table to the warehouse, records the model version and feature timestamp, and sends only eligible customers to the campaign platform. Outcomes are returned to the warehouse for monitoring and future retraining.

Outcome and learning points

The case demonstrates that a high-performing algorithm is not automatically the best business solution. Correct temporal design prevents leakage; capacity-aware ranking connects prediction to action; calibration supports value estimation; and governance records make the process auditable. Performance should be monitored for drift in feature distributions, churn prevalence, response rates and subgroup outcomes.

12. Challenges, ethics and governance

Supervised models inherit defects in historical data. A target may reflect unequal access to services rather than genuine need. Protected attributes may be excluded while proxy variables, such as location or purchasing pattern, reproduce their effects. Fairness cannot be guaranteed by removing one column.

Governance should specify legitimate purpose, data minimization, retention, access control, lineage, reproducibility, human review and appeal mechanisms. Privacy-preserving practices include aggregation, pseudonymization and restricted use of sensitive features, although pseudonymization does not eliminate re-identification risk. Automated decisions with significant effects may require explanation, documentation and human oversight under applicable law.

Common technical failures include leakage, target drift, unrepresentative samples, missing-not-at-random values, feedback loops and unmonitored degradation. Monitoring should cover data quality, prediction distributions, calibration, error rates, fairness indicators and business outcomes. Retraining should be triggered by evidence and governed change procedures rather than by an arbitrary schedule alone.

13. Contemporary developments and best practice

Automated machine-learning systems can search model and preprocessing choices, but automation does not replace problem definition, causal reasoning or governance. Explainability tools such as permutation importance and local attribution can assist investigation, but they are not proofs of causality and may be unstable under correlated features. Generative AI can support feature documentation and analyst workflows, yet generated code and interpretations require independent validation.

A strong professional workflow is to begin with a meaningful baseline, define the decision and cost structure, prevent leakage, use appropriate validation, compare diverse but justified algorithms, calibrate where probabilities drive action, test subgroup and temporal robustness, document assumptions, obtain stakeholder review, deploy with monitoring and reassess whether prediction is preferable to an intervention experiment. Where the objective is to decide what action causes an outcome, predictive association alone is insufficient; causal inference or randomized experimentation may be required.

Chapter Summary

Supervised learning estimates relationships between features and known outcomes for prediction or classification. Linear regression is interpretable and effective for approximately linear numerical relationships; logistic regression estimates binary probabilities through the log-odds; decision trees represent rule-based nonlinear structure; Bayesian methods combine priors and likelihoods; k-nearest neighbours uses local similarity; support vector machines optimize separating margins; ensembles reduce variance or build additive predictive strength; and neural networks learn layered nonlinear representations.

Reliable practice requires sound target definition, leakage-resistant data preparation, suitable validation, task-specific metrics, calibration, complexity control and comparison with simple baselines. Business deployment must account for capacity, costs, fairness, privacy, explainability, drift and operational monitoring. The best model is therefore not necessarily the most complex or accurate in isolation, but the one that produces reliable, governable and valuable decisions.

Key Terms

Supervised learning: Learning a mapping from labelled inputs to known outcomes.

Regression: Prediction of a numerical response.

Classification: Assignment of observations to categorical classes.

Feature: An input variable used by a predictive model.

Target: The outcome that the model learns to predict.

Loss function: A measure of discrepancy between observed and predicted outcomes.

Regularization: A complexity penalty used to reduce overfitting.

Logistic regression: A model that represents binary-event log-odds as a linear function of predictors.

Decision tree: A recursive rule-based partition of the feature space.

Gini impurity: A measure of class mixture within a node.

Bayes’ theorem: A rule for updating prior probabilities using evidence.

Naïve Bayes: A Bayesian classifier using conditional independence of features given the class.

k-nearest neighbours: A similarity-based method using nearby training observations.

Support vector machine: A margin-based classifier or regressor.

Ensemble: A combination of multiple predictive models.

Boosting: Sequential construction of models that reduce a chosen loss.

Calibration: Agreement between predicted probabilities and observed frequencies.

Data leakage: Unintended use of information unavailable at prediction time.

Concept drift: Change over time in the data-generating relationship.

Confusion matrix: A table of classification outcomes by predicted and actual class.

Review Questions

Short Answer Questions

1. Distinguish regression from classification and give one business example of each.
2. Why must transformations such as imputation and scaling be fitted only on training data?
3. Explain the interpretation of e^βⱼ in logistic regression.
4. State one strength and one limitation of decision trees.
5. Why is accuracy potentially misleading for a highly imbalanced classification problem?

Long Answer Questions

1. Compare linear regression, logistic regression and decision trees in terms of assumptions, interpretability, nonlinear structure and overfitting.
2. Explain Bayes’ theorem and the naïve conditional-independence assumption. Discuss when naïve Bayes may nevertheless perform well.
3. Describe how bagging, random forests, boosting and stacking differ, including their principal risks.
4. Explain the role of training, validation and test data in model development, including the special requirements of temporal data.
5. Discuss how ethics, privacy, fairness, calibration and monitoring should influence the deployment of a customer-risk model.

Analytical / Application Questions

1. A fraud model has excellent accuracy but detects very few fraudulent transactions. Which metrics, validation design and threshold strategy should be investigated, and why?
2. A retailer has 50,000 structured customer records and limited computing resources. Propose a model-comparison plan involving a baseline, logistic regression, a tree ensemble and calibration. Justify the evaluation criteria.
3. A churn model performs well overall but poorly for a small geographic group. Analyse possible causes and propose technical, data-governance and managerial responses.

Multiple Choice Questions with Answers

Question 1

Which metric is especially appropriate when the positive class is rare and the organization cares about retrieved positive cases?

A. Precision–recall analysis

B. Mean squared error only

C. R-squared

D. Variance inflation factor

Correct Answer: A. Precision–recall analysis

Question 2

In logistic regression, e^βⱼ represents which quantity, holding other variables constant?

A. The additive change in probability

B. The multiplicative change in odds for a one-unit increase in xⱼ

C. The residual standard deviation

D. The classification accuracy

Correct Answer: B. The multiplicative change in odds for a one-unit increase in xⱼ

Question 3

What does a decision-tree impurity decrease measure?

A. The increase in the number of features

B. The reduction in weighted child-node impurity relative to the parent

C. The distance between two observations

D. The number of support vectors

Correct Answer: B. The reduction in weighted child-node impurity relative to the parent

Question 4

What assumption is central to naïve Bayes?

A. All classes have equal prior probability

B. Features are conditionally independent given the class

C. The target is continuous

D. Errors have constant variance

Correct Answer: B. Features are conditionally independent given the class

Question 5

Why is feature scaling usually important for k-nearest neighbours?

A. It makes the target categorical

B. It prevents large-unit features from dominating distance calculations

C. It removes the need to select k

D. It guarantees zero classification error

Correct Answer: B. It prevents large-unit features from dominating distance calculations

Question 6

In a soft-margin SVM, what does C primarily control?

A. The number of classes

B. The trade-off between margin width and training violations

C. The number of nearest neighbours

D. The prior probability of a class

Correct Answer: B. The trade-off between margin width and training violations

Question 7

Which ensemble method builds models sequentially so later learners address previous errors?

A. Bagging

B. Boosting

C. Random subsampling without aggregation

D. Nearest-neighbour voting

Correct Answer: B. Boosting

Question 8

What is data leakage in supervised learning?

A. Loss of a database connection

B. Use of information unavailable at prediction time during model development

C. Random variation in the target

D. A model with too few parameters

Correct Answer: B. Use of information unavailable at prediction time during model development

Question 9

A model has high recall but low precision. What does this generally mean?

A. It finds many actual positives but also flags many negatives as positive

B. It finds no positive cases

C. It predicts only the majority class

D. Its regression residuals are normally distributed

Correct Answer: A. It finds many actual positives but also flags many negatives as positive

Question 10

Which situation most strongly favours chronological rather than random validation?

A. Independent observations with no time structure

B. A prediction task in which future behaviour must be predicted from past records

C. A dataset containing only one observation

D. A classification task with no target variable

Correct Answer: B. A prediction task in which future behaviour must be predicted from past records