7. Classification and Predictive Modelling

Introduction and Context

Classification and predictive modelling are central tasks in data mining. They transform historical observations into models that can assign categories, estimate probabilities, or predict future outcomes. In a data warehouse, transactional and operational data are commonly integrated, cleaned, and organized for analysis. Classification uses this prepared data to answer questions such as: Which customers are likely to default? Is a medical image benign or malignant? Will a machine fail during the next production cycle? Which email messages should be treated as spam?

A classification problem predicts a discrete class label, whereas regression predicts a continuous numerical value. In both cases, the underlying objective is generalization: learning patterns from a training set that remain useful for previously unseen cases. A model that merely memorizes its training data has little practical value. Consequently, predictive modelling involves not only selecting an algorithm but also defining the target, preparing data, validating performance, controlling complexity, and interpreting results.

This chapter examines major classification approaches, including decision trees, rule-based classifiers, logistic regression, Bayesian methods, k-nearest neighbours, support vector machines, ensemble methods, and neural networks. It also considers model comparison, overfitting, interpretability, responsible use, and the relationship between predictive accuracy and organizational decision-making.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Distinguish classification from regression and descriptive data mining.

2. Explain the principles and assumptions of major classification algorithms.

3. Construct and interpret decision trees, rules, probabilistic models, and linear classifiers.

4. Select suitable algorithms according to data characteristics and business requirements.

5. Apply validation, regularization, pruning, and ensemble methods to reduce overfitting.

6. Evaluate classifiers using appropriate performance measures rather than accuracy alone.

7. Discuss interpretability, fairness, privacy, governance, and operational risks.

8. Compare predictive models in an applied organizational scenario.

The Predictive Modelling Problem

Let a data set contain observations represented by a feature vector \(x=(x_1,x_2,\ldots,x_p)\) and a target class \(y\). A classifier estimates a function \(f(x)\) that maps the features to a class, or estimates posterior probabilities such as \(P(y=c\mid x)\). In binary classification, the classes may be represented as 0 and 1; in multiclass classification, one observation may belong to one of several mutually exclusive categories. Multi-label classification permits an observation to have several labels simultaneously.

A labelled data set is generally divided into training, validation, and test partitions. The training set is used to estimate model parameters. The validation set supports choices such as hyperparameter selection and threshold adjustment. The test set is used only for final evaluation. When data are limited, k-fold cross-validation repeatedly divides the training data into folds, trains on some folds, and validates on the remaining fold. Test information must not influence preprocessing or model selection, because such leakage produces an overly optimistic estimate of performance.

The distinction between parameters and hyperparameters is important. Parameters are learned from data, such as regression coefficients or tree split points. Hyperparameters are set by the analyst, such as tree depth, the value of k in k-nearest neighbours, or the penalty strength in support vector machines. A sound modelling workflow includes target definition, data profiling, preprocessing, feature engineering, model training, validation, error analysis, interpretation, deployment, and monitoring.

Important Terminology

A feature is an input variable used for prediction, while a label or target is the outcome to be predicted. A classifier produces a predicted class; a score or probability expresses confidence or estimated likelihood. The decision threshold converts a probability into a class. The confusion matrix records true positives, true negatives, false positives, and false negatives. A model’s capacity is its ability to represent complex relationships. High capacity can reduce bias but increase variance and overfitting.

In binary classification, precision is the proportion of predicted positives that are correct, while recall, or sensitivity, is the proportion of actual positives detected. Specificity is the proportion of actual negatives correctly identified. The F1-score is the harmonic mean of precision and recall. The receiver operating characteristic curve displays the trade-off between true-positive and false-positive rates at different thresholds; its area, ROC-AUC, measures ranking ability across thresholds. For strongly imbalanced data, precision-recall analysis may be more informative than ROC analysis.

Families of Classification Models

Decision Trees: Recursive Partitioning

A decision tree represents decisions as a sequence of tests. Internal nodes contain conditions on features, branches represent outcomes of those conditions, and leaves contain class predictions or class probabilities. The tree is built by selecting a split that makes resulting nodes more homogeneous. Common impurity measures include entropy and the Gini index:

\[

Entropy(S)=-\sum_c p_c\log_2(p_c),

\]

where \(p_c\) is the proportion of observations in class \(c\). The Gini impurity is \(1-\sum_c p_c^2\). Information gain compares parent impurity with the weighted impurity of child nodes.

Trees are attractive because they accommodate numerical and categorical variables, capture nonlinear relationships, and require little scaling. Their paths can be explained in ordinary language. However, unrestricted trees can become highly specific to training observations. Pre-pruning limits depth, minimum leaf size, or the number of splits. Post-pruning first grows a larger tree and then removes branches that do not provide sufficient validated improvement. A small change in data can also produce a substantially different tree, making individual trees unstable.

Rule-Based Classification

Rule-based classifiers express knowledge as IF–THEN statements, for example: IF missed payments exceed two AND income is below a threshold, THEN classify as high risk. Rules may be extracted from trees, generated directly through sequential covering, or supplied by domain experts. Sequential covering repeatedly identifies a rule that covers many positive examples with relatively few negative examples, removes covered cases, and continues.

Rules are often easier for domain experts to audit than numerical models. They can also encode policy constraints explicitly. Important issues include rule coverage, accuracy, conflicts between rules, and the default class used when no rule applies. A very large rule set may be as difficult to understand as a complex statistical model, and rigid thresholds may perform poorly when the environment changes.

Logistic Regression: Probabilistic Linear Classification

Logistic regression models the probability of a binary outcome as

\[

P(y=1\mid x)=\frac{1}{1+e^{-(\beta_0+\beta_1x_1+\cdots+\beta_px_p)}}.

\]

Although the probability is nonlinear in the input to the sigmoid function, the log-odds are linear in the features. Coefficients indicate the direction and magnitude of association, conditional on the other variables. Exponentiating a coefficient gives an odds ratio, although interpretation requires care when variables are correlated, scaled unusually, or affected by confounding.

Logistic regression is a strong baseline because it is relatively fast, statistically established, and interpretable. It works particularly well when the decision boundary is approximately linear. Polynomial terms, interactions, transformations, and regularization can extend its usefulness. L1 regularization can produce sparse models by driving some coefficients to zero; L2 regularization shrinks coefficients and usually handles correlated predictors more smoothly. Logistic regression should not be interpreted as proving causation merely because a coefficient is statistically significant.

Bayesian Classification

Bayesian methods combine prior knowledge with observed evidence. Bayes’ theorem states:

\[

P(y\mid x)=\frac{P(x\mid y)P(y)}{P(x)}.

\]

Naïve Bayes assumes that features are conditionally independent given the class. This assumption is often unrealistic, but the method can nevertheless perform well, especially for high-dimensional text data. For a document, word frequencies provide evidence for classes such as spam or legitimate mail. Smoothing prevents an unseen word from making an entire class probability zero. Gaussian, multinomial, and Bernoulli versions are selected according to the nature of the features.

More flexible Bayesian networks represent dependencies among variables through a directed graph. They can provide a useful probabilistic explanation, but learning their structure and estimating reliable probabilities may be computationally demanding. Bayesian methods are especially valuable when prior information, uncertainty quantification, or incremental updating is important.

k-Nearest Neighbours

The k-nearest neighbours (k-NN) method stores labelled training instances. To classify a new observation, it identifies the k closest observations according to a distance measure and assigns the majority class, optionally weighting nearer neighbours more heavily. The method is non-parametric and can represent irregular decision boundaries.

The choice of distance, feature scaling, and k is crucial. A variable measured in thousands can dominate a variable measured in fractions unless features are normalized or standardized. A small k produces flexible, low-bias predictions but is sensitive to noise; a large k smooths the boundary and may underfit. In high-dimensional spaces, distances become less discriminative, a problem known as the curse of dimensionality. Prediction can also be expensive because many stored observations must be searched.

Support Vector Machines

A support vector machine seeks a separating hyperplane with the largest margin between classes. The observations nearest to the boundary are the support vectors. A soft-margin formulation allows some violations and balances margin width against classification errors through a penalty parameter. Kernel functions, such as the radial basis function kernel, implicitly map data into a higher-dimensional feature space and permit nonlinear boundaries.

SVMs can be effective for high-dimensional data and moderate sample sizes. They generally require feature scaling, and their parameters—particularly the penalty and kernel parameters—need validation. Kernel models may be difficult to explain to nontechnical stakeholders, and training can become expensive for very large data sets. Linear SVMs are often preferable when the number of features is very large, as in text classification.

Ensemble Learning

Ensemble methods combine several models to obtain more reliable predictions. Bagging trains models on bootstrap samples and averages or votes over their predictions, reducing variance. Random forests add random feature selection to bagged decision trees, usually improving diversity and robustness. Boosting builds models sequentially, giving greater attention to observations previously predicted poorly. Gradient boosting optimizes a loss function through a sequence of weak learners; modern implementations can model complex interactions with strong predictive performance.

Ensembles often outperform a single model, but their complexity can make explanation harder. Feature importance measures may be unstable or misleading when predictors are correlated. Stacking combines predictions from different model families through a meta-model, but it must be implemented with carefully separated validation predictions to avoid leakage. In safety-sensitive contexts, a small gain in accuracy may not justify a substantial loss of transparency.

Neural Networks

A neural network consists of layers of weighted units. An artificial neuron computes a weighted sum, applies an activation function, and passes the result forward. During training, backpropagation calculates gradients of a loss function, and an optimization method updates the weights. Hidden layers allow networks to learn representations and nonlinear relationships.

Multilayer perceptrons are suitable for structured tabular data, while convolutional networks are prominent for images and sequential architectures are used for temporal or language data. Neural networks typically need sufficient data, careful regularization, and substantial computational resources. Dropout, weight decay, early stopping, data augmentation, and batch normalization can improve generalization. Their internal representations are rarely as directly interpretable as a short tree or a sparse logistic model, so post-hoc explanation methods must be used cautiously.

Choosing and Comparing Models

No algorithm is universally best. Model choice should follow the data, decision context, and cost of errors. A transparent logistic model or shallow tree may be appropriate for credit explanations or clinical triage, whereas an image-recognition task may justify a neural network. Text classification may favour linear SVMs or naïve Bayes. Large tabular data sets often benefit from gradient-boosted trees.

Performance evaluation must reflect the operational objective. If failing to detect a fraudulent transaction is much more costly than investigating a legitimate transaction, recall may be prioritized and a lower threshold may be selected. If investigation capacity is limited, precision may matter more. Accuracy is misleading when one class is rare: a classifier that labels every case negative can appear accurate while detecting no positives.

Cross-validation should preserve class proportions through stratification when appropriate. Time-dependent data require temporal validation: training on future records to predict the past is an invalid simulation of deployment. Grouped observations, such as several transactions from one customer, should be split by group when leakage between related records is possible. Confidence intervals, calibration plots, and error analysis provide additional evidence beyond a single score.

Overfitting, Generalization, and Interpretability

Overfitting occurs when a model captures random variation or idiosyncrasies of the training data rather than stable relationships. Underfitting occurs when the model is too simple to represent important structure. The bias–variance perspective describes the trade-off: increasing complexity generally reduces bias but may increase variance. Regularization, pruning, early stopping, feature selection, dimensionality reduction, and more training data can improve generalization.

Interpretability has several forms. Global interpretability describes overall model behaviour, such as coefficient signs or average feature importance. Local interpretability explains an individual prediction, for example by identifying features that moved a case toward a class. Counterfactual explanations state what change might alter a prediction, but such explanations should be feasible, lawful, and causally meaningful. Explanations generated after training are not automatically faithful to the model and should be validated.

Data quality is often more important than algorithmic sophistication. Missing values, inconsistent labels, measurement error, sampling bias, and target leakage can invalidate the model. Feature engineering should be performed within the training folds when it uses estimated quantities. A deployed model also requires monitoring for drift in feature distributions, class prevalence, relationships, and performance.

Applications and Professional Use

Classification supports customer churn intervention, credit assessment, intrusion detection, predictive maintenance, disease screening, document routing, fraud detection, quality inspection, and recommendation systems. In organizations, the model is only one component of a decision process. A risk score may trigger human review, a maintenance prediction may schedule an inspection, and a fraud alert may temporarily block a transaction.

Managers should ask whether the prediction changes an action, whether the action benefits from early warning, and what errors cost. Technical teams should document data provenance, feature definitions, model versions, validation design, thresholds, and known limitations. Governance teams should establish approval procedures, access controls, audit logs, monitoring responsibilities, and processes for contesting automated decisions.

Illustrative Case Study: Predicting Loan Delinquency

Background

A fictional regional bank has several years of customer and loan records in its analytical warehouse. It wants to identify accounts at risk of becoming seriously delinquent within the next six months so that advisers can offer restructuring or financial counselling.

Situation and Problem

The target class is delinquency within six months. Candidate features include repayment history, utilization ratio, income band, loan age, number of recent applications, and employment stability. Only a small proportion of accounts become delinquent, so accuracy alone would conceal poor detection. The bank also needs reasons that can be communicated to customers and reviewed by compliance officers.

Analysis and Approach

The data team first defines the prediction date and ensures that every feature was available on that date. It removes post-delinquency variables, which would cause target leakage, and splits records chronologically. Missing values are handled within the training process. A regularized logistic regression, a shallow decision tree, a random forest, and a gradient-boosted model are compared using stratified measures within the training period and tested on a later period.

The boosted model produces the highest recall at the bank’s chosen review capacity, while logistic regression is easier to explain. Analysts examine calibration, subgroup performance, false-positive cases, and stability across time. Rather than automatically rejecting customers, the bank uses the model to prioritize human outreach. A threshold is selected to fit the number of advisers available, and cases near the threshold receive additional review.

Strategy and Outcome

The bank deploys the model with a documented scorecard, explanation fields, monitoring dashboards, and an appeal process. Performance is reviewed monthly. When economic conditions change and delinquency prevalence rises, the threshold and calibration are reassessed rather than silently allowing the original policy to continue.

Learning Points

The most accurate model is not necessarily the most suitable model. Temporal validation is essential when predicting future events. Class imbalance requires decision-oriented metrics. Excluding leakage, examining fairness, and preserving human accountability are as important as selecting an algorithm.

Risks, Ethics, and Governance

Predictive models can reproduce historical discrimination when training labels reflect unequal access, biased enforcement, or past institutional decisions. Fairness cannot be reduced to one universal metric: demographic parity, equal opportunity, equalized odds, and calibration impose different requirements that may conflict. The appropriate analysis depends on the domain, legal framework, causal assumptions, and affected communities.

Privacy risks arise when organizations combine apparently harmless attributes to infer sensitive information. Data minimization, purpose limitation, secure storage, access control, retention policies, and consent or lawful basis are essential. Sensitive decisions should provide meaningful notice and, where appropriate, human review and an avenue for correction. Analysts must avoid presenting probabilistic predictions as certain facts.

Common mistakes include evaluating on training data, tuning on the test set, using accuracy for imbalanced classes, ignoring calibration, treating correlation as causation, deploying without monitoring, and assuming that feature importance proves fairness or causality. Automated decisions also create accountability questions: someone must own the model, investigate failures, and authorize changes.

Current Developments and Best Practice

Current predictive systems increasingly combine structured warehouse data with text, images, sensor streams, and foundation-model representations. Automated machine-learning tools can search preprocessing and algorithm configurations, but automation does not remove the need for sound target definition, leakage control, fairness assessment, and expert review. Explainable and responsible AI practices are becoming integrated into model risk management.

Effective practice includes the following principles:

• define the decision, target window, and error costs before training;
• establish a leakage-resistant validation design;
• create a simple, interpretable baseline;
• compare models using metrics connected to operational consequences;
• assess calibration, subgroup performance, robustness, and drift;
• document data, assumptions, features, thresholds, and limitations;
• use human oversight for consequential decisions; and
• monitor the deployed system and provide a controlled path for updating it.

Chapter Summary

Classification predicts discrete outcomes from labelled data and forms a bridge between data warehousing and operational decision-making. Decision trees and rule systems provide readable decision paths; logistic regression gives a strong probabilistic and interpretable baseline; Bayesian methods represent uncertainty and prior knowledge; k-nearest neighbours uses local similarity; support vector machines construct margin-based boundaries; ensembles improve predictive stability and accuracy; and neural networks learn complex representations. Each method involves trade-offs involving data requirements, computational cost, interpretability, and robustness.

Reliable predictive modelling depends on more than algorithm selection. Proper data preparation, leakage-resistant validation, suitable metrics, threshold selection, regularization, calibration, and monitoring determine whether a model generalizes. In professional settings, fairness, privacy, accountability, and explainability must be considered alongside predictive performance. The most appropriate model is therefore the one that supports a legitimate decision effectively, transparently, and sustainably.

Key Terms

Classification: Prediction of a discrete class or category.

Predictive modelling: Learning a model from historical data to estimate outcomes for new cases.

Feature: An input variable used by a predictive model.

Label: The observed target outcome associated with a training example.

Overfitting: Learning training-specific noise that reduces performance on unseen data.

Regularization: A technique that penalizes model complexity to improve generalization.

Decision tree: A hierarchical sequence of feature-based tests ending in predictions.

Entropy: An impurity measure based on the uncertainty of class membership.

Rule-based classifier: A classifier represented by IF–THEN decision rules.

Logistic regression: A model that estimates class probability through the logistic function.

Naïve Bayes: A Bayesian classifier assuming conditional independence of features given the class.

k-nearest neighbours: A local method that predicts from the labels of nearby observations.

Support vector machine: A margin-maximizing classifier that may use kernel functions.

Ensemble: A combination of multiple predictive models.

Random forest: An ensemble of randomized, bootstrapped decision trees.

Boosting: Sequential construction of models that focus on previous errors.

Precision: The proportion of predicted positive cases that are actually positive.

Recall: The proportion of actual positive cases correctly identified.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Data leakage: Use of information unavailable at prediction time or improperly shared across validation stages.

Review Questions

Short Answer Questions

1. Distinguish classification from regression.

2. What is the purpose of a validation set?

3. Define precision and recall.

4. Why must features be scaled for many k-nearest-neighbours applications?

5. What is data leakage, and why is it harmful?

Long Answer Questions

1. Explain how a decision tree is constructed and discuss pre-pruning and post-pruning.

2. Compare logistic regression, naïve Bayes, and support vector machines in terms of assumptions, interpretability, and applications.

3. Explain bagging, random forests, boosting, and stacking.

4. Discuss the bias–variance trade-off and methods for controlling overfitting.

5. Describe a complete, governance-aware workflow for deploying a classification model.

Analytical / Application Questions

1. A fraud classifier has 99.5% accuracy but detects only 20% of fraudulent transactions. Diagnose the evaluation problem and propose better measures and threshold decisions.

2. A bank must choose between a highly accurate boosted model and a slightly less accurate logistic model for customer credit intervention. Develop a decision framework for selecting the model.

3. A model performs well in cross-validation but poorly after deployment during an economic crisis. Analyze possible causes and recommend monitoring and remedial actions.

22. Multiple Choice Questions with Answers

Question 1

Which validation design is most appropriate when a model predicts future customer delinquency from historical records?

A. Randomly mix all records before splitting

B. Train on later records and test on earlier records

C. Train on earlier records and test on a later time period

D. Evaluate only on the training records

Correct Answer: C. Train on earlier records and test on a later time period

Question 2

A classifier identifies 90 of 100 predicted positive cases correctly, while 150 positive cases actually exist. What are its precision and recall?

A. Precision 90%, recall 60%

B. Precision 60%, recall 90%

C. Precision 90%, recall 40%

D. Precision 40%, recall 60%

Correct Answer: A. Precision 90%, recall 60%

Question 3

Which technique primarily reduces the variance of decision trees by averaging predictions from many bootstrapped trees?

A. Logistic regression

B. Bagging

C. Pruning a single tree

D. Naïve Bayes

Correct Answer: B. Bagging

Question 4

Why can an unrestricted decision tree perform poorly on unseen data?

A. It cannot model categorical variables

B. It always assumes linear relationships

C. It may memorize training-specific noise

D. It cannot produce class probabilities

Correct Answer: C. It may memorize training-specific noise

Question 5

In logistic regression, exponentiating a coefficient most directly gives the change in which quantity for a one-unit feature increase, holding other variables constant?

A. Odds ratio

B. Entropy

C. Euclidean distance

D. Gini impurity

Correct Answer: A. Odds ratio

Question 6

Which issue is especially important when applying k-nearest neighbours to features measured on very different scales?

A. Kernel selection

B. Distance domination by large-scale variables

C. Prior probability estimation

D. Tree branch pruning

Correct Answer: B. Distance domination by large-scale variables

Question 7

What is the defining objective of a basic support vector machine?

A. Maximize the number of model parameters

B. Find the hyperplane with the largest class-separating margin

C. Assume all features are conditionally independent

D. Use only the nearest single observation

Correct Answer: B. Find the hyperplane with the largest class-separating margin

Question 8

A spam classifier uses word frequencies and assumes that words are conditionally independent given the class. Which method is being used?

A. Naïve Bayes

B. Random forest

C. k-nearest neighbours

D. Multilayer perceptron

Correct Answer: A. Naïve Bayes

Question 9

Why is accuracy often unsuitable as the sole metric for rare-event fraud detection?

A. Accuracy cannot be computed from a confusion matrix

B. A classifier can achieve high accuracy by ignoring the rare positive class

C. Accuracy measures only calibration

D. Accuracy requires continuous targets

Correct Answer: B. A classifier can achieve high accuracy by ignoring the rare positive class

Question 10

Which practice best prevents target leakage during feature engineering?

A. Use information generated after the prediction date

B. Compute preprocessing quantities using the complete data set before splitting

C. Restrict features to information available at prediction time and fit transformations within training folds

D. Select features using the test-set outcomes

Correct Answer: C. Restrict features to information available at prediction time and fit transformations within training folds