Classification Models

Introduction

Classification is a supervised learning task in which a model learns from labelled examples and assigns one of a finite set of classes to a new observation. A class may represent a category such as fraudulent or legitimate, diseased or healthy, approved or rejected, or one of several species. Classification is central to data science because many organisational decisions are categorical even when the underlying data are numerical, textual, temporal, or visual.

A classification solution has two related outputs. The first is a predicted class, such as “default”. The second is often a score or estimated probability expressing the model’s confidence. These outputs must not be confused: a probability is an estimate conditional on the model, data, and assumptions; it is not automatically a guarantee that the prediction is correct.

This chapter develops five foundational classifiers: logistic regression, k-nearest neighbours, naïve Bayes, decision trees, and support vector machines (SVMs). They represent different approaches to classification. Logistic regression models class probability directly; k-nearest neighbours uses local similarity; naïve Bayes applies Bayes’ theorem with a conditional-independence assumption; decision trees partition the feature space through rules; and SVMs seek a boundary with a large margin.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish binary, multiclass, and multilabel classification;
• explain how the five major classification models produce predictions;
• interpret probabilities, decision boundaries, margins, and class labels;
• select suitable evaluation metrics and decision thresholds;
• implement a reliable classification workflow and avoid data leakage;
• compare model strengths, limitations, interpretability, and computational requirements;
• analyse an applied classification problem and communicate its risks responsibly.

Foundations of Classification

Let an observation be represented by a feature vector x = (x₁, x₂, …, x_p), where p is the number of predictors. In supervised learning, the training data are pairs (x_i, y_i), where y_i is the known target for observation i. For binary classification, y_i ∈ {0,1}. A classifier learns a function that maps features to a class or a score:

Here, ŷ is the predicted class. A probabilistic classifier may instead estimate P(Y = 1 | X = x). A threshold converts that probability into a class prediction. With threshold t, the common rule is:

The default threshold is often 0.5, but it should reflect the costs of errors, class prevalence, and operational requirements.

Types of classification

• Binary classification: exactly two classes, such as positive and negative.
• Multiclass classification: one of more than two mutually exclusive classes, such as low, medium, or high risk.
• Multilabel classification: an observation may belong to several classes simultaneously, such as a document tagged “finance” and “regulation”.
• Ordinal classification: classes have an order, such as poor, satisfactory, and excellent. Treating ordinal classes as ordinary nominal classes may discard useful information.

A decision boundary is the set of feature values at which the predicted class changes. A linear boundary in two dimensions can be written as:

where w₀ is an intercept and w₁, w₂ are feature weights. A model may have a linear boundary in the original feature space but a nonlinear boundary after transformations or feature expansion.

Confusion matrix and error types

For binary classification, predictions can be summarised as follows:

Important metrics are:

Accuracy is the proportion of all predictions that are correct. Precision answers, “Among predicted positives, how many are positive?” Recall, also called sensitivity or true-positive rate, answers, “Among actual positives, how many were detected?” Specificity measures the true-negative rate. These formulas assume the relevant denominators are non-zero.

The F1 score is the harmonic mean of precision and recall:

It is useful when both types of positive-class performance matter, but it does not incorporate true negatives and should not be treated as a universal measure. For imbalanced data, accuracy can be misleading. A classifier that labels every transaction as legitimate may achieve high accuracy while detecting no fraud.

Logistic Regression

Despite its name, logistic regression is primarily a classification model. It begins with a linear score:

The score is transformed into a probability using the logistic, or sigmoid, function:

The output lies strictly between 0 and 1. The inverse transformation is the logit:

Thus, logistic regression assumes that the log-odds of the positive class are a linear function of the predictors. It does not assume that the probability itself changes linearly with every feature.

Interpretation

For a one-unit increase in x_j, holding other variables constant, β_j is the change in log-odds. The quantity e^(β_j) is the corresponding odds ratio. An odds ratio above 1 indicates increased odds, while one below 1 indicates decreased odds. Interpretation requires attention to units, coding, interactions, and the difference between odds and probability.

Worked example. Suppose a fitted model gives z = −2 + 0.8x, where x = 3. Then z = −2 + 0.8(3) = 0.4. Therefore:

The estimated positive-class probability is approximately 0.599. With threshold 0.5, the predicted class is 1; with threshold 0.7, it is 0. The model has not changed—only the decision rule has changed.

Logistic regression parameters are commonly estimated by maximum likelihood rather than ordinary least squares. Regularisation is often added to reduce overfitting. L2 regularisation penalises large squared coefficients, while L1 regularisation penalises absolute coefficient values and can produce exact zero coefficients. The strength of regularisation must be selected using validation data, not the test set.

Logistic regression is suitable when a reasonably interpretable, efficient baseline is needed and the relationship is approximately linear on the log-odds scale. It may underperform when boundaries are strongly nonlinear unless appropriate transformations, interactions, or engineered features are added.

k-Nearest Neighbours

k-nearest neighbours (k-NN) is an instance-based method. To classify a new point, it finds the k training observations closest to it and assigns the majority class among those neighbours. A common distance is Euclidean distance:

This formula is appropriate for numerical features measured on comparable scales. If feature scales differ substantially, a high-range feature can dominate the distance. Standardisation commonly transforms a feature using:

where μ and σ are the training-set mean and standard deviation. The same training-set transformation must be applied to validation, test, and future data.

For binary classification, the proportion of positive neighbours can serve as a local probability estimate:

This is a neighbourhood proportion, not necessarily a calibrated probability.

A small k produces flexible, irregular boundaries and is sensitive to noise. A large k produces smoother boundaries but may overlook local structure. The value of k should be selected through cross-validation. Distance weighting can give closer neighbours more influence. k-NN is simple and effective for modest, well-scaled datasets, but prediction can be slow because distances to many training records may be computed at prediction time. It also suffers in high dimensions, where observations tend to become similarly distant; this is part of the curse of dimensionality.

Naïve Bayes

Bayes’ theorem relates posterior and prior probabilities:

Here, c is a class, P(Y = c) is the prior probability, P(x | Y = c) is the likelihood, and P(x) is the evidence. Since P(x) is the same for every candidate class, classification can compare:

Naïve Bayes assumes conditional independence of features given the class:

The assumption is often unrealistic—for example, words in a document may be related—but the method can still classify effectively. Variants match feature types: Gaussian naïve Bayes for continuous features, multinomial naïve Bayes for count data such as word frequencies, and Bernoulli naïve Bayes for binary features.

A zero count can make a product of probabilities zero. Laplace smoothing avoids this for categorical or count models. With smoothing parameter α > 0, estimated category probabilities commonly take the form:

where N_jvc is the count for value v in class c, N_jc is the relevant class count, and K is the number of possible values for that feature. Implementations often use log probabilities to prevent numerical underflow when many small probabilities are multiplied.

Naïve Bayes trains quickly, works well with sparse text data, and can perform well with limited training data. Its probability estimates may be poorly calibrated, especially when features are correlated. Calibration methods such as Platt scaling or isotonic regression can be evaluated when reliable probabilities are operationally important.

Decision Trees

A decision tree recursively splits data using questions such as “Is income greater than a threshold?” Each leaf produces a class prediction, often the majority class, and may report the class proportions in that leaf as estimated probabilities.

A common split criterion is Gini impurity:

where S is a node, C is the number of classes, and p_c is the proportion of observations in node S belonging to class c. Impurity is zero when a node contains one class only. Entropy is another criterion:

By convention, a term with p_c = 0 contributes zero. A split is attractive when it reduces weighted impurity. For a parent node split into child nodes S₁, …, S_m, the weighted impurity is:

The split with the greatest reduction in impurity is selected, subject to stopping rules or pruning.

Trees capture nonlinear relationships and interactions without requiring feature scaling. Their rules are easy to communicate, but a fully grown tree can overfit. Maximum depth, minimum samples per leaf, minimum samples for splitting, and pruning control complexity. A single tree can be unstable: small changes in training data may produce a different structure. Ensembles such as random forests and gradient-boosted trees improve predictive performance, but they reduce the simplicity of a single-tree explanation.

Tree probabilities are empirical class proportions in leaves and may be poorly calibrated, particularly in small leaves. A probability of 0.8 from a leaf means that approximately 80% of training observations reaching that leaf belonged to the class; it does not establish an 80% universal frequency without suitable validation.

Support Vector Machines

For two classes coded as y_i ∈ {−1,+1}, a linear SVM seeks a separating hyperplane:

The geometric margin is related to 1 / ||w||. In the soft-margin formulation, the optimisation problem is:

Here, ξ_i are slack variables allowing violations, C > 0 controls the trade-off between a wide margin and training violations, and n is the number of observations. A large C penalises violations strongly and may create a narrower, more complex boundary; a small C allows more violations and emphasises a wider margin.

Only observations near the boundary, called support vectors, directly determine the fitted boundary. SVMs can model nonlinear boundaries through a kernel. The radial basis function kernel is commonly written:

where γ > 0 controls how rapidly similarity decreases with distance. A large γ produces local, flexible influence; a small value produces smoother influence. Both C and γ should be tuned with cross-validation.

SVMs are sensitive to feature scale, so numerical features should generally be standardised. The raw SVM decision score is not automatically a probability. If probabilities are required, a calibration procedure must be fitted and validated. SVMs can be powerful in medium-sized, high-dimensional spaces, but kernel training may become computationally expensive for very large datasets, and the resulting model may be difficult to explain to nontechnical stakeholders.

Comparing the Models

No model is universally best. Model selection should consider predictive performance, latency, interpretability, data volume, missing values, class imbalance, probability requirements, and the consequences of errors.

A Reliable Classification Workflow

A professional workflow separates model development from final evaluation.

1. Define the decision and target. Specify the positive class, prediction time, acceptable errors, and whether probabilities or labels are required.
2. Inspect the data. Check class frequencies, missingness, duplicates, outliers, leakage, and feature availability at prediction time.
3. Create a split. Use training, validation or cross-validation, and a final test set. For temporal data, use a time-ordered split rather than random shuffling. Stratification can preserve class proportions when appropriate.
4. Build a preprocessing pipeline. Fit imputation, scaling, encoding, and feature selection only on training folds. Applying information from the full dataset before splitting causes leakage.
5. Establish a baseline. Compare models with a simple majority-class or domain baseline and a regularised logistic regression baseline.
6. Tune using cross-validation. Select hyperparameters such as k, tree depth, regularisation strength, C, or γ using a metric aligned with the problem.
7. Evaluate once on held-out data. Report a confusion matrix and suitable metrics with uncertainty where possible. Do not repeatedly optimise against the test set.
8. Check calibration and threshold. A threshold should reflect costs and capacity, not merely convention.
9. Analyse subgroups and stability. Examine performance across relevant populations, time periods, and operating conditions.
10. Monitor after deployment. Track drift in features, class prevalence, calibration, error rates, and data quality.

Threshold selection and costs

Suppose a false positive costs C_FP and a false negative costs C_FN. If the predicted probability of a positive class is p, and the decision is made between positive and negative actions, a positive action is preferred when its expected cost is lower. Under a simplified equal-benefit setting, the threshold is:

This expression assumes calibrated probabilities, mutually exclusive actions, and that correct decisions have zero cost while the stated error costs are the relevant losses. It should not be applied blindly when costs, benefits, or operational constraints are more complex. For example, if a missed disease case is much more costly than an unnecessary follow-up, a lower threshold may be justified.

Applied Case: Illustrative Loan-Default Screening

Background and problem

A lender wants to identify applications requiring additional review. The target is default within twelve months. Available predictors include debt-to-income ratio, previous repayment history, income band, loan amount, and employment duration. This is an illustrative case, not a claim about any particular lender.

Analysis

The team first defines default as the positive class and confirms that every feature would have been available at application time. A later payment-status field is removed because it would leak the outcome. The data are split chronologically so that earlier applications train the model and later applications provide a realistic test. Numerical features are imputed and standardised within each training fold; categorical features are encoded within the same pipeline.

Because defaults are less common than non-defaults, accuracy is not selected as the only measure. Recall is important because missed defaults create financial risk, while precision matters because manual reviews are limited. The team also evaluates the precision–recall curve, calibration, and subgroup performance.

Approach and solution

Regularised logistic regression provides an interpretable baseline. A decision tree supplies understandable rules, while an SVM and k-NN are considered as alternatives. Naïve Bayes is tested but is less attractive if the predictors are strongly dependent. Hyperparameters are selected using stratified cross-validation within the training period. The final threshold is chosen using the cost of review, the estimated cost of missed defaults, and review capacity rather than automatically using 0.5.

Suppose a test confusion matrix is TP = 180, FP = 120, FN = 60, and TN = 1,640. Then:

The model correctly classifies 91.0% of cases, identifies 75.0% of observed defaults, and is correct on 60.0% of applications it flags. The precision result indicates that many flagged applications will not default, but that may be acceptable if review is relatively inexpensive.

Governance and outcome

Before deployment, the lender validates probability calibration, documents the feature pipeline, tests performance across legally relevant groups, and provides an appeal or human-review process. Protected characteristics should not be used improperly, and removing them does not guarantee fairness because correlated variables may act as proxies. The model should support consistent review rather than silently determine a person’s access to credit. Performance and data drift should be monitored after deployment, with a documented process for retraining and rollback.

Learning points

The case illustrates that a technically accurate classifier may still be operationally unsuitable. Target definition, temporal validity, leakage prevention, threshold choice, calibration, fairness, explainability, and monitoring are part of the classification solution—not afterthoughts.

Challenges, Ethics, and Contemporary Practice

Classification systems can reproduce historical discrimination when labels reflect unequal treatment or when features encode structural disadvantage. Important controls include documenting data provenance, testing subgroup error rates, assessing calibration by group where appropriate, minimising unnecessary sensitive data, and providing meaningful explanations and contestability. Privacy principles require collecting and retaining only data justified by a legitimate purpose, securing them appropriately, and respecting applicable law.

Data drift occurs when feature distributions change; concept drift occurs when the relationship between features and outcomes changes. A model trained on one economic period may become unreliable in another. Monitoring should therefore include input distributions, missingness, prediction distributions, delayed labels, calibration, and subgroup metrics.

Contemporary practice increasingly combines interpretable baselines with ensembles, automated hyperparameter search, probability calibration, and model cards or similar documentation. Explainability tools can describe local or global patterns, but an explanation of model behaviour is not necessarily a causal explanation. Machine learning identifies predictive associations; it does not, by itself, establish that changing a feature will change the outcome.

Best Practices

• Define the target, prediction horizon, and decision costs before modelling.
• Use leakage-resistant temporal or stratified evaluation appropriate to the problem.
• Put preprocessing and the classifier in one reproducible pipeline.
• Scale distance- and margin-based methods using training data only.
• Compare against simple baselines and report more than one metric.
• Distinguish scores, calibrated probabilities, and thresholded labels.
• Tune hyperparameters and thresholds without contaminating the final test set.
• Inspect errors, subgroup performance, calibration, and stability.
• Prefer the simplest model that meets performance and governance requirements.
• Monitor deployed systems and define retraining, review, and rollback procedures.

Chapter Summary

Classification assigns observations to categorical outcomes using labelled training data. A probability estimate and a class label are different outputs, and the threshold connecting them should reflect error costs and operational constraints. Logistic regression models log-odds linearly and offers interpretable coefficients. k-nearest neighbours predicts from local similarity and requires careful scaling and choice of k. Naïve Bayes uses Bayes’ theorem with a conditional-independence assumption and is particularly effective for many text problems. Decision trees create recursive, interpretable partitions but require complexity control. SVMs seek a maximum-margin boundary and can represent nonlinear relationships through kernels, although scaling and tuning are important.

Reliable classification requires appropriate data splitting, leakage-resistant preprocessing, cross-validation, metric selection, calibration, threshold analysis, subgroup assessment, and post-deployment monitoring. Ethical and professional practice recognises that predictive performance alone does not determine whether a classification system is suitable, fair, lawful, or safe.

Key Terms

Classification: Predicting one or more categorical outcomes from input features.

Binary classification: Classification involving two possible classes.

Multiclass classification: Classification involving more than two mutually exclusive classes.

Probability estimate: A model-based estimate of the likelihood of a class conditional on observed inputs.

Decision threshold: A score or probability cutoff used to convert an estimate into a class label.

Decision boundary: The region separating different predicted classes in feature space.

Confusion matrix: A table of true and false positive and negative predictions.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are detected.

Specificity: The proportion of actual negatives correctly identified.

Logistic regression: A model that represents binary log-odds as a function of predictors.

Odds ratio: The multiplicative change in odds associated with a one-unit predictor change, holding other variables constant.

k-nearest neighbours: A classifier based on the classes of the closest training observations.

Naïve Bayes: A probabilistic classifier using Bayes’ theorem and conditional feature independence.

Decision tree: A hierarchical sequence of feature-based splits ending in prediction leaves.

Gini impurity: A measure of class mixture within a tree node.

Support vector machine: A margin-based classifier determined principally by boundary-near observations.

Support vector: A training observation that influences the SVM boundary.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Data leakage: The improper use of information unavailable at prediction time or reserved for evaluation.

Review Questions

Short Answer Questions

1. What is the difference between a predicted probability and a predicted class?
2. Why should features usually be scaled before applying k-nearest neighbours?
3. State the conditional-independence assumption made by naïve Bayes.
4. What do precision and recall measure?
5. Why can a fully grown decision tree overfit?

Long Answer Questions

1. Explain logistic regression, including the sigmoid function, log-odds interpretation, thresholding, and regularisation.
2. Compare k-nearest neighbours, naïve Bayes, and decision trees in terms of assumptions, training, prediction, and limitations.
3. Explain the soft-margin SVM objective and the roles of the parameters C and γ.
4. Design a leakage-resistant workflow for training and evaluating a binary classifier on temporal business data.
5. Discuss why accuracy, calibration, fairness, and monitoring should all be considered when deploying a classifier.

Analytical / Application Questions

1. A medical screening model has high accuracy but low recall. Analyse why this may be unsafe and recommend evaluation and threshold changes.
2. A k-NN model performs well during training but poorly on a test set. Diagnose possible causes and propose at least three corrective actions.
3. In the loan-default case, explain how changing the decision threshold would affect TP, FP, FN, precision, recall, and the workload of human reviewers.

Multiple Choice Questions with Answers

Question 1

Which transformation converts a logistic-regression linear score into a value between 0 and 1?

A. The sigmoid function

B. The Euclidean distance

C. The Gini impurity

D. The hinge loss

Correct Answer: A. The sigmoid function

Question 2

A classifier produces TP = 80 and FN = 20. What is its recall?

A. 20%

B. 40%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 3

Why is feature scaling especially important for k-nearest neighbours?

A. It forces all classes to have equal size

B. It prevents large-unit features from dominating distance calculations

C. It removes the need to select k

D. It converts a nonlinear boundary into a linear one

Correct Answer: B. It prevents large-unit features from dominating distance calculations

Question 4

Which assumption is characteristic of naïve Bayes?

A. All classes have equal prior probability

B. The decision boundary must be linear

C. Features are conditionally independent given the class

D. Every feature must be normally distributed

Correct Answer: C. Features are conditionally independent given the class

Question 5

A decision-tree node contains observations from only one class. What is its Gini impurity?

A. 0

B. 0.25

C. 0.5

D. 1

Correct Answer: A. 0

Question 6

In a soft-margin SVM, increasing C generally does what?

A. Ignores all training violations

B. Penalises margin violations more strongly

C. Automatically standardises features

D. Guarantees calibrated probabilities

Correct Answer: B. Penalises margin violations more strongly

Question 7

A model flags 300 applications, of which 180 default. What is its precision?

A. 40%

B. 60%

C. 75%

D. 90%

Correct Answer: B. 60%

Question 8

Which practice is an example of data leakage?

A. Selecting a threshold using validation data

B. Fitting a scaler on the training fold and applying it to the validation fold

C. Using a post-outcome variable as a predictor at application time

D. Reporting a confusion matrix on a held-out test set

Correct Answer: C. Using a post-outcome variable as a predictor at application time

Question 9

What is the main effect of choosing a very small k in k-nearest neighbours?

A. A smoother boundary with lower variance

B. A more locally flexible and noise-sensitive boundary

C. Removal of the curse of dimensionality

D. Guaranteed probability calibration

Correct Answer: B. A more locally flexible and noise-sensitive boundary

Question 10

If false positives cost 1 unit and false negatives cost 4 units, the simplified cost-based threshold is t = 1/(1+4). What is t?

A. 0.05

B. 0.20

C. 0.40

D. 0.80

Correct Answer: B. 0.20