Supervised Learning: Classification

Introduction and Context

Classification is a supervised learning task in which a model learns from labelled examples and assigns new observations to one of two or more predefined categories. Examples include identifying a fraudulent transaction, classifying an email as spam or legitimate, diagnosing a disease from clinical measurements, and recognizing an image as belonging to a particular object class.

In supervised learning, each training observation consists of input features and a known target label. If the feature vector for observation i is xᵢ and its class label is yᵢ, the learning problem is to estimate a function that maps features to a class prediction:

Here, X is the feature space and Y is the set of possible classes. A classification model may produce either a discrete class prediction or a probability for each class. Probabilities are especially useful when decisions involve risk, cost, or adjustable thresholds.

Classification differs from regression, whose target is numerical and usually continuous. However, both tasks involve selecting an appropriate representation, fitting a model, validating its generalization, and evaluating performance on unseen data.

Learning Objectives

After studying this chapter, you should be able to:

• explain binary, multiclass, and multilabel classification;
• prepare data for classification and avoid information leakage;
• apply logistic regression, k-nearest neighbors, decision trees, random forests, and support vector machines;
• interpret decision boundaries and model assumptions;
• construct and interpret a confusion matrix;
• calculate accuracy, precision, recall, F1-score, and ROC-AUC;
• select classification thresholds according to practical costs;
• compare models using suitable validation procedures; and
• identify limitations, ethical risks, and deployment considerations.

Classification Problems and Terminology

A binary classification problem has two classes, such as defective and non-defective. A multiclass problem has more than two mutually exclusive classes, such as low, medium, and high risk. In multilabel classification, one observation may belong to several classes simultaneously; a news article, for example, may be tagged as both technology and business.

Important terms include:

• Feature: an input variable used by the model.
• Target or label: the class to be predicted.
• Training set: data used to estimate model parameters.
• Validation set: data used to select models or hyperparameters.
• Test set: held-out data used for final performance estimation.
• Decision boundary: the boundary separating regions assigned to different classes.
• Hyperparameter: a setting selected before or during training, such as k in k-nearest neighbors.
• Class imbalance: a substantial difference in the number of observations per class.
• Probability threshold: a chosen probability above which an observation is assigned to a class, commonly 0.5 for binary classification but not necessarily optimal.

A good workflow begins with a clearly defined target and an operational meaning for errors. In medical screening, a false negative may be more harmful than a false positive. In an automated advertising system, the relative costs may be reversed.

Preparing Data for Classification

Before fitting a model, inspect missing values, outliers, duplicated records, class frequencies, and the measurement scales of features. Categorical variables generally require encoding, while numerical variables may need scaling.

Scaling is particularly important for distance-based and margin-based methods. A common standardization is

where x is an observed feature value, μ is the training-set mean, and σ is the training-set standard deviation. The transformation is applied using parameters calculated from the training data only. Applying test-set information during preprocessing causes data leakage and produces an overly optimistic evaluation.

A typical split is training data for fitting, validation data or cross-validation for selection, and a final test set for unbiased assessment. For imbalanced classification, use stratified splitting so that class proportions are approximately preserved.

Logistic Regression

Model and Interpretation

Logistic regression is a linear classification model that estimates the probability of a binary outcome. It models the log-odds as a linear function of the features:

Here, p is the probability of the positive class, β₀ is the intercept, βⱼ is the coefficient for feature xⱼ, and p in the subscript is the number of features. Solving for the probability gives the sigmoid function:

The output is between 0 and 1 and can therefore be interpreted as an estimated probability, subject to model calibration and data quality. With a threshold of 0.5, the positive class is selected when p ≥ 0.5. Because the sigmoid equals 0.5 when z = 0, the default decision boundary is the hyperplane

The boundary is linear in the original feature space. A coefficient βⱼ represents the change in log-odds for a one-unit increase in xⱼ while other features remain constant. The odds multiplier is e^βⱼ. This interpretation assumes the feature coding and linearity of log-odds are appropriate.

Training Objective

Parameters are commonly estimated by minimizing binary cross-entropy (log loss):

Here, n is the number of training examples, yᵢ is 0 or 1, and pᵢ is the predicted probability of class 1. The logarithm strongly penalizes confident incorrect predictions. Regularization may be added to reduce overfitting. L1 regularization can encourage coefficients to become exactly zero; L2 regularization shrinks coefficients toward zero.

Logistic regression is attractive because it is fast, interpretable, and often effective when the classes are approximately linearly separable in the chosen feature representation. It may perform poorly when important relationships are nonlinear or involve complex interactions not represented in the features.

k-Nearest Neighbors

k-nearest neighbors (k-NN) classifies a new observation using the labels of the k most similar training observations. For a numerical feature vector, Euclidean distance is commonly used:

The notation compares a new vector x with training vector xᵢ. The majority class among the k nearest observations becomes the prediction. A probability can be estimated as the fraction of neighbors belonging to each class.

Small values of k create flexible boundaries but can be sensitive to noise. Large values produce smoother boundaries but may obscure local patterns. The best value is usually selected by validation. Features must be scaled when their units differ, because an unscaled high-magnitude feature can dominate distance calculations.

k-NN has little explicit training cost but can be computationally expensive at prediction time and requires storing the training data. It is less effective in very high-dimensional spaces, where distances become less informative—a challenge known as the curse of dimensionality.

Decision Trees

A decision tree recursively divides the feature space using questions such as “Is age ≤ 40?” Each terminal node, or leaf, receives a class prediction, often based on the majority class in that leaf. A tree therefore produces a series of axis-aligned decision boundaries.

A common impurity measure is the Gini impurity:

where K is the number of classes and pₖ is the proportion of observations from class k in a node. Gini impurity is zero when a node contains only one class. Entropy is another measure:

A split is useful when it reduces impurity. If a parent node has impurity I and its child nodes have impurities Iₘ with weights wₘ, the weighted impurity after splitting is

The reduction I − I_split is the impurity decrease. Tree-growing algorithms choose splits that produce large decreases, subject to constraints such as maximum depth, minimum samples per leaf, or minimum impurity decrease.

Trees handle nonlinear relationships, interactions, and mixed feature types well. They do not require feature scaling and are relatively easy to explain. An unconstrained tree can overfit by creating leaves that describe noise. Pruning or restricting complexity improves generalization. Tree probabilities may be poorly calibrated, especially when leaves contain few observations.

Random Forests

A random forest combines many decision trees. Each tree is trained on a bootstrap sample of the training data, and each split considers a randomly selected subset of features. For classification, the forest usually predicts the class receiving the most votes; class probabilities may be estimated from the proportion of tree votes.

Random feature selection reduces correlation among trees. Averaging many diverse, moderately accurate trees generally reduces variance compared with a single tree. Important hyperparameters include the number of trees, maximum depth, minimum leaf size, and number of candidate features per split.

Random forests often provide strong performance with limited preprocessing and can model nonlinear interactions. However, they are less transparent than a single tree, may be memory-intensive, and their probability estimates should be checked if decisions depend on accurate risk probabilities. Feature importance measures can be misleading when predictors have different scales, cardinalities, or correlations. Permutation importance and partial dependence plots can support interpretation, but neither proves causation.

Support Vector Machines

A support vector machine (SVM) seeks a decision boundary with a large margin between classes. For a linearly separable binary problem, a hyperplane can be written as

where w is a vector of feature weights and b is the intercept. The geometric margin is related to 1 ÷ ‖w‖; maximizing the margin tends to improve robustness.

Real data are rarely perfectly separable. A soft-margin SVM permits violations and balances margin width against classification errors. The parameter C controls this trade-off: a large C penalizes violations strongly and may fit training data closely, whereas a small C allows more violations and stronger regularization.

Nonlinear boundaries can be obtained through kernels. The radial basis function (RBF) kernel is commonly expressed as

where γ controls how rapidly similarity decreases with distance. Large γ produces highly local influence and potentially complex boundaries; small γ produces smoother boundaries. SVMs usually require feature scaling and careful hyperparameter tuning. They can be powerful for medium-sized datasets but become less convenient for very large datasets. Standard SVM outputs are not automatically calibrated probabilities; calibration may be applied when probabilities are required.

Comparing Decision Boundaries

The geometry of a model helps explain its behaviour:

No model is universally best. The appropriate choice depends on data size, feature type, interpretability needs, latency, calibration requirements, and the cost of errors.

Confusion Matrices and Classification Metrics

For binary classification, predictions can be summarized as follows:

The total number of observations is N = TP + FP + FN + TN. The principal metrics are:

Accuracy is the proportion correctly classified. It can be misleading when one class is rare.

Precision answers: among predicted positives, how many are truly positive?

Recall, also called sensitivity or true-positive rate, answers: among actual positives, how many were found?

Specificity is the true-negative rate.

F1 is the harmonic mean of precision and recall. It is defined when the denominator is nonzero and gives a balanced summary when both types of performance matter. It does not include true negatives and therefore should not be treated as a complete measure in every application.

Worked Example

Suppose a screening model produces TP = 36, FP = 9, FN = 4, and TN = 51. Then N = 100.

Accuracy = (36 + 51) ÷ 100 = 0.87, or 87%.

Precision = 36 ÷ (36 + 9) = 36 ÷ 45 = 0.80, or 80%.

Recall = 36 ÷ (36 + 4) = 36 ÷ 40 = 0.90, or 90%.

F1 = 2 × (0.80 × 0.90) ÷ (0.80 + 0.90) ≈ 0.847.

The model detects most actual positives, but one in five positive predictions is false. Whether this is acceptable depends on the cost of follow-up investigations and missed cases.

ROC-AUC and Threshold Selection

A classifier that produces scores or probabilities can use different thresholds. Lowering the threshold generally increases recall and false-positive rate; raising it generally decreases them, although the exact changes depend on the data.

The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate across thresholds:

The area under this curve, ROC-AUC, measures ranking ability across all possible thresholds. An AUC of 0.5 corresponds, in expectation, to random ranking, while 1.0 represents perfect ranking. AUC is threshold-independent, but it does not indicate whether predicted probabilities are calibrated, nor does it directly encode the cost of errors. With highly imbalanced classes, a precision–recall curve may be more informative.

Threshold selection should reflect consequences. If the cost of a false negative is C_FN and the cost of a false positive is C_FP, a simple expected-cost decision rule predicts positive when the expected cost of doing so is lower than predicting negative. If predicted probabilities are calibrated, equal action costs, and the only costs are these two error costs, the positive decision threshold can be expressed as

This formula assumes the costs are nonnegative, the class probability is calibrated, and there are no additional benefits, intervention costs, capacity constraints, or changing base rates. For example, if C_FP = 1 and C_FN = 4, then t = 1 ÷ 5 = 0.20. A lower threshold is justified because missed positives are more costly. In practice, select the threshold on validation data and report its effects on the confusion matrix and relevant metrics.

Applied Case: Predicting Customer Loan Default

Background and Problem

This illustrative case concerns a financial institution that wants to identify applicants at high risk of default. Features include income, debt-to-income ratio, repayment history, account age, and recent missed payments. The target is whether a customer defaults within twelve months.

A careless system might optimize accuracy. If only 5% of customers default, a model predicting “no default” for everyone achieves 95% accuracy but has zero recall for default cases. This demonstrates why evaluation must begin with the business and social consequences of errors.

Analysis and Approach

The institution first defines the target observation period and removes information unavailable at application time. It separates training, validation, and test data chronologically to reduce leakage from future information. Numerical variables are standardized for logistic regression, k-NN, and SVM. The tree-based models do not require scaling, although a consistent preprocessing pipeline is still useful.

The team trains logistic regression as an interpretable baseline, a pruned decision tree, a random forest, k-NN, and an RBF-kernel SVM. Hyperparameters are selected using stratified cross-validation on the training data. The test set is used only once for final comparison.

Solution and Threshold Decision

Because failing to identify a likely defaulter is costly, the team does not automatically use a 0.5 threshold. It examines precision–recall values at several thresholds and estimates the operational cost of manual review. A threshold of 0.25 increases recall sufficiently to meet the risk department’s requirement, while keeping the review workload within capacity.

The final report includes the confusion matrix, recall, precision, F1-score, ROC-AUC, calibration information, and performance across demographic groups. The model is not used as an automatic rejection mechanism. Instead, it supports trained staff, who can review additional evidence and provide an appeal route.

Learning Points

The case illustrates five principles: accuracy alone is inadequate under imbalance; leakage can arise from features recorded after the prediction time; thresholds are decision-policy choices rather than fixed truths; probability calibration matters when scores guide resource allocation; and fairness must be evaluated alongside aggregate performance.

Validation, Generalization, and Best Practice

Use pipelines that combine preprocessing and modelling so that transformations are fitted separately within each training fold. Use stratified cross-validation for classification, and use grouped or time-based splits when observations from the same person or future periods would otherwise appear in both training and validation data.

Compare models using confidence intervals or repeated validation where feasible, not merely a single score. Keep a final test set untouched until model selection is complete. Examine performance by class and relevant subgroups. Check calibration with reliability diagrams or calibration metrics when probabilities drive decisions. Monitor performance after deployment because data distributions, user behaviour, and class prevalence can change.

Challenges, Ethics, and Governance

Classification systems can reproduce historical discrimination if labels reflect unequal treatment or if features act as proxies for protected characteristics. Removing a sensitive feature does not guarantee fairness. Assess data provenance, label quality, subgroup error rates, and the consequences of unequal false-positive or false-negative rates.

Important governance practices include documenting the intended use, limitations, training data, evaluation protocol, threshold policy, and responsible owner. Protect personal data through data minimization, access controls, retention limits, and appropriate anonymization. Provide explanations suited to the audience, but avoid claiming that feature importance proves causation. In high-impact contexts, human oversight, contestability, audit trails, and regulatory compliance are essential.

Contemporary practice increasingly combines predictive performance with uncertainty estimation, model monitoring, fairness assessment, and responsible machine-learning operations. Automated retraining should not be permitted to change decisions without validation and approval controls.

Chapter Summary

Classification learns a mapping from features to discrete classes or class probabilities. Logistic regression provides an interpretable linear probability model through log-odds; k-NN uses local similarity; decision trees partition feature space with interpretable rules; random forests reduce variance by aggregating randomized trees; and SVMs seek large-margin boundaries, with kernels enabling nonlinear separation. Each method has assumptions and operational trade-offs.

Confusion matrices provide the foundation for accuracy, precision, recall, specificity, and F1-score. ROC-AUC evaluates ranking over thresholds, while precision–recall analysis is often especially useful for rare positive classes. Thresholds should be selected using validated performance, calibrated probabilities where appropriate, and the real costs of errors. Reliable classification also requires careful splitting, leakage prevention, appropriate preprocessing, subgroup evaluation, documentation, privacy protection, and post-deployment monitoring.

Key Terms

Classification: Predicting a discrete class or category.

Binary classification: Classification with two possible classes.

Multiclass classification: Classification with more than two mutually exclusive classes.

Feature: An input variable used for prediction.

Label: The known target class associated with an observation.

Decision boundary: A surface separating regions assigned to different classes.

Logistic regression: A model that represents binary log-odds as a linear function of features.

Sigmoid function: A function mapping real-valued scores to values between 0 and 1.

k-nearest neighbors: A method that predicts using labels of nearby training observations.

Decision tree: A recursive set of feature-based decision rules.

Random forest: An ensemble of randomized decision trees.

Support vector machine: A margin-based classifier that can use kernel functions.

Hyperparameter: A model setting selected outside the fitted parameters.

Confusion matrix: A table of correct and incorrect classification outcomes.

Precision: The proportion of predicted positives that are actual positives.

Recall: The proportion of actual positives correctly identified.

F1-score: The harmonic mean of precision and recall.

ROC-AUC: The area under the receiver operating characteristic curve.

Threshold: A score or probability cutoff used to make a class decision.

Data leakage: Use of information during training that would not be available at prediction time.

Review Questions

5 Short Answer Questions

1. What is the difference between binary and multiclass classification?
2. Why should features be scaled before applying k-nearest neighbors?
3. What does recall measure in a binary confusion matrix?
4. State one advantage and one limitation of random forests.
5. Why can accuracy be misleading for an imbalanced dataset?

5 Long Answer Questions

1. Explain logistic regression, including the sigmoid function, log-odds, and the meaning of a linear decision boundary.
2. Compare decision trees, random forests, and support vector machines in terms of boundaries, preprocessing, interpretability, and overfitting.
3. Explain how a confusion matrix supports the calculation and interpretation of precision, recall, specificity, accuracy, and F1-score.
4. Describe a sound training, validation, and testing workflow for a classification project.
5. Discuss why threshold selection should be treated as a decision problem rather than an automatic use of 0.5.

3 Analytical / Application Questions

1. A fraud detector has very few fraudulent cases. Which metrics and validation strategy would you use, and why?
2. A medical screening model has high recall but low precision. Explain the practical consequences and propose two ways to improve the system or decision process.
3. Design a classification approach for loan default prediction. Specify preprocessing, at least two candidate models, evaluation measures, threshold considerations, and ethical safeguards.

Multiple Choice Questions with Answers

Question 1

Which model directly represents binary log-odds as a linear combination of features?

A. Logistic regression

B. k-nearest neighbors

C. Random forest

D. Decision tree

Correct Answer: A. Logistic regression

Question 2

A classifier has TP = 30, FP = 10, FN = 5, and TN = 55. What is its recall?

A. 0.60

B. 0.75

C. 0.857

D. 0.90

Correct Answer: C. 0.857

Question 3

Why is feature scaling especially important for k-nearest neighbors?

A. It removes all missing values

B. It prevents a large-unit feature from dominating distance

C. It converts a nonlinear boundary into a linear one

D. It guarantees perfect accuracy

Correct Answer: B. It prevents a large-unit feature from dominating distance

Question 4

What does a random forest primarily achieve by averaging many diverse trees?

A. It eliminates the need for labels

B. It reduces variance compared with a single tree

C. It guarantees calibrated probabilities

D. It makes every boundary linear

Correct Answer: B. It reduces variance compared with a single tree

Question 5

For a binary classifier, what does precision measure?

A. The proportion of actual positives that were found

B. The proportion of all observations classified correctly

C. The proportion of predicted positives that are truly positive

D. The proportion of actual negatives correctly rejected

Correct Answer: C. The proportion of predicted positives that are truly positive

Question 6

If false negatives are much more costly than false positives, which threshold change will generally increase recall?

A. Lower the probability threshold

B. Raise the probability threshold

C. Remove the validation set

D. Use accuracy as the threshold

Correct Answer: A. Lower the probability threshold

Question 7

Which model seeks a decision boundary with a large margin between classes?

A. k-nearest neighbors

B. Support vector machine

C. Naive majority classifier

D. Standardization

Correct Answer: B. Support vector machine

Question 8

What is a major danger of using information from the test set during preprocessing?

A. It causes data leakage and optimistic evaluation

B. It always reduces the number of features to zero

C. It guarantees underfitting

D. It changes a binary task into regression

Correct Answer: A. It causes data leakage and optimistic evaluation

Question 9

For calibrated probabilities with false-positive cost 1 and false-negative cost 4, the simple cost-based threshold is what value?

A. 0.10

B. 0.20

C. 0.40

D. 0.80

Correct Answer: B. 0.20

Question 10

What does ROC-AUC primarily evaluate?

A. Ranking ability across classification thresholds

B. The number of features in the dataset

C. Whether probabilities are perfectly calibrated

D. The training time of the model

Correct Answer: A. Ranking ability across classification thresholds