Classification Models

Introduction

Classification is the supervised learning task of assigning an observation to one of a finite set of classes. Examples include identifying fraudulent transactions, classifying medical tests as positive or negative, routing customer requests, and recognizing objects in images. A classifier may produce a class label, a score, or an estimated probability. These outputs are related but not interchangeable.

This chapter examines five foundational classifiers: logistic regression, k-nearest neighbours (KNN), naïve Bayes, decision trees, and support vector machines (SVMs). Each represents a different modelling principle. Logistic regression models log-odds; KNN relies on local similarity; naïve Bayes applies Bayes’ theorem with conditional-independence assumptions; decision trees partition the feature space; and SVMs maximize a geometric margin.

Model comparison must extend beyond accuracy. Probability quality, decision thresholds, computational cost, interpretability, class imbalance, data type, and error consequences all affect the appropriate choice.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain the mathematical principles of five major classification models.

2. Implement classifiers for numerical and categorical data.

3. Interpret probability estimates, decision scores, decision boundaries, and thresholds.

4. Compare classifiers using validation data and suitable performance measures.

5. Prevent data leakage by separating training, validation, and test functions.

6. Select thresholds according to asymmetric error costs and operational constraints.

7. Evaluate limitations, fairness risks, and governance requirements in classification systems.

1. Classification Foundations

Let an observation be represented by a feature vector x ∈ ℝᵖ and its class by y. In binary classification, y ∈ {0,1}. A classifier learns a function from features to either a predicted class or a score:

ŷ = f(x).

A probabilistic classifier estimates

p(x) = P(Y = 1 | X = x).

Given a threshold t, the probability is converted into a class decision:

ŷ = 1 if p(x) ≥ t, and ŷ = 0 otherwise.

The conventional threshold t = 0.5 is not universally optimal. It is appropriate only under particular assumptions about probability calibration, class priorities, and decision costs.

1.1 Decision boundaries

A decision boundary separates regions assigned to different classes. For a probability classifier using threshold t, the boundary consists of points satisfying

P(Y = 1 | X = x) = t.

A boundary may be linear, piecewise linear, curved, or highly irregular. Logistic regression without nonlinear feature transformations produces a linear boundary. Trees produce axis-aligned regions. KNN can generate irregular local boundaries, while kernel SVMs can represent nonlinear boundaries.

1.2 Scores, probabilities, and labels

A decision score preserves ranking information but need not be a probability. For example, an SVM’s signed distance-like decision value indicates which side of the separating surface an observation lies on. It does not automatically represent P(Y = 1 | X = x).

A probability must lie between zero and one and should ideally be calibrated. Calibration means that, among observations assigned probability 0.70, approximately 70% belong to the positive class over repeated comparable cases.

A class label contains less information than either a score or probability because it depends on a chosen threshold.

2. Logistic Regression

Despite its name, logistic regression is primarily a classification model. It models the conditional probability of a binary outcome through the logistic function.

2.1 Model formulation

For features x₁,…,xₚ, define the linear predictor

z = β₀ + β₁x₁ + ··· + βₚxₚ.

The positive-class probability is

P(Y = 1 | x) = σ(z) = 1/(1 + e⁻ᶻ).

The odds are p/(1 − p). Taking logarithms gives

log[p/(1 − p)] = β₀ + β₁x₁ + ··· + βₚxₚ.

Thus, a one-unit increase in xⱼ, holding other variables constant, multiplies the odds by e^{βⱼ}. This interpretation concerns odds, not probability. The probability change is not constant because the logistic curve is nonlinear.

2.2 Estimation and loss

For independent observations (xᵢ,yᵢ), with yᵢ ∈ {0,1}, the likelihood is

L(β) = ∏ᵢ pᵢ^{yᵢ}(1 − pᵢ)^{1−yᵢ}.

Maximizing this likelihood is equivalent to minimizing binary cross-entropy:

−∑ᵢ [yᵢ log(pᵢ) + (1 − yᵢ)log(1 − pᵢ)].

Regularization limits coefficient magnitude and helps control overfitting. L2 regularization adds λ∑ⱼβⱼ², whereas L1 regularization adds λ∑ⱼ|βⱼ| and may produce zero coefficients. The intercept is commonly excluded from the penalty, although implementation conventions vary.

2.3 Worked probability and boundary

Suppose

z = −0.4 + 0.8x.

For x = 2,

z = −0.4 + 0.8(2) = 1.2,

p = 1/(1 + e⁻¹·²) ≈ 0.7685.

At threshold 0.5, the observation is classified positive. The 0.5 boundary satisfies z = 0:

−0.4 + 0.8x = 0,

so x = 0.5.

For a general threshold t, the boundary satisfies

z = log[t/(1 − t)].

At t = 0.8, z = log(4) ≈ 1.3863, giving x ≈ 2.2329. Raising the threshold therefore reduces the set of observations classified positive.

2.4 Strengths and limitations

Logistic regression is computationally efficient, supports probability estimation, and offers interpretable coefficients. However, its basic boundary is linear. Interactions, transformations, or basis functions are needed when log-odds are nonlinear. Strong multicollinearity may destabilize unregularized coefficients, and coefficient interpretation can become difficult after extensive feature engineering.

For multiclass problems, common extensions are one-versus-rest classification and multinomial logistic regression. The latter uses the softmax function to produce class probabilities summing to one.

3. K-Nearest Neighbours

KNN is an instance-based method. It stores training observations rather than estimating a fixed parametric equation.

For a new point x, the algorithm:

1. Computes its distance from training observations.

2. Selects the k closest observations.

3. Predicts by majority vote or distance-weighted vote.

An estimated class probability can be obtained from the fraction of neighbours in a class. If 7 of 10 neighbours are positive, the unweighted estimate is 0.70. Such estimates can be coarse and poorly calibrated, particularly when k is small.

3.1 Distance and scaling

Euclidean distance is

d(x,a) = √[∑ⱼ(xⱼ − aⱼ)²].

Because distance depends on scale, numerical variables normally require standardization or another justified transformation. Without scaling, a variable measured in thousands may dominate one measured between zero and one.

Categorical variables cannot be inserted directly into Euclidean distance as arbitrary integer codes because numerical spacing would have no valid meaning. One-hot encoding, Hamming distance, Gower distance, or specialized mixed-data methods may be used.

3.2 Choosing k

Small k produces flexible boundaries with low bias and high variance. Large k smooths the boundary, increasing bias and potentially reducing variance. The value must be selected using validation or cross-validation rather than the test set. Ties should be resolved by a predefined rule; odd k reduces but does not eliminate ties in binary classification.

3.3 Computational characteristics

Training is inexpensive, but prediction may be costly because distances to many stored observations must be computed. Approximate nearest-neighbour indexes can improve scalability. Performance also deteriorates in high-dimensional spaces because distances become less informative, a phenomenon associated with the curse of dimensionality.

4. Naïve Bayes

Naïve Bayes applies Bayes’ theorem:

P(Y = c | x) = P(x | Y = c)P(Y = c)/P(x).

For classification, P(x) is common to all candidate classes, so prediction uses

ŷ = argmax_c P(Y = c)P(x | Y = c).

The naïve conditional-independence assumption gives

P(x | Y = c) = ∏ⱼ P(xⱼ | Y = c).

Consequently,

ŷ = argmax_c P(Y = c)∏ⱼP(xⱼ | Y = c).

Products of many small probabilities can underflow numerically. Implementations therefore sum logarithms:

log P(Y = c) + ∑ⱼ log P(xⱼ | Y = c).

4.1 Major variants

Gaussian naïve Bayes assumes each numerical feature follows a class-conditional normal distribution:

Xⱼ | Y = c ~ Normal(μⱼc, σ²ⱼc).

Multinomial naïve Bayes is suitable for non-negative counts, such as word frequencies. Bernoulli naïve Bayes models binary indicators. Categorical naïve Bayes models discrete categories.

For mixed numerical and categorical data, a principled mixed model may combine Gaussian likelihoods for continuous variables with categorical likelihoods for categorical variables. The combined log-score is

log P(Y = c) + ∑_{j∈N} log f_G(xⱼ | c) + ∑_{j∈C} log P(xⱼ | c),

where N and C index numerical and categorical variables respectively. This requires software that supports heterogeneous likelihoods or a custom estimator.

An alternative implementation discretizes numerical features into bins and one-hot encodes all features before applying a discrete naïve Bayes model. This is convenient but loses within-bin numerical information. Applying Gaussian naïve Bayes indiscriminately to one-hot indicators is generally a misspecification because binary indicators are not normally distributed.

4.2 Smoothing and limitations

An unseen category can produce a zero likelihood and force the entire product to zero. Additive smoothing addresses this problem. With Laplace smoothing, one count is added to each category count.

Naïve Bayes often performs well in text classification and high-dimensional sparse settings. Its probability estimates may nevertheless be overconfident because correlated features violate the conditional-independence assumption. Strong classification accuracy does not guarantee calibrated probabilities.

5. Decision Trees

A decision tree recursively partitions the feature space. Each internal node applies a rule, each branch represents an outcome of that rule, and each leaf produces a class or class distribution.

For binary classes, the Gini impurity of a node with positive-class proportion p is

G = 1 − p² − (1 − p)² = 2p(1 − p).

If p = 0.5, G = 0.5. If all observations belong to one class, G = 0.

Entropy is another impurity measure:

H = −∑_c p_c log₂(p_c),

with 0 log₂0 defined by continuity as zero.

A candidate split is assessed by the weighted impurity of its child nodes:

I_split = (n_L/n)I_L + (n_R/n)I_R.

The impurity reduction is the parent impurity minus this weighted value.

5.1 Worked split calculation

A parent node contains 20 observations: 10 positive and 10 negative. Its Gini impurity is 0.5. A split creates:

• Left node: 8 positive and 2 negative, so G_L = 1 − 0.8² − 0.2² = 0.32.
• Right node: 2 positive and 8 negative, so G_R = 0.32.

The weighted child impurity is

(10/20)(0.32) + (10/20)(0.32) = 0.32.

The impurity reduction is

0.50 − 0.32 = 0.18.

5.2 Controlling complexity

Unrestricted trees can fit small irregularities and noise. Controls include maximum depth, minimum samples per leaf, minimum impurity reduction, and post-pruning. These settings should be selected using validation data.

Trees can model nonlinear relationships and interactions without feature scaling. They are easy to visualize when shallow. However, small data changes can produce different structures, and deep trees often generalize poorly. Leaf probabilities are empirical class proportions and may be unreliable in small leaves.

Categorical inputs require attention. Some libraries support categorical splits directly; others require one-hot or ordinal encoding. Arbitrary ordinal encoding can create artificial order unless the algorithm explicitly treats codes as nominal categories.

6. Support Vector Machines

A linear SVM seeks a separating hyperplane

wᵀx + b = 0

with a large margin between classes. For labels yᵢ ∈ {−1,+1}, the soft-margin primal problem is

min_{w,b,ξ} ½||w||² + C∑ᵢξᵢ

subject to

yᵢ(wᵀxᵢ + b) ≥ 1 − ξᵢ, and ξᵢ ≥ 0.

The slack variables ξᵢ permit margin violations. The parameter C controls their penalty. A larger C penalizes violations more strongly and usually fits the training data more closely, potentially with a narrower margin. A smaller C permits more violations and increases regularization.

Only selected training observations—support vectors—determine the fitted boundary. Feature scaling is important because the margin and kernel distances depend on feature magnitudes.

6.1 Kernel SVMs

The kernel trick allows nonlinear boundaries through similarity functions. A common radial basis function kernel is

K(x,z) = exp(−γ||x − z||²),

where γ > 0 controls the locality of influence. Large γ can create highly flexible boundaries, while small γ produces smoother influence. Both C and γ require validation-based selection.

6.2 Scores and probabilities

The SVM decision function provides a signed score rather than a direct probability. Probability estimates can be obtained through calibration methods such as Platt scaling or isotonic regression. Calibration must be learned using held-out predictions or properly nested cross-validation; fitting a calibrator and evaluating it on the same observations produces optimistic results.

SVMs can be effective in high-dimensional spaces but may be computationally expensive on very large datasets. Kernel models are also less transparent than logistic regression or shallow trees.

7. Comparing Model Behaviour

The five models encode different assumptions:

• Logistic regression assumes that class log-odds are linear in the supplied features unless nonlinear terms are added.
• KNN assumes nearby observations tend to share labels.
• Naïve Bayes assumes class-conditional feature independence and specified feature distributions.
• Decision trees assume useful predictions can be formed through recursive partitions.
• SVMs seek a high-margin boundary, with kernels extending this principle nonlinearly.

No classifier is universally best. Validation performance, probability quality, inference speed, explainability, data volume, and deployment constraints must be evaluated together.

8. Performance Measures and Threshold Selection

For a positive class, the confusion matrix contains true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN).

Accuracy = (TP + TN)/(TP + TN + FP + FN)

Precision = TP/(TP + FP)

Recall = TP/(TP + FN)

Specificity = TN/(TN + FP)

F1 = 2(Precision × Recall)/(Precision + Recall)

These measures require nonzero denominators. Software and reports should state how undefined cases are handled.

8.1 Threshold effects

Lowering the positive threshold generally increases the number of positive predictions. On a fixed dataset, recall cannot decrease when the threshold is lowered, although precision may increase or decrease depending on the newly included cases. Raising the threshold generally reduces positive predictions and cannot increase recall.

The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate across thresholds. The precision–recall curve is often more informative when the positive class is rare. ROC area under the curve measures ranking ability, not calibration and not performance at one chosen threshold.

Log loss evaluates probabilities:

Log loss = −(1/n)∑ᵢ[yᵢlog(pᵢ) + (1−yᵢ)log(1−pᵢ)].

The Brier score is the mean squared probability error:

Brier = (1/n)∑ᵢ(pᵢ − yᵢ)².

Both reward probability quality, although they emphasize errors differently.

8.2 Cost-sensitive thresholding

Suppose a false positive costs C_FP and a false negative costs C_FN, while correct decisions have zero incremental cost. If p is a calibrated probability, predicting positive has expected cost C_FP(1−p), and predicting negative has expected cost C_FNp. Predict positive when

C_FP(1−p) ≤ C_FNp,

which gives

p ≥ C_FP/(C_FP + C_FN).

This formula applies only under the stated two-error-cost assumptions. Additional intervention costs, benefits, capacity constraints, or unequal correct-decision utilities require a revised decision analysis.

For example, if C_FP = 1 and C_FN = 4, the threshold is 1/(1+4) = 0.20. The low threshold reflects the greater cost of missing a positive case.

9. Mixed-Data Implementation and Reliable Model Selection

Real datasets commonly contain continuous variables, counts, binary indicators, nominal categories, and missing values. Preprocessing must be learned from training data only. A pipeline ensures that imputation, scaling, encoding, and model fitting are repeated correctly during validation.

A sound three-way design is:

1. Training set: fit model parameters and preprocessing.

2. Validation set: compare models, tune hyperparameters, calibrate probabilities, and select thresholds.

3. Test set: perform one final evaluation after all choices are fixed.

Evaluating every candidate repeatedly on the test set turns it into an informal validation set and biases the reported performance.

The following binary-classification example handles mixed data. Numerical features are median-imputed and standardized for distance- and margin-based models. Categorical features are mode-imputed and one-hot encoded. For naïve Bayes, numerical variables are discretized and then one-hot encoded, producing non-negative indicator features suitable for MultinomialNB.

```python

import numpy as np

import pandas as pd

from sklearn.base import clone

from sklearn.compose import ColumnTransformer

from sklearn.pipeline import Pipeline

from sklearn.impute import SimpleImputer

from sklearn.preprocessing import OneHotEncoder, StandardScaler, KBinsDiscretizer

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

from sklearn.neighbors import KNeighborsClassifier

from sklearn.naive_bayes import MultinomialNB

from sklearn.tree import DecisionTreeClassifier

from sklearn.svm import SVC

from sklearn.metrics import log_loss, roc_auc_score, f1_score

X is a DataFrame and y is a binary Series coded 0/1.

num_cols = X.select_dtypes(include="number").columns.tolist()

cat_cols = X.select_dtypes(exclude="number").columns.tolist()

Reserve the test set before any model comparison.

X_dev, X_test, y_dev, y_test = train_test_split(

X, y, test_size=0.20, stratify=y, random_state=42

)

X_train, X_val, y_train, y_val = train_test_split(

X_dev, y_dev, test_size=0.25, stratify=y_dev, random_state=42

) # 60% train, 20% validation, 20% test overall

numeric = Pipeline([

("impute", SimpleImputer(strategy="median")),

("scale", StandardScaler())

])

categorical = Pipeline([

("impute", SimpleImputer(strategy="most_frequent")),

("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False))

])

standard_prep = ColumnTransformer([

("num", numeric, num_cols),

("cat", categorical, cat_cols)

])

Discrete representation for MultinomialNB.

nb_numeric = Pipeline([

("impute", SimpleImputer(strategy="median")),

("bins", KBinsDiscretizer(

n_bins=5, encode="onehot-dense", strategy="quantile",

subsample=None, random_state=42

))

])

nb_categorical = Pipeline([

("impute", SimpleImputer(strategy="most_frequent")),

("encode", OneHotEncoder(handle_unknown="ignore", sparse_output=False))

])

nb_prep = ColumnTransformer([

("num", nb_numeric, num_cols),

("cat", nb_categorical, cat_cols)

])

models = {

"logistic": Pipeline([

("prep", standard_prep),

("model", LogisticRegression(max_iter=2000))

]),

"knn": Pipeline([

("prep", standard_prep),

("model", KNeighborsClassifier(n_neighbors=11))

]),

"naive_bayes": Pipeline([

("prep", nb_prep),

("model", MultinomialNB(alpha=1.0))

]),

"tree": Pipeline([

("prep", standard_prep),

("model", DecisionTreeClassifier(

max_depth=6, min_samples_leaf=20, random_state=42

))

]),

"svm": Pipeline([

("prep", standard_prep),

("model", SVC(C=1.0, kernel="rbf", probability=True,

random_state=42))

])

}

Compare candidates on validation data only.

validation_results = {}

for name, estimator in models.items():

fitted = clone(estimator).fit(X_train, y_train)

p_val = fitted.predict_proba(X_val)[:, 1]

validation_results[name] = {

"log_loss": log_loss(y_val, p_val),

"roc_auc": roc_auc_score(y_val, p_val)

}

Predeclared selection rule: lowest validation log loss.

best_name = min(validation_results,

key=lambda n: validation_results[n]["log_loss"])

Select the operating threshold using validation predictions.

best_validation_model = clone(models[best_name]).fit(X_train, y_train)

p_val = best_validation_model.predict_proba(X_val)[:, 1]

thresholds = np.linspace(0.10, 0.90, 81)

best_threshold = max(

thresholds,

key=lambda t: f1_score(y_val, (p_val >= t).astype(int))

)

Refit the chosen model on all development data.

final_model = clone(models[best_name]).fit(X_dev, y_dev)

Use the untouched test set once for the final performance estimate.

p_test = final_model.predict_proba(X_test)[:, 1]

y_test_pred = (p_test >= best_threshold).astype(int)

final_test_results = {

"log_loss": log_loss(y_test, p_test),

"roc_auc": roc_auc_score(y_test, p_test),

"f1": f1_score(y_test, y_test_pred)

}

```

The threshold criterion should be chosen before inspecting test performance. F1 is appropriate only when its balance of precision and recall matches the application. Healthcare, finance, and safety systems often require explicit cost, recall, or capacity constraints instead.

For robust hyperparameter tuning, cross-validation may replace the single training–validation split within the development data. The test set must still remain untouched. If calibration and hyperparameter tuning are both conducted, nested procedures or separate calibration data may be necessary.

10. Illustrative Case: Hospital Readmission Screening

Background

This fictional case illustrates model selection for mixed clinical and administrative data. A hospital wishes to identify patients at elevated risk of unplanned readmission within 30 days. Features include age, laboratory measurements, prior admissions, diagnosis group, insurance category, discharge destination, and whether follow-up was scheduled.

Problem

Only 12% of patients are readmitted. Missing a high-risk patient may prevent timely support, but excessive false alerts can overwhelm the care-management team. The hospital requires interpretable risk estimates and can contact only 200 patients per week.

Analysis and approach

Data are divided by patient to prevent the same patient appearing across partitions. A chronological split is also considered because deployment concerns future admissions. Numerical variables are median-imputed; categorical variables are mode-imputed and one-hot encoded. Transformations are fitted within training folds.

Logistic regression, KNN, discrete naïve Bayes, a pruned decision tree, and an RBF SVM are compared using development-data cross-validation. Primary measures are log loss and precision–recall area. Calibration plots assess probability reliability. The untouched test period is not accessed during model selection.

Logistic regression provides stable, reasonably calibrated estimates and clinically reviewable effects. The SVM has slightly better ranking but needs probability calibration and is harder to explain. KNN is sensitive to local density and slower at prediction. The tree is interpretable but less stable. Naïve Bayes is fast but overconfident because several utilization variables are correlated.

Rather than using 0.5, the hospital chooses a threshold corresponding to the 200 highest predicted risks each week. This is a capacity-constrained policy, not a fixed universal probability threshold. The threshold is estimated from validation-period score distributions and monitored after deployment.

Outcome

After model and policy selection are complete, logistic regression is refitted on all development data and evaluated once on the held-out test period. The final report includes discrimination, calibration, recall among the 200 contacted patients, subgroup performance, and confidence intervals. Deployment begins as a decision-support pilot rather than an autonomous denial or discharge system.

Learning points

1. Mixed clinical data require type-appropriate preprocessing.

2. Class imbalance makes accuracy insufficient.

3. The best ranking model may not provide the best probabilities or governance profile.

4. Thresholds should reflect intervention costs and capacity.

5. Patient-level and temporal separation can be more realistic than random record splitting.

6. Final test data should be used only after the model and policy are fixed.

11. Challenges, Governance, and Contemporary Practice

11.1 Distribution shift

Relationships learned from historical data may change because of economic conditions, user behaviour, measurement systems, policy, or population composition. Performance and calibration should be monitored after deployment. Retraining must itself be governed and validated.

11.2 Fairness and proxy variables

A model may produce unequal false-positive or false-negative rates across demographic groups. Removing protected attributes does not eliminate discrimination because other variables may act as proxies. Fairness analysis should consider the application, affected groups, legal duties, and potential trade-offs among fairness definitions.

11.3 Explainability and contestability

Coefficient interpretations, tree paths, feature-attribution methods, and example-based explanations can support review, but explanations must not be mistaken for causal conclusions. Individuals affected by high-impact decisions may require notice, human review, and a process for correcting inaccurate data.

11.4 Privacy and security

Classification systems may process sensitive personal information. Data minimization, access controls, retention limits, audit logs, and lawful processing are essential. Models can also be attacked through adversarial inputs, data poisoning, or attempts to infer training records.

11.5 Modern developments

Ensembles such as random forests and gradient-boosted trees often outperform a single decision tree, while deep neural networks dominate some unstructured-data applications. Nevertheless, the foundational models in this chapter remain important baselines. Contemporary practice increasingly emphasizes calibrated uncertainty, interpretable deployment policies, automated pipeline testing, model cards, drift monitoring, and human oversight.

12. Best-Practice Checklist

A defensible classification project should:

• define the positive class and decision objective clearly;
• split data in a way that reflects deployment and prevents entity leakage;
• fit every preprocessing step using training data only;
• represent numerical and categorical variables appropriately;
• tune models and thresholds using validation data or cross-validation;
• keep the test set untouched until one final evaluation;
• compare discrimination, calibration, threshold metrics, and resource demands;
• report class prevalence and uncertainty alongside point estimates;
• assess subgroup performance, privacy, explainability, and potential harm;
• monitor data quality, drift, calibration, and operational outcomes after deployment.

Chapter Summary

Classification models transform features into class scores, probabilities, or labels. Logistic regression models linear log-odds and supports interpretable probability estimation. KNN classifies through local similarity but depends strongly on scaling, dimensionality, and the choice of k. Naïve Bayes combines prior probabilities with class-conditional feature likelihoods and requires distributional assumptions appropriate to each data type. Decision trees create recursive partitions and capture nonlinear interactions, although unrestricted trees are unstable and prone to overfitting. SVMs maximize a geometric margin and can use kernels for nonlinear classification, but their scores require calibration before being interpreted as probabilities.

Decision boundaries depend on the model and the classification threshold. Threshold selection should reflect error costs, class imbalance, operational capacity, or specified performance requirements. Probability calibration and ranking ability are distinct properties. Reliable comparison requires training and validation procedures that prevent leakage, while an untouched test set provides one final estimate after all model and threshold choices are fixed. Responsible deployment additionally requires fairness assessment, privacy protection, monitoring, documentation, and human oversight.

Key Terms

Calibration: Agreement between predicted probabilities and observed event frequencies.

Classification threshold: The score or probability cutoff used to assign a class label.

Conditional independence: The naïve Bayes assumption that features are independent after conditioning on the class.

Confusion matrix: A table of true-positive, false-positive, true-negative, and false-negative counts.

Cross-entropy: A loss function that penalizes inaccurate probability predictions.

Decision boundary: The set of feature values at which the predicted class changes.

Decision tree: A classifier that recursively partitions observations using feature-based rules.

Entropy: An impurity measure based on uncertainty in class proportions.

Gini impurity: A measure of class mixture commonly used to select tree splits.

Kernel: A function used by an SVM to represent similarity in an implicit feature space.

K-nearest neighbours: A classifier that predicts from the labels of nearby training observations.

Logistic regression: A model that represents class log-odds as a linear function of features.

Margin: The separation between an SVM decision boundary and influential training observations.

Naïve Bayes: A family of probabilistic classifiers based on Bayes’ theorem and conditional independence.

One-hot encoding: Representation of categorical levels using separate binary indicator variables.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives correctly identified.

Regularization: A penalty or constraint used to control model complexity.

Support vector: A training observation that influences an SVM boundary.

Test set: Data reserved for final evaluation after model and policy selection.

Review Questions

Short Answer Questions

1. Distinguish a decision score, a probability estimate, and a class label.

2. Why is feature scaling important for KNN and SVMs?

3. What does e^{βⱼ} represent in binary logistic regression?

4. State the conditional-independence assumption used by naïve Bayes.

5. Why should the test set not be used to choose a classification threshold?

Long Answer Questions

1. Derive the binary logistic probability function from the log-odds equation and explain coefficient interpretation.

2. Compare the assumptions, decision boundaries, computational properties, and interpretability of the five classifiers in this chapter.

3. Explain how tree impurity, stopping rules, and pruning affect the bias–variance trade-off.

4. Discuss discrimination, calibration, and threshold-dependent performance as distinct aspects of classifier evaluation.

5. Design a model-selection workflow that uses preprocessing, cross-validation, calibration, threshold selection, and one final test evaluation without leakage.

Analytical / Application Questions

1. A fraud model assigns a probability of 0.35. A false positive costs $20 and a false negative costs $80, with zero incremental cost for correct decisions. Derive the cost-minimizing threshold under these assumptions and determine the predicted class.

2. A dataset contains age, annual income, occupation, region, and several missing values. Propose type-appropriate preprocessing for all five classifiers and identify any model-specific modifications.

3. A classifier produces TP = 42, FP = 18, TN = 880, and FN = 60. Calculate accuracy, precision, recall, specificity, and F1, then discuss whether the model is suitable for a high-consequence screening task.

Multiple Choice Questions with Answers

Question 1

A classifier estimates P(Y = 1 | x) = 0.62 and uses a threshold of 0.70. What class is predicted?

A. Class 0

B. Class 1

C. Both classes

D. No class can be assigned

Correct Answer: A. Class 0

Question 2

In logistic regression, what does e raised to the coefficient βj represent, holding other variables constant?

A. The constant change in probability

B. The multiplicative change in the odds for a one-unit increase in xj

C. The model's classification threshold

D. The additive change in accuracy

Correct Answer: B. The multiplicative change in the odds for a one-unit increase in xj

Question 3

Why are numerical features commonly standardized before applying KNN?

A. To make all class frequencies equal

B. To convert probabilities into labels

C. To prevent large-scale features from dominating distance calculations

D. To ensure the decision boundary is linear

Correct Answer: C. To prevent large-scale features from dominating distance calculations

Question 4

Which expression is maximized by a naïve Bayes classifier when predicting class c?

A. P(Y = c) times the product of the class-conditional feature likelihoods

B. The Euclidean distance to class c

C. The margin width of class c

D. The Gini impurity of class c

Correct Answer: A. P(Y = c) times the product of the class-conditional feature likelihoods

Question 5

What is the Gini impurity of a binary tree node containing equal proportions of the two classes?

A. 0.00

B. 0.25

C. 0.50

D. 1.00

Correct Answer: C. 0.50

Question 6

In a soft-margin SVM, what is the usual effect of increasing C?

A. Margin violations are penalized more strongly

B. All support vectors are removed

C. Feature scaling becomes unnecessary

D. Decision scores become calibrated probabilities

Correct Answer: A. Margin violations are penalized more strongly

Question 7

Which dataset should be used to compare candidate models and choose hyperparameters?

A. The untouched final test set

B. The validation data or development-data cross-validation

C. Future production observations

D. Only misclassified training observations

Correct Answer: B. The validation data or development-data cross-validation

Question 8

Which approach provides a valid discrete naïve Bayes representation for mixed numerical and categorical data?

A. Assign arbitrary integer distances to nominal categories

B. Discretize numerical variables and one-hot encode the resulting bins and categories

C. Apply a logarithm to every category name

D. Treat missing values as probability estimates

Correct Answer: B. Discretize numerical variables and one-hot encode the resulting bins and categories

Question 9

A classifier produces 2 true positives and 8 false positives. What is its precision?

A. 0.20

B. 0.25

C. 0.80

D. 4.00

Correct Answer: A. 0.20

Question 10

Which statement best describes a well-calibrated classifier?

A. It always predicts the majority class correctly

B. It has a perfectly linear decision boundary

C. Among cases assigned probability 0.70, approximately 70% are positive over comparable cases

D. It uses a threshold of exactly 0.50

Correct Answer: C. Among cases assigned probability 0.70, approximately 70% are positive over comparable cases