Classification and Decision Models

Introduction and Context

Many business questions have a categorical outcome: whether a customer will churn, whether a transaction is fraudulent, whether an applicant will default, whether a lead will convert, or whether a product review is positive. Classification is the supervised-learning task of assigning observations to predefined classes. A classification model estimates either a class label, a probability of class membership, or both.

Classification is not identical to decision-making. A model may estimate that a loan applicant has a 0.30 probability of default, but the organization must still decide whether to approve the loan. That decision depends on a threshold, the relative costs of false approvals and false rejections, operational capacity, and sometimes expected profit. This chapter therefore treats classification as a two-stage process:

1. estimate class probabilities or scores from data; and
2. convert those estimates into actions using an explicit decision rule.

The distinction is central to responsible business analytics. A model can have excellent ranking performance but produce poor operational decisions if its threshold, calibration, or cost assumptions are unsuitable.

Learning Objectives

By the end of this chapter, you should be able to:

• explain binary and multiclass classification;
• build and interpret logistic regression, k-nearest-neighbor, naïve Bayes, and support-vector models;
• distinguish probabilities, scores, predicted labels, and decisions;
• construct and interpret a confusion matrix;
• calculate accuracy, precision, recall, specificity, F1 score, ROC-AUC, and related measures;
• assess probability calibration;
• select thresholds using expected cost or expected profit;
• recognize model assumptions, limitations, fairness risks, and deployment requirements.

1. Classification Foundations

Let an observation be represented by a feature vector x = (x₁, x₂, …, x_p), where p is the number of predictors. Let Y be the target class. In binary classification, Y ∈ {0, 1}; for example, 1 may represent fraud and 0 legitimate activity. A model estimates

The output may be:

• a probability, such as 0.82;
• a score, which may not be on a probability scale;
• a predicted class, such as 1; or
• a business action, such as investigate, approve, or contact.

A default threshold of 0.50 predicts class 1 when the estimated probability is at least 0.50. However, 0.50 is not a universal rule. If missing a fraudulent transaction is much more costly than investigating a legitimate one, a lower threshold may be appropriate.

Training, validation, and test data

A dataset should generally be divided into training, validation, and test sets. The training set estimates model parameters. The validation set supports hyperparameter and threshold selection. The test set provides a final, untouched estimate of generalization performance. In time-dependent business data, random splitting can cause leakage from the future into the past; chronological or rolling validation is usually more appropriate.

Preprocessing must be fitted using training data only. For example, a mean used to standardize a feature must be calculated from the training set, then applied unchanged to validation and test observations. This prevents information leakage.

Class imbalance

If only 1% of transactions are fraudulent, a model that predicts “legitimate” for every transaction has 99% accuracy but no fraud-detection value. Evaluation must therefore consider the positive-class rate, the costs of errors, and metrics such as precision, recall, and area under the precision–recall curve.

2. Logistic Regression

Logistic regression is a discriminative model that directly estimates the probability of a binary outcome. It assumes that the log-odds of the event are a linear function of the predictors:

where p(x) = P(Y = 1 | X = x), β₀ is the intercept, βⱼ is the coefficient for predictor xⱼ, and the predictors are measured on their stated scales. Solving for p(x) gives the logistic function:

The formula always produces a value strictly between 0 and 1. The model is estimated commonly by maximum likelihood, selecting coefficients that make the observed class labels most plausible. For independent observations, the likelihood is

where n is the number of observations, yᵢ ∈ {0,1}, and pᵢ is the predicted probability for observation i. In practice, the log-likelihood is optimized, often with regularization.

A one-unit increase in xⱼ changes the log-odds by βⱼ, holding other variables constant. The corresponding odds ratio is exp(β_j). Thus, if βⱼ = 0.7, the odds are multiplied by approximately 2.01 for a one-unit increase, assuming the model specification is appropriate. An odds ratio is not a probability increase; the probability effect depends on the starting probability.

Regularization

With L2 regularization, the optimization objective commonly penalizes large coefficients:

where λ ≥ 0 controls penalty strength. Larger λ generally reduces variance but can increase bias. The intercept is typically not penalized, and software conventions differ in the scaling of the penalty. L1 regularization uses λ∑|β_j|, which can set some coefficients exactly to zero and thereby perform feature selection.

Logistic regression is attractive because it is fast, interpretable, and naturally probabilistic. Its main limitations are the linearity of the log-odds assumption, sensitivity to influential observations and separation, and dependence on thoughtful feature engineering. Nonlinear effects can be represented using transformations, interactions, splines, or other models.

3. k-Nearest Neighbors

The k-nearest-neighbor (k-NN) method is a nonparametric, instance-based classifier. To classify a new point, it identifies the k training observations closest to it and assigns the majority class. A probability estimate may be calculated as the fraction of neighbors in class 1:

where Nₖ(x) is the set of k nearest observations and I(·) is an indicator equal to 1 when its condition is true and 0 otherwise.

The common Euclidean distance is

It is appropriate when numeric features are meaningfully comparable and their scales have been standardized or otherwise addressed. A feature measured in thousands can dominate a feature measured in percentages if scaling is ignored. Categorical variables require appropriate encodings and distance definitions.

Small k values create flexible boundaries but may be sensitive to noise. Large k values smooth the boundary but may miss local patterns. k is selected using validation, and odd values can reduce ties in binary classification. Distance weighting can give nearer observations greater influence.

The method has minimal training cost but potentially high prediction cost because distances to many observations may be computed. It is also vulnerable to the curse of dimensionality: as dimensions increase, observations become relatively farther apart and local neighborhoods become less informative. k-NN is often useful as a transparent benchmark, but its probability estimates may require calibration.

4. Naïve Bayes

Naïve Bayes applies Bayes’ theorem while assuming that features are conditionally independent given the class. For class c and feature vector x,

The denominator is common across classes when comparing them. Therefore classification can use the proportional score

The prior P(Y=c) represents the prevalence of class c. The likelihood terms describe feature behavior within that class. For text, a multinomial version often uses word counts; for continuous features, a Gaussian version may assume each feature has a class-specific normal distribution.

A practical issue is zero probability. If a feature value never appears in a class during training, the product becomes zero. Laplace smoothing replaces counts with adjusted counts; for a categorical feature with V possible values, an estimate may be formed as

where α > 0 is a smoothing parameter. The independence assumption is often false, yet naïve Bayes can perform well, especially with high-dimensional sparse text. Its probabilities may be poorly calibrated because the model treats correlated evidence as independent.

5. Support-Vector Machines

A support-vector machine (SVM) seeks a separating boundary with a large margin. For a linear classifier, the decision function is

where w is a vector of coefficients and b is an intercept. The predicted class is determined by the sign of f(x), while the magnitude is a margin-based score rather than automatically a probability.

For separable data, the margin is maximized by minimizing (1/2)||w||² subject to correct separation. Real data often overlap, so soft-margin SVM introduces slack variables ξᵢ:

subject to

Here yᵢ is coded −1 or +1, ξᵢ measures margin violation, and C controls the trade-off between a wide margin and training errors. Large C penalizes violations more strongly and may produce a more complex boundary.

Kernel methods allow nonlinear boundaries by replacing inner products with a kernel function. A radial basis function kernel is often expressed as

where γ controls how rapidly similarity declines with distance. Both C and γ should be tuned using validation. Features generally need scaling because SVM distances and margins are scale-sensitive.

SVMs can be powerful in high-dimensional settings, but their raw scores are not probabilities. Platt scaling, isotonic regression, or another calibration method can transform scores into estimated probabilities using separate validation data. Calibration must not be assessed on the same data used to fit the calibration transformation.

6. Comparing the Four Models

Model selection should reflect the business problem. A slightly less accurate model may be preferable if its probabilities are well calibrated, its decisions are explainable, or its monitoring requirements are manageable. Validation should compare models on metrics relevant to the intended action rather than accuracy alone.

7. Confusion Matrices and Threshold-Dependent Metrics

For binary classification, outcomes are organized as follows:

From these counts, with total N = TP + FP + FN + TN:

Accuracy is the proportion classified correctly, but it can be misleading under imbalance.

Precision is the proportion of predicted positives that are truly positive. It matters when investigations or interventions are expensive.

Recall, also called sensitivity or true-positive rate, is the proportion of actual positives detected. It matters when missed positives are costly.

Specificity is the true-negative rate. The false-positive rate is

The F1 score is the harmonic mean of precision and recall:

It is useful when both precision and recall matter, but it ignores true negatives and does not incorporate monetary costs.

Worked example: threshold consequences

Suppose a validation sample contains 1,000 customers, of whom 100 churn. At a selected threshold, the model identifies 70 churners and incorrectly flags 90 non-churners. Thus TP = 70, FN = 30, FP = 90, and TN = 810.

Accuracy = (70 + 810)/1,000 = 0.88, or 88%.

Precision = 70/(70 + 90) = 0.4375, or 43.75%.

Recall = 70/(70 + 30) = 0.70, or 70%.

Specificity = 810/(810 + 90) = 0.90, or 90%.

The model is therefore much better at finding churners than at ensuring every flagged customer will churn. Whether this is acceptable depends on the cost and benefit of retention activity.

8. ROC-AUC, Precision–Recall, and Ranking

A classifier can produce scores for many possible thresholds. A receiver operating characteristic (ROC) curve plots TPR against FPR as the threshold varies. The ROC-AUC is the area under this curve and can be interpreted, under standard tie handling, as the probability that a randomly selected positive receives a higher score than a randomly selected negative. AUC = 0.5 corresponds approximately to random ranking; AUC = 1 represents perfect ranking.

ROC-AUC is threshold-independent and useful for comparing ranking quality, but it does not specify the operating threshold or monetary consequences. With rare positives, the precision–recall curve is often more informative because it focuses on positive-class performance. Its baseline precision is related to the positive prevalence.

AUC should be calculated on held-out data. It does not prove that probabilities are accurate, nor does it guarantee good performance after deployment if the population or behavior changes.

9. Calibration and Probability Quality

A model is calibrated if, among observations assigned probability q, approximately q proportion actually belong to the positive class. For example, among 1,000 cases assigned probabilities near 0.20, about 200 should be positive if the model is well calibrated.

Calibration can be assessed with reliability diagrams, calibration curves, and summary measures such as the Brier score:

The Brier score applies to binary outcomes, where lower values are better. It combines aspects of calibration and discrimination, so it should not be interpreted as a pure calibration measure. Reliability diagrams group predictions into probability bins and compare average predicted probability with observed event frequency. Calibration can vary across customer segments; overall calibration can conceal subgroup miscalibration.

Calibration is especially important when decisions depend on expected cost or profit. A ranking model with distorted probabilities may still have a high AUC but produce systematically poor financial estimates.

10. Thresholds, Expected Cost, and Expected Profit

Let a decision rule classify an observation as positive when p(x) ≥ t, where t is the threshold. Changing t changes TP, FP, FN, and TN. The appropriate threshold depends on the action and the consequences.

Suppose the costs of a false positive and false negative are C_FP and C_FN, respectively. A sample-level expected error cost is

If each positive intervention costs C_A and each successfully treated positive yields benefit B, an expected-profit framework can include intervention and outcome terms. A simple realized-profit expression is

where B_TP is the benefit associated with a true positive action. Organizations must define these quantities carefully and avoid counting the same cost twice.

For a one-time binary action with calibrated probability p, assume acting yields benefit B if the person is positive and costs C if negative, while not acting yields zero incremental value. The expected value of acting is

Act when EV(action) > 0. Solving gives

This threshold is valid under the stated assumptions: calibrated probabilities, a common benefit B, a common cost C, and zero value for inaction. If treatment costs, delayed effects, capacity limits, or heterogeneous benefits exist, the rule must be adapted.

Threshold example

Suppose a retention offer produces a benefit of $120 when a customer would otherwise churn and costs $30 when sent to a customer who would not churn. The threshold is

$30/($120 + $30) = 0.20.

A calibrated probability above 0.20 justifies the offer under this simplified model. This does not mean every customer above 0.20 should receive an offer: contact capacity, customer fatigue, fairness, and uncertainty in the estimated values must also be considered.

11. Illustrative Case: Retail Fraud Investigation

Background and problem

A fictional online retailer processes 200,000 orders per month. Historically, 1.5% are fraudulent. Every order receives a risk score. The fraud team can manually investigate only 3,000 orders monthly. A false negative creates an average loss of $180, while a false positive consumes approximately $8 of review capacity and may delay a legitimate order.

Analysis

The analytics team compares logistic regression, k-NN, naïve Bayes, and an RBF-kernel SVM. The training period precedes the validation period chronologically. Numeric variables are scaled within a training pipeline. Text and categorical variables are encoded without using future information. Hyperparameters are selected on validation data, and the final test period is held out until model and threshold selection are complete.

ROC-AUC is used to compare ranking, precision–recall performance is examined because fraud is rare, and calibration curves are reviewed because expected losses will be estimated. The SVM produces the highest ranking AUC, but its raw scores are calibrated using isotonic regression on a separate validation subset. Logistic regression has slightly lower AUC but better interpretability and comparable precision among the 3,000 highest-risk orders.

Approach and solution

The team selects the threshold corresponding to the investigation capacity, rather than automatically using 0.50. It estimates expected loss using the validation confusion matrix and tests sensitivity to the assumed $180 loss and $8 review cost. It also establishes a policy for borderline scores: orders just below the threshold are not automatically rejected; they receive normal processing unless an independent rule triggers review.

Outcome and governance

On the untouched test period, the selected policy detects most fraud within capacity while maintaining an acceptable legitimate-order delay rate. Monitoring includes monthly fraud prevalence, precision among investigated orders, calibration by payment method and region, review backlog, and changes in customer complaints. Access to model features is restricted, and investigators can record reasons for overriding model recommendations.

Learning points

The case illustrates that the best classifier by AUC is not necessarily the best operational choice. Capacity determines a feasible threshold; calibration supports financial estimation; temporal validation reflects deployment; and governance requires monitoring both model performance and business side effects.

12. Practical Implementation Workflow

A robust classification project commonly follows these steps:

1. Define the target, prediction time, intervention, and success measure.
2. Establish the positive class and document its business meaning.
3. Check label quality, missingness, leakage, duplicates, and sampling bias.
4. Split data according to the deployment timeline or sampling design.
5. Build preprocessing and modeling as one reproducible pipeline.
6. Fit simple baselines, including prevalence and logistic regression.
7. Tune k-NN, naïve Bayes variants, SVM parameters, or regularization using validation.
8. Compare discrimination, calibration, subgroup performance, latency, explainability, and cost.
9. Select a threshold using capacity, expected cost, profit, or policy requirements.
10. Evaluate once on untouched test data.
11. Deploy with versioning, monitoring, rollback, and documented responsibility.

13. Limitations, Ethics, and Contemporary Practice

Classification models learn patterns from historical labels. If past decisions were biased, labels may encode discrimination. Features that appear neutral can act as proxies for protected characteristics. Removing a protected attribute does not guarantee fairness because correlated variables may remain.

Relevant checks include performance by subgroup, false-positive and false-negative disparities, calibration by subgroup, equalized-error considerations, and analysis of the operational impact of interventions. These criteria can conflict; no single fairness metric is universally correct. The organization must define the harm it is trying to prevent and involve affected stakeholders.

Privacy requires data minimization, purpose limitation, access control, retention rules, and appropriate legal bases. High-impact uses may require explanation, human review, contestability, and documentation under applicable regulation. Sensitive information should not be used merely because it improves AUC.

Contemporary practice increasingly emphasizes reproducible pipelines, model cards, data sheets, probability calibration, uncertainty analysis, drift detection, and monitoring for label delay. Automated machine learning can efficiently compare models, but it does not replace target definition, causal reasoning, cost specification, or governance. Generative AI may assist with coding and documentation, but generated code and analytical claims require testing and expert review.

Best Practices

• Treat prediction and decision as separate design problems.
• Use time-aware validation when deployment predicts the future.
• Report the positive prevalence and a baseline model.
• Use precision, recall, calibration, and cost—not accuracy alone.
• Tune thresholds on validation data and lock them before testing.
• Calibrate scores when expected-value decisions require probabilities.
• Reassess assumptions when costs, prevalence, or capacity change.
• Document feature definitions, exclusions, data lineage, and model version.
• Monitor subgroup outcomes and investigate unexpected disparities.
• Provide human escalation and override procedures for consequential decisions.

Chapter Summary

Classification models estimate class membership or a ranking score, while decision models convert estimates into actions. Logistic regression models a linear relationship in log-odds and offers interpretable probabilities. k-NN classifies according to nearby observations and requires careful scaling. Naïve Bayes uses Bayes’ theorem with a conditional-independence assumption and is often effective for sparse data. SVMs maximize a separating margin and can model nonlinear boundaries through kernels, but their scores require calibration if probabilities are needed.

Confusion matrices provide TP, FP, FN, and TN, from which accuracy, precision, recall, specificity, F1, and related measures are calculated. ROC-AUC evaluates ranking across thresholds, whereas calibration evaluates whether predicted probabilities correspond to observed frequencies. Thresholds should be chosen using operational capacity, expected cost, expected profit, or policy requirements. Sound practice combines statistical evaluation with temporal validation, leakage prevention, fairness analysis, privacy protection, monitoring, and human governance.

Key Terms

Classification: Assignment of observations to predefined categories.

Binary classification: Classification with two possible classes.

Probability estimate: Estimated likelihood of membership in a class.

Threshold: Cutoff used to convert a score or probability into a predicted class.

Logistic regression: Model expressing binary-event log-odds as a function of predictors.

Odds ratio: Multiplicative change in odds associated with a one-unit predictor change.

Regularization: Penalty used to control model complexity.

k-nearest neighbors: Classifier based on the labels of nearby training observations.

Distance metric: Rule for measuring separation between observations.

Naïve Bayes: Bayesian classifier using conditional independence of features given class.

Smoothing: Adjustment preventing zero probability estimates.

Support-vector machine: Maximum-margin classifier.

Kernel: Function enabling similarity-based nonlinear modeling.

Confusion matrix: Table of actual and predicted binary outcomes.

Precision: Proportion of predicted positives that are truly positive.

Recall: Proportion of actual positives detected.

ROC-AUC: Area under the receiver operating characteristic curve.

Calibration: Agreement between predicted probabilities and observed frequencies.

Expected cost: Probability-weighted or realized monetary cost of decisions.

Data leakage: Use of information unavailable at prediction time.

Review Questions

5 Short Answer Questions

1. Why can accuracy be misleading when the positive class is rare?
2. Interpret a logistic-regression coefficient of 0.5 in terms of log-odds and odds ratio.
3. Why should features generally be scaled for k-NN and SVM models?
4. What is the difference between precision and recall?
5. Why is calibration important for expected-profit decisions?

5 Long Answer Questions

1. Compare logistic regression, k-NN, naïve Bayes, and SVM in terms of assumptions, interpretability, computational characteristics, and business use.
2. Explain how a confusion matrix changes as the classification threshold changes, and discuss the consequences for precision and recall.
3. Describe a leakage-resistant workflow for developing a churn-classification model.
4. Explain ROC-AUC and why it should not be treated as a complete measure of business value.
5. Discuss ethical, privacy, and fairness risks in automated credit or fraud classification.

3 Analytical / Application Questions

1. A campaign has benefit B = $100 for a correctly targeted positive customer and cost C = $25 for targeting a negative customer. Under the simple expected-value assumptions in this chapter, calculate the probability threshold for action and explain two reasons an organization might still choose a different operational threshold.
2. A test set has TP = 45, FP = 15, FN = 30, and TN = 910. Calculate accuracy, precision, recall, and specificity, then identify which metric would be most relevant if missing a positive case is especially costly.
3. Design an evaluation plan for a fraud model where fraud prevalence changes seasonally and investigators have fixed monthly capacity. Specify the data split, metrics, calibration approach, threshold-selection method, and monitoring indicators.

Multiple Choice Questions with Answers

Question 1

Which quantity does logistic regression model as a linear function of the predictors?

A. The probability itself on an unrestricted scale

B. The log-odds of the positive class

C. The Euclidean distance to the nearest observation

D. The ROC-AUC directly

Correct Answer: B. The log-odds of the positive class

Question 2

A binary classifier has TP = 70, FP = 30, FN = 20, and TN = 880. What is its recall?

A. 70%

B. 70/100 = 70%

C. 70/90 ≈ 77.8%

D. 880/910 ≈ 96.7%

Correct Answer: B. 70/100 = 70%

Question 3

Why is feature scaling particularly important for k-nearest neighbors?

A. It guarantees that all classes have equal prevalence

B. It prevents the target variable from being observed

C. It prevents large-unit features from dominating distance calculations

D. It converts categorical variables into probabilities automatically

Correct Answer: C. It prevents large-unit features from dominating distance calculations

Question 4

What assumption distinguishes naïve Bayes from a general Bayes classifier?

A. All features have identical means

B. Features are conditionally independent given the class

C. The classes must have equal prior probabilities

D. The decision boundary must be linear

Correct Answer: B. Features are conditionally independent given the class

Question 5

In an SVM, what does a larger value of C generally do?

A. Penalizes margin violations more strongly

B. Forces all features to have unit variance

C. Converts scores directly into calibrated probabilities

D. Reduces the number of training observations

Correct Answer: A. Penalizes margin violations more strongly

Question 6

A model predicts 80 positives, of which 60 are truly positive. What is its precision?

A. 60/80 = 75%

B. 60/100 = 60%

C. 80/60 ≈ 133.3%

D. 20/80 = 25%

Correct Answer: A. 60/80 = 75%

Question 7

What does a well-calibrated prediction of 0.30 mean for a sufficiently large comparable group?

A. Exactly 30% of every subgroup must be positive

B. Approximately 30% of the group should be positive

C. The model’s ROC-AUC must equal 0.30

D. The classification threshold must be 0.30

Correct Answer: B. Approximately 30% of the group should be positive

Question 8

Under the chapter’s simple action-value assumptions, with benefit B = $120 and negative-case cost C = $30, what is the action threshold?

A. 0.20

B. 0.25

C. 0.30

D. 0.80

Correct Answer: A. 0.20

Question 9

Which statement about ROC-AUC is correct?

A. It is calculated only at a single fixed threshold

B. It measures whether predicted probabilities are perfectly calibrated

C. It evaluates ranking across possible thresholds

D. It is always more informative than precision when positives are rare

Correct Answer: C. It evaluates ranking across possible thresholds

Question 10

Why should the test set remain untouched during model and threshold selection?

A. To increase the number of training observations

B. To provide an unbiased final estimate of generalization performance

C. To ensure that the test set has no positive cases

D. To make regularization unnecessary

Correct Answer: B. To provide an unbiased final estimate of generalization performance