Classification and Probability Estimation

Introduction

Classification is a central predictive-analytics task in which observations are assigned to categories. A bank may classify an application as likely to default or not; a retailer may classify a customer as likely to churn or remain; a medical service may classify a case as requiring follow-up or not. In each example, the analyst is not merely predicting a label. A useful model estimates the probability of each outcome, supports a decision threshold, and communicates the consequences of errors.

This chapter focuses on binary and multiclass classification, especially logistic regression and related methods. It also develops the connection between probabilities, odds, decision thresholds, confusion matrices, precision, recall, F1 score, ROC-AUC, and calibration. These concepts are important because a model can rank cases well while producing poorly calibrated probabilities, or achieve high accuracy while failing the group or outcome that matters most.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish classification from regression and explain the role of estimated probabilities.
2. Interpret odds, log-odds, logistic-regression coefficients, and odds ratios.
3. Build and interpret binary and multiclass classification models.
4. Construct confusion matrices and calculate accuracy, precision, recall, specificity, and F1 score.
5. Explain threshold selection, ROC curves, ROC-AUC, precision–recall analysis, and calibration.
6. Evaluate classification models using appropriate validation procedures.
7. Recognize limitations, ethical risks, and governance requirements in classification projects.

1. Classification as a Predictive-Analytics Problem

Let an observation have predictor vector [[M:x = (x_1, x_2, …, x_p)]] and outcome [[M:Y]]. In binary classification, [[M:Y ∈ {0,1}]]. The model estimates

The probability is often more informative than a hard class label. For example, probabilities of 0.51 and 0.99 may lead to the same class under a 0.50 threshold, but they represent very different levels of evidence and may justify different interventions.

A classification system usually has three stages:

1. Probability estimation: estimate the conditional probability of each class.
2. Decision rule: convert probabilities into actions using a threshold or a cost-sensitive rule.
3. Evaluation: assess discrimination, calibration, operational usefulness, fairness, and stability.

Classification differs from regression in its target. Regression estimates a numerical quantity such as revenue; classification estimates membership in a category. Nevertheless, many classification methods are statistical models of conditional distributions, and their outputs can be used in forecasting, targeting, screening, and resource allocation.

Important terminology

• Class: a possible outcome category.
• Positive class: the outcome designated as the event of interest, coded conventionally as 1.
• Negative class: the non-event, coded conventionally as 0.
• Score: a model output used to rank observations; it may or may not be a probability.
• Threshold: a cutoff used to convert a probability into a predicted class.
• Discrimination: the ability to rank positive cases above negative cases.
• Calibration: agreement between predicted probabilities and observed event frequencies.

The choice of positive class is substantive, not merely technical. In fraud detection, fraud may be positive. In loan approval, however, “default” is often the positive class because it is the adverse event being detected.

2. Probability, Odds, and Log-Odds

For an event with probability [[M:p]], the odds are

where [[M:0 < p < 1]]. Odds express the ratio of the probability that an event occurs to the probability that it does not occur. If [[M:p = 0.20]], the odds are [[M:0.20/0.80 = 0.25]], or 1 to 4. Conversely,

The log-odds, or logit, is

The logit maps probabilities from the interval (0, 1) to all real numbers. This makes it possible to model log-odds as a linear function of predictors while ensuring that the resulting probability remains between 0 and 1.

A probability of 0.50 has odds of 1 and log-odds of 0. A probability greater than 0.50 has positive log-odds; a probability less than 0.50 has negative log-odds. Odds are not probabilities: odds of 2 mean a probability of 2/3, not 2.

3. Logistic Regression

3.1 The model

Binary logistic regression models the log-odds of the positive outcome as a linear combination of predictors:

Equivalently, the predicted probability is

Here, [[M:β_0]] is the intercept, [[M:β_j]] is the coefficient for predictor [[M:x_j]], and [[M:exp(·)]] is the exponential function. The model assumes that predictors have a linear and additive relationship with the log-odds, not necessarily with the probability itself.

The coefficients are usually estimated by maximum likelihood. For independent observations indexed by [[M:i = 1,…,n]], with observed outcomes [[M:y_i ∈ {0,1}]] and predicted probabilities [[M:p_i]], the likelihood is

In practice, software maximizes the log-likelihood

This formulation assumes correctly specified independent observations, a suitable outcome definition, and predictors measured without unacceptable leakage. Regularization may be added when the number of predictors is large or multicollinearity is substantial.

3.2 Interpreting coefficients

Holding all other predictors constant, a one-unit increase in [[M:x_j]] changes the log-odds by [[M:β_j]]. Exponentiating gives the odds ratio:

An odds ratio greater than 1 indicates increased odds; an odds ratio below 1 indicates decreased odds. An odds ratio of 1 indicates no multiplicative change in odds.

Suppose a model is

where age is measured in years and complaints is the number of complaints. Holding complaints constant, one additional year of age multiplies the odds by [[M:exp(0.08) ≈ 1.083]], or approximately 8.3%. One additional complaint multiplies the odds by [[M:exp(0.70) ≈ 2.01]]. These are changes in odds, not percentage-point changes in probability.

Worked Example: Estimating a probability

Assume a customer is 30 years old and has one complaint. The linear predictor is

Therefore,

The estimated probability is approximately 52.5%. At a threshold of 0.50, the customer is classified as positive; at a threshold of 0.60, the customer is classified as negative. The probability itself has not changed—only the decision rule has changed.

3.3 Assumptions and practical extensions

Logistic regression does not require normally distributed predictors, equal predictor variances, or a normally distributed error term. Important practical assumptions and checks include:

• observations should be independent, unless dependence is explicitly modeled;
• the logit should be approximately linear in continuous predictors;
• extreme multicollinearity should be investigated;
• influential observations and complete or quasi-complete separation should be examined;
• the sample should contain sufficient events for the model complexity;
• missing-data handling should be conducted within the validation design.

Nonlinearity can be addressed with transformations, splines, polynomial terms, or carefully chosen interactions. Regularized logistic regression commonly uses a penalized objective. For example, a generic objective is

for lasso regularization, or

for ridge regularization. Here [[M:λ ≥ 0]] controls penalty strength. The intercept is commonly not penalized. Regularization can reduce variance, but it does not automatically remove bias or guarantee causal interpretation.

4. Multiclass Classification

When the outcome has [[M:K > 2]] categories, the analyst may use one-vs-rest, one-vs-one, or a direct multiclass model.

Multinomial logistic regression

Choose class [[M:K]] as a reference category. For each other class [[M:k = 1,…,K−1]], model

The corresponding class probabilities are

for [[M:k = 1,…,K−1]], and

where [[M:η_k]] is the linear predictor for class [[M:k]]. The probabilities are nonnegative and sum to 1. A coefficient compares the log-odds of class [[M:k]] with the reference class, holding other predictors constant.

In one-vs-rest classification, a separate binary model is fitted for each class. This can be practical and flexible, but independently estimated probabilities may not always form a coherent probability distribution without additional normalization or calibration. Multiclass performance can be summarized with a confusion matrix, macro-averaged metrics, micro-averaged metrics, or class-specific results.

5. From Probabilities to Decisions

A binary decision rule with threshold [[M:t]] is

The threshold should reflect the costs of errors, operational capacity, and the consequences of action. If a false negative costs [[M:C_FN]] and a false positive costs [[M:C_FP]], and these are the only costs, classify as positive when

This result assumes the probabilities are calibrated, the costs are correctly specified, and action is available for every case. If the positive action has a benefit or there are capacity constraints, a broader decision analysis is required.

For example, if a missed positive case costs 9 units and a false alarm costs 1 unit, the cost-based threshold is [[M:1/(1+9) = 0.10]]. This does not mean that 0.10 is universally optimal. It means that, under the stated assumptions, a relatively low threshold is justified because false negatives are much more costly.

6. Confusion Matrices and Performance Measures

For binary classification, predictions can be arranged as follows:

The total number of observations is [[M:N = TP + TN + FP + FN]]. The principal measures are:

Accuracy is the proportion classified correctly. Precision answers: among predicted positives, what proportion are truly positive? Recall, also called sensitivity or true-positive rate, answers: among actual positives, what proportion were detected? Specificity is the true-negative rate. F1 is the harmonic mean of precision and recall and is defined when both are meaningful and their sum is nonzero.

Worked Example: Confusion-matrix metrics

Suppose a fraud model produces [[M:TP = 80]], [[M:FP = 20]], [[M:FN = 40]], and [[M:TN = 860]]. Then [[M:N = 1000]].

• Accuracy = (80 + 860)/1000 = 0.94, or 94%.
• Precision = 80/(80 + 20) = 0.80, or 80%.
• Recall = 80/(80 + 40) = 0.667, or 66.7%.
• Specificity = 860/(860 + 20) = 0.977, or 97.7%.
• F1 = 2(0.80 × 0.667)/(0.80 + 0.667) ≈ 0.727.

The high accuracy is partly explained by the large number of true negatives. If detecting fraud is the priority, recall and the cost of missed fraud deserve close attention.

Class imbalance makes accuracy particularly dangerous. If only 1% of transactions are fraudulent, a model that predicts “not fraud” for every transaction has 99% accuracy but zero recall for fraud. The baseline prevalence, class-specific metrics, and business costs should therefore accompany accuracy.

7. ROC Curves, Precision–Recall Curves, and AUC

Changing the threshold changes the confusion matrix. A receiver operating characteristic (ROC) curve plots the true-positive rate against the false-positive rate across thresholds:

The area under the ROC curve, ROC-AUC, can be interpreted as the probability that a randomly selected positive observation receives a higher score than a randomly selected negative observation, subject to handling of ties. An AUC of 0.5 corresponds to random ranking in an idealized setting; 1.0 represents perfect ranking. AUC measures ranking discrimination, not calibration and not necessarily operational value.

Precision–recall curves plot precision against recall across thresholds. They are often more informative than ROC curves when the positive class is rare because precision directly reflects false discoveries among predicted positives. The baseline precision for an uninformative system is related to positive prevalence, although the exact curve depends on the scoring procedure.

A model with a higher ROC-AUC may still be inferior at the operating threshold required by an organization. Analysts should report threshold-specific confusion matrices and metrics, not AUC alone.

8. Probability Calibration

A calibrated model satisfies the approximate condition that, among observations assigned probability [[M:q]], the event occurs at frequency [[M:q]]. Thus, among 1,000 cases assigned probability 0.20, approximately 200 should be positive in a sufficiently large and representative group.

Calibration can be assessed using reliability diagrams, calibration intercepts and slopes, and proper scoring rules. Two common probability scores are:

and the logarithmic loss

The Brier score ranges from 0 upward for binary outcomes, with lower values better. Log loss strongly penalizes confident incorrect predictions and requires probabilities strictly between 0 and 1 in its unmodified form.

Calibration may deteriorate when the deployment population differs from the training population, when prevalence changes, or when a flexible model overfits. Platt scaling and isotonic regression are common post-training calibration approaches. Calibration must be fitted on data separate from the data used to train the underlying model, often through nested cross-validation or a dedicated calibration set.

Discrimination and calibration are distinct. A model can rank cases correctly but systematically overstate risk. Conversely, a model can be well calibrated but have limited ability to distinguish individuals. A responsible evaluation reports both.

9. Model Development and Validation Workflow

A professional classification project normally follows these steps:

1. Define the decision and outcome. Specify the prediction time, positive class, action, time horizon, and consequences of errors.
2. Audit the data. Check missingness, duplicates, label quality, class prevalence, measurement timing, and potential leakage.
3. Partition appropriately. Use a temporal split for time-dependent deployment where possible. Use stratification when random splitting is appropriate and the outcome is imbalanced.
4. Create a baseline. Compare against prevalence, a simple rules-based system, or an intercept-only model.
5. Preprocess within training data. Fit imputation, scaling, encoding, feature selection, and resampling procedures without using validation information.
6. Fit candidate models. Logistic regression offers interpretability; tree ensembles, support-vector methods, and neural networks may capture nonlinear structure.
7. Tune and validate. Use cross-validation or a time-based validation scheme. Keep a final test set untouched until model selection is complete.
8. Assess discrimination, calibration, and stability. Include confidence intervals or resampling-based uncertainty where feasible.
9. Select an operating threshold. Consider costs, capacity, fairness, and intervention effectiveness.
10. Monitor deployment. Track data drift, prevalence, calibration, error rates, and performance across relevant groups.

Preprocessing is part of the model. For example, calculating a mean for imputation using the entire dataset leaks information from validation observations. The same principle applies to target encoding, feature selection, and oversampling. Pipelines that fit transformations only on training folds reduce this risk.

10. Illustrative Case: Customer Churn Intervention

Background

A fictional subscription company wants to identify customers likely to cancel within the next 30 days. The retention team can contact only 2,000 of 20,000 customers each week. A contact has a financial cost, while failing to contact a customer who would otherwise cancel has a larger expected loss.

Problem

The company initially uses a 0.50 threshold. Because churn prevalence is 8%, few customers exceed this threshold. The resulting recall is low, and the retention team believes the model is missing too many vulnerable customers.

Analysis

Analysts define churn using a future 30-day window and exclude variables recorded after the prediction date. Predictors include tenure, recent usage decline, service contacts, payment history, and plan type. Logistic regression provides an interpretable baseline, while a gradient-boosted model provides a nonlinear comparison. Both models are evaluated using a temporal split: earlier months for training and a later month for testing.

The boosted model produces higher ROC-AUC, but its raw probabilities are overconfident. Calibration on a separate validation period improves the Brier score. At a threshold of 0.20, recall rises substantially, but the number of predicted positives exceeds the team’s capacity. The company therefore ranks customers by calibrated probability and contacts the top 2,000, while monitoring the realized churn rate and treatment response.

Approach and solution

The final process uses calibrated probabilities, a capacity-based selection rule, and an experiment to estimate whether contact actually reduces churn. The model is not credited for customers who were predicted accurately but would not have responded to intervention. A randomized holdout group is retained to measure incremental business impact.

Outcome

The model identifies a high-risk segment with useful lift over the baseline. However, performance varies across plan types because usage data are less complete for some customers. The company adds missingness indicators, reviews fairness across groups, and schedules monthly calibration checks.

Learning points

• A threshold should reflect capacity and intervention economics, not convention alone.
• A high-risk prediction is not the same as a guaranteed treatment benefit.
• Temporal validation is important when customer behavior and product conditions change.
• Calibration matters when probabilities guide prioritization.
• Monitoring and experimentation are necessary after deployment.

11. Ethical, Governance, and Managerial Considerations

Classification can allocate credit, insurance, employment opportunities, medical attention, or public resources. Errors may therefore have unequal consequences. Governance should address:

• Purpose limitation: use data for a defined, legitimate purpose.
• Transparency: document variables, training population, outcome definition, threshold, and limitations.
• Data protection: minimize personal data, control access, and retain data only as justified.
• Fairness: compare error rates, calibration, selection rates, and benefits across relevant groups, while recognizing that fairness criteria can conflict.
• Human oversight: provide review and appeal mechanisms for consequential decisions.
• Accountability: assign ownership for approval, monitoring, incident response, and model retirement.
• Security: protect features, predictions, and model interfaces from unauthorized use or manipulation.

A protected attribute should not simply be removed and assumed to eliminate unfairness. Correlated variables can reproduce historical disadvantage, and excluding the attribute may prevent meaningful fairness auditing. Legal requirements differ by jurisdiction; analysts must consult applicable privacy, anti-discrimination, consumer-protection, and sector-specific rules.

Probability estimates should not be represented as objective facts. They are conditional estimates based on data, definitions, assumptions, and a population. In managerial communication, decision-makers should receive uncertainty, operating conditions, and known failure modes rather than only a performance headline.

12. Contemporary Developments and Best Practices

Modern classification systems increasingly combine interpretable statistical baselines with more flexible machine-learning models. Explainability tools can help describe influential features, but an explanation of model association is not automatically a causal explanation. Generative AI may assist with feature documentation, data-quality investigation, or analyst workflow, but its outputs require verification and should not silently determine labels or consequential decisions.

Best practices include:

• establish a meaningful baseline before adopting complex methods;
• report prevalence and class-specific metrics;
• separate ranking quality, calibration, and decision utility;
• select thresholds using explicit costs, capacity, and stakeholder input;
• validate using data that resemble deployment conditions;
• use confidence intervals or repeated resampling to communicate uncertainty;
• inspect subgroup and intersectional performance;
• document data lineage, model versions, and threshold changes;
• monitor drift, calibration, and realized outcomes after deployment;
• reassess whether the intervention itself remains effective.

Chapter Summary

Classification estimates the probability that an observation belongs to one or more classes and then applies a decision rule. Logistic regression models binary log-odds as a linear function of predictors and converts them to probabilities through the logistic function. Coefficients describe changes in log-odds; exponentiated coefficients are odds ratios.

Multinomial logistic regression extends the framework to multiple classes, while one-vs-rest and other approaches provide alternative strategies. A threshold converts probabilities into decisions, and its selection should reflect error costs, capacity, and consequences. Confusion matrices support calculation of accuracy, precision, recall, specificity, and F1, but no single metric is sufficient in every context.

ROC-AUC measures ranking discrimination, whereas calibration evaluates whether predicted probabilities correspond to observed frequencies. Brier score and log loss assess probabilistic quality. Sound projects define the outcome carefully, prevent leakage, validate realistically, monitor deployment, and address fairness, privacy, transparency, and accountability. The most useful classifier is not necessarily the one with the highest AUC; it is the one whose probability estimates and decisions create reliable, responsible value in the intended setting.

Key Terms

Classification: Prediction of a categorical outcome.

Binary classification: Classification with two possible outcome classes.

Multiclass classification: Classification with more than two possible classes.

Positive class: The outcome designated as the event of interest.

Probability estimate: A numerical estimate of the likelihood of an outcome conditional on observed predictors.

Odds: The ratio of event probability to non-event probability.

Logit: The natural logarithm of the odds.

Logistic regression: A model that relates predictors linearly to binary log-odds.

Odds ratio: The multiplicative change in odds associated with a predictor change.

Threshold: A cutoff used to convert a probability into a predicted class.

Confusion matrix: A table comparing actual and predicted classes.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positives correctly detected.

Specificity: The proportion of actual negatives correctly identified.

F1 score: The harmonic mean of precision and recall.

ROC-AUC: The area under the receiver operating characteristic curve.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Brier score: Mean squared error between predicted probabilities and binary outcomes.

Data leakage: Use of information during training that would not be available at prediction time.

Class imbalance: A substantial difference in the prevalence of outcome classes.

Review Questions

Short Answer Questions

1. What is the difference between a probability estimate and a classification decision?
2. Define odds and explain how they differ from probability.
3. What does an odds ratio of 0.70 mean, holding other variables constant?
4. Distinguish precision from recall.
5. Why can accuracy be misleading for a highly imbalanced outcome?

Long Answer Questions

1. Explain the logistic-regression model, including its logit form, probability form, assumptions, and coefficient interpretation.
2. Compare ROC-AUC and calibration. Explain why a model may perform well on one and poorly on the other.
3. Discuss how threshold selection should incorporate error costs, operational capacity, and stakeholder objectives.
4. Describe a robust workflow for developing and validating a classification model.
5. Evaluate the ethical and governance issues that arise when classification is used for consequential decisions.

Analytical / Application Questions

1. A model produces 120 true positives, 30 false positives, 40 false negatives, and 810 true negatives. Calculate accuracy, precision, recall, specificity, and F1 score, and explain which metric would matter most if missed positive cases were very costly.
2. A calibrated model assigns a customer a 0.18 probability of default. A false positive costs 2 units and a false negative costs 8 units. Using the cost-based threshold formula, determine the decision and explain one assumption behind your calculation.
3. A churn model has excellent ROC-AUC but assigns probabilities near 0.90 to cases that churn only 0.60 of the time. Diagnose the problem, recommend an evaluation method, and propose a practical remedy before deployment.

Multiple Choice Questions with Answers

Question 1

Which quantity is modeled as a linear function of predictors in standard binary logistic regression?

A. The probability itself

B. The odds ratio

C. The log-odds of the positive outcome

D. The confusion-matrix accuracy

Correct Answer: C. The log-odds of the positive outcome

Question 2

If the probability of an event is 0.20, what are its odds?

A. 0.05

B. 0.20

C. 0.25

D. 0.80

Correct Answer: C. 0.25

Question 3

A model has TP = 80 and FP = 20. What is its precision?

A. 0.667

B. 0.80

C. 0.88

D. 0.95

Correct Answer: B. 0.80

Question 4

Which metric measures the proportion of actual positive cases that are detected?

A. Precision

B. Specificity

C. Recall

D. Accuracy

Correct Answer: C. Recall

Question 5

What does ROC-AUC primarily measure?

A. Probability calibration

B. Ranking discrimination

C. The cost of false positives

D. The prevalence of the positive class

Correct Answer: B. Ranking discrimination

Question 6

A calibrated model assigns probability 0.30 to a large group. Approximately what event frequency should be observed in that group?

A. 0.03

B. 0.30

C. 0.70

D. 1.30

Correct Answer: B. 0.30

Question 7

If a false positive costs 1 unit and a false negative costs 9 units, what cost-based threshold follows from [[M:t = C_FP/(C_FP + C_FN)]]?

A. 0.01

B. 0.10

C. 0.50

D. 0.90

Correct Answer: B. 0.10

Question 8

Which practice best prevents preprocessing leakage during cross-validation?

A. Fit transformations using the full dataset before splitting

B. Fit transformations separately within each training fold

C. Use the test labels to select features

D. Tune the threshold on the final test set

Correct Answer: B. Fit transformations separately within each training fold

Question 9

Why can a classifier with 99% accuracy be unhelpful when the positive class prevalence is 1%?

A. Accuracy cannot be calculated for binary outcomes

B. It may predict every case as negative and detect no positives

C. Accuracy is always lower than recall

D. The positive class is too common

Correct Answer: B. It may predict every case as negative and detect no positives

Question 10

What is a principal interpretation of an odds ratio of exp(0.70) for a one-unit predictor increase?

A. The probability increases by exactly 70 percentage points

B. The odds are multiplied by approximately 2.01

C. The odds decrease by approximately 70%

D. The model becomes perfectly calibrated

Correct Answer: B. The odds are multiplied by approximately 2.01