Classification Models

Introduction and context

Classification is the task of assigning an observation to one of a finite set of categories. In business analytics, the categories may be churn or no churn, high-risk or low-risk, fraudulent or legitimate, or respond or do not respond. Classification models are central to predictive analytics because many managerial decisions are categorical even when the underlying evidence is numerical.

A classification model estimates the likelihood of each possible outcome from observed features. The model output is commonly a probability or score. A business rule then converts that output into an action. For example, a bank may review an application when the estimated probability of default exceeds a chosen threshold, while a retailer may send an offer when the estimated probability of response is sufficiently high.

This distinction is important: a model estimates risk, but a decision rule determines what the organisation does. The best classification model is therefore not necessarily the one with the highest accuracy. It is the one that produces reliable, useful and governable decisions under the organisation’s costs, benefits, constraints and ethical obligations.

Learning objectives

By the end of this chapter, you should be able to:

• explain binary, multiclass and multilabel classification;
• interpret probabilities and coefficients from logistic regression;
• construct and interpret decision trees;
• distinguish training, validation and test performance;
• calculate and interpret a confusion matrix and common classification metrics;
• select a threshold using business consequences rather than accuracy alone;
• recognise class imbalance, overfitting, leakage and calibration problems;
• apply classification methods to churn, credit risk, fraud and response propensity;
• evaluate ethical, regulatory and operational implications of automated classification.

1. The classification problem

Let an observation be represented by a feature vector x = (x₁, x₂, ..., x_p), where the x values are explanatory variables such as tenure, income, transaction amount or number of complaints. Let Y be the target class. In binary classification, Y ∈ {0,1}. A model estimates P(Y=1|x), the conditional probability that the positive class occurs given the observed features.

Examples include:

The target definition must include a time horizon. “Churn” is incomplete unless the analyst specifies, for example, whether churn means cancellation within 30, 60 or 90 days. Likewise, credit default may mean a payment more than 90 days overdue within 12 months. A poorly defined target produces a model that may be statistically correct but operationally meaningless.

Types of classification

• Binary classification: two mutually exclusive classes.
• Multiclass classification: one of more than two mutually exclusive classes, such as low, medium or high risk.
• Multilabel classification: an observation may belong to several classes simultaneously, such as a customer interested in both travel and insurance.
• Ordinal classification: classes have a meaningful order, such as poor, satisfactory and excellent service.

This chapter concentrates on binary classification, while noting how the methods extend to other settings.

2. From probability to decision

A probability is not itself a decision. Suppose a model estimates a 0.40 probability of default. Whether this applicant is declined depends on the cost of approving a likely default, the opportunity cost of rejecting a good customer, regulatory requirements and the institution’s risk appetite.

For a binary decision, define:

• TP: true positives, correctly identified positive cases;
• TN: true negatives, correctly identified negative cases;
• FP: false positives, negative cases incorrectly classified as positive;
• FN: false negatives, positive cases incorrectly classified as negative.

A threshold t converts a predicted probability p into a class prediction:

Here ŷ is the predicted class, p is the estimated probability of class 1, and t is the decision threshold, with 0 ≤ t ≤ 1. The commonly used threshold 0.5 is not universally appropriate. Lowering t generally identifies more positive cases, increasing sensitivity but also usually increasing false positives.

Confusion matrix

Important metrics are:

Accuracy is the proportion of all predictions that are correct. Precision answers, “Of the cases predicted positive, how many are truly positive?” Recall answers, “Of the truly positive cases, how many were found?” Specificity measures the proportion of true negatives correctly rejected. F1 is the harmonic mean of precision and recall and is useful when both matter, although it does not include true negatives or monetary costs.

These metrics require nonzero denominators. They are threshold-dependent except for metrics such as the area under the receiver operating characteristic curve (ROC AUC), which evaluates ranking over many thresholds. ROC AUC can be interpreted as the probability that a randomly selected positive case receives a higher score than a randomly selected negative case, subject to the usual handling of ties.

Worked example: threshold consequences

Assume a fraud model is evaluated on 1,000 transactions. There are 100 fraudulent transactions. At a selected threshold, it identifies 80 frauds and incorrectly flags 45 legitimate transactions. Thus TP = 80, FN = 20, FP = 45 and TN = 855.

Accuracy is (80 + 855) ÷ 1,000 = 0.935, or 93.5%.

Precision is 80 ÷ (80 + 45) = 0.64, or 64.0%.

Recall is 80 ÷ (80 + 20) = 0.80, or 80.0%.

Specificity is 855 ÷ (855 + 45) = 0.95, or 95.0%.

The model is effective at finding fraud, but 36% of flagged transactions are legitimate. If every alert requires a costly manual investigation, this false-positive rate may be operationally unacceptable even though accuracy is high.

3. Logistic regression

Logistic regression is a supervised classification method that models the probability of a binary outcome. It is widely used because it is comparatively interpretable, computationally efficient and suitable for probability estimation.

A linear predictor is defined as:

where β_0 is the intercept, β_j is the coefficient for feature x_j, and p is the number of features. Logistic regression transforms z into a probability using the logistic function:

The probability is always between 0 and 1. The equivalent log-odds form is:

The left side is the logit, or natural logarithm of the odds. Odds are p ÷ (1 − p). The coefficients are additive on the log-odds scale. Holding other variables constant, increasing x_j by one unit multiplies the odds by e^β_j. Thus e^β_j is the odds ratio associated with a one-unit increase in x_j, provided the model specification and units remain fixed.

Interpretation example

Suppose a churn model contains monthly complaints and contract length. The coefficient for complaints is 0.30. The odds ratio is e^0.30 ≈ 1.35. Holding other variables constant, one additional complaint multiplies the estimated odds of churn by approximately 1.35, or increases the odds by about 35%. This is not the same as saying that the probability increases by 35 percentage points. The change in probability depends on the customer’s starting probability and the values of all predictors.

If contract length is represented by a binary variable with coefficient −0.80, the odds ratio is approximately 0.45. Customers in that contract category have estimated churn odds about 55% lower than the reference category, conditional on the other model variables.

Estimation and assumptions

Coefficients are commonly estimated by maximum likelihood. The method chooses coefficients that make the observed outcomes most plausible under the Bernoulli model. For observation i, with outcome y_i ∈ {0,1} and predicted probability p_i, the likelihood contribution is:

The full likelihood is the product of all contributions, assuming observations are conditionally independent:

In practice, software maximises the log-likelihood. Logistic regression does not require normally distributed predictors or equal variances, but important assumptions and considerations include:

• the outcome is appropriately defined as binary for the model;
• observations are independent, or dependence is explicitly modelled;
• the log-odds relationship is approximately linear in continuous predictors;
• severe multicollinearity is avoided or managed;
• the sample contains sufficient events for the model complexity;
• there is no complete or quasi-complete separation, where predictors perfectly distinguish classes.

Categorical predictors are generally represented using indicator variables with a stated reference category. Continuous variables may require transformations, splines or interaction terms when their relationship with log-odds is not approximately linear.

Strengths and limitations

Logistic regression offers transparent coefficients, probability outputs, straightforward implementation and a strong baseline for many business problems. It is especially valuable where explanations, auditability and regulatory review matter.

Its limitations include sensitivity to misspecification, difficulty representing complex nonlinear relationships without feature engineering, and possible instability with highly correlated features or rare events. A statistically significant coefficient does not establish causation. It describes association conditional on the included variables.

4. Decision trees

A decision tree recursively partitions observations using rules such as “tenure < 12 months” or “transaction amount > 500.” Each internal node contains a split, each branch represents an outcome of the split, and each leaf provides a predicted class or probability.

A tree chooses splits that make the resulting groups more homogeneous. Two common impurity measures are Gini impurity and entropy. If a node contains class proportions p_1, p_2, ..., p_K, its Gini impurity is:

For binary classification, maximum Gini impurity occurs when the classes are equally mixed, while impurity is zero when the node contains one class only. Entropy is:

The logarithm base 2 expresses entropy in bits. A candidate split is assessed by the weighted impurity of its child nodes. If parent node P is divided into child nodes j with n_j observations, the weighted impurity is:

where I_j is the impurity of child j and n_P is the number of observations in the parent. The preferred split often minimises I_split, or equivalently maximises impurity reduction.

Tree construction and pruning

A fully grown tree can memorise training data. Controls such as maximum depth, minimum observations per leaf, minimum split size and maximum number of leaves reduce overfitting. Cost-complexity pruning balances fit and tree size:

Here R(T) is the tree’s empirical error or impurity-based risk, |T| is the number of terminal leaves, and α ≥ 0 penalises complexity. The precise implementation depends on the software and loss function.

Trees are easy to communicate, can capture nonlinearities and interactions automatically, and require little scaling of numerical features. However, individual trees can be unstable: small data changes may produce different structures. Their probability estimates may also be poorly calibrated, particularly in small leaves. Deep trees are especially prone to overfitting.

Random forests and gradient-boosted trees improve predictive performance by combining many trees. A random forest averages trees built from bootstrap samples and randomly selected feature subsets. Boosting builds trees sequentially, with later trees focusing on errors made earlier. These ensembles often perform strongly on structured business data, but they require careful validation and may be less transparent than a single tree. Feature importance measures should not automatically be interpreted as causal effects.

5. Preparing data for classification

Good modelling begins before algorithm selection.

Target and observation design

Define the observation date, prediction horizon and information-available date. Features must be available at the time the decision is made. A churn model predicting next month’s churn must not use a cancellation request recorded after the prediction date.

Missing values

Missingness may be informative, but it must be investigated. Options include imputation, explicit “missing” categories, missingness indicators or models that handle missing values directly. The imputation method must be fitted on training data and then applied to validation and test data to avoid leakage.

Categorical and numerical variables

Logistic regression commonly requires categorical encoding. Standardisation is not essential for the mathematical validity of ordinary logistic regression, but it can assist regularisation and comparison across variables. Tree methods generally do not require standardisation because splits depend on ordering rather than scale.

Class imbalance

If 1% of transactions are fraudulent, a classifier that predicts “legitimate” for every transaction achieves 99% accuracy while finding no fraud. Remedies include threshold adjustment, class-weighted loss, carefully designed resampling and metrics such as precision, recall, PR AUC and cost-weighted performance. Resampling must occur within the training process; validation and test sets should preserve the intended deployment distribution unless a justified alternative evaluation design is used.

Leakage

Data leakage occurs when the model uses information that would not be available at prediction time or information directly derived from the target. Leakage can produce impressive validation results and poor operational performance. Temporal leakage, target encoding performed before splitting, and post-outcome variables are common examples.

6. Evaluation and validation

A sound evaluation design separates model development from final assessment.

• Training data are used to fit model parameters.
• Validation data are used to select algorithms, features, hyperparameters and thresholds.
• Test data are reserved for an unbiased final estimate.

For small datasets, cross-validation repeatedly partitions the training data into fitting and validation portions. In time-dependent business data, a temporal split is often more realistic than random cross-validation: earlier observations are used for training and later observations for evaluation.

Performance should be reported with uncertainty where feasible, through confidence intervals, repeated resampling or performance across time periods and important customer segments. A model that performs well overall but poorly for a material subgroup requires investigation.

Calibration

A model is calibrated when predicted probabilities correspond to observed frequencies. Among cases assigned a probability near 0.70, approximately 70% should experience the outcome over repeated comparable groups. Discrimination and calibration are different: a model can rank cases well but produce probabilities that are systematically too high or too low. Calibration plots, reliability tables and methods such as Platt scaling or isotonic regression may be used, with calibration performed on data separate from model fitting.

Choosing a metric

Metric selection should follow the decision context:

Lift compares the response rate in a selected scored group with the overall response rate. Gains and lift are useful for ranking campaign prospects, but they do not prove that contacting a customer caused the response. For that purpose, treatment-control experiments or uplift modelling are more appropriate.

7. Threshold selection using business consequences

Suppose a positive prediction leads to an action. Let C_FP be the cost of a false positive, C_FN the cost of a false negative, and assume correct classifications have zero incremental cost for simplicity. If the estimated probability of a positive outcome is p, the expected cost of predicting positive is:

The expected cost of predicting negative is:

Predict positive when the first cost is no greater than the second:

Solving gives the cost-based threshold:

This formula assumes calibrated probabilities, known and constant costs, and no additional benefits or constraints. If a true-positive benefit, true-negative benefit, treatment effect, capacity limit or differentiated customer value exists, expected utility should be modelled directly rather than relying on this simplified threshold.

Worked threshold example

Suppose a missed fraud costs 900 monetary units and an unnecessary investigation costs 100. The cost-based threshold is 100 ÷ (100 + 900) = 0.10. Under the stated assumptions, a transaction with estimated fraud probability of at least 10% should be investigated. The low threshold reflects the high cost of missing fraud. In practice, investigation capacity may require ranking alerts and selecting only the highest-risk cases.

For churn, the relevant comparison may be expected retained margin from an intervention minus intervention cost, not simply the probability of churn. A customer with 0.70 churn probability may still be a poor target if the customer has low value or cannot be influenced. This is why propensity-to-churn and uplift are distinct analytical problems.

8. Applied case: illustrative telecom churn programme

Background

A fictional subscription company, Northstar Mobile, experiences increasing monthly cancellations. Management wants a model to prioritise customers for retention offers. The retention team can contact only 8,000 of 100,000 customers each month.

Problem definition

The target is cancellation within the next 30 days, defined using account status at the monthly scoring date. Features include tenure, plan type, recent complaints, service interruptions, payment history, usage change and recent price changes. Only information recorded before the scoring date is eligible.

Analysis and approach

The analytics team first establishes a temporal split: earlier months for training, a later month for validation and the most recent month for testing. Logistic regression provides an interpretable baseline. A pruned decision tree is developed for communication with managers, while a gradient-boosted tree is tested for additional predictive power.

The team evaluates recall, precision, calibration, lift and expected financial value. It does not select a threshold of 0.5 automatically. Instead, it ranks customers by predicted risk and tests the top 8,000 because contact capacity is fixed. It then conducts a randomised retention experiment within the selected group: some customers receive the offer and others form a control group.

Outcome

The model identifies a high-risk segment, but the experiment shows that many high-risk customers would leave regardless of the offer. A smaller segment with both high churn risk and positive incremental response produces the greatest retained margin. Northstar therefore uses the classification model for risk prioritisation and the experiment to estimate treatment value. It monitors calibration monthly because pricing changes and competitor actions alter customer behaviour.

Learning points

1. A precise target and prediction date are essential.
2. Capacity constraints can make ranking more relevant than a universal threshold.
3. Predicting churn is not the same as predicting who can be persuaded to stay.
4. Experimental evidence is needed to estimate causal impact.
5. Model governance includes monitoring drift, subgroup performance and financial outcomes.

9. Ethical, legal and governance considerations

Classification can allocate opportunities, impose restrictions or trigger surveillance. Credit and insurance decisions may be subject to specific laws, supervisory expectations and requirements for notice or explanation. Organisations should identify the lawful basis for data use, minimise unnecessary personal data, protect sensitive information and retain auditable records of model versions and decisions.

Fairness must be assessed in context. Relevant measures may include group-specific selection rates, false-positive rates, false-negative rates, calibration and equal opportunity. These measures can conflict mathematically, particularly when groups have different base rates. No single fairness metric resolves the underlying policy question. Analysts should involve compliance, domain experts and affected stakeholders rather than treating fairness as a purely technical adjustment.

Proxy discrimination is possible even when protected attributes are excluded: location, language, purchasing patterns or device information may correlate with protected characteristics. Removing a protected variable does not guarantee fairness. Conversely, protected attributes may be needed for auditing, subject to lawful and carefully controlled use.

Human review is not automatically a safeguard. Reviewers may over-trust model scores or apply inconsistent standards. Effective governance specifies escalation rules, documentation, appeal processes, access controls, monitoring responsibilities and procedures for correcting data or decisions.

10. Contemporary practice and best practices

Modern classification systems commonly combine interpretable baselines with ensemble models, probability calibration, automated monitoring and explainability tools. Explainability methods can describe model behaviour, but they do not prove causality or guarantee fairness. Generative AI may assist with documentation or analyst workflows, but the predictive model, data lineage and decision controls still require conventional validation.

Recommended practice includes:

• define the outcome, horizon, action and success measure before modelling;
• use a simple interpretable baseline;
• prevent leakage through time-aware feature design and preprocessing pipelines;
• preserve a genuinely untouched test set;
• report confusion-matrix metrics at operationally relevant thresholds;
• assess calibration, drift and subgroup performance;
• choose thresholds using costs, benefits, capacity and customer value;
• document assumptions, exclusions, versions and limitations;
• monitor outcomes after deployment and establish retraining criteria;
• test interventions experimentally when causal impact matters.

Challenges and limitations

Classification models are limited by historical data, measurement error, changing behaviour and omitted variables. A model may reproduce past organisational bias. Rare events make estimates unstable. Correlated records can create overly optimistic validation if the same customer appears in both training and test sets. Concept drift occurs when the relationship between features and outcomes changes; data drift occurs when the feature distribution changes. Operational changes, such as a new fraud-control policy, can also alter the labels used for future training.

A model score should therefore be treated as decision support within a controlled process, not as an unquestionable truth. Clear ownership, monitoring and periodic review are as important as algorithm selection.

Chapter Summary

Classification models predict categorical outcomes and support decisions in churn, credit, fraud and marketing. Logistic regression estimates binary probabilities through the logistic function and offers interpretable log-odds coefficients. Decision trees use recursive rules and impurity reduction; pruning and ensemble methods help manage overfitting. Evaluation requires confusion-matrix measures, discrimination, calibration and validation designs that reflect deployment.

The central managerial principle is that probability estimation and action selection are separate. Thresholds should reflect false-positive and false-negative consequences, capacity, customer value and operational constraints. Accuracy alone is often misleading, especially with imbalanced outcomes. Sound practice also requires precise target definitions, leakage prevention, temporal validation, fairness assessment, privacy protection, documentation and post-deployment monitoring.

Key Terms

Classification: Assignment of observations to predefined categories.

Binary classification: Classification involving two possible outcome classes.

Multiclass classification: Classification involving more than two mutually exclusive classes.

Logistic regression: A model that estimates a binary outcome probability through the logistic function.

Logit: The natural logarithm of the odds of an event.

Odds ratio: The multiplicative change in odds associated with a one-unit predictor change.

Decision tree: A sequence of feature-based rules ending in predicted classes or probabilities.

Gini impurity: A measure of class mixture in a node.

Entropy: An impurity measure based on information content.

Overfitting: Learning noise or idiosyncrasies of training data rather than general patterns.

Confusion matrix: A table of actual and predicted classes.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives correctly identified.

Specificity: The proportion of actual negatives correctly identified.

Calibration: Agreement between predicted probabilities and observed frequencies.

Threshold: A cutoff used to convert a score or probability into an action or class.

Class imbalance: A substantial difference in the frequency of outcome classes.

Data leakage: Use of information unavailable at prediction time or derived from the target.

Concept drift: Change over time in the relationship between predictors and outcomes.

Uplift: The incremental effect of an intervention relative to an appropriate control condition.

Review Questions

5 Short Answer Questions

1. What is the difference between a predicted probability and a classification threshold?
2. Interpret an odds ratio of 1.40 in a logistic regression model.
3. Why can accuracy be misleading for fraud detection?
4. What is the purpose of pruning a decision tree?
5. Distinguish calibration from discrimination.

5 Long Answer Questions

1. Explain logistic regression using the probability and log-odds formulations, including the interpretation of coefficients.
2. Compare logistic regression and decision trees in terms of interpretability, assumptions, nonlinear relationships and overfitting.
3. Describe a rigorous train-validation-test workflow for a time-dependent churn problem.
4. Explain how class imbalance affects evaluation and identify suitable responses.
5. Discuss the ethical and governance issues involved in automated credit classification.

3 Analytical / Application Questions

1. A fraud model has 70 true positives, 30 false negatives, 50 false positives and 850 true negatives. Calculate accuracy, precision, recall and specificity. Which metric would you prioritise if missed fraud is much more costly than investigation?
2. A screening decision has a false-positive cost of 120 monetary units and a false-negative cost of 480. Using the simplified cost-based threshold, calculate the threshold and explain the assumptions behind it.
3. Design a classification programme for a retailer’s response-propensity problem. Specify the target horizon, features, validation approach, threshold or capacity rule, success metric and one experiment needed to test whether targeting causes incremental sales.

Multiple Choice Questions with Answers

Question 1

Which quantity does logistic regression directly estimate for a binary outcome?

A. The conditional probability of the positive class

B. The number of observations in the training set

C. The impurity of every decision-tree node

D. The cost of a false positive

Correct Answer: A. The conditional probability of the positive class

Question 2

A model identifies 80 true positives and 20 false negatives. What is its recall?

A. 20%

B. 40%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 3

Why might a threshold lower than 0.5 be appropriate in fraud detection?

A. It guarantees perfect calibration

B. The cost of missing fraud may be much greater than the cost of investigation

C. It removes all false positives

D. It makes the target variable continuous

Correct Answer: B. The cost of missing fraud may be much greater than the cost of investigation

Question 4

Which measure is used to assess class mixture within a decision-tree node?

A. Gini impurity

B. Mean absolute error only

C. R-squared

D. Variance inflation factor only

Correct Answer: A. Gini impurity

Question 5

What is data leakage?

A. Randomly splitting data into training and test sets

B. Using information unavailable at prediction time

C. Standardising numerical variables

D. Removing duplicate rows

Correct Answer: B. Using information unavailable at prediction time

Question 6

If C_FP = 100 and C_FN = 900, what is the simplified cost-based threshold?

A. 0.01

B. 0.10

C. 0.50

D. 0.90

Correct Answer: B. 0.10

Question 7

Which statement best describes precision?

A. The proportion of actual positives that were found

B. The proportion of predicted positives that are truly positive

C. The proportion of actual negatives that were rejected

D. The proportion of all predictions that are correct

Correct Answer: B. The proportion of predicted positives that are truly positive

Question 8

What is the main purpose of a test set?

A. To fit the model coefficients

B. To select every feature repeatedly

C. To provide a final estimate on data not used for model development

D. To increase the number of positive labels

Correct Answer: C. To provide a final estimate on data not used for model development

Question 9

A calibrated group of cases receives predicted probability 0.70. What does calibration imply over repeated comparable groups?

A. Exactly 70% of every individual case will be positive

B. Approximately 70% of the group should experience the outcome

C. The model has 70% accuracy at every threshold

D. The model has no false negatives

Correct Answer: B. Approximately 70% of the group should experience the outcome

Question 10

Why is a churn model not necessarily sufficient to choose retention targets?

A. Churn models cannot use customer data

B. High churn risk does not establish that an intervention will change the outcome

C. Decision trees cannot rank customers

D. Logistic regression cannot estimate probabilities

Correct Answer: B. High churn risk does not establish that an intervention will change the outcome