Classification and Propensity Modelling

Introduction

Classification is a supervised learning task in which a model assigns an observation to one of a set of categories. In business analytics, the categories may represent churn versus retention, loan default versus repayment, response versus non-response, or fraudulent versus legitimate activity. Propensity modelling is closely related: it estimates the likelihood that an individual, account, transaction, or organisation will exhibit a specified behaviour.

A classification model therefore has two outputs. First, it may estimate a probability, such as the probability that a customer will churn. Second, it converts that probability into an operational decision by applying a threshold. This distinction is central. A probability estimate of 0.62 does not automatically mean that a customer should be contacted; the appropriate action depends on campaign cost, expected benefit, capacity, and the consequences of incorrect decisions.

This chapter develops three practical approaches—logistic regression, decision trees, and probability-based classification—and explains evaluation, threshold selection, class imbalance, and misclassification costs.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish classification from regression and propensity estimation;
• explain and interpret logistic regression probabilities and odds;
• construct and interpret a simple decision tree;
• evaluate classifiers using a confusion matrix and suitable metrics;
• select decision thresholds using business costs and benefits;
• identify the effects of class imbalance and apply appropriate remedies;
• assess applications in churn, credit risk, marketing response, and fraud;
• recognise ethical, governance, and operational risks in classification systems.

1. The classification problem

Let X denote a vector of predictor variables and Y denote a categorical outcome. In binary classification, Y has two possible values, conventionally 0 and 1. A model estimates:

Here, p(X) is the conditional probability of class 1 for an observation with predictors X. The estimate is meaningful only relative to a defined outcome, time horizon, population, and observation unit. For example, “probability of churn” must specify whether churn means no purchase for 90 days and whether the unit is a customer or an account.

A threshold t converts probability into a predicted class:

The commonly used threshold of 0.50 is not a universal rule. It is appropriate only under particular assumptions, including comparable consequences of the two error types and a suitable probability model. In many business problems, the threshold should be chosen from costs, benefits, or operational capacity.

Important terminology

• Positive class: the outcome treated as 1, such as churn or fraud.
• Negative class: the outcome treated as 0.
• Propensity score: an estimated probability of a specified outcome or response.
• Feature/predictor: an input variable used by the model.
• Label/target: the observed outcome used for training.
• Score: a model output, often a probability or ranking value.
• Calibration: agreement between predicted probabilities and observed frequencies.
• Discrimination: the ability to rank positive cases above negative cases.

2. The confusion matrix and performance measures

After applying a threshold, predictions can be summarised in a confusion matrix.

From these four counts, several measures are calculated:

Precision answers: “Among cases predicted positive, how many were actually positive?” Recall, also called sensitivity, answers: “Among all actual positives, how many were detected?” Specificity measures the proportion of actual negatives correctly rejected. Each denominator must be non-zero.

The F1 score is the harmonic mean of precision and recall:

It is useful when both precision and recall matter, but it ignores true negatives and does not directly represent financial value. Accuracy can be seriously misleading when classes are imbalanced. If only 2% of transactions are fraudulent, predicting “legitimate” for every transaction gives 98% accuracy but detects no fraud.

A receiver operating characteristic (ROC) curve plots recall against the false-positive rate as the threshold varies. The area under this curve, ROC-AUC, measures ranking discrimination across thresholds. For highly rare positives, a precision–recall curve is often more informative because it focuses on positive detection and the associated false positives. AUC values should not be interpreted as probabilities of business success; they measure a particular statistical property of ranking.

3. Logistic regression for propensity estimation

Logistic regression models the log odds of a binary outcome as a linear function of predictors:

Equivalently, the predicted probability is:

Here, p is the estimated probability of class 1, β₀ is the intercept, βᵢ is the coefficient for predictor xᵢ, and e is the base of the natural logarithm. The logistic transformation ensures that predicted probabilities lie between 0 and 1.

Interpretation

A coefficient is additive on the log-odds scale. Holding other variables constant, increasing xᵢ by one unit multiplies the odds by:

This quantity is the odds ratio. An odds ratio above 1 indicates higher odds of the positive outcome; an odds ratio below 1 indicates lower odds. The odds ratio is not the same as a probability difference. The probability impact depends on the starting probability and the values of the other predictors.

For a binary predictor, an odds ratio of 1.50 means that the odds are estimated to be 50% higher for the group coded 1 than for the reference group, conditional on the model variables. It does not mean that probability rises by 50 percentage points.

Worked example: churn propensity

Suppose a model is:

where A is the number of unresolved service complaints and S equals 1 when the customer experienced a service failure. For a customer with A = 3 and S = 1:

Thus:

The estimated churn propensity is approximately 25.7%. The coefficient for service failure has odds ratio e⁰.90 ≈ 2.46, meaning that, holding the other model variables constant, a service failure is associated with approximately 2.46 times the odds of churn. This is an association, not necessarily a causal effect.

Assumptions and limitations

Logistic regression does not require predictors to be normally distributed, but important conditions remain. Observations should be appropriately independent, the outcome should be correctly defined, and continuous predictors should have an approximately linear relationship with the log odds unless transformations or nonlinear terms are included. Strong multicollinearity can make coefficients unstable. Very small samples, complete separation, missing data, and leakage can also produce unreliable estimates.

Categorical variables require suitable coding, usually with one reference category. Interaction terms may be necessary where the effect of one variable depends on another. Regularisation, such as L1 or L2 penalties, can reduce overfitting, particularly when there are many predictors. Coefficients should be interpreted with domain knowledge and uncertainty measures, not treated as unquestionable causal evidence.

4. Decision trees

A decision tree recursively partitions observations using questions about predictors. A terminal node produces a class prediction or an estimated class probability. For example, a churn tree might first ask whether recent complaints exceed a threshold, then whether usage has declined, and finally whether contract type is monthly or annual.

A common splitting criterion is Gini impurity:

where pⱼ is the proportion of observations in a node belonging to class j, and m is the number of classes. For binary classification, a node containing equal proportions of both classes has Gini impurity 0.5; a pure node has impurity 0.

Entropy is another criterion:

The value 0 log₂ 0 is defined as 0 by continuity. A split is selected when it produces sufficient impurity reduction, subject to constraints such as maximum depth, minimum leaf size, and minimum samples for a split.

If a leaf contains 30 observations, 6 of which are positive, its estimated positive probability is:

A tree may be highly interpretable, but an unrestricted tree can overfit. Pruning, cross-validation, depth constraints, and minimum leaf sizes help control this risk. Trees handle nonlinear relationships and interactions naturally and require little feature scaling. However, a single tree can be unstable: small changes in training data may create a different structure. Tree probabilities may also be poorly calibrated, so calibration should be checked when probabilities drive decisions.

5. Probability-based classification and threshold design

Probability-based classification separates estimation from action. A model estimates p, while a decision-maker chooses whether to act. Consider an intervention with benefit B if the person is positive and contacted, cost C_FP if a negative person is contacted, and cost C_FN if a positive person is not contacted. If the action is taken, its expected net value, ignoring constants associated with no action, is:

Action is preferred when this exceeds the value of no action, which in this simplified setting has a false-negative cost of C_FN:

Rearranging gives the threshold:

This expression assumes that the probability is calibrated, the stated costs are appropriate, and the action has the same benefit and cost structure for all cases. If there is a fixed intervention cost already included in C_FP, it should be represented there. In practice, organisations often estimate expected profit directly for a range of thresholds and select the threshold with the highest validated value.

For example, if the benefit of correctly contacting a positive customer is £80, the false-positive cost is £10, and the false-negative cost is £20, then:

The low threshold reflects the high value of finding positives. It may nevertheless generate too many contacts if staff capacity is limited. In that situation, a top-k policy—contacting the highest-scoring 5%—may be more appropriate than a fixed probability threshold.

6. Class imbalance

Class imbalance occurs when one class is substantially more common than another. It is common in fraud, equipment failure, default, and disease screening. Imbalance can cause a model to prioritise the majority class and can make accuracy misleading.

Good practice includes:

1. Use suitable metrics. Report recall, precision, specificity, balanced accuracy, PR-AUC, and financial outcomes where relevant.
2. Use stratified validation. Preserve class proportions in folds, while ensuring that time-dependent problems are validated chronologically.
3. Apply class weights or cost-sensitive learning. Give greater training importance to costly or rare observations.
4. Resample carefully. Oversampling the minority class or undersampling the majority class must occur within training folds only. Otherwise, information can leak into validation data.
5. Tune the threshold. The best threshold is usually not 0.50.
6. Inspect calibration. Oversampling changes the apparent class prevalence and may require probability correction or post-training calibration.

Synthetic oversampling can be useful, but synthetic cases may be implausible, especially with categorical or highly structured transactional data. Resampling is not a substitute for defining the business decision correctly.

7. Applications

Churn

The target may be cancellation within 30 days. Predictors can include tenure, recent usage, complaints, payment failures, and contract type. The intervention may be a retention offer. Evaluation should consider incremental retention, offer cost, customer value, and the possibility that high-risk customers would have stayed without intervention. A propensity model predicts risk; it does not prove that contacting a customer will prevent churn. Uplift modelling or controlled experiments is needed to estimate treatment effects.

Credit and operational risk

A risk model may estimate probability of default within 12 months. Decisions can include approval, pricing, credit limits, or manual review. False negatives may expose the lender to loss, while false positives may unfairly exclude applicants. Models require documentation, monitoring, fairness testing, adverse-action explanations where required, and careful treatment of rejected applications in later model development.

Marketing response

A response model estimates the likelihood of purchase after an offer. Ranking customers by propensity can improve campaign efficiency, but high propensity does not necessarily mean high incremental response. A customer likely to buy without an offer may be less valuable to target than a persuadable customer. Randomised holdout groups are important for evaluating incremental impact.

Fraud detection

Fraud systems often combine a probability score with rules and investigation capacity. The positive class is rare, labels may arrive late, and fraud patterns change rapidly. Precision at the investigation capacity, monetary loss prevented, investigator workload, and false declines are often more useful than accuracy.

8. Illustrative case: subscription-service churn

Background. A subscription provider wants to reduce monthly cancellations. It has 40,000 historical customer records, including usage, tenure, payment events, complaints, plan type, and a label indicating cancellation within the following 30 days.

Problem. Only 8% of customers churn. The retention team can contact 1,600 customers per month. A blanket campaign is expensive, while missing a high-value customer is costly.

Analysis. Analysts first define the prediction date and ensure that every feature was available before that date. They remove post-cancellation information and split the data chronologically: earlier months for training, a later month for validation, and the most recent month for final testing. This avoids optimistic results caused by future information or temporal leakage.

They fit logistic regression for interpretability and a constrained decision tree for nonlinear patterns. Both models are evaluated using precision–recall curves, calibration plots, recall at the 1,600-customer capacity, and expected campaign value. The tree finds an interaction between falling usage and payment failure; logistic regression represents the same pattern using indicator and interaction variables.

Approach and solution. The final model ranks customers by predicted churn probability. The operating rule is top-1,600 rather than probability ≥ 0.50. A randomised holdout group receives no retention contact, allowing the business to estimate incremental retention rather than merely observed retention among contacted customers. Model scores are calibrated on validation data and monitored monthly.

Outcome. In this illustrative scenario, the campaign produces more retained contribution per contact than blanket targeting, but the model is not treated as a complete solution. The business separately monitors whether offers are being concentrated among vulnerable groups, whether customer complaints are resolved, and whether a retention offer is genuinely beneficial.

Learning points. Correct time alignment is as important as algorithm choice. Capacity determines the action rule. Propensity is not causation. A test-and-control design is needed to assess intervention impact, and governance must continue after deployment.

9. Model development and governance

A professional workflow includes:

• define the decision, target, horizon, and unit of analysis;
• audit data quality, missingness, duplicates, and label reliability;
• prevent target leakage and use time-aware validation where appropriate;
• establish a simple baseline before complex modelling;
• compare discrimination, calibration, stability, cost, and operational capacity;
• document features, exclusions, thresholds, training data, and known limitations;
• monitor data drift, outcome drift, calibration, subgroup performance, and realised value;
• establish human review, appeal, override, and incident procedures.

Fairness deserves explicit attention. A model can have different error rates across groups because of data quality, historical inequality, or differing prevalence. Removing a sensitive attribute does not guarantee fairness because correlated variables may retain group information. Organisations should define the relevant fairness objective, assess legal obligations, check for disparate impact, and involve affected stakeholders. Accuracy improvements do not justify unlawful discrimination, deceptive use, or intrusive data collection.

Privacy controls include data minimisation, access restrictions, retention limits, secure processing, and transparent explanations. Automated decisions may require notification, human intervention, or reasons for adverse outcomes depending on the jurisdiction and application. A score should not be used outside the population and time period for which it was validated without reassessment.

10. Contemporary developments and best practice

Ensemble methods such as random forests and gradient-boosted trees often improve predictive performance by combining many trees, although their additional complexity can reduce transparency. Explainability methods can describe model behaviour, but a feature-attribution explanation is not automatically a causal explanation. Modern machine-learning systems should therefore be evaluated not only by accuracy but also by calibration, robustness, fairness, latency, security, and maintainability.

Best practice is to begin with a clearly defined decision and a transparent baseline. Use logistic regression when interpretability and stable probability estimates are valuable; use trees when nonlinear interactions and operational rules are important; and compare models using a validation design that reflects deployment. Select thresholds using validated costs, benefits, capacity, and risk tolerance. Recalibrate and retrain only under controlled governance, retaining versioned data, code, and performance records.

Chapter Summary

Classification assigns observations to categories, while propensity modelling estimates the probability of a specified outcome. Logistic regression models log odds as a linear function of predictors and provides interpretable odds ratios. Decision trees partition observations into progressively more homogeneous nodes and can represent nonlinear relationships and interactions, but they require controls against overfitting. A probability estimate becomes an operational decision only after a threshold or ranking rule is selected.

Confusion-matrix measures include accuracy, precision, recall, specificity, and F1. ROC-AUC assesses ranking discrimination, whereas precision–recall analysis is often more informative for rare positives. Class imbalance requires appropriate metrics, stratified or time-aware validation, cost-sensitive learning, careful resampling, and threshold selection. Churn, risk, response, and fraud applications differ in their costs, label quality, timing, and governance requirements. Finally, propensity is not causation: interventions should be evaluated with experiments or suitable causal designs, and deployed systems require monitoring, fairness assessment, privacy protection, and documented accountability.

Key Terms

Classification: Assigning an observation to one of two or more outcome categories.

Propensity model: A model estimating the probability of a specified behaviour or outcome.

Binary outcome: A target with two possible classes, commonly 0 and 1.

Logistic regression: A regression model that estimates binary-outcome probabilities through the logit link.

Log odds: The logarithm of odds, where odds equal probability divided by one minus probability.

Odds ratio: The multiplicative change in odds associated with a one-unit predictor change.

Decision tree: A recursive sequence of predictor-based splits ending in prediction leaves.

Gini impurity: A measure of class mixture within a tree node.

Threshold: A probability cut-off used to convert a score into a class decision.

Confusion matrix: A table of true and false positive and negative predictions.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positives that are detected.

Specificity: The proportion of actual negatives correctly identified.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Class imbalance: A substantial difference in the frequency of outcome classes.

Misclassification cost: The financial, operational, or social consequence of an incorrect classification.

Data leakage: Use of information unavailable at prediction time or derived from the target.

Concept drift: Change over time in relationships between predictors and outcomes.

Uplift modelling: Estimation of the incremental effect of an intervention on an individual or group.

Review Questions

5 Short Answer Questions

1. What is the difference between a propensity estimate and a final classification decision?
2. Interpret an odds ratio of 0.70 in a logistic regression model.
3. Why can accuracy be misleading for fraud detection?
4. State the difference between precision and recall.
5. Give two methods for controlling overfitting in a decision tree.

5 Long Answer Questions

1. Explain logistic regression, including the log-odds equation and interpretation of coefficients.
2. Compare logistic regression and decision trees for a churn-modelling project.
3. Discuss how class imbalance affects model evaluation and model training.
4. Explain why threshold selection should reflect business costs and operational capacity.
5. Describe a complete and governed workflow for deploying a credit-risk classifier.

3 Analytical / Application Questions

1. A fraud model detects 90 fraudulent transactions, misses 10, and incorrectly flags 180 legitimate transactions. Calculate recall and precision, and explain which additional business information is required before setting a threshold.
2. A retention campaign has limited capacity. Design an evaluation plan that distinguishes churn prediction from the incremental effect of contacting customers.
3. A decision tree performs well on a random test split but poorly on the next month’s data. Diagnose at least three possible causes and propose corrective actions.

Multiple Choice Questions with Answers

Question 1

Which expression represents the probability produced by a binary logistic regression model?

A. p = 1 / (1 + e^(−z))

B. p = β₀ + β₁x

C. p = z²

D. p = log(z)

Correct Answer: A. p = 1 / (1 + e^(−z))

Question 2

A model predicts 80 customers as positive, of whom 20 are truly positive. What is precision?

A. 20%

B. 25%

C. 60%

D. 80%

Correct Answer: B. 25%

Question 3

Which measure answers the question: among all actual positive cases, what proportion did the model detect?

A. Precision

B. Specificity

C. Recall

D. Accuracy

Correct Answer: C. Recall

Question 4

Why is a 0.50 probability threshold not always appropriate?

A. Probabilities cannot be estimated below 0.50

B. Business costs and benefits of errors may be unequal

C. Decision trees cannot use thresholds

D. Logistic regression produces only rankings

Correct Answer: B. Business costs and benefits of errors may be unequal

Question 5

A pure decision-tree node contains observations from only one class. Its Gini impurity is:

A. 0

B. 0.25

C. 0.50

D. 1

Correct Answer: A. 0

Question 6

Which metric is often especially informative when the positive class is rare?

A. Raw accuracy only

B. Precision–recall analysis

C. Mean absolute error

D. R-squared

Correct Answer: B. Precision–recall analysis

Question 7

In the illustrative churn case, why was a chronological split preferred?

A. It guarantees perfect accuracy

B. It prevents all class imbalance

C. It better reflects prediction on future customers and reduces temporal leakage

D. It eliminates the need for a test set

Correct Answer: C. It better reflects prediction on future customers and reduces temporal leakage

Question 8

An odds ratio greater than 1 for a predictor indicates that, holding other model variables constant, the predictor is associated with:

A. Lower odds of the positive outcome

B. Higher odds of the positive outcome

C. Exactly the same probability for every case

D. A guaranteed causal effect

Correct Answer: B. Higher odds of the positive outcome

Question 9

Which practice helps prevent leakage during oversampling?

A. Oversample before splitting into training and validation data

B. Oversample the complete dataset repeatedly

C. Oversample within training folds only

D. Oversample only the final test set

Correct Answer: C. Oversample within training folds only

Question 10

Why is a randomised holdout group useful in a retention campaign?

A. It proves the propensity model is causal

B. It estimates incremental intervention impact

C. It makes all customers equally likely to churn

D. It removes the need to monitor fairness

Correct Answer: B. It estimates incremental intervention impact