From Prediction to Decisions

Introduction

Machine-learning systems are often introduced as prediction engines: they estimate demand, customer churn, credit default, fraud risk, or the probability that a customer will respond to an offer. In business, however, prediction is rarely the final objective. Managers must decide which customers to contact, how much inventory to order, which claims to investigate, or whether a policy should be introduced. These decisions involve costs, benefits, constraints, uncertainty, and sometimes ethical consequences.

A predictive model estimates what is likely to happen. A decision system determines what should be done. Moving from one to the other requires four connected disciplines:

1. Experimentation, which measures the effect of an intervention under controlled conditions.
2. Causal inference, which estimates what would happen under alternative actions.
3. Optimization, which selects actions subject to objectives and constraints.
4. Prescriptive analytics, which combines predictions, causal effects, and optimization to recommend action.

This chapter develops these connections and shows why a highly accurate model may still produce poor business decisions. It also explains how to design A/B tests, distinguish correlation from causation, translate scores into resource allocation, and govern automated decisions responsibly.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish predictive, causal, and prescriptive questions;
• design and interpret a basic randomized A/B test;
• explain confounding, selection bias, treatment effects, and counterfactuals;
• calculate incremental business value from model scores and treatment effects;
• formulate resource-allocation problems using objective functions and constraints;
• evaluate decisions under uncertainty, fairness, operational limits, and regulatory requirements;
• communicate recommendations together with assumptions, risks, and monitoring plans.

1. Three Questions: What Will Happen, What Caused It, and What Should We Do?

A predictive question asks: What is likely to happen? For example, which customers are likely to cancel within 30 days? A model may estimate a probability such as p_i = P(Y_i = 1 | X_i), where Y_i is the outcome and X_i is the observed information about customer i.

A causal question asks: What difference would an intervention make? For example, would a retention discount prevent cancellation? The relevant quantity is not merely the cancellation probability; it is the difference between the outcome if the customer receives the discount and the outcome if the same customer does not receive it.

A prescriptive question asks: Which action should be taken, for whom, and when? The answer depends on predicted outcomes, incremental effects, financial value, resource constraints, and risk preferences.

A central danger is confusing high risk with high responsiveness. A customer may be very likely to churn but unlikely to change behavior because of an offer. Conversely, a moderate-risk customer may be highly persuadable. Targeting should therefore focus on incremental impact, not simply predicted risk.

2. From Scores to Business Value

Suppose a model assigns each customer a score. A score is not automatically a decision rule. The organization needs a threshold, an action, and a value model.

For customer i, let:

• p_i = predicted probability of the relevant outcome;
• v_i = financial value if the outcome is achieved;
• c_i = cost of taking the action;
• r_i = risk-adjusted penalty or expected loss;
• a_i = decision, where 1 means take the action and 0 means do not.

A simple expected net value is:

This formula applies when p_i represents the probability that the action produces the valued outcome and when value and costs are expressed in the same currency and time period. A rational rule under a simple independent-decision setting is to act when ENV_i > 0. In practice, a budget or capacity constraint may require selecting only the highest-value cases.

For interventions, the more useful quantity is the incremental effect. Let p_i¹ be the probability of success if treated and p_i⁰ the probability without treatment. The conditional incremental effect is:

If the financial benefit per additional success is v_i, treatment cost is c_i, and other expected costs are r_i, then:

The rule is appropriate when treatment effects are estimated reliably and the action does not materially affect other customers or future periods. Effects may be positive, zero, or negative. A model that ranks only p_i¹ can mistakenly target people who would succeed anyway.

Worked Example: Retention Targeting

A subscription company estimates that a £20 retention offer costs £4 per customer. Each prevented cancellation contributes £60 in future contribution margin. For Customer A, the estimated cancellation probability is 0.70 without the offer and 0.62 with it. Thus:

τ_A = 0.62 - 0.70 = -0.08

The reduction in cancellation probability is 0.08. Expected incremental benefit is 0.08 × £60 = £4.80. Net incremental value is £4.80 - £4.00 = £0.80. The offer is financially attractive before considering contact constraints and estimation uncertainty.

Customer B has probabilities 0.90 without the offer and 0.88 with it. The incremental benefit is 0.02 × £60 = £1.20, so the net value is -£2.80. Although Customer B is at higher risk, the offer should not be made under this value model. This illustrates the difference between risk targeting and uplift targeting.

3. Experimentation and A/B Testing

The Logic of Randomization

An A/B test randomly assigns eligible units—customers, transactions, stores, or web sessions—to a treatment group and a control group. The treatment group receives the intervention; the control group receives the existing experience or a placebo. Randomization aims to make the groups comparable, on average, before treatment. Therefore, differences in average outcomes can be attributed to treatment more credibly than differences observed in ordinary operational data.

Let Y_i(1) denote the potential outcome for unit i if treated and Y_i(0) the potential outcome if untreated. The individual treatment effect is:

Only one potential outcome can be observed for each unit. This is the fundamental missing-data problem of causal inference. A randomized experiment estimates the average treatment effect (ATE):

In a simple experiment with independent observations, an estimator is:

where Ȳ_T and Ȳ_C are the sample mean outcomes for treatment and control. For a binary outcome, these are conversion or success rates. The estimate is most credible when assignment is genuinely random, treatment is delivered as assigned, outcomes are measured consistently, and interference between units is limited.

Designing an A/B Test

A professional test begins with a decision, not with a dashboard. The design should specify:

1. Decision and hypothesis. Example: “Introducing one-click checkout will increase completed purchases without increasing refunds.”
2. Unit of randomization. A customer may be preferable to a page view when repeated exposures could contaminate results.
3. Treatment and control. Define exactly what changes and what remains constant.
4. Primary outcome. Select one principal metric, such as contribution margin per eligible customer, rather than relying on whichever metric happens to improve.
5. Guardrail metrics. Monitor complaints, refunds, service contacts, fraud, and latency.
6. Sample size and duration. Consider baseline rate, minimum practically important effect, desired power, significance level, variance, seasonality, and expected attrition.
7. Analysis plan. Define exclusions, subgroups, stopping rules, and statistical methods before inspecting results.
8. Operational safeguards. Establish rollback criteria and data-quality checks.

For a binary outcome, a rough equal-sized two-arm sample-size calculation uses:

Here, n is the required number per group, p̄ is an anticipated baseline proportion used for planning, Δ is the minimum absolute difference to detect, α is the two-sided Type I error rate, β is the Type II error rate, and z denotes a standard-normal quantile. This is an approximation for independent observations with equal group sizes; clustering, unequal allocation, repeated users, and attrition require adjustment.

Statistical and Practical Significance

A small p-value indicates that the observed data would be relatively unusual under a specified null hypothesis. It does not establish that the effect is large, valuable, or permanent. A confidence interval communicates estimation precision. Management should also evaluate:

• absolute and relative effect size;
• expected financial value;
• implementation and opportunity costs;
• heterogeneous effects across customers;
• short-term versus long-term outcomes;
• uncertainty and downside risk.

Repeatedly checking results and stopping when a preferred outcome appears inflates false-positive risk unless sequential testing methods or pre-specified stopping rules are used. Running many variants or metrics also increases the probability of finding a seemingly positive result by chance.

4. Correlation, Causation, and Observational Data

Correlation describes association. Causation concerns how an outcome would change under an intervention. Correlation alone may arise from:

• Confounding: a third variable affects both the supposed cause and the outcome;
• Selection bias: the observed sample differs systematically from the target population;
• Reverse causality: the outcome influences the explanatory variable;
• Measurement error: variables are recorded inaccurately;
• Interference: one unit’s treatment affects another unit’s outcome;
• Time trends: both variables change because of seasonality or common shocks.

For example, customers who receive a service call may have higher retention than customers who do not. This does not prove that the call caused retention, because service agents may select customers who already appear motivated to stay. Alternatively, agents may call customers in severe difficulty, producing a misleading negative association even if calls help.

A causal diagram can clarify assumptions. If X influences both treatment T and outcome Y, then conditioning on or adjusting for X may be necessary. However, adjusting for a mediator—something caused by treatment—or a collider—a variable influenced by two other variables—can create bias. Causal analysis therefore requires subject-matter reasoning, not merely an automated list of correlations.

When randomization is unavailable, analysts may use regression adjustment, matching, inverse-probability weighting, difference-in-differences, instrumental variables, regression discontinuity, or synthetic controls. Each method depends on assumptions. For example, difference-in-differences commonly requires parallel trends: absent treatment, treated and comparison groups would have followed similar trends. A credible observational study should state its identification strategy, test relevant pre-treatment patterns, examine sensitivity, and avoid presenting assumptions as facts.

5. Heterogeneous Treatment Effects and Uplift Modeling

The ATE may conceal important differences. A marketing offer may help price-sensitive customers, have no effect on indifferent customers, and annoy customers who value simplicity. The conditional average treatment effect (CATE) is:

It describes the expected treatment effect for units with characteristics x. Uplift models attempt to estimate and rank these effects. Common conceptual groups are:

These categories are conceptual rather than directly observed. Estimating them requires randomized data or strong causal assumptions. Heterogeneous-effect models should be evaluated on policy value or out-of-sample uplift, not only ordinary prediction accuracy.

6. Optimization: Converting Estimates into Actions

Once expected value has been estimated, optimization determines which feasible actions maximize an objective. A general binary allocation model is:

Here, v_i is expected net value for unit i, c_i is resource consumption, B is the available budget, and a_i indicates selection. The model assumes additive values and known or treated-as-known estimates. If all costs are equal, selecting the largest values is optimal. If costs differ, value per unit of resource may be useful as a heuristic, but the exact solution is a knapsack problem and should be solved with appropriate optimization software when scale or complexity warrants it.

Business constraints may include contact capacity, inventory, geography, staffing, fairness, risk limits, minimum service levels, and legal restrictions. A policy can be represented as a mapping from information to action: π(X) → A. The best policy is not necessarily the policy with the most accurate model; it is the policy with the highest reliable value under constraints.

Worked Allocation Example

A sales team can contact two of three leads. Estimated incremental net values are £18, £12, and £9, while contact times are 3, 2, and 1 minutes. If capacity is 3 minutes, selecting Lead 1 yields £18. Selecting Leads 2 and 3 uses 3 minutes and yields £21, so the second combination is better despite excluding the individually highest-value lead. This small example shows why resource constraints can change the ranking rule.

Under uncertainty, a robust decision may maximize expected value while limiting downside. One simple objective is:

where E[V(a)] is expected value, Var[V(a)] is variance, and λ ≥ 0 represents risk aversion. This is a modeling choice, not a universal law. It is appropriate only when variance is a meaningful proxy for decision risk and when the organization can justify the risk parameter.

7. Prescriptive Analytics in Practice

A prescriptive system typically contains five layers:

1. Data layer: collects outcomes, treatments, costs, constraints, and context.
2. Prediction and causal layer: estimates baseline outcomes and incremental effects.
3. Decision layer: optimizes actions under constraints.
4. Execution layer: delivers recommendations through CRM, supply-chain, pricing, or workflow systems.
5. Learning and governance layer: measures outcomes, detects drift, audits impacts, and updates the policy.

A useful implementation separates three metrics:

• Model metric: discrimination, calibration, or forecast error;
• Policy metric: value produced by the recommended action;
• Business metric: profit, retention, service quality, customer welfare, or strategic outcome.

Calibration is particularly important when probabilities are used financially. A calibrated model means that among cases assigned probability 0.30, the event occurs approximately 30% of the time over a suitable population. Discrimination and calibration are different: a model may rank cases well while producing poorly scaled probabilities.

8. Illustrative Case: A Retailer’s Retention Policy

Background

A fictional online retailer loses subscribers after delivery delays and price increases. It has a churn model ranking customers by predicted 60-day cancellation risk. The company plans to send a £5 voucher to 100,000 customers.

Problem

The existing approach targets the highest-risk customers. Finance observes that voucher expenditure is rising faster than retained contribution margin. The analytics team must determine whether the policy creates incremental value.

Analysis

The team runs a randomized experiment among eligible customers. Half receive the voucher and half receive the standard communication. The primary outcome is 60-day retained contribution margin per eligible customer; churn is a secondary outcome. The team records voucher cost, later purchases, complaints, delivery experience, and customer tenure.

The results show that the voucher reduces churn among medium-risk customers by 6 percentage points, but among the highest-risk customers by only 1 percentage point. Some high-risk customers were already planning to leave because of unresolved delivery problems. The test also finds a small increase in support contacts.

The team estimates incremental net value for each segment. For medium-risk customers, the expected contribution from prevented churn exceeds voucher cost and expected support cost. For highest-risk customers, it does not. A constrained policy therefore targets the medium-risk segment first, while routing customers with delivery problems to operational recovery rather than discounting.

Approach and Solution

The retailer combines an uplift model with an allocation model. The uplift model estimates treatment effects using the experiment. The allocation model selects customers subject to the voucher budget and contact capacity. The company also creates a holdout group for ongoing monitoring and repeats the test when customer behavior or offer design changes.

Outcome

The policy produces higher incremental contribution per voucher than risk-only targeting. It also reveals that operational service recovery may be more effective than a price incentive for some customers.

Learning Points

First, predictive risk is not incremental response. Second, the correct outcome is business value rather than a single conversion rate. Third, experimentation can reveal segment-specific effects. Fourth, a decision policy must include constraints and alternative actions. Finally, the policy should be monitored because treatment effects may change over time.

9. Ethics, Governance, and Responsible Decisions

Automated decisions can distribute opportunities and burdens. A model may recommend fewer loans, offers, inspections, or services to a protected group even when protected attributes are excluded, because proxy variables encode unequal social conditions. Removing sensitive variables does not guarantee fairness and may make auditing harder.

Responsible practice includes:

• defining the legitimate business purpose;
• collecting only necessary data;
• documenting data provenance and model assumptions;
• assessing disparate impact and error differences where legally and ethically relevant;
• testing whether treatment effects differ across groups;
• providing human review for high-stakes cases;
• enabling explanations, contestability, and correction;
• protecting confidential and personal information;
• monitoring performance after deployment;
• retaining an audit trail of model, policy, and threshold changes.

Fairness criteria can conflict. Equal selection rates, equal error rates, calibration, and equal treatment effects are not generally all achievable simultaneously, especially when base rates differ. Analysts should not choose a metric mechanically; they should engage legal, domain, and affected-community stakeholders and document the rationale.

A/B tests also require ethics. Users should not be exposed to unreasonable risk merely because randomization is statistically useful. Consent, privacy, vulnerable populations, potential harms, and the possibility of stopping a harmful treatment must be considered. In regulated sectors, statistical evidence does not replace requirements for transparency, documentation, human oversight, and lawful processing.

10. Challenges and Limitations

Data and measurement

Outcomes may be delayed, incomplete, or affected by changes in definitions. Revenue may not equal profit, and short-term conversion may reduce long-term loyalty. The target variable must represent the decision objective.

External validity

An experiment conducted in one region, season, channel, or customer segment may not generalize. Replication and post-deployment monitoring are essential.

Interference and feedback

Marketing actions can affect competitors, inventory, or other customers. A model-driven policy can change the population on which it is later trained, producing feedback loops.

Nonstationarity

Prices, regulations, customer preferences, and economic conditions change. Treatment effects and cost assumptions require periodic review.

Uncertainty in estimates

Estimated uplift and value are uncertain. Decisions based on point estimates alone may over-target noisy cases. Confidence intervals, shrinkage, sensitivity analysis, scenario analysis, and conservative thresholds can reduce overconfidence.

Operational feasibility

A mathematically optimal policy may be impossible to execute if staff cannot contact customers, systems cannot apply the recommendation, or frontline employees do not trust it. Implementation constraints belong in the model, not as an afterthought.

Best-Practice Workflow

1. Define the decision, eligible population, action alternatives, and objective.
2. Translate the objective into measurable outcomes and financial or social value.
3. Establish a causal strategy, preferably randomization where feasible.
4. Estimate baseline outcomes and incremental treatment effects separately.
5. Quantify uncertainty, costs, constraints, and potential harms.
6. Optimize the policy rather than merely ranking predictions.
7. Test the policy against a control or benchmark.
8. Monitor model quality, policy value, fairness, drift, and operational outcomes.
9. Document assumptions, ownership, approval, rollback, and review dates.
10. Update the system when evidence or conditions change.

Chapter Summary

Prediction estimates likely outcomes; causal inference estimates changes caused by actions; prescriptive analytics selects actions that best meet an objective under constraints. A/B tests use random assignment to make treatment and control groups comparable, while the potential-outcomes framework explains why individual treatment effects cannot be observed directly. Correlation may be misleading because of confounding, selection, reverse causality, measurement problems, and interference.

For business targeting, incremental effect is usually more valuable than risk alone. Expected net value combines treatment effect, financial benefit, cost, and risk. Optimization converts these estimates into feasible allocations subject to budgets, capacity, service, and governance constraints. Reliable decision systems evaluate not only model accuracy but also policy value, calibration, uncertainty, fairness, privacy, and long-term outcomes. The strongest analytics practice is therefore an iterative cycle of modeling, experimentation, decision, monitoring, and responsible revision.

Key Terms

Predictive analytics: Estimating likely future or unknown outcomes.

Causal inference: Estimating the effect of an intervention or exposure.

Prescriptive analytics: Recommending actions using predictions, causal effects, objectives, and constraints.

A/B test: A randomized comparison between treatment and control conditions.

Potential outcome: The outcome a unit would experience under a specified treatment condition.

Average treatment effect (ATE): The average difference between treated and untreated potential outcomes.

Conditional average treatment effect (CATE): The expected treatment effect for units with specified characteristics.

Counterfactual: The unobserved alternative outcome for the same unit under another action.

Confounding: A factor associated with both treatment and outcome that distorts causal interpretation.

Selection bias: Distortion caused by systematic differences in who enters or remains in an analysis.

Uplift modeling: Estimating heterogeneous incremental effects to target interventions.

Persuadable: A unit expected to respond because of treatment.

Expected net value: Expected benefit minus action costs and other expected losses.

Policy: A rule mapping available information to an action.

Objective function: A mathematical representation of what an optimization problem seeks to maximize or minimize.

Constraint: A limit or requirement restricting feasible decisions.

Calibration: Agreement between predicted probabilities and observed frequencies.

Guardrail metric: A measure monitored to prevent unacceptable side effects.

Interference: A situation in which one unit’s treatment affects another unit’s outcome.

Policy value: The expected outcome or benefit generated by applying a decision policy.

Review Questions

Short Answer Questions

1. How does a causal question differ from a predictive question?
2. Why might targeting the highest-risk customers be inferior to targeting customers with the highest uplift?
3. What is the purpose of randomization in an A/B test?
4. Define confounding and give one business example.
5. What is the difference between a model metric and a policy metric?

Long Answer Questions

1. Explain the potential-outcomes framework and discuss why individual treatment effects cannot normally be observed directly.
2. Describe the stages involved in designing a credible A/B test, including primary outcomes, guardrails, sample size, and stopping rules.
3. Compare risk-based targeting with uplift-based targeting and explain when each might be appropriate.
4. Explain how optimization converts estimated values into resource-allocation decisions under budget and capacity constraints.
5. Discuss the ethical and governance requirements for deploying automated prescriptive decisions in a high-stakes business context.

Analytical / Application Questions

1. A firm estimates that an intervention changes a customer’s success probability from 0.25 to 0.35. The financial contribution from an additional success is £40, and treatment costs £2. Calculate the expected incremental net value and state whether treatment is attractive before considering capacity constraints.
2. Three customers have expected net values of £20, £15, and £12, requiring 4, 2, and 1 minutes respectively. If a team has 3 minutes, identify the best feasible allocation and explain why selecting the single highest-value customer is not optimal.
3. A bank observes that customers receiving financial counselling have lower default rates than customers who do not receive counselling. Identify at least two reasons this association may not be causal and propose a suitable evaluation design.

Multiple Choice Questions with Answers

Question 1

Which question is primarily causal?

A. Which customers are likely to churn?

B. What is next month’s demand forecast?

C. Would a retention offer reduce churn?

D. Which products have the highest margin?

Correct Answer: C. Would a retention offer reduce churn?

Question 2

If treatment changes success probability from 0.25 to 0.35, what is the incremental effect?

A. 0.10

B. 0.25

C. 0.35

D. 0.60

Correct Answer: A. 0.10

Question 3

What is the main purpose of random assignment in an A/B test?

A. To guarantee a positive result

B. To make treatment and control groups comparable on average

C. To eliminate sampling variation completely

D. To increase the treatment group’s size

Correct Answer: B. To make treatment and control groups comparable on average

Question 4

A customer has a 0.08 treatment effect, a £60 benefit per additional success, and a £4 treatment cost. What is the expected incremental net value before other costs?

A. £0.80

B. £4.80

C. £8.00

D. £64.00

Correct Answer: A. £0.80

Question 5

Which situation is an example of confounding?

A. A randomized treatment assignment

B. A third variable affects both treatment receipt and the outcome

C. A perfectly calibrated probability model

D. A fixed budget constraint

Correct Answer: B. A third variable affects both treatment receipt and the outcome

Question 6

What does uplift modeling attempt to estimate?

A. Only the probability of an outcome without treatment

B. The average cost of data collection

C. Incremental treatment effects across units or segments

D. The number of features in a model

Correct Answer: C. Incremental treatment effects across units or segments

Question 7

In an allocation model, what does a budget constraint do?

A. It restricts the feasible set of decisions

B. It guarantees maximum revenue

C. It removes uncertainty from predictions

D. It converts correlation into causation

Correct Answer: A. It restricts the feasible set of decisions

Question 8

Which is a guardrail metric for a new marketing intervention?

A. The model’s file size

B. Customer complaints or refund rates

C. The number of model features only

D. The analyst’s preferred threshold

Correct Answer: B. Customer complaints or refund rates

Question 9

A model can rank customers accurately but produce probabilities that are systematically too high. Which property is deficient?

A. Calibration

B. Randomization

C. Interference

D. Optimization feasibility

Correct Answer: A. Calibration

Question 10

Why can a high-risk customer be a poor target for an intervention?

A. High-risk customers are never eligible for treatment

B. The customer may have a small incremental response and low net value

C. Risk scores cannot be used in business

D. Treatment effects are always identical

Correct Answer: B. The customer may have a small incremental response and low net value