Prediction-Driven Decision-Making

Introduction

Predictive analytics estimates what is likely to happen. Decision-making determines what should be done. The two activities are related but not identical. A model may estimate that a customer has a 0.72 probability of cancelling a subscription, yet that estimate does not by itself determine whether the firm should offer a discount, make a service call, or take no action. The appropriate action depends on costs, benefits, capacity, constraints, fairness, timing, and the consequences of errors.

Prediction-driven decision-making is the disciplined conversion of model outputs into operational or strategic choices. It includes setting decision thresholds, ranking cases, estimating expected value, incorporating asymmetric error costs, analysing alternative scenarios, and testing interventions through experimentation. It also requires clear distinction between predictive, causal, and prescriptive conclusions.

A central principle is:

A prediction describes a likely outcome; a decision selects an action under uncertainty.

This chapter develops a framework for moving from predictions to defensible decisions.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish predictive, causal, and prescriptive analytics.
2. Convert probabilities or scores into actions using thresholds and rankings.
3. Calculate expected value and compare alternative actions.
4. Design cost-sensitive decision rules.
5. Use scenario analysis to examine uncertainty and constraints.
6. Explain how experimentation supports causal decision-making.
7. Identify ethical, governance, and operational risks in automated decisions.
8. Evaluate a decision policy rather than evaluating a model in isolation.

From Model Output to Business Action

A predictive model commonly produces one of three outputs:

• A class, such as fraud or not fraud.
• A probability, such as the estimated probability of default.
• A continuous forecast, such as expected daily demand.

A decision system adds an action rule. A simple binary rule is:

Here, p̂ is the predicted probability of an event and τ is the decision threshold. The threshold is not a universal property of the model. It reflects the organization’s objectives, error costs, available resources, and risk tolerance.

For example, a hospital may use a lower threshold for identifying patients who require further screening when missing a serious condition is costly. A marketing team may use a higher threshold when contacting customers is expensive and the expected benefit is small.

Predictive, Causal, and Prescriptive Analytics

These forms of analytics answer different questions:

A high-risk customer is not necessarily a customer who will respond to an intervention. Risk prediction concerns the outcome without necessarily identifying the effect of treatment. Prescriptive decisions require information about both likely outcomes and the consequences of available actions.

A useful potential-outcomes representation considers two possible outcomes for an individual: Y(1) if treated and Y(0) if not treated. The individual treatment effect is:

The notation is conceptual because both potential outcomes for the same individual cannot normally be observed at the same time. Randomized experiments estimate average effects by comparing suitable treatment and control groups. A predictive model alone generally estimates risk, not τᵢ.

Threshold Decisions

Choosing a Threshold

Suppose a model predicts whether a transaction is fraudulent. If the predicted probability exceeds a threshold, the transaction is reviewed. Lowering the threshold usually identifies more suspicious transactions, increasing sensitivity but also increasing the number of legitimate transactions sent for review. Raising it has the opposite effect.

Threshold choice should consider:

• The cost of a false positive.
• The cost of a false negative.
• The benefit of correctly identifying an event.
• Review capacity and service-level requirements.
• Legal or safety requirements.
• The consequences of delayed action.
• Differences in error rates across groups.

Accuracy alone is often a poor decision metric, particularly when events are rare. A fraud system that labels every transaction legitimate may have high accuracy but no operational value.

Cost-Based Thresholds

Let:

• p = predicted probability that the event occurs;
• B = benefit of taking action when the event occurs;
• C_A = cost of taking action;
• C_M = cost of missing the event when no action is taken.

If action is taken, expected net value may be represented as:

If no action is taken, expected net value is:

Action is preferred when:

Rearranging gives:

This expression applies only when the quantities are defined consistently as monetary or utility values and when the model probability is sufficiently calibrated for the decision population. A simpler special case occurs when action costs C_A and prevents a loss L if the event occurs. Then action is worthwhile when:

or equivalently p > C_A/L. These formulas should guide, not mechanically replace, managerial judgment. Benefits may be uncertain, costs may be nonlinear, and actions may affect future outcomes.

Worked Example: Screening for Fraud

A payment processor reviews transactions predicted to be fraudulent. A review costs $4. If fraud is missed, the expected loss is $120. Assume review prevents the loss when fraud is present and produces no additional benefit. The break-even probability is:

Therefore, reviewing transactions with predicted fraud probability above approximately 3.33% is economically justified under these assumptions. If review capacity is limited, the firm cannot review every transaction above this threshold; it must rank cases and select those with the highest expected value until capacity is exhausted.

Ranking and Resource Allocation

Thresholding answers whether a case qualifies for action. Ranking answers which qualifying cases should be handled first. Ranking is particularly important when personnel, inventory, credit, or time is limited.

For a case i, a simple priority score can be expected loss:

where pᵢ is the predicted probability of an adverse event and Lᵢ is its estimated loss. A case with a lower probability may deserve higher priority if its potential loss is substantially larger.

For an intervention with benefit Bᵢ and cost Cᵢ, expected net value is:

This score is appropriate only if Bᵢ represents the benefit conditional on the relevant event and the action is capable of producing that benefit. If people respond differently to an action, a churn-risk score may be inferior to a response or uplift score.

Ranking policies should also account for:

• Capacity limits.
• Time windows.
• Dependencies between cases.
• Minimum service obligations.
• Geographic or operational constraints.
• Equity and access requirements.

A purely numerical ranking can be inappropriate when the highest-scoring cases systematically exclude protected or underserved populations. Constraints should be made explicit rather than hidden in informal overrides.

Expected Value and Decision Trees

Expected value combines possible outcomes with their probabilities. For mutually exclusive outcomes j = 1, …, n, the expected value of an action is:

where pⱼ is the probability of outcome j, Vⱼ(a) is the value of action a under that outcome, and the probabilities sum to 1. The values must use a common unit, such as dollars, profit, lives saved, or a specified utility measure.

Worked Example: Inventory Decision

A retailer must decide whether to order 100 additional units. Demand is either high or low. The probability of high demand is 0.60 and low demand is 0.40. If the order is made, profit is $3,000 under high demand and a loss of $1,000 under low demand. If the order is not made, profit is $1,200 under high demand and $800 under low demand.

For ordering:

For not ordering:

Ordering has the higher expected value by $360. This conclusion depends on the probabilities, profit estimates, and assumption that the alternatives and outcomes have been specified correctly. A risk-averse manager may reject the order despite its higher expected value because the downside is substantial.

Expected Value of Information

Additional information has value when it changes decisions in a way that improves outcomes. The expected value of perfect information is:

This is an upper bound on what the organization should pay for perfect information. Real information is imperfect, so its value cannot exceed EVPI. The concept helps managers judge whether more data collection, testing, or sensing is economically justified.

Scenario Analysis and Sensitivity

Models and decisions rely on assumptions. Scenario analysis examines how a decision changes when important assumptions vary. A scenario may alter demand, costs, probabilities, response rates, interest rates, or operational capacity.

A useful approach is to construct:

• A base case using the most plausible assumptions.
• An optimistic case with favourable but credible conditions.
• A downside case with adverse conditions.
• A stress case designed to test resilience under severe conditions.

Sensitivity analysis changes one or more inputs and observes the effect on the decision. If small changes reverse the recommendation, the decision is sensitive and may require additional information or a staged commitment.

For example, suppose an intervention costs $20 per customer and yields $100 if successful. The break-even response probability is 0.20 because:

If the estimated response rate is 0.23, the expected value is $3 per customer. However, if the plausible range is 0.15 to 0.30, the decision is uncertain: the intervention loses $5 per customer at 0.15 and gains $10 at 0.30. Management might run a pilot rather than immediately scaling the programme.

Scenario analysis does not convert assumptions into facts. Its purpose is to expose dependence on assumptions and support robust decisions.

Experimentation and Causal Learning

When an organization wants to know whether an action causes an outcome, experimentation is often the strongest practical design. In a randomized controlled experiment, eligible units are randomly assigned to treatment or control groups. Randomization helps balance observed and unobserved characteristics in expectation.

If the average outcome in treatment is Ȳ_T and the average outcome in control is Ȳ_C, the estimated average treatment effect is:

The estimate is meaningful when treatment assignment is appropriately randomized, outcomes are measured consistently, interference is limited or addressed, and attrition does not introduce serious bias. Statistical significance is not the same as business significance; the effect must also justify its cost and operational risk.

Experimentation should be designed around the decision. Important elements include:

1. A clearly defined eligible population.
2. A treatment that can actually be delivered.
3. A control or comparison condition.
4. A primary outcome and measurement window.
5. Sample-size and power considerations.
6. Monitoring for harm and imbalance.
7. A pre-specified decision rule for continuation, modification, or stopping.

A model can be used to target an experiment, but targeting should not compromise random assignment. After evidence is obtained, predictive models may help identify heterogeneous treatment effects, although such estimates require adequate sample size and careful validation.

Policy Evaluation Rather Than Model Evaluation

A model is not the final product. A decision policy maps information into actions. It should therefore be evaluated using measures such as:

• Net financial value.
• Cost per successful intervention.
• Service-level performance.
• Avoided loss.
• Customer or employee outcomes.
• Calibration and subgroup reliability.
• Error costs and complaint rates.
• Stability over time.

Offline model performance may not predict operational impact. For example, a model can improve area under the ROC curve but create little additional value if the selected cases are already handled by staff. Conversely, a modest model may be valuable if it improves prioritization in a high-cost process.

Monitoring should compare predicted probabilities with observed frequencies. A calibrated model producing probabilities near 0.70 should correspond to outcomes occurring approximately 70% of the time in comparable groups and periods. Calibration can deteriorate because of population change, policy changes, data drift, or strategic responses by customers.

Illustrative Case: Retention Offers at a Subscription Company

Background

A fictional subscription company experiences customer cancellations. It develops a model predicting cancellation within the next 60 days. The company considers offering a $15 service credit to selected customers. A retained customer generates an estimated contribution margin of $80 during the relevant period.

Problem

The company initially proposes sending offers to everyone with a predicted cancellation probability above 0.50. This policy is questionable because a high-risk customer may cancel despite the offer, while a low-risk customer may remain without receiving it. The company needs to determine whom to contact and whether the offer causes retention.

Analysis

The firm estimates that the credit costs $15. In a preliminary randomized pilot, customers receiving the offer show a 0.10 higher retention rate than comparable control customers. The estimated incremental expected contribution per contacted customer is:

On these assumptions, the offer loses $7 per customer. However, the overall estimate may conceal heterogeneous effects. The pilot analysis finds an estimated incremental retention effect of 0.35 for a segment with service problems and 0.05 for a segment with voluntary switching behaviour.

For the first segment:

For the second segment:

Approach and Solution

The company should not use cancellation risk alone. It should combine risk prediction with evidence about treatment response. It can prioritize customers who have both meaningful cancellation risk and a plausible positive response to the offer. Before scaling, it should repeat or extend the experiment across relevant customer segments, monitor complaints, and assess whether the credit is being distributed fairly.

If capacity allows only 10,000 offers, customers should be ranked by expected incremental value rather than by cancellation probability alone. The policy should include a minimum margin, exclusion rules for customers already receiving a discount, and a review process for unusual cases.

Outcome

In this illustrative case, the company avoids a costly blanket campaign and adopts a targeted pilot. It learns that prediction of risk is insufficient for selecting treatment recipients. The decision requires an estimate of incremental response and a complete accounting of treatment cost.

Learning Points

• High predicted risk does not imply high treatment effectiveness.
• Randomized experiments support causal conclusions.
• Expected value connects effects to financial consequences.
• Policies should be tested in operation, not judged only by model metrics.

Ethical, Legal, and Governance Considerations

Prediction-driven decisions can affect credit, employment, insurance, education, health, and access to services. Organizations should therefore establish governance before deployment.

Key practices include:

• Define the legitimate purpose of the decision.
• Use data that are relevant, proportionate, and lawfully obtained.
• Document data sources, model versions, assumptions, thresholds, and overrides.
• Test performance and error rates across relevant groups.
• Provide human review where consequences are significant.
• Offer notice, explanation, appeal, or correction mechanisms when required.
• Monitor for proxy discrimination and feedback loops.
• Restrict access to sensitive information.
• Retain audit logs and assign accountability.

Fairness cannot be reduced to one universal metric. Different fairness criteria may conflict, especially when base rates differ. The organization must identify the relevant harm, legal requirements, and decision context. Human involvement is not automatically protective: a poorly supervised reviewer may reproduce or amplify model bias. Governance must address the whole process, including data collection, model development, threshold selection, deployment, and appeal.

Technology and Contemporary Practice

Modern decision systems increasingly combine machine-learning models, optimization, workflow software, and generative AI. These tools can improve speed but introduce additional risks. Generative systems may produce unsupported explanations, inconsistent recommendations, or leakage of confidential information. Optimization systems may maximize a stated objective while ignoring unmeasured harms.

Best practice is to separate components conceptually and operationally:

1. The predictive layer estimates outcomes.
2. The causal or experimental layer estimates intervention effects where possible.
3. The decision layer applies objectives, constraints, and policies.
4. The monitoring layer evaluates outcomes and detects drift.
5. The governance layer provides oversight, documentation, and appeal.

This separation improves accountability because it makes clear whether an error arose from prediction, causal inference, valuation, or implementation.

Best-Practice Checklist

Before deploying a prediction-driven decision policy, ask:

1. What decision is being made, and who is affected?
2. Is the output a prediction, a causal estimate, or a recommendation?
3. Are the probabilities calibrated for the deployment population?
4. What are the costs of false positives and false negatives?
5. Is a threshold or ranking more appropriate?
6. Are capacity, timing, and legal constraints explicit?
7. What is the expected value under plausible scenarios?
8. Could an experiment improve knowledge before scaling?
9. How will performance, fairness, drift, and harm be monitored?
10. Who is accountable for the final decision?

Chapter Summary

Prediction-driven decision-making converts model outputs into actions by combining probabilities with objectives, costs, constraints, and consequences. Thresholds are appropriate when an action rule depends on crossing a specified level; rankings are useful when resources are limited. Expected value provides a structured way to compare actions, provided that probabilities, benefits, costs, and units are defined consistently. Scenario and sensitivity analysis reveal how robust a recommendation is to uncertain assumptions.

Predictive analytics estimates what is likely to occur, causal analytics estimates what changes because of an intervention, and prescriptive analytics selects an action or policy. A risk model should not automatically be used to target treatment because high risk does not necessarily imply high responsiveness. Experiments provide stronger evidence about causal effects, while policy evaluation measures operational value rather than model performance alone. Ethical governance requires attention to fairness, transparency, privacy, accountability, monitoring, and the possibility of harm.

Key Terms

Predictive analytics: Analysis that estimates likely future or unknown outcomes.

Causal analytics: Analysis concerned with the effects of interventions or exposures.

Prescriptive analytics: Analysis that recommends actions under objectives and constraints.

Decision threshold: A cutoff used to convert a score or probability into an action.

Ranking: Ordering cases by priority or expected value.

False positive: An action or classification triggered when the target event is absent.

False negative: Failure to act or classify when the target event is present.

Expected value: Probability-weighted value across possible outcomes.

Expected loss: Probability-weighted potential loss.

Expected net value: Expected benefit minus expected cost.

Calibration: Agreement between predicted probabilities and observed frequencies.

Sensitivity analysis: Examination of how results change when inputs vary.

Scenario analysis: Evaluation of decisions under alternative combinations of assumptions.

Experiment: A structured comparison used to estimate the effect of an intervention.

Average treatment effect: The average difference in outcomes caused by treatment relative to control.

Policy: A rule mapping information and conditions to an action.

Capacity constraint: A limit on available resources for implementing decisions.

Uplift: The incremental effect of an intervention relative to a comparison condition.

Model drift: Deterioration or change in model performance caused by changing conditions.

Decision governance: Processes for accountability, oversight, documentation, and control.

Review Questions

Short Answer Questions

1. Why is a predictive probability not, by itself, a decision?
2. What factors should influence the choice of a decision threshold?
3. Distinguish expected loss from expected net value.
4. Why might ranking be preferable to thresholding when capacity is limited?
5. What is the difference between predictive and causal analytics?

Long Answer Questions

1. Explain how a firm could select a fraud-review threshold using error costs and operational capacity.
2. Discuss the assumptions and limitations of expected-value analysis in managerial decision-making.
3. Explain why a high-risk customer is not necessarily a good target for a retention intervention.
4. Describe how scenario analysis and sensitivity analysis can improve strategic decisions.
5. Evaluate the ethical and governance requirements for an automated high-impact decision system.

Analytical / Application Questions

1. An intervention costs $12 and produces a contribution of $60 when it successfully prevents an adverse event. What minimum success probability makes the intervention worthwhile? Explain the assumptions behind your calculation.
2. A model ranks customers by cancellation probability, but a randomized pilot shows that treatment effects differ substantially by segment. Design a decision policy that uses the evidence appropriately.
3. A predictive model improves its area under the ROC curve but produces no increase in profit after deployment. Identify at least three possible explanations and propose a monitoring or experimental response.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes predictive from prescriptive analytics?

A. Predictive analytics estimates likely outcomes, while prescriptive analytics selects actions.

B. Predictive analytics always proves causation, while prescriptive analytics measures accuracy.

C. Predictive analytics applies only to financial data, while prescriptive analytics applies only to operations.

D. Predictive analytics eliminates uncertainty, while prescriptive analytics creates uncertainty.

Correct Answer: A. Predictive analytics estimates likely outcomes, while prescriptive analytics selects actions.

Question 2

A review action costs $4 and prevents an expected loss of $120 when fraud is present. Under the stated assumptions, above what predicted fraud probability is review economically justified?

A. 0.33%

B. 3.33%

C. 30%

D. 33.3%

Correct Answer: B. 3.33%

Question 3

Why is ranking especially useful in a decision process?

A. It guarantees causal validity.

B. It removes the need to estimate costs.

C. It helps allocate limited resources to higher-priority cases.

D. It ensures that all cases receive identical treatment.

Correct Answer: C. It helps allocate limited resources to higher-priority cases.

Question 4

If an action has benefit B when an event occurs and costs C_A to implement, which expression represents its expected net value under the simple model?

A. pB − C_A

B. p + B − C_A

C. B ÷ p − C_A

D. (1 − p)B + C_A

Correct Answer: A. pB − C_A

Question 5

In the inventory example, what was the expected value of ordering?

A. $360

B. $800

C. $1,040

D. $1,400

Correct Answer: D. $1,400

Question 6

What does a randomized controlled experiment primarily help estimate?

A. The causal effect of an intervention

B. The model’s file size

C. The number of variables in a dataset

D. The guaranteed outcome for every individual

Correct Answer: A. The causal effect of an intervention

Question 7

A model is well calibrated when:

A. It has 100% classification accuracy.

B. Its predicted probabilities correspond approximately to observed frequencies.

C. It uses the largest possible number of features.

D. It produces the same prediction for every group.

Correct Answer: B. Its predicted probabilities correspond approximately to observed frequencies.

Question 8

Why can a high predicted cancellation risk be insufficient for targeting a retention offer?

A. Risk models cannot produce probabilities.

B. High-risk customers may not be responsive to the intervention.

C. Retention offers never have a cost.

D. Causal effects are always identical across customers.

Correct Answer: B. High-risk customers may not be responsive to the intervention.

Question 9

What is the purpose of sensitivity analysis?

A. To hide uncertainty from decision-makers

B. To determine whether a recommendation changes when assumptions vary

C. To guarantee that a model is unbiased

D. To replace all operational monitoring

Correct Answer: B. To determine whether a recommendation changes when assumptions vary

Question 10

Which practice is most appropriate for a high-impact automated decision system?

A. Deploying without documentation if accuracy is high

B. Removing all human review regardless of context

C. Monitoring outcomes, subgroup performance, drift, and avenues for appeal

D. Using the same threshold permanently even when conditions change

Correct Answer: C. Monitoring outcomes, subgroup performance, drift, and avenues for appeal