Evidence-Based Decision-Making and Experimentation

Introduction

Business analytics creates value only when evidence improves a decision. Descriptive dashboards, predictive models and large-scale data platforms do not automatically produce better outcomes. Managers must still define the decision, identify alternatives, assess uncertainty, compare consequences and learn from results. Evidence-based decision-making is therefore both an analytical discipline and a managerial practice.

This chapter examines rational and bounded-rationality models, decision trees, scenario analysis, causal reasoning, A/B testing and sensitivity analysis. It also addresses cognitive bias, ethical risks and the common error of interpreting correlation as causation. The emphasis is practical: an analyst should be able to translate an ambiguous business question into a defensible decision process, select an appropriate design, communicate uncertainty and recommend action.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish rational, bounded-rational and intuitive decision-making.
2. Construct and interpret decision trees using expected monetary value.
3. Develop scenarios and test the sensitivity of a recommendation.
4. Explain correlation, confounding, counterfactuals and causal effects.
5. Design and interpret an ethical A/B test.
6. Recognise cognitive bias, statistical error and governance risks.
7. Apply an evidence-to-action workflow in a business setting.

1. What Makes a Decision Evidence-Based?

An evidence-based decision links five elements:

1. Decision question: What choice must be made, by whom and by when?
2. Alternatives: What feasible actions are available?
3. Evidence: What data, research, experiments and expert knowledge are relevant?
4. Uncertainty: Which assumptions, estimates and unknown events could change the conclusion?
5. Action and learning: What will be implemented, monitored and revised?

Evidence includes more than numerical data. Internal transaction records, controlled experiments, customer research, operational knowledge and credible external studies can all be useful. However, evidence must be relevant to the decision and evaluated for quality, bias, timeliness and applicability.

A useful distinction is between descriptive, predictive and prescriptive analytics. Descriptive analytics reports what has happened; predictive analytics estimates what may happen; prescriptive analytics evaluates what should be done. A highly accurate prediction may still be strategically irrelevant if managers cannot act on it or if the action has unacceptable costs.

The evidence-to-action cycle

A robust cycle is:

Frame → Measure → Analyse → Decide → Implement → Evaluate → Learn

Framing prevents analysts from answering the wrong question. Measurement requires valid definitions and reliable data. Analysis should match the decision and its uncertainty. Evaluation compares observed outcomes with an appropriate benchmark or counterfactual. Learning converts the result into improved policy, process or future experimentation.

2. Rationality, Bounded Rationality and Intuition

The rational model

The classical rational model assumes that a decision-maker:

• knows the objective;
• identifies all important alternatives;
• has reliable information about consequences;
• assigns preferences or utilities consistently; and
• chooses the alternative that maximises expected value.

This model is valuable as a benchmark. It clarifies what complete analysis would require, even though its assumptions are rarely fully satisfied in organisations.

If an alternative a can lead to outcomes o with probabilities p(o|a), its expected utility is:

Here, EU(a) is expected utility, p(o|a) is the probability of outcome o given action a, and U(o) is the decision-maker’s utility for that outcome. The probabilities must sum to one across mutually exclusive and collectively exhaustive outcomes. Expected monetary value is a special case in which utility is measured directly in money and risk preferences are ignored.

For a simple investment with a 0.6 probability of earning £80,000 and a 0.4 probability of losing £20,000:

The EMV is not a promise that £40,000 will be received. It is a probability-weighted long-run average under the stated assumptions.

Bounded rationality

Herbert Simon’s concept of bounded rationality recognises that decision-makers face limited time, information, computational capacity and attention. Rather than optimise across every possible alternative, they often satisfice: they search until they find an option that meets a sufficiently acceptable standard.

Bounded rationality does not mean irrationality. It means that rational behaviour is constrained by the decision environment. A manager may select a supplier that meets quality, compliance and price thresholds rather than evaluate every supplier worldwide. This can be efficient when additional analysis is costly and the consequences of a marginally better option are small.

Intuition and expertise

Intuition is a rapid judgement based on pattern recognition. It can be valuable where feedback is frequent, reliable and similar cases recur, such as experienced maintenance diagnosis. It is less reliable in volatile environments with delayed feedback, political incentives or rare events. Analytics should not automatically replace expertise; it should make assumptions visible and create disciplined opportunities for challenge.

Decision hygiene

Good decision hygiene separates:

• the quality of the decision process from the luck of the outcome;
• forecasts from preferences;
• facts from assumptions;
• evidence from interpretation; and
• reversible from irreversible commitments.

A poor process can produce a favourable result by chance, while a sound process can produce an unfavourable result because an uncertain event occurred.

3. Decision Trees and Expected Value

A decision tree represents sequential choices and uncertain events. A square denotes a decision node, a circle denotes a chance node and terminal branches show outcomes. The tree is evaluated from right to left: calculate expected values at chance nodes, then select the best available action at decision nodes.

Consider a retailer deciding whether to launch a premium product. A launch costs £30,000. If demand is high, the net contribution after launch cost is £100,000; if demand is low, it is −£30,000. The probability of high demand is 0.4.

If not launching produces £0, the launch has a higher EMV under risk-neutral assumptions. Yet the recommendation depends on the quality of the probability estimate, the completeness of the outcomes and the organisation’s risk tolerance.

Value of information

Information has value when it can change an action and improve expected outcomes. The expected value of perfect information (EVPI) is:

In the example, perfect information would lead to launching when demand is high and not launching when demand is low. Its expected value is:

Therefore:

A market study costing £10,000 could be economically justified if it provided sufficiently reliable information and did not delay the opportunity. Perfect information is an upper bound; real research is imperfect. The expected value of sample information is normally lower than EVPI and should be estimated using the possible research results and their effects on the decision.

Practical cautions

Decision trees can create false precision. Probabilities may be subjective, outcomes may omit reputational or regulatory costs, and branches may not be independent. Analysts should record the source and confidence of each probability, use ranges where appropriate and test how much the decision depends on each assumption.

4. Scenario Analysis and Sensitivity Analysis

Scenario analysis

Scenario analysis explores coherent combinations of assumptions rather than predicting one certain future. A strategic scenario is not simply a best, average and worst number; it describes a plausible environment, such as rapid adoption, regulated stagnation or supply disruption.

A scenario should state:

• key external drivers;
• assumptions about customers, competitors, costs and regulation;
• operational implications;
• leading indicators; and
• decisions that remain robust across scenarios.

Scenarios are especially useful for long-term investments where probabilities are difficult to justify. They should not be presented as forecasts unless probabilities are explicitly estimated.

Sensitivity analysis

Sensitivity analysis changes one or more inputs to determine whether the recommendation is robust. For a project with revenue R, variable cost Cᵥ, fixed cost C𝒇 and investment I, a simplified profit measure is:

If revenue is £500,000, variable cost is £280,000, fixed cost is £120,000 and investment is £50,000:

A one-way sensitivity test might reduce revenue by 10%:

The recommendation is therefore highly sensitive to revenue. A two-way analysis could vary both revenue and variable cost, while a probabilistic Monte Carlo analysis would assign distributions to uncertain inputs and simulate many combinations. Simulation results should be reported as distributions or decision probabilities, not as falsely precise certainties.

A useful decision rule is the break-even value. If profit is zero when revenue equals R, then:

This equation applies to the simplified accounting model above and assumes costs are measured on the same time basis and no taxes, financing costs or residual value are included.

5. Causal Reasoning: From Association to Action

Correlation is not causation

Correlation describes statistical association. It does not establish that changing one variable will change another. A third variable may cause both, reverse causality may operate, the association may be coincidental or the data may be selected in a biased way.

For variables X and Y, Pearson’s correlation is commonly written as:

Here, cov(X,Y) is covariance and σₓ and σᵧ are standard deviations. The coefficient ranges from −1 to +1 and measures linear association, not causal effect. It is undefined when either variable has zero variance and can be distorted by outliers or non-linear relationships.

For example, ice-cream sales and swimming accidents may rise together because hot weather increases both. Weather is a confounder. Reducing ice-cream sales would not necessarily reduce accidents.

Counterfactual thinking

A causal effect compares what happened under treatment with what would have happened to the same unit at the same time without treatment. The latter is the counterfactual, which cannot normally be observed directly. A credible research design approximates it using randomisation, a comparison group, before-and-after data with suitable assumptions or other quasi-experimental methods.

The average treatment effect can be expressed as:

Here, Y(1) is the potential outcome under treatment, Y(0) is the potential outcome without treatment and E denotes the population average. The notation emphasises why causal inference is difficult: for each unit, only one potential outcome is observed.

Common causal designs

• Randomised experiment: treatment is assigned by chance, balancing observed and unobserved factors in expectation.
• Difference-in-differences: compares changes over time between treatment and comparison groups, requiring a credible parallel-trends assumption.
• Regression discontinuity: exploits a treatment rule based on a threshold, assuming units near the threshold are comparable.
• Instrumental variables: uses a variable that affects treatment but influences the outcome only through treatment; valid instruments are difficult to find.
• Matched observational analysis: constructs comparable groups using measured characteristics, but cannot remove unmeasured confounding.

Causal claims should therefore state the design, population, estimand, assumptions and limitations.

6. A/B Testing and Experimentation

An A/B test randomly assigns eligible units to a control condition, A, or a treatment condition, B. The purpose is to estimate the causal effect of changing one intervention while holding other conditions as constant as possible.

Experiment design

A professional experiment requires:

1. A measurable decision and primary outcome.
2. A precisely defined eligible population.
3. A treatment that differs from control in the intended way.
4. Random assignment and protection against contamination.
5. A pre-specified analysis plan.
6. Adequate duration to cover normal cycles.
7. Guardrail metrics for harm, quality and compliance.
8. A decision rule agreed before results are seen.

Examples of primary outcomes include conversion rate, average order value, retention or time to resolution. Secondary outcomes provide context. Guardrail metrics might include complaints, refunds, latency, accessibility or unsubscribes.

Estimating a difference in conversion rates

Suppose pᴬ and pᴮ are conversion rates in control and treatment groups. The estimated absolute effect is:

The relative lift is:

If control conversion is 8% and treatment conversion is 9.2%, the absolute improvement is 1.2 percentage points and relative lift is 15%.

For two independent proportions, a large-sample standard error for the difference is approximately:

where nᴬ and nᴮ are group sizes. This approximation requires independent observations and sufficiently large expected counts. Clustered users, repeated observations and low event rates may require specialised methods.

A 95% confidence interval is often written as:

The 1.96 value relies on a normal approximation. A confidence interval describes the performance of a repeated-sampling procedure; it is not a 95% probability statement about a fixed parameter after the data have been observed.

Power and sample size

Power is the probability of detecting a specified effect under a chosen significance threshold, assuming that effect is real. Sample-size planning requires a baseline rate, minimum practically important effect, significance level, desired power and allocation ratio. Statistical significance alone does not establish commercial importance. A tiny improvement may be statistically detectable but economically irrelevant, while a meaningful improvement may be uncertain in a small sample.

Experimentation errors

Repeatedly checking results and stopping when a desired p-value appears inflates false-positive risk. Testing many outcomes or segments creates multiple-comparison problems. Changing the treatment during the test, allowing users to switch groups or running concurrent experiments on the same outcome can compromise interpretation. Pre-registration, sequential-testing methods, clear stopping rules and version-controlled analysis reduce these risks.

7. Cognitive Bias and Analytical Error

Evidence-based practice must address both data limitations and human judgement. Common biases include:

Analytical errors include leakage, selection bias, non-response bias, missing-not-at-random data, inappropriate aggregation and changing definitions. A model can be technically accurate yet operationally unfair if historical outcomes reflect discriminatory practices. Analysts should document provenance, inclusion rules, transformations, model limitations and known exclusions.

8. Ethics, Governance and Responsible Experimentation

Experiments involving customers or employees require more than statistical validity. Organisations should consider informed notice, consent where appropriate, privacy, security, fairness, accessibility and the possibility of harm. Personalisation experiments can create differential prices or opportunities; employee experiments may affect livelihood and raise power imbalances.

Governance controls include:

• lawful and proportionate data collection;
• data minimisation and retention limits;
• role-based access and audit logs;
• ethical review for sensitive interventions;
• pre-defined harm thresholds;
• transparent escalation and rollback procedures; and
• human oversight for high-impact decisions.

An experiment should not be used to justify conduct that would be unacceptable if disclosed. Statistical validity cannot legitimise manipulation, discrimination or privacy intrusion. AI systems add risks including opaque recommendations, feedback loops, automation bias and model drift. Human review should focus on meaningful challenge rather than merely approving an automated output.

9. Applied Case: Personalised Delivery-Time Messaging

The following is an illustrative case.

Background

A national online retailer observed that customers shown a narrow delivery window appeared to complete checkout more often than customers shown a broad window. The commercial team proposed replacing “Delivery in 2–4 days” with “Delivery by Tuesday”.

Problem

The observed association might reflect customer mix: mobile users, high-value products or particular regions could both receive narrow windows and convert more often. Management therefore needed causal evidence, not a dashboard correlation.

Analysis and approach

The analytics team defined checkout conversion as the primary outcome and cancellation, late delivery and customer complaints as guardrails. Eligible customers were randomly assigned at session level to:

• Control: existing broad delivery window.
• Treatment: precise delivery date when operational confidence exceeded a defined threshold.

The team excluded orders already subject to exceptional logistics conditions and recorded the analysis plan before launch. Randomisation was checked using region, device and customer type. The test ran through two complete weekly cycles to cover weekday effects.

Suppose the control group contained 50,000 sessions with 4,000 purchases, giving an 8.0% conversion rate. The treatment group contained 50,000 sessions with 4,600 purchases, giving a 9.2% rate. The estimated absolute effect was 1.2 percentage points and the relative lift was 15%.

Using the approximate standard error:

An approximate 95% interval is:

Thus the estimated improvement is about 0.85 to 1.55 percentage points under the stated assumptions. The team also found that late-delivery complaints increased by 0.3 percentage points, concentrated in low-confidence regions.

Solution and outcome

The retailer did not roll out the treatment universally. It deployed precise dates only where the logistics system met the confidence threshold and retained the broad window elsewhere. Monitoring continued after rollout because treatment effects could change during peak seasons.

Learning points

1. A/B testing converted an association into a more credible causal estimate.
2. Guardrail metrics prevented conversion gains from hiding service failure.
3. Segmentation supported a conditional policy rather than an overly broad average.
4. Statistical significance was not the only decision criterion; operational reliability and customer trust mattered.
5. An experiment is a management intervention, not merely a statistical exercise.

10. A Professional Workflow for Evidence-Based Decisions

A practical workflow is:

1. State the decision: identify owner, deadline, alternatives and consequences.
2. Define success: specify primary, secondary and guardrail outcomes.
3. Map the causal story: identify intervention, outcome, confounders and mechanisms.
4. Assess available evidence: distinguish internal data, external evidence, assumptions and expert judgement.
5. Choose the design: decision tree, scenario analysis, observational study, experiment or combination.
6. Quantify uncertainty: use intervals, distributions, sensitivity analysis and explicit confidence levels.
7. Check robustness: vary key assumptions and test alternative explanations.
8. Review bias and ethics: examine affected groups, privacy, fairness and possible harms.
9. Decide and document: record the rationale, alternatives rejected and conditions for reversal.
10. Evaluate and learn: compare outcomes with the counterfactual or benchmark and update the decision process.

The final report should be concise but auditable. It should include the question, data period, population, method, assumptions, effect estimates, uncertainty, limitations, recommendation and monitoring plan. Decision-makers need to know not only what the analysis concludes, but also when that conclusion should no longer be trusted.

Challenges and Limitations

Evidence-based decision-making cannot eliminate uncertainty. Data may be incomplete, delayed or generated by processes that change over time. Experimental results may not generalise beyond the tested population, treatment intensity or period. Organisational constraints may prevent randomisation. Expected monetary value can undervalue catastrophic downside, and averages can conceal distributional effects.

Causal analysis also depends on assumptions that cannot always be verified. A statistically significant result may be a false positive, a negligible effect or an artefact of measurement. Conversely, a non-significant result does not prove no effect; it may reflect low power or excessive noise. Professional judgement remains necessary, but it should be explicit, challengeable and informed by evidence.

Chapter Summary

Evidence-based decision-making connects a clearly framed choice with relevant evidence, explicit uncertainty and accountable action. The rational model provides a useful benchmark through expected utility, while bounded rationality explains why managers satisfice under constraints. Decision trees calculate probability-weighted consequences and can assess the value of additional information.

Scenario analysis explores coherent alternative futures, whereas sensitivity analysis tests whether a recommendation changes when assumptions vary. Correlation measures association but cannot by itself establish causation. Counterfactual reasoning, randomised experiments and credible quasi-experimental designs strengthen causal claims. A/B tests require careful population definition, randomisation, pre-specified outcomes, adequate duration and guardrail metrics.

Bias, data quality, privacy, fairness and governance are integral to analytics rather than optional additions. Strong analysts communicate effect size and uncertainty, distinguish statistical from practical significance, test robustness and design decisions that can be monitored and reversed. The ultimate purpose of experimentation is organisational learning: making better decisions under uncertainty over time.

Key Terms

A/B test: A randomised comparison of a control condition and a treatment condition.

Average treatment effect: The average causal difference between treatment and no treatment across a defined population.

Bounded rationality: Decision-making constrained by limited information, time and cognitive capacity.

Causal effect: The change in an outcome attributable to an intervention under a specified counterfactual.

Confounder: A variable associated with both the explanatory variable and outcome that can distort their relationship.

Confidence interval: An interval produced by a statistical procedure intended to contain the parameter at a stated long-run frequency.

Counterfactual: The outcome that would have occurred for the same unit under an alternative condition.

Decision tree: A representation of sequential decisions, chance events and outcomes.

Expected monetary value: The probability-weighted average monetary outcome of an action.

Expected utility: The probability-weighted average of utility across possible outcomes.

Experiment: A structured intervention designed to estimate effects by comparison.

Guardrail metric: A measure used to detect unacceptable side effects or harm.

Intuition: Rapid judgement based on pattern recognition and experience.

Randomisation: Assignment by chance to reduce systematic differences between groups.

Relative lift: The estimated change divided by the control value.

Scenario analysis: Examination of outcomes under coherent alternative sets of assumptions.

Sensitivity analysis: Testing how results change when input assumptions vary.

Satisficing: Selecting an option that meets an acceptable threshold rather than maximising an objective.

Statistical significance: Evidence, under a specified model and threshold, that is inconsistent with a null hypothesis.

Survivorship bias: Error caused by analysing only entities that remain visible or successful.

Review Questions

Short Answer Questions

1. What are the five elements of an evidence-based decision?
2. Explain the difference between expected monetary value and expected utility.
3. Why does correlation not establish causation?
4. What is the purpose of a guardrail metric in an A/B test?
5. Define bounded rationality and satisficing.

Long Answer Questions

1. Compare the rational model with bounded rationality and discuss when each is useful in business analytics.
2. Explain how a decision tree is constructed and evaluated, including the role of expected value of information.
3. Compare scenario analysis with sensitivity analysis and give an example of when each should be used.
4. Describe the design, analysis and governance requirements of a credible A/B test.
5. Discuss how cognitive bias and data-quality problems can undermine evidence-based management.

Analytical / Application Questions

1. A project has a 0.3 probability of producing £120,000 and a 0.7 probability of losing £20,000. Calculate its expected monetary value and explain why a risk-averse manager might reject it.
2. An online service records higher retention among customers receiving a new feature, but those customers were selected by account managers. Identify plausible causal threats and propose a stronger evaluation design.
3. A treatment increases conversion from 6% to 6.6% but also increases complaint rates from 1% to 1.8%. Recommend a decision framework that incorporates commercial value, uncertainty and customer harm.

Multiple Choice Questions with Answers

Question 1

Which statement best describes bounded rationality?

A. Decision-makers always maximise expected utility with complete information

B. Decision-makers choose randomly because analysis is impossible

C. Decision-makers make purposeful choices under limits of information, time and cognition

D. Decision-makers rely exclusively on statistical models

Correct Answer: C. Decision-makers make purposeful choices under limits of information, time and cognition

Question 2

A project has a 0.4 probability of earning £100,000 and a 0.6 probability of losing £20,000. What is its expected monetary value?

A. £28,000

B. £40,000

C. £52,000

D. £80,000

Correct Answer: A. £28,000

Question 3

What is the primary purpose of a decision tree?

A. To prove that every outcome is predictable

B. To represent sequential choices, uncertainties and consequences

C. To replace all managerial judgement

D. To measure correlation only

Correct Answer: B. To represent sequential choices, uncertainties and consequences

Question 4

Which situation is the clearest example of confounding?

A. A randomised treatment increases sales

B. Two variables have no observed association

C. Hot weather increases both ice-cream sales and swimming accidents

D. A measurement is recorded in pounds rather than kilograms

Correct Answer: C. Hot weather increases both ice-cream sales and swimming accidents

Question 5

If control conversion is 8% and treatment conversion is 9.2%, what is the relative lift?

A. 1.2%

B. 8.0%

C. 9.2%

D. 15%

Correct Answer: D. 15%

Question 6

Which practice most directly reduces false-positive risk from repeatedly checking an A/B test?

A. Stopping whenever any segment improves

B. Pre-specifying stopping rules or using a valid sequential method

C. Removing the control group midway

D. Testing more outcomes without adjustment

Correct Answer: B. Pre-specifying stopping rules or using a valid sequential method

Question 7

What does a confidence interval quantify in a conventional repeated-sampling interpretation?

A. The probability that the observed sample is correct

B. The long-run coverage performance of a statistical procedure

C. The probability that the null hypothesis is true

D. The guaranteed range of all future observations

Correct Answer: B. The long-run coverage performance of a statistical procedure

Question 8

Which technique tests whether a recommendation changes when assumptions vary?

A. Sensitivity analysis

B. Randomisation

C. Data encryption

D. Deduplication

Correct Answer: A. Sensitivity analysis

Question 9

Why is a guardrail metric important in experimentation?

A. It guarantees statistical significance

B. It detects harmful or unacceptable side effects

C. It replaces the primary outcome

D. It eliminates the need for random assignment

Correct Answer: B. It detects harmful or unacceptable side effects

Question 10

Which statement about correlation is correct?

A. A strong correlation always proves a causal effect

B. Correlation measures association and may be affected by confounding

C. Correlation can only be positive

D. Correlation is identical to expected utility

Correct Answer: B. Correlation measures association and may be affected by confounding