12. Experimentation, Causal Reasoning and Decision Analytics

Introduction and Context

Data science creates value not merely by describing what has happened or predicting what may happen, but by helping organisations decide what to do. This distinction is central to advanced analytics. A predictive model may identify customers likely to cancel a subscription; it does not, by itself, establish that a discount will prevent cancellation. Similarly, a correlation between advertising expenditure and sales does not prove that increasing advertising will produce a corresponding increase in revenue. Decisions require evidence about consequences, not only associations.

Experimentation and causal reasoning address this problem. They ask questions such as: What would have happened if a different action had been taken? Did a policy cause the observed change? Which intervention produces the greatest benefit at acceptable cost? Decision analytics extends these questions by combining causal estimates with forecasts, constraints, preferences, and optimisation.

This chapter develops a practical and analytical framework for designing A/B tests, evaluating quasi-experiments, interpreting causal effects, and translating evidence into managerial decisions. It also shows how predictive, prescriptive, and optimisation approaches complement one another. The central principle is that a sound decision system must distinguish three propositions: what is associated with an outcome, what changes the outcome, and what action should be selected given objectives and constraints.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish descriptive, predictive, causal, prescriptive, and optimisation questions.

2. Explain counterfactual reasoning, treatment effects, and the conditions required for causal interpretation.

3. Design and evaluate a randomised A/B test, including units, outcomes, sample size, randomisation, and statistical inference.

4. Select and critically assess quasi-experimental designs such as difference-in-differences, regression discontinuity, instrumental variables, and matching.

5. Recognise threats including confounding, selection bias, interference, attrition, novelty effects, and multiple testing.

6. Combine causal estimates, predictive models, business constraints, and optimisation methods for managerial decisions.

7. Assess ethical, governance, and operational requirements for experimentation in organisations.

From Prediction to Intervention

A useful starting point is to classify analytical questions. Descriptive analytics asks what happened; predictive analytics estimates what is likely to happen; causal analytics estimates how an outcome would change under an intervention; prescriptive analytics recommends an action; and optimisation selects the best feasible action according to an objective function.

These categories are related but not interchangeable. Suppose a bank predicts that a customer has a high probability of default. The prediction can support monitoring, but it does not reveal whether a repayment reminder, a lower interest rate, or financial counselling would reduce default. That is a causal question. If several interventions are effective, the bank must compare their costs, risks, and capacity requirements. This is a prescriptive problem. If staff time, regulatory constraints, and budget limits must also be respected, the problem becomes one of constrained optimisation.

The distinction can be formalised through potential outcomes. For a unit i, let Yi(1) denote the outcome if the unit receives a treatment or intervention and Yi(0) the outcome if it does not. The individual treatment effect is

τi = Yi(1) − Yi(0).

Only one of these potential outcomes is observed for each unit. The unobserved outcome is the counterfactual: what would have happened to the same unit under the alternative condition. The fundamental problem of causal inference is therefore not simply estimating an association; it is reconstructing a missing counterfactual.

Common causal estimands include the average treatment effect (ATE),

ATE = E[Y(1) − Y(0)],

and the average treatment effect on the treated (ATT),

ATT = E[Y(1) − Y(0) | T = 1],

where T indicates treatment assignment. The ATE concerns a defined target population, whereas the ATT concerns those who actually received treatment. In practice, analysts must state the population, intervention, comparison, outcome, and time horizon explicitly. This is often called the PICO structure: population, intervention, comparison, and outcome.

Designing an A/B Test

A/B testing is a controlled experiment in which eligible units are randomly assigned to a control condition or one of one or more treatment conditions. It is widely used in digital products, marketing, healthcare operations, financial services, and public policy. Randomisation is valuable because, in expectation, it balances both observed and unobserved characteristics between groups. Consequently, differences in average outcomes can be attributed to treatment, provided the experiment is properly implemented.

Begin with a decision, not a metric

An experiment should begin with a decision that may change as a result of the evidence. A weak objective is “increase engagement.” A stronger objective is “determine whether the redesigned onboarding flow increases completed registrations without increasing early account closure.” The latter identifies a primary outcome, a potential adverse outcome, and a managerial decision rule.

The treatment must be operationally precise. “Improved service” is not a treatment definition; “a response-time guarantee displayed during checkout” is more suitable. The control condition must also be described. A test can compare a new intervention with the current process, no intervention, or another active alternative. The choice affects interpretation.

Unit of randomisation and unit of analysis

The randomisation unit is the entity assigned to treatment: a user, household, store, hospital, account, or geographic region. The analysis unit may differ. For example, users may be assigned to a product version, while outcomes are measured as transactions. Randomising individuals when users influence one another can cause contamination. In such settings, cluster randomisation, such as assigning entire stores or regions, may be more appropriate, although it generally requires a larger sample.

A related concern is interference. The potential outcome of one unit may depend on the treatment received by another unit. A referral promotion is an obvious example: treating one customer may affect another customer’s behaviour. The standard potential-outcomes framework often assumes no interference, an assumption known as part of SUTVA (stable unit treatment value assumption). When it is implausible, network effects and spillovers should be measured or explicitly modelled.

Outcomes, guardrails, and measurement

Experiments commonly use a primary outcome, secondary outcomes, and guardrail metrics. The primary outcome is selected before analysis and determines the principal test of success. Secondary outcomes provide additional insight, while guardrails detect harmful effects. For an e-commerce experiment, purchase conversion may be primary; average order value, refund rate, customer complaints, and page latency may be secondary or guardrail measures.

Metrics should be defined with a numerator, denominator, observation window, and eligibility rule. “Conversion” might mean a completed payment within seven days among users who entered the checkout page. Ambiguous metrics permit selective reporting and make replication difficult. The measurement system should be tested before launch, including event logging, timestamp consistency, identity resolution, and treatment exposure records.

Randomisation and implementation fidelity

Random assignment should be generated independently of expected outcomes. Blocked or stratified randomisation can improve balance when important characteristics, such as geography or customer segment, are known in advance. Allocation should remain concealed where operationally possible. Analysts should inspect baseline balance, but balance checks are diagnostic rather than a reason to repeatedly re-randomise until groups look identical.

Implementation fidelity concerns whether assigned treatment was actually received. A user may be assigned to a new interface but fail to load it because of a technical error. The intention-to-treat effect compares groups according to assignment and preserves the benefit of randomisation. The treatment-on-the-treated effect concerns actual exposure, but estimating it requires additional assumptions and can be biased if exposure is related to behaviour. For most managerial experiments, intention-to-treat is the primary estimand.

Sample size and statistical power

Before an experiment begins, analysts should specify a minimum practically important effect, significance level, desired power, baseline variance or conversion rate, and expected attrition. Power is the probability of detecting an effect of a specified size when that effect exists. A statistically significant result can be commercially trivial if the sample is enormous; a practically important effect may fail to reach significance in a small sample.

For a binary outcome, sample-size calculations depend on baseline conversion, the minimum detectable uplift, allocation ratio, significance threshold, and power. For continuous outcomes, the relevant quantities include standard deviation and expected mean difference. Cluster-randomised studies require an inflation factor because observations within a cluster are correlated. A useful design should also estimate the cost of running the experiment and the value of information it may produce.

Analysis and interpretation

The simplest analysis compares mean outcomes between treatment and control. For a continuous outcome, the estimated effect is the difference in group means. For a binary outcome, analysts may report absolute risk difference, relative risk, and odds ratio. Absolute differences are often more useful for decisions: an increase from 10% to 11% is a one-percentage-point improvement, even though the relative increase is 10%.

Confidence intervals express sampling uncertainty around an estimate. A narrow interval indicates greater precision, not necessarily absence of bias. A p-value addresses compatibility with a null hypothesis under a specified statistical model; it does not measure the probability that the treatment is beneficial or the size of its business value. Interpretation should therefore include effect size, uncertainty, costs, and practical significance.

Pre-specification is important. If analysts examine many outcomes, segments, time periods, and model specifications, some apparently significant findings may occur by chance. Multiple-testing procedures, such as false discovery rate control, may be appropriate. Sequential monitoring is permissible when the design includes an appropriate sequential testing method; repeatedly checking an ordinary p-value and stopping when it becomes significant inflates the false-positive rate.

Quasi-Experimental Reasoning

Randomised experiments are not always feasible. Legal, ethical, logistical, or financial constraints may prevent random allocation. In these circumstances, quasi-experimental designs seek to approximate the counterfactual using naturally occurring variation, institutional rules, or carefully selected comparison groups. Their credibility depends more heavily on assumptions that must be explained and assessed.

Difference-in-differences

Difference-in-differences (DiD) compares the change over time in a treated group with the change over the same period in a comparison group:

Effect = (Ytreated, after − Ytreated, before) − (Ycontrol, after − Ycontrol, before).

The key assumption is parallel trends: in the absence of treatment, the treated and comparison groups would have followed similar trends. Historical data can be used to examine pre-treatment trends, although similar trends do not prove that future trends would have remained parallel. Modern DiD analyses may use event-study graphs, staggered adoption methods, and cluster-robust uncertainty estimates. Treatment timing, anticipation, spillovers, and concurrent policies must be considered.

Regression discontinuity

Regression discontinuity (RD) applies when treatment is assigned according to a threshold, such as eligibility for a grant when a score exceeds a cutoff. Units just above and below the threshold may be comparable, allowing a local treatment effect to be estimated. The design requires continuity of potential outcomes around the threshold and no precise manipulation of the assignment variable. The result is generally local: it may not apply to units far from the cutoff.

Instrumental variables

An instrumental variable (IV) affects treatment assignment but has no direct effect on the outcome except through treatment. For example, random encouragement may influence participation in a programme without directly changing the outcome. A valid instrument requires relevance, exclusion, and appropriate independence assumptions. IV often identifies a local average treatment effect for compliers—units whose treatment status responds to the instrument—rather than an effect for the entire population. Because exclusion is difficult to defend, IV findings require careful institutional reasoning, not only statistical diagnostics.

Matching and weighting

Matching methods construct a comparison group with similar observed characteristics, using exact matching, propensity scores, or other distance measures. Weighting methods reweight observations to create balance across covariates. These approaches can reduce observed confounding but cannot control for unmeasured variables. Analysts should assess covariate balance, overlap, missing data, sensitivity to specifications, and the plausibility of no unmeasured confounding. Matching is not a substitute for a clear causal design.

Synthetic controls and interrupted time series

A synthetic control combines several untreated units to create a weighted comparison for one treated unit, such as a city adopting a policy. An interrupted time-series design examines whether the level or slope of an outcome changes after an intervention. Both designs can be useful for policy evaluation, but concurrent shocks, anticipation, seasonality, and changes in measurement may produce misleading effects.

Causal Graphs and Assumption Management

Directed acyclic graphs (DAGs) provide a visual language for causal assumptions. Nodes represent variables and arrows represent proposed causal relationships. A confounder is a common cause of treatment and outcome. For example, customer intent may influence both exposure to a promotional message and purchase, creating a misleading association. Adjustment for appropriate pre-treatment confounders can block non-causal backdoor paths.

Adjustment can also create bias. Conditioning on a mediator, which lies on the pathway from treatment to outcome, removes part of the effect when estimating a total effect. Conditioning on a collider, a variable caused by both treatment and another cause of the outcome, can open a spurious association. DAGs therefore help analysts decide what to measure and adjust for rather than relying on automated variable selection alone.

A strong causal claim states its assumptions. These may include consistency, well-defined treatment versions, exchangeability or no unmeasured confounding, positivity or adequate treatment overlap, no interference, correct temporal ordering, and reliable measurement. Sensitivity analysis examines how strong an unmeasured confounder would need to be to overturn a conclusion. Transparency about assumptions is more scientifically valuable than presenting a causal estimate without qualification.

Heterogeneous Effects and Targeting

Average effects can conceal important differences. A treatment may help new customers but harm experienced customers, or benefit high-risk patients while having little effect on low-risk patients. Conditional average treatment effects describe how effects vary with characteristics. However, subgroup analysis is vulnerable to small samples, multiple testing, and data-driven narratives.

Predictive models can estimate the probability of an outcome, whereas uplift or individualised treatment-effect models estimate the difference between outcomes under alternative actions. In practice, treatment-effect models should be evaluated using honest holdout data, policy value, calibration, and stability—not merely predictive accuracy. Targeting should also consider fairness: a group with a lower estimated effect should not automatically be denied a beneficial service if access decisions create inequity or violate policy.

Decision Analytics: From Evidence to Action

Causal evidence is an input to decision analysis rather than the final decision. A prescriptive framework combines estimated effects with outcome value, intervention cost, operational capacity, risk tolerance, and organisational objectives. A simple incremental net value for action a can be expressed as

Net Value(a) = Expected Benefit(a) − Expected Cost(a) − Expected Harm(a).

If treatment changes an outcome by τ(x) for a person with characteristics x, and the monetary value per unit outcome is v(x), a targeting policy may treat the person when v(x)τ(x) exceeds the incremental cost, subject to ethical and operational constraints. This rule is simplified: uncertainty, delayed effects, interactions, budget limits, and fairness constraints may materially change the decision.

Optimisation formalises selection under constraints. A linear programme may maximise expected profit:

max Σi [bi xi − ci xi]

subject to Σi ci xi ≤ Budget, capacity constraints, eligibility rules, and fairness requirements,

where xi indicates whether or how strongly an action is applied. Integer, nonlinear, robust, and stochastic optimisation may be required when decisions are discrete, effects interact, or uncertainty is substantial. The objective function should reflect the organisation’s actual priorities. Optimising a poorly chosen metric can produce efficient but undesirable behaviour.

A practical decision workflow is therefore: define the decision and estimand; generate or obtain causal evidence; estimate heterogeneous effects where justified; translate outcomes into value; incorporate constraints and uncertainty; compare policies; deploy with monitoring; and learn through subsequent experimentation. This creates a feedback loop rather than a one-time analysis.

Illustrative Case: A Retention Intervention at a Subscription Platform

Background

Consider an illustrative subscription platform experiencing increased cancellations among monthly customers. The company proposes an automated message offering a temporary service benefit to customers identified as potentially at risk. A predictive churn model can rank customers, but management needs to know whether the benefit actually reduces cancellation and whether its cost is justified.

Situation and design

The team defines the eligible population as active monthly customers who have used the service during the previous 30 days and have not previously received the benefit. Customers are randomly assigned within risk strata to either the current retention process or the proposed message and benefit. The primary outcome is cancellation within 30 days. Secondary outcomes include subsequent usage, renewal revenue, and support contacts. Guardrails include complaints and benefit abuse.

The team chooses intention-to-treat analysis and pre-specifies a minimum practically important reduction of one percentage point in cancellation. It estimates sample size using the baseline cancellation rate and desired power, checks event instrumentation, and runs the test for a complete billing cycle to avoid partial exposure.

Analysis

Suppose the treatment group shows a lower cancellation rate than the control group, with a confidence interval that excludes zero. The effect is larger among high-risk customers, but the benefit cost is also higher because these customers receive a more expensive incentive. The team therefore calculates incremental retained contribution rather than treating every reduction in churn as equally valuable. It also examines whether customers would have renewed without the intervention, since unnecessary incentives reduce profitability.

A naïve policy would offer the benefit to every predicted high-risk customer. A causal targeting analysis instead estimates incremental benefit. Some high-risk customers are “sure things” who would renew anyway; others are unlikely to renew even with treatment. The economically attractive segment consists of customers whose behaviour is changed by the intervention at a cost below the value of retention.

Decision and outcome

Management approves a constrained rollout to the segment with positive expected net value, subject to a monthly budget and a fairness review. A holdout group is retained for ongoing measurement because treatment effects may decay as customers become familiar with the message. Monitoring detects a later increase in complaints, leading to a revised message and a new experiment.

Learning points

The case demonstrates that prediction is useful for prioritising measurement and targeting, but prediction alone cannot establish intervention value. It also shows why primary outcomes, guardrails, economic value, heterogeneity, holdouts, and iterative experimentation are essential to a responsible decision system.

Challenges, Risks, and Common Errors

One common error is confusing correlation with causation. A variable may predict an outcome because it is a proxy for a causal factor, a consequence of the outcome, or a selection mechanism. Another is post-treatment adjustment: controlling for variables affected by treatment can distort the estimated total effect.

Selection bias arises when participation or treatment assignment is related to expected outcomes. Attrition can be especially damaging when dropout differs between groups. Novelty and learning effects may make an intervention appear stronger or weaker than it will be after routine adoption. Seasonality, concurrent campaigns, technical outages, and policy changes can contaminate comparisons.

Operational errors are equally important. Treatment leakage, inconsistent eligibility, duplicate identities, logging failures, and changes to the control condition undermine internal validity. External validity may be limited when an experiment is conducted on a narrow population, during an unusual period, or in an artificial environment. Transporting an effect to another setting requires reasoning about differences in population, treatment delivery, baseline risk, and institutional context.

Decision errors include optimising a proxy, ignoring uncertainty, treating statistical significance as business importance, and applying average effects to individuals without evidence. A model can also create feedback loops: an intervention changes future data, which then changes model predictions and subsequent treatment assignment. Continuous monitoring and periodic re-evaluation are therefore required.

Ethics, Governance, and Responsible Experimentation

Experiments affect real people and may distribute benefits, burdens, or opportunities unequally. Organisations should establish a legitimate purpose, minimise risk, protect privacy, and provide appropriate disclosure where required. Sensitive experiments in health, credit, employment, education, or public services may require formal review and additional legal safeguards.

Consent requirements vary by context and jurisdiction, but the absence of a consent form does not eliminate ethical responsibility. Deception, differential access, emotional manipulation, and exclusion of vulnerable groups require careful justification. Fairness should be evaluated not only in outcome rates but also in treatment access, error rates, burden, and the distribution of benefits.

Governance should document the hypothesis, protocol, estimand, population, treatment, analysis plan, deviations, data sources, code versions, and decision rule. Access controls, data minimisation, secure handling, and retention limits protect participants. An independent review or experimentation committee can assess high-risk designs. Results should be reported honestly, including null effects, adverse outcomes, uncertainty, and deviations from the original plan.

Technology and Emerging Practice

Modern experimentation platforms automate assignment, feature delivery, metric computation, sample-size monitoring, and dashboard reporting. Feature flags permit controlled release, while data pipelines connect exposure logs with outcome systems. These tools improve speed but do not solve identification problems; an automated platform can reliably measure a poorly designed experiment.

Machine learning is increasingly used for heterogeneous treatment effects, sequential experimentation, contextual bandits, and adaptive allocation. Bandits can allocate more traffic to apparently successful alternatives, potentially improving short-term reward, but they complicate inference and may under-explore uncertain options. Causal representation learning and structural models aim to combine domain knowledge with high-dimensional data, although their conclusions remain dependent on assumptions.

Synthetic data, privacy-preserving computation, and federated analytics may support experimentation where direct data sharing is restricted. Digital twins and simulation can evaluate policies before deployment, but simulated results should not be treated as empirical causal evidence without validation. The emerging standard is therefore not “more automation,” but auditable systems that combine statistical design, domain expertise, governance, and continuous learning.

Best Practices for Professional Use

A credible experimentation and decision-analytics programme should:

• define the decision, population, intervention, comparator, outcome, and time horizon before analysis;
• select the correct unit of randomisation and account for interference and clustering;
• pre-specify primary outcomes, guardrails, analysis methods, and stopping rules;
• distinguish intention-to-treat effects from effects among actual recipients;
• report effect sizes and confidence intervals alongside p-values;
• test assumptions using balance checks, pre-trends, overlap diagnostics, placebo tests, and sensitivity analysis;
• separate exploratory subgroup discovery from confirmatory evaluation;
• convert causal effects into expected value while including costs, risks, capacity, and uncertainty;
• retain holdout groups when long-term effects or adaptation are plausible;
• document decisions, data lineage, model versions, and deviations;
• monitor fairness, privacy, safety, and unintended consequences after deployment.

Chapter Summary

Experimentation and causal reasoning enable data scientists to move from observed associations to defensible claims about interventions. The counterfactual framework defines treatment effects by comparing potential outcomes, although only one potential outcome is observed for each unit. Randomised A/B tests create credible comparisons through assignment, while quasi-experimental methods use time, thresholds, instruments, or constructed controls when randomisation is unavailable.

Good experiments require a clear decision, precise treatment and control definitions, appropriate randomisation units, valid outcome measurement, sufficient power, pre-specified analyses, and attention to interference, attrition, multiple testing, and implementation fidelity. Causal graphs and explicit assumptions help analysts avoid inappropriate adjustment and clarify the limits of evidence.

Causal estimates become managerial value only when combined with predictions, costs, constraints, fairness considerations, and optimisation. Predictive models identify risk, causal models estimate incremental effects, prescriptive analytics compares policies, and optimisation selects feasible actions. Responsible practice requires transparency, privacy, ethical review, monitoring, and honest reporting of uncertainty and harm.

Key Terms

Counterfactual: The outcome that would have occurred for a unit under an alternative treatment condition.

Potential outcome: An outcome defined under a particular treatment or intervention state.

Average treatment effect (ATE): The average difference between treated and untreated potential outcomes in a target population.

Average treatment effect on the treated (ATT): The average causal effect among units that received treatment.

Randomisation: Assignment of units to treatment conditions using a chance mechanism.

Intention-to-treat: Estimating effects according to assigned treatment, regardless of actual exposure.

Treatment-on-the-treated: The effect among units that actually received or complied with treatment.

Interference: A situation in which one unit’s outcome depends on another unit’s treatment.

SUTVA: Assumptions of stable treatment versions and no interference in a potential-outcomes analysis.

Guardrail metric: A measure used to detect harmful or undesirable effects of an intervention.

Statistical power: The probability of detecting a specified effect when that effect exists.

Difference-in-differences: A design comparing changes over time between treated and comparison groups.

Parallel trends: The assumption that treated and comparison groups would have followed similar trends without treatment.

Regression discontinuity: A design exploiting treatment assignment at a known threshold.

Instrumental variable: A variable that affects treatment but affects the outcome only through treatment under stated assumptions.

Confounder: A variable influencing both treatment assignment and the outcome.

Collider: A variable caused by two other variables, conditioning on which can create bias.

Heterogeneous treatment effect: Variation in causal effect across people, groups, or contexts.

Prescriptive analytics: Analytics that recommends actions based on expected consequences and objectives.

Robust optimisation: Optimisation designed to perform acceptably under uncertainty about inputs or model assumptions.

Review Questions

Short Answer Questions

1. Why does a predictive model not by itself establish that an intervention will change an outcome?

2. Define the average treatment effect and distinguish it from the average treatment effect on the treated.

3. What is the purpose of a guardrail metric in an A/B test?

4. State the parallel-trends assumption in a difference-in-differences design.

5. What is the difference between a confounder and a mediator?

Long Answer Questions

1. Explain how an A/B test should be designed from the initial managerial decision through analysis and reporting.

2. Compare randomised experiments with difference-in-differences, regression discontinuity, instrumental variables, and matching.

3. Discuss how causal graphs help analysts decide which variables to adjust for and which variables may introduce bias.

4. Explain how predictive, causal, prescriptive, and optimisation analytics can be integrated in a customer-retention problem.

5. Discuss the ethical, governance, and operational requirements for experimentation in high-impact domains.

Analytical / Application Questions

1. A retailer assigns stores in different regions to a new pricing policy, but the treated regions were already experiencing faster sales growth. Which assumptions and diagnostic analyses would you examine before using difference-in-differences?

2. A bank’s churn model identifies customers likely to leave, and a retention offer reduces churn among those contacted. Design an analysis to determine whether the offer creates incremental profit rather than merely rewarding customers who would have stayed.

3. A public agency assigns a subsidy to applicants scoring above a threshold. Explain how you would assess a regression-discontinuity design, identify its local interpretation, and investigate manipulation or other threats to validity.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a causal question from a predictive question?

A. A causal question estimates what would happen under an intervention, whereas a predictive question estimates what is likely to happen.

B. A causal question always requires a neural network, whereas a predictive question requires regression.

C. A causal question describes past events, whereas a predictive question describes only current events.

D. A causal question is qualitative, whereas a predictive question is quantitative.

Correct Answer: A. A causal question estimates what would happen under an intervention, whereas a predictive question estimates what is likely to happen.

Question 2

In a randomised A/B test, the primary value of random assignment is that it tends to:

A. Guarantee that every individual receives the better treatment.

B. Balance observed and unobserved characteristics between groups in expectation.

C. Eliminate sampling variation from the estimated effect.

D. Make the treatment effect identical for all subgroups.

Correct Answer: B. Balance observed and unobserved characteristics between groups in expectation.

Question 3

A company compares changes in sales before and after a policy in treated stores with changes in untreated stores. Which assumption is central to a difference-in-differences interpretation?

A. The treated stores must have exactly the same sales level as control stores before treatment.

B. The policy must affect every store identically.

C. Treated and control stores would have followed parallel trends without the policy.

D. No outcome data may be collected after treatment.

Correct Answer: C. Treated and control stores would have followed parallel trends without the policy.

Question 4

Why can conditioning on a collider produce a biased causal estimate?

A. It removes all treatment variation from the data.

B. It can open a non-causal association between variables that caused the collider.

C. It guarantees that treatment is randomly assigned.

D. It converts a continuous outcome into a binary outcome.

Correct Answer: B. It can open a non-causal association between variables that caused the collider.

Question 5

A treatment reduces cancellation by 2 percentage points, but its cost exceeds the contribution gained from retained customers. What is the most appropriate managerial conclusion?

A. Deploy the treatment because statistical significance always implies profitability.

B. Reject the treatment solely because the effect is not zero.

C. Evaluate incremental net value and avoid deployment if expected costs exceed benefits.

D. Apply the treatment only to customers with the highest predicted cancellation probability, without further analysis.

Correct Answer: C. Evaluate incremental net value and avoid deployment if expected costs exceed benefits.

Question 6

Which analysis most directly preserves the benefits of randomisation when some assigned users do not actually receive the treatment?

A. Intention-to-treat analysis

B. Post-treatment subgroup analysis only

C. Analysis of users who voluntarily opened the message

D. Adjustment for every variable measured after assignment

Correct Answer: A. Intention-to-treat analysis

Question 7

In a regression-discontinuity design, the estimated effect is usually interpreted as:

A. A universal effect for every population and context.

B. A local effect for units near the assignment threshold.

C. A purely correlational effect with no assumptions.

D. An effect that requires treatment to be randomly assigned across the entire population.

Correct Answer: B. A local effect for units near the assignment threshold.

Question 8

Which example best illustrates interference between units?

A. A customer’s purchase depends only on their own assigned webpage.

B. A patient’s outcome is measured with random recording error.

C. A referral promotion given to one customer changes the behaviour of their friends.

D. A store has different sales in different months.

Correct Answer: C. A referral promotion given to one customer changes the behaviour of their friends.

Question 9

What is the main danger of repeatedly examining many metrics and stopping an experiment when any one becomes significant?

A. It necessarily reduces the sample size to zero.

B. It increases the probability of false-positive findings.

C. It guarantees a negative treatment effect.

D. It prevents the calculation of confidence intervals.

Correct Answer: B. It increases the probability of false-positive findings.

Question 10

Which sequence best represents the relationship among advanced analytics approaches?

A. Optimisation predicts outcomes, prediction establishes causality, and causality describes historical data.

B. Prediction estimates likely outcomes, causal analysis estimates intervention effects, and optimisation selects feasible actions.

C. Prescriptive analytics replaces the need for objectives and constraints.

D. Causal analysis and optimisation are identical because both use numerical models.

Correct Answer: B. Prediction estimates likely outcomes, causal analysis estimates intervention effects, and optimisation selects feasible actions.