Experimentation and Incrementality

Introduction

Advertising analytics increasingly requires an answer to a causal question: What additional outcome was produced by the campaign? Descriptive metrics such as impressions, clicks, conversions and revenue show what happened among exposed audiences, but they do not by themselves show what would have happened without advertising. That unobserved alternative is the counterfactual.

Incrementality is the difference between the outcome observed under advertising and the outcome that would have occurred in its absence. Experimentation provides the most credible framework for estimating this difference. It supports decisions about budget allocation, audience strategy, channel selection, creative effectiveness and return on advertising spend (ROAS).

This chapter examines A/B tests, randomized controlled trials, geo-experiments, holdouts, lift studies and quasi-experimental designs. It also explains confidence intervals, statistical significance, effect size, practical relevance, power, threats to validity and the governance requirements of causal measurement.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish correlation, attribution and incrementality.
2. Design an appropriate randomized or quasi-experimental study.
3. Calculate and interpret incremental conversion, revenue and return on incremental investment.
4. Explain confidence intervals, p-values, statistical power and effect size.
5. Identify threats to internal and external validity.
6. Interpret geo-experiments, holdouts and lift studies for managerial decisions.
7. Assess the ethical, privacy and operational implications of experimentation.

1. From Attribution to Causal Measurement

Attribution assigns credit for an observed conversion to one or more marketing contacts. For example, a last-click model may credit a purchase to a search ad because the search click occurred immediately before conversion. Attribution is useful for reporting customer journeys, but it can over-credit channels that reach people who were already likely to buy.

Causal measurement asks a stronger question: did exposure to the advertising change the probability, value or timing of the outcome? Let Y_i(1) denote the potential outcome for person, household, region or other unit i if treated with advertising, and Y_i(0) the potential outcome for the same unit without advertising. The individual treatment effect is:

The fundamental problem of causal inference is that both potential outcomes cannot normally be observed for the same unit at the same time. An experiment estimates an average treatment effect by constructing a credible comparison group.

The average treatment effect (ATE) is:

Here, E denotes the population mean. The formula applies when treatment and control outcomes are defined consistently and the target population is specified. In advertising, Y may be a binary conversion indicator, number of orders, revenue, profit or a customer-retention measure.

For treated units only, the average treatment effect on the treated (ATT) is:

where D = 1 identifies units receiving treatment. In a properly randomized experiment, the estimated treatment effect generally identifies the ATE for the experimental population; in observational studies, the ATT may be easier to identify than the ATE under suitable assumptions.

Incremental versus observed outcomes

Suppose a campaign produces 12,000 conversions. If 9,000 would have occurred without advertising, the incremental conversions are 3,000. The campaign’s observed conversions are not equivalent to its causal contribution.

For a treated group and a control group, the simple difference in means is:

where Ȳ_T is the mean outcome in treatment and Ȳ_C is the mean outcome in control. In a randomized experiment, Δ estimates the ATE when randomization is implemented correctly and interference is limited.

For conversion rates, incremental lift is often reported as:

where p_T and p_C are treatment and control conversion rates. Relative lift is undefined when p_C = 0 and can appear large when the baseline is very small. Absolute lift, p_T − p_C, should therefore always be reported alongside relative lift.

2. Experimental Designs for Advertising

2.1 A/B tests and randomized controlled trials

An A/B test randomly assigns eligible units to two or more conditions. In a basic advertising test, the treatment group is eligible to receive the campaign and the control group is deliberately withheld from it. Randomization makes treatment assignment independent of measured and unmeasured pre-treatment characteristics in expectation.

A randomized controlled trial (RCT) follows the same logic but may involve more formal protocols, multiple treatment arms, prespecified hypotheses, power calculations and governance procedures. The unit of randomization may be:

• an individual user;
• a cookie, device or logged-in account;
• a household;
• a store or sales territory;
• a postcode, municipality or media market; or
• a time period.

The unit must match the mechanism of interference. If household members share media exposure, randomizing individuals may contaminate the control group. If advertising is purchased at regional level, geographic randomization is more appropriate.

2.2 User-level holdouts

A holdout reserves a randomly selected portion of an eligible audience from advertising. A common design might assign 90% to treatment and 10% to control. The holdout estimates the outcome that would have occurred without the campaign.

Advantages include operational simplicity, direct measurement and compatibility with digital platforms. Limitations include spillover, imperfect suppression, cookie loss, cross-device identity problems and the possibility that the holdout is exposed through other channels.

The treatment effect depends on the contrast actually delivered. If the treatment group receives only 70% of intended impressions, the experiment estimates the effect of that delivered exposure, not necessarily the effect of the planned media schedule.

2.3 Intent-to-treat and treatment-on-the-treated effects

The intent-to-treat (ITT) effect compares units according to their randomized assignment, regardless of whether treatment was successfully delivered:

Z is assignment status. ITT is usually the primary estimate because it preserves the benefits of randomization and reflects real operational effectiveness.

A treatment-on-the-treated (TOT) estimate attempts to measure the effect among units actually exposed. Removing non-exposed units from the analysis can introduce selection bias. Under specific instrumental-variable assumptions, assignment Z may be used as an instrument for actual exposure D. A simplified Wald estimator is:

This interpretation requires relevance, exclusion, independence and a monotonicity condition. It identifies a local effect for compliers rather than necessarily the effect for every audience member.

2.4 Geo-experiments

Geo-experiments randomize or otherwise select geographic units such as cities, regions or designated market areas. Treatment geographies receive advertising while control geographies do not, or receive a different level of investment.

They are useful when media buying, retail availability or outdoor advertising operates at geographic scale. They reduce user-level identity problems and can capture cross-device effects. Their disadvantages include a small number of units, geographic heterogeneity, spillover between neighbouring areas and correlated shocks such as weather, local events or competitor activity.

A pre-period design should compare trends before treatment. If treatment and control regions had substantially different trajectories before launch, a simple post-period comparison may be misleading.

2.5 Pre–post and difference-in-differences designs

A difference-in-differences (DiD) estimator compares the change in treatment areas with the change in control areas:

The key identifying assumption is parallel trends: in the absence of treatment, the average change in the treatment group would have been the same as the average change in the control group. Pre-treatment data cannot prove this assumption, but it can provide evidence about its plausibility.

A regression representation is:

where Y_it is the outcome for unit i at time t; Treat_i indicates the treatment group; Post_t indicates the post-treatment period; and δ estimates DiD under the parallel-trends assumption. α, β and γ are fixed terms, and ε_it is the residual.

With multiple periods and units, analysts may add unit fixed effects and time fixed effects:

α_i controls for time-invariant unit differences, while λ_t controls for shocks common to all units. Standard errors must account for dependence within units and, often, clustering at the treatment-assignment level.

2.6 Synthetic controls and matched markets

When only one or a few regions receive treatment, a synthetic control can construct a weighted combination of untreated regions whose pre-treatment outcome closely resembles the treated region. The post-treatment gap between the treated region and its synthetic counterpart estimates an effect, subject to the quality of the match and the assumption that untreated regions provide a valid counterfactual.

Matched-market designs pair regions using historical sales, category penetration, demographics, media costs and pre-period trends. Matching improves comparability but does not replace randomization. Unobserved differences and later divergence remain possible.

3. Planning an Experiment

A credible experiment begins before campaign launch.

3.1 Define the estimand

The estimand is the precise quantity to be estimated. Examples include:

• incremental probability of purchase within 14 days;
• incremental revenue per eligible customer within 30 days;
• incremental profit per exposed household;
• incremental store visits during the campaign; or
• change in brand consideration measured through a survey.

The estimand should specify the population, treatment, comparison, outcome, measurement window and analysis method. Vague objectives such as “measure campaign performance” create ambiguity at the interpretation stage.

3.2 Choose the randomization unit and exposure rule

The randomization unit should be stable, identifiable and aligned with the intervention. Analysts should document eligibility, exclusion rules, frequency caps, treatment allocation, suppression procedures and exposure windows.

Randomization is not the same as random delivery. A platform may assign a user to treatment, but auctions, budgets and eligibility rules may determine whether an impression is actually delivered. The analysis must distinguish assignment from exposure.

3.3 Determine the primary outcome

A primary outcome should be closely connected to the campaign objective and measured reliably. Secondary outcomes can include visits, sign-ups, average order value, unsubscribes and complaints. Excessive testing of many outcomes increases the chance of finding a seemingly significant result by chance.

3.4 Power and minimum detectable effect

Statistical power is the probability of detecting an effect of a specified size if that effect truly exists. The minimum detectable effect (MDE) is the smallest effect the design can detect with chosen power and significance level.

For a two-sided comparison of two independent proportions with equal sample sizes, an approximate per-group sample size is:

Here, n is the sample size per group; p̄ is the planning conversion rate averaged across groups; Δ is the absolute conversion-rate difference to be detected; α is the two-sided Type I error rate; β is the Type II error rate; and z_q is the qth standard-normal quantile. This approximation assumes independent observations, a sufficiently large sample for the normal approximation and equal allocation.

For example, suppose the baseline conversion rate is 4%, the desired absolute lift is 0.8 percentage points, α = 0.05 and power = 0.80. Using z_(0.975) ≈ 1.96 and z_(0.80) ≈ 0.84, p̄ ≈ 0.044. Then:

n ≈ [2 × 0.044 × 0.956 × (1.96 + 0.84)²] ÷ 0.008²

The numerator is approximately 0.693, and the denominator is 0.000064, producing approximately 10,830 units per group. The result is a planning approximation, not a guarantee. Clustering, noncompliance, attrition, repeated observations and multiple treatment arms may require a larger sample.

For clustered geo-experiments, the effective sample size is reduced by within-cluster similarity. A commonly used design-effect approximation is:

where m is the average number of observations per cluster and ρ is the intracluster correlation coefficient. The approximate individual-level sample requirement is multiplied by DE. In practice, the number of clusters is often more important than the number of people inside each cluster.

4. Estimation and Statistical Interpretation

4.1 Difference in conversion rates

Suppose treatment has x_T conversions among n_T eligible units and control has x_C conversions among n_C units:

The estimated absolute effect is p_T − p_C. Under independent assignment and sufficiently large samples, an approximate standard error is:

An approximate 95% confidence interval is:

This interval is appropriate as a large-sample approximation. For small counts or rare events, analysts should use methods such as exact intervals, Wilson intervals or a suitable regression model rather than relying mechanically on the normal approximation.

4.2 Worked example: conversion lift

An online retailer randomly assigns 50,000 eligible customers to treatment and 50,000 to control. Treatment produces 2,750 conversions; control produces 2,500.

p_T = 2,750 ÷ 50,000 = 0.055 = 5.5%

p_C = 2,500 ÷ 50,000 = 0.050 = 5.0%

Absolute lift = 0.055 − 0.050 = 0.005, or 0.5 percentage points.

Relative lift = 0.005 ÷ 0.050 = 0.10, or 10%.

The standard error is approximately:

SE = √[(0.055 × 0.945 ÷ 50,000) + (0.050 × 0.950 ÷ 50,000)]

SE ≈ 0.00142.

The 95% confidence interval is:

0.005 ± 1.96 × 0.00142 = 0.005 ± 0.00278.

Thus, the interval is approximately 0.0022 to 0.0078, or 0.22 to 0.78 percentage points. Because the interval excludes zero, the result is statistically distinguishable from no effect at the 5% level under the stated assumptions. The practical value depends on margin, campaign cost, customer quality and whether the result generalizes beyond the experiment.

4.3 Revenue and incremental return

If the outcome is revenue per eligible unit, the effect is the difference in mean revenue. If the treatment creates an estimated incremental revenue R_inc and campaign cost is C, incremental return on advertising spend is:

If profit is available, incremental return on investment is more decision-relevant:

where Π_inc is incremental contribution profit before campaign cost. Revenue should not be treated as profit unless margin, returns, discounts and variable costs have been addressed.

Suppose the conversion experiment above covers 50,000 treatment units and the estimated absolute lift is 0.005. Incremental conversions are approximately:

50,000 × 0.005 = 250.

If contribution profit per incremental order is £18, incremental profit is £4,500. If campaign cost is £3,000, then iROI = (£4,500 − £3,000) ÷ £3,000 = 0.50, or 50%. This conclusion depends on the validity of the incremental conversion estimate and the profit assumption.

4.4 Significance, uncertainty and practical relevance

A p-value is the probability, under a specified null model, of observing a result at least as incompatible with the null as the result obtained. It is not the probability that the null hypothesis is true, nor is it the probability that the campaign “worked.”

A confidence interval communicates both direction and precision. A narrow interval around a small effect may be statistically persuasive but commercially unimportant. A wide interval that includes meaningful positive and negative effects indicates decision uncertainty, not proof of no effect.

Effect size should be reported in units decision-makers understand: percentage points, incremental orders, revenue per customer, profit, cost per incremental acquisition or changes in brand measures. Statistical significance is influenced strongly by sample size; practical relevance depends on economics and strategic priorities.

4.5 Multiple comparisons and sequential monitoring

Testing many audiences, creatives, outcomes or time windows increases the probability of at least one false positive. Prespecification, hierarchical testing, false-discovery-rate procedures or familywise error adjustments can reduce this risk.

Repeatedly checking results and stopping when significance appears can inflate Type I error unless a sequential-testing framework is used. Interim monitoring should therefore be planned, documented and governed.

5. Lift Studies and Survey-Based Experiments

A lift study may measure incremental brand awareness, consideration, preference, purchase intent or ad recall. Eligible respondents are randomized to an exposed or control condition, often using a platform-based survey. The difference in response rates estimates lift under the study design.

Survey lift can complement sales experiments when the purchase cycle is long or sales are infrequent. However, self-reported outcomes may be affected by response bias, demand effects, panel composition and imperfect exposure measurement. A positive brand-lift result does not automatically establish incremental sales or profit.

A strong design links the survey population to the campaign’s target audience, uses neutral wording, reports response rates and distinguishes assignment from self-reported exposure. Where possible, survey outcomes should be triangulated with behavioural measures.

6. Quasi-Experimental Methods

Quasi-experiments are used when random assignment is infeasible, unethical or operationally impossible. They attempt to reproduce the counterfactual using design and modelling assumptions.

Important methods include:

• Difference-in-differences: compares changes across treated and untreated units.
• Interrupted time series: models whether the outcome level or trend changes after intervention.
• Regression discontinuity: compares units just above and below a treatment threshold, assuming units cannot precisely manipulate the threshold and potential outcomes vary smoothly there.
• Synthetic control: creates a weighted comparison from untreated units.
• Propensity-score methods: balance observed covariates associated with treatment, but cannot automatically remove unmeasured confounding.
• Instrumental variables: use an external assignment mechanism related to treatment but affecting the outcome only through treatment, subject to strong assumptions.

Observational regression alone does not establish causality. A model can control for measured variables while leaving bias from unmeasured intent, prior engagement, sales capacity or platform optimization.

7. A Detailed Illustrative Case: Regional Streaming-Service Campaign

Background

A fictional streaming service planned a six-week video campaign to increase paid subscriptions in a mature market. The media team expected a strong platform-reported ROAS, but many targeted households had previously visited the website and were likely to subscribe without advertising.

Problem

The company needed to estimate incremental subscriptions and determine whether the campaign should be expanded. Individual-level suppression was unreliable because households used multiple devices, so a geo-experiment was selected.

Analysis and approach

Forty comparable regions were identified using two years of subscription history, population, broadband penetration, competitor intensity and media cost. Twenty regions were randomly assigned to treatment and twenty to control. Treatment regions received the campaign; control regions were excluded from the principal media placements. The primary outcome was new paid subscriptions per 10,000 households during the six-week period. The analysis plan was approved before results were inspected.

Pre-period trends were examined. Treatment and control regions had similar average subscription rates and parallel trajectories, although two regions experienced unusual sports events during the test and were flagged for sensitivity analysis. The analysis used a DiD model with region and week fixed effects and standard errors clustered by region.

Outcome

Treatment regions generated an average of 118 subscriptions per 10,000 households during the campaign, compared with 105 in the pre-period. Control regions generated 109 during the campaign, compared with 103 in the pre-period.

The DiD estimate was:

(118 − 105) − (109 − 103) = 13 − 6 = 7 subscriptions per 10,000 households.

If the treatment regions contained 2,000,000 households, the estimated incremental subscriptions were:

2,000,000 ÷ 10,000 × 7 = 1,400.

Suppose contribution profit per new subscription, after expected early churn, was £24. Estimated incremental contribution profit was £33,600. With campaign cost of £28,000, incremental profit after advertising cost was £5,600 and iROI was £5,600 ÷ £28,000 = 20%.

The team then repeated the model excluding the two regions affected by unusual events. The estimate fell to 5 subscriptions per 10,000 households, with a confidence interval that included zero. The appropriate conclusion was not that the campaign definitively failed, but that the evidence was sensitive to regional shocks and insufficiently precise for confident expansion.

Learning points

1. A high platform-attributed subscription count need not imply high incremental subscriptions.
2. Region-level experiments require attention to pre-trends and external shocks.
3. A point estimate should be accompanied by uncertainty and sensitivity analysis.
4. Profit-based evaluation is more useful than revenue-only reporting.
5. A decision may appropriately be “test again with more regions or a longer period” rather than immediate scale-up or cancellation.

8. Validity, Bias and Operational Failure

Internal validity

Internal validity concerns whether the estimated effect is causal for the experimental population. Threats include:

• Selection bias: assignment or analysis differs systematically between groups.
• Contamination: control units receive treatment through another route.
• Noncompliance: assigned treatment is not delivered or control is accidentally exposed.
• Spillover: treatment affects control units through social, household or geographic networks.
• Attrition and identity loss: outcomes are observed differently across groups.
• Instrumentation changes: tracking or conversion definitions change during the test.
• Seasonality and concurrent campaigns: other events coincide with treatment.
• Novelty and carryover: effects differ during launch from those after sustained exposure.

External validity

External validity concerns generalization. Results may not transport to another season, region, audience, budget level, creative execution, auction environment or competitive situation. Treatment effects can be heterogeneous: a campaign may work for new customers but merely retarget existing demand among loyal customers.

SUTVA and interference

Many causal designs rely on a version of the stable unit treatment value assumption (SUTVA): one unit’s outcome is not changed by another unit’s treatment, and treatment is well-defined. Advertising often violates this through word of mouth, shared households, social platforms and geographic spillover. Analysts should select clusters, use buffer zones, measure cross-exposure and interpret results as effects of the full treatment environment where appropriate.

9. Governance, Ethics and Responsible Practice

Experimentation must respect privacy, consent requirements, platform policies and applicable data-protection law. Randomized withholding of advertising is generally less problematic than withholding essential services, but sensitive categories require heightened safeguards.

Best practice includes:

• collect only necessary data;
• use pseudonymous identifiers and access controls;
• document retention and deletion rules;
• avoid discriminatory targeting or exclusion;
• assess whether treatment changes access to essential products;
• disclose material experimentation where required;
• prevent re-identification through small-cell reporting;
• preserve an audit trail of assignment, exposure and analysis decisions; and
• separate exploratory analyses from confirmatory conclusions.

AI systems can support audience construction, anomaly detection, power simulation and heterogeneous-treatment-effect analysis. They do not remove the need for randomization, identification assumptions or human review. Automated optimization can also create post-treatment selection: if an algorithm reallocates impressions toward users who respond well, analysing only delivered impressions may bias the apparent effect.

10. Best-Practice Workflow

A professional experimentation workflow is:

1. Define the decision and causal estimand.
2. State the treatment, control condition, population and outcome window.
3. Select the randomization unit and assess interference.
4. Estimate baseline rates, MDE, sample size and test duration.
5. Register hypotheses, primary outcomes and stopping rules.
6. Implement assignment, suppression and exposure logging.
7. Audit balance, delivery, contamination and measurement before analysing outcomes.
8. Estimate effects using an appropriate model and uncertainty interval.
9. Report absolute effect, relative lift, effect size, statistical evidence and economics.
10. Conduct sensitivity, subgroup and robustness analyses without disguising exploratory work as confirmatory evidence.
11. Decide whether to scale, modify, repeat or stop the campaign.
12. Archive data, code, assumptions and governance approvals.

Chapter Summary

Experimentation estimates advertising incrementality by comparing observed outcomes with a credible counterfactual. Randomized A/B tests and holdouts offer strong internal validity, while geo-experiments and quasi-experimental methods are valuable when individual randomization is impractical. The estimand, unit of assignment, outcome definition and exposure rule must be specified before launch.

Treatment effects may be expressed as absolute lift, relative lift, incremental conversions, revenue or profit. Confidence intervals communicate uncertainty; p-values address compatibility with a null model; and effect size and iROI support managerial interpretation. Statistical significance does not guarantee practical importance.

Valid conclusions require attention to power, contamination, interference, noncompliance, parallel trends, clustering, multiple comparisons and generalizability. Ethical experimentation requires privacy protection, fairness, transparency, responsible use of AI and a documented analytical audit trail.

Key Terms

Incrementality: The additional outcome caused by advertising relative to the relevant counterfactual.

Counterfactual: The outcome that would have occurred under an alternative treatment condition.

Potential outcome: An outcome a unit would experience under a specified treatment state.

Average treatment effect (ATE): The average causal effect across a defined population.

Average treatment effect on the treated (ATT): The average causal effect among treated units.

A/B test: A randomized comparison between two conditions.

Holdout: A randomly selected group withheld from treatment.

Intent-to-treat (ITT): The effect of assignment to treatment, regardless of actual exposure.

Treatment-on-the-treated (TOT): An effect estimate based on actual treatment receipt under additional assumptions.

Geo-experiment: An experiment conducted by assigning geographic units to conditions.

Difference-in-differences: An estimator comparing changes over time between treatment and control groups.

Parallel trends: The assumption that treatment and control would have changed similarly without treatment.

Effect size: The magnitude of an estimated treatment effect.

Confidence interval: An interval procedure expressing sampling uncertainty around an estimate.

Statistical power: The probability of detecting a specified effect under a chosen design.

Minimum detectable effect: The smallest effect a design is intended to detect at chosen power and significance level.

Contamination: Unintended exposure of control units to treatment.

Spillover: An effect of one unit’s treatment on another unit.

Incremental ROAS: Incremental revenue divided by campaign cost.

Incremental ROI: Incremental profit after campaign cost divided by campaign cost.

Review Questions

Short Answer Questions

1. Why is an attributed conversion not necessarily an incremental conversion?
2. Define the average treatment effect and identify its main components.
3. What is the purpose of a holdout group?
4. Explain the difference between absolute lift and relative lift.
5. State the parallel-trends assumption in difference-in-differences analysis.

Long Answer Questions

1. Compare user-level A/B tests, holdouts and geo-experiments in terms of validity, operational requirements and sources of bias.
2. Explain how statistical significance, confidence intervals and practical relevance should jointly inform campaign decisions.
3. Discuss the assumptions and limitations of difference-in-differences estimation.
4. Describe a complete process for planning, executing and reporting an advertising incrementality experiment.
5. Evaluate the ethical, privacy and governance issues involved in experimentation with digital advertising audiences.

Analytical / Application Questions

1. A treatment group has 80,000 eligible users and 4,800 conversions. A control group has 80,000 eligible users and 4,400 conversions. Calculate the conversion rates, absolute lift, relative lift and estimated incremental conversions in the treatment group. Interpret the result without claiming statistical significance unless sufficient uncertainty information is supplied.
2. A campaign is tested across ten treatment regions and ten control regions. Treatment sales rise after launch, but treatment regions had already been growing faster before launch. Which identification assumption is threatened, and what additional analyses or design changes would you recommend?
3. A platform reports a high ROAS for a retargeting campaign, but a randomized holdout shows only a small incremental conversion effect. Explain at least three reasons for the discrepancy and recommend a budget decision framework.

Multiple Choice Questions with Answers

Question 1

Which quantity most directly represents the causal effect of advertising for unit i?

A. Y_i(1) − Y_i(0)

B. Y_i(1) ÷ Y_i(0)

C. Y_i(1) + Y_i(0)

D. Y_i(0) − Y_i(1) only when treatment is profitable

Correct Answer: A. Y_i(1) − Y_i(0)

Question 2

A treatment group converts at 5.5% and a control group at 5.0%. What is the relative lift?

A. 0.5%

B. 5%

C. 10%

D. 110%

Correct Answer: C. 10%

Question 3

What is the primary purpose of random assignment?

A. To guarantee a large effect

B. To make treatment assignment independent of potential outcomes in expectation

C. To eliminate all measurement error

D. To ensure every assigned unit receives an impression

Correct Answer: B. To make treatment assignment independent of potential outcomes in expectation

Question 4

Which assumption is central to a basic difference-in-differences design?

A. Treatment and control have identical post-treatment outcomes

B. Treatment and control have equal sample sizes

C. Parallel trends in the absence of treatment

D. Every control unit must be exposed to advertising

Correct Answer: C. Parallel trends in the absence of treatment

Question 5

In the worked conversion example, what was the estimated absolute lift?

A. 0.05 percentage points

B. 0.22 percentage points

C. 0.50 percentage points

D. 10 percentage points

Correct Answer: C. 0.50 percentage points

Question 6

What does a 95% confidence interval primarily communicate?

A. The probability that the null hypothesis is true

B. The precision and uncertainty associated with an estimate under the chosen procedure

C. The probability that every customer was exposed

D. The campaign’s guaranteed future profit

Correct Answer: B. The precision and uncertainty associated with an estimate under the chosen procedure

Question 7

Which issue occurs when control users receive campaign exposure through another channel?

A. Contamination

B. Power inflation

C. Parallel treatment

D. Exact randomization

Correct Answer: A. Contamination

Question 8

Why can statistical significance fail to imply managerial importance?

A. Significance is unrelated to sample size

B. A statistically detectable effect may be too small to justify its cost

C. Confidence intervals always exclude zero

D. Practical relevance is determined only by the p-value

Correct Answer: B. A statistically detectable effect may be too small to justify its cost

Question 9

If incremental contribution profit is £4,500 and campaign cost is £3,000, what is incremental ROI?

A. 20%

B. 33.3%

C. 50%

D. 150%

Correct Answer: C. 50%

Question 10

Which statement about intent-to-treat analysis is correct?

A. It excludes all units that did not receive an impression

B. It compares units according to randomized assignment

C. It necessarily estimates the effect only among compliers

D. It is valid only for geographic experiments

Correct Answer: B. It compares units according to randomized assignment