Experimentation and Causal Analytics

Introduction

Data analytics is often used to identify relationships: users who receive a recommendation may purchase more, regions with higher advertising expenditure may report greater sales, and patients who use a treatment may experience better outcomes. Such relationships are useful for prediction, but they do not by themselves establish causation. A managerial decision—launching a product feature, changing a price, allocating a public budget, or adopting a medical intervention—requires a different question: What would have happened if the intervention had not occurred?

Experimentation and causal analytics provide the conceptual and statistical tools for answering this counterfactual question. Randomized experiments, particularly A/B tests, create comparable treatment and control groups by design. Quasi-experimental methods estimate causal effects when random assignment is infeasible, unethical, or unavailable. These methods include difference-in-differences, regression discontinuity, instrumental variables, matching, and synthetic controls.

At postgraduate level, causal analysis should be understood not merely as a collection of statistical techniques, but as a combination of research design, assumptions, measurement, statistical estimation, operational governance, and decision analysis. A sophisticated model cannot compensate for a poorly defined intervention, contaminated control group, selective measurement, or an implausible identification assumption.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish descriptive association, prediction, and causal effect.

2. Define treatment, outcome, unit of analysis, estimand, and counterfactual.

3. Design and evaluate online A/B tests, including randomization, power, metrics, and experimentation risks.

4. Estimate and interpret average treatment effects and related causal quantities.

5. Select and critically assess quasi-experimental designs.

6. Diagnose confounding, selection bias, interference, spillovers, and other threats to validity.

7. Use causal evidence responsibly in product, organizational, and public-policy decisions.

1. From Correlation to Causation

1.1 Association is not intervention evidence

An association exists when two variables vary together. For example, customers exposed to a premium offer may have higher revenue than unexposed customers. Several explanations are possible:

• The offer caused customers to spend more.
• High-value customers were more likely to receive the offer.
• A third variable, such as prior engagement, influenced both exposure and spending.
• The exposure and outcome were measured incorrectly.
• Customers changed behavior because they knew they were being observed.

Prediction focuses on estimating an outcome from available information. Causal analysis focuses on how the outcome would change under a specified intervention. A predictive model may accurately identify likely purchasers while providing no reliable estimate of the effect of sending an email campaign.

1.2 The counterfactual framework

Let unit \(i\) represent a customer, employee, patient, school, or geographic area. Let \(T_i=1\) indicate treatment and \(T_i=0\) indicate control. Define:

• \(Y_i(1)\): the potential outcome for unit \(i\) if treated;
• \(Y_i(0)\): the potential outcome for unit \(i\) if not treated.

The individual treatment effect is:

\[

\tau_i=Y_i(1)-Y_i(0).

\]

Only one potential outcome is observed for each unit. This is the fundamental problem of causal inference: the untreated outcome for a treated unit, or the treated outcome for a control unit, is unobserved. Causal methods attempt to construct a credible estimate of this missing counterfactual.

The average treatment effect (ATE) is:

\[

ATE=E[Y(1)-Y(0)]=E[Y(1)]-E[Y(0)].

\]

The expectation is over the target population. The ATE is meaningful only when treatment, outcome, population, and time horizon are clearly specified. Other estimands include the average treatment effect on the treated (ATT),

\[

ATT=E[Y(1)-Y(0)\mid T=1],

\]

and the average treatment effect on the untreated (ATU). The ATT may differ from the ATE when treatment effects vary across individuals.

1.3 Causal diagrams and assumptions

Directed acyclic graphs (DAGs) represent assumed causal relationships. A confounder affects both treatment assignment and outcome. For example, prior engagement may influence both the probability of receiving a recommendation and future purchases. Conditioning on a suitable set of pre-treatment confounders can block non-causal backdoor paths between treatment and outcome.

Three assumptions are central in many observational analyses:

1. Consistency: the observed outcome under the received treatment equals the corresponding potential outcome. This requires treatment versions to be sufficiently well defined.

2. Conditional exchangeability: after controlling for covariates \(X\), treatment assignment is independent of potential outcomes: \(Y(1),Y(0)\perp T\mid X\). This is also called no unmeasured confounding.

3. Positivity: for relevant values of \(X\), every unit has a nonzero probability of receiving each treatment: \(0<P(T=1\mid X)<1\).

In a randomized experiment, exchangeability is created by randomization rather than assumed conditional on measured variables. In observational research, exchangeability is generally a substantive claim that cannot be proven solely from the dataset.

2. Designing Randomized Experiments

2.1 The logic of an A/B test

An A/B test randomly assigns eligible units to version A (control) or version B (treatment). Randomization makes treatment assignment independent, in expectation, of both observed and unobserved baseline characteristics. Therefore, differences in average outcomes can be attributed to treatment, provided implementation and measurement are valid.

For a binary outcome, the difference in sample proportions is:

\[

\widehat{ATE}=\bar{Y}_1-\bar{Y}_0,

\]

where \(\bar{Y}_1\) and \(\bar{Y}_0\) are the observed treatment and control means. For a continuous outcome, the same difference-in-means estimator applies. Its interpretation depends on the randomization unit and the population represented by the experiment.

A valid experiment requires more than random allocation. The researcher must specify:

• the research question and estimand;
• eligible population and unit of randomization;
• treatment variants and control condition;
• primary and secondary outcomes;
• experiment duration and stopping rule;
• minimum practically important effect;
• sample-size and power requirements;
• exclusion, missing-data, and analysis rules;
• safeguards for users and affected stakeholders.

2.2 Unit of randomization and interference

The unit of randomization is the entity assigned to treatment. It may be a user, session, account, store, classroom, or geographic region. Randomizing sessions when treatment affects users across sessions can produce contamination. Randomizing individuals in a social network can create interference, where one user’s treatment affects another user’s outcome. The standard potential-outcomes framework often assumes no interference between units, commonly called the stable unit treatment value assumption (SUTVA). This assumption must be considered rather than silently adopted.

Cluster randomization is appropriate when members of a group interact or when treatment is delivered at group level. It usually reduces effective sample size because outcomes within a cluster are correlated. The intracluster correlation coefficient (ICC) measures within-cluster similarity. A common approximation for the design effect is:

\[

DE=1+(m-1)\rho,

\]

where \(m\) is average cluster size and \(\rho\) is the ICC. The required individual-level sample size is approximately multiplied by this design effect. This expression assumes roughly equal cluster sizes and a simple cluster-randomized design; unequal sizes and stratification require more careful planning.

2.3 Outcomes and metrics

A primary metric should be closely connected to the decision and measured consistently in both arms. Product experiments often use conversion, retention, revenue, latency, or error rate. A good metric is:

• sensitive to the proposed mechanism;
• difficult to manipulate without genuine improvement;
• measured over a suitable time horizon;
• interpretable in business or policy terms;
• unlikely to be distorted by exposure or logging changes.

Guardrail metrics detect harms that a primary metric may conceal. For example, a recommendation algorithm may increase clicks while increasing returns, complaints, or inequality. Multiple outcomes create multiplicity concerns: testing many metrics or subgroups increases the chance of at least one apparently significant result by chance. Primary outcomes, correction procedures, and exploratory analyses should be specified in advance where possible.

2.4 Sample size, power, and minimum effect

Statistical significance is not the same as practical importance. Experiment planning begins with a minimum detectable or minimum practically important effect, baseline variance or rate, significance level \(\alpha\), desired power \(1-\beta\), and allocation ratio.

For an individually randomized comparison of two means with equal group sizes, a commonly used planning approximation is:

\[

n_{group}\approx \frac{2\sigma^2(z_{1-\alpha/2}+z_{1-\beta})^2}{\delta^2},

\]

where \(\sigma^2\) is the assumed common outcome variance, \(\delta\) is the target absolute difference, and \(z_q\) is the q-th standard-normal quantile. This formula assumes independent observations, equal allocation, approximately normal sampling behavior, and a two-sided test. Binary outcomes require formulas involving baseline proportions; clustered or sequential designs require adjustments.

A very small effect can be statistically significant with a huge sample, while a valuable effect can fail to reach significance in an underpowered experiment. Confidence intervals should therefore accompany p-values. A confidence interval describes sampling uncertainty under the model and design; it does not automatically incorporate every source of operational or measurement uncertainty.

2.5 Implementation threats

Important threats include:

• Noncompliance: assigned users do not receive the assigned treatment.
• Attrition: outcome data are missing differentially across arms.
• Contamination: control users receive treatment indirectly.
• Novelty and learning effects: behavior changes temporarily after launch.
• Instrumentation changes: logging differs between variants.
• Peeking: repeated significance checks inflate false-positive risk unless sequential methods are used.
• Sample-ratio mismatch: observed assignment proportions differ unexpectedly from planned proportions, indicating allocation or logging problems.

The intention-to-treat (ITT) effect compares units according to assignment, regardless of compliance. ITT preserves the benefit of randomization and is usually the appropriate estimate of the effect of offering or assigning treatment. A per-protocol analysis estimates effects among adherent units but can introduce selection bias because adherence may be related to outcomes. Instrumental-variable methods can sometimes estimate a local average treatment effect for compliers, under additional assumptions.

3. Estimating and Interpreting Treatment Effects

3.1 Difference in means and regression adjustment

In a randomized experiment, the unadjusted difference in means is transparent and robust. Regression can improve precision by adjusting for pre-treatment covariates:

\[

Y_i=\alpha+\tau T_i+\gamma'X_i+\varepsilon_i.

\]

Here, \(Y_i\) is the outcome, \(T_i\) is treatment assignment, \(X_i\) contains pre-treatment covariates, and \(\tau\) is interpreted as the adjusted treatment effect under the model. In a properly randomized study, adjustment is not required for unbiasedness, but prognostic covariates can reduce variance. Post-treatment variables should generally not be adjusted for when estimating the total treatment effect, because they may be mediators or colliders.

For binary outcomes, risk difference, risk ratio, and odds ratio answer different questions. If treatment and control risks are \(p_1\) and \(p_0\), respectively:

\[

RD=p_1-p_0,\qquad RR=\frac{p_1}{p_0},\qquad OR=\frac{p_1/(1-p_1)}{p_0/(1-p_0)}.

\]

The risk difference is often most directly useful for decisions. Odds ratios can exaggerate apparent differences when outcomes are common and are frequently misinterpreted as risk ratios.

3.2 Heterogeneous treatment effects

The ATE can conceal important variation. Treatment may help new users but harm experienced users, or benefit high-risk patients while producing negligible effects for others. Heterogeneity can be studied through pre-specified subgroup analyses, interaction models, causal trees, Bayesian hierarchical models, or heterogeneous-treatment-effect learners.

Subgroup analysis must be justified by a plausible mechanism and adequate sample size. Searching many subgroups after observing results produces unstable findings and false discoveries. Individualized treatment rules also require validation: a model that predicts who responds may not correctly estimate who benefits causally.

3.3 Mediation and mechanisms

A treatment may affect an outcome through intermediate variables. For example, a faster interface may increase completed transactions by reducing abandonment. Mediation analysis distinguishes total, direct, and indirect effects, but it requires stronger assumptions than total-effect estimation. Conditioning on a mediator can block part of the treatment effect or create bias. Mechanistic analysis should therefore be planned separately from the primary total-effect analysis.

4. Quasi-Experimental Designs

Quasi-experiments are used when treatment is not randomly assigned. Their credibility depends on a transparent assignment mechanism and assumptions that make the counterfactual plausible.

4.1 Difference-in-differences

Difference-in-differences (DiD) compares changes over time in a treated group with changes in an untreated comparison group. With two groups and two periods:

\[

\widehat{\tau}_{DiD}=(\bar{Y}_{T,post}-\bar{Y}_{T,pre})-(\bar{Y}_{C,post}-\bar{Y}_{C,pre}).

\]

The key identifying assumption is parallel trends: absent treatment, the treated and comparison groups would have experienced the same average change. Pre-treatment trends can provide supporting evidence, but they cannot prove parallel trends after treatment. Researchers should inspect event-study plots, investigate anticipatory behavior, and account for time-varying confounders.

In settings with many treatment times, conventional two-way fixed-effects estimators can be misleading when treatment effects vary across groups or time. Modern DiD approaches explicitly address staggered adoption and heterogeneous effects.

4.2 Regression discontinuity

Regression discontinuity (RD) exploits a treatment rule based on whether a running variable crosses a threshold. For example, applicants with a score above 70 may receive a grant. Units just below and above the cutoff can be comparable if they cannot precisely manipulate the running variable.

The discontinuity in outcomes at the cutoff estimates a local average treatment effect. A sharp RD has deterministic assignment at the threshold; a fuzzy RD has a discontinuous probability of treatment and is often estimated using the threshold indicator as an instrument. Validity checks include continuity of baseline covariates, absence of sorting around the cutoff, appropriate bandwidth and functional form, and sufficient observations near the threshold. RD estimates may not generalize far from the cutoff.

4.3 Instrumental variables

An instrumental variable \(Z\) affects treatment \(T\) but affects outcome \(Y\) only through treatment. The core assumptions are:

• relevance: \(Z\) changes treatment probability;
• independence: \(Z\) is independent of potential outcomes, often conditional on covariates;
• exclusion: \(Z\) has no direct path to the outcome except through treatment;
• often, monotonicity: the instrument does not make some units less likely to receive treatment while making others more likely.

Under these assumptions, an instrumental-variable estimate commonly identifies a local average treatment effect (LATE) for compliers, not necessarily the ATE. Weak instruments produce unstable estimates and misleading conventional inference. Exclusion is usually the most difficult assumption to defend substantively.

4.4 Matching and weighting

Propensity-score methods estimate the probability of treatment given observed covariates:

\[

e(X)=P(T=1\mid X).

\]

Units can be matched or weighted using \(e(X)\), inverse-probability weights, or related balancing methods. These methods can reduce imbalance in measured pre-treatment covariates. They do not solve unmeasured confounding, and extreme propensity scores create unstable weights. Covariate balance should be assessed after matching or weighting; predictive accuracy of the propensity model is not the primary criterion.

4.5 Synthetic controls and interrupted time series

A synthetic control constructs a weighted combination of untreated units to approximate the treated unit’s pre-intervention trajectory. It is useful when one region, organization, or product is treated and a credible donor pool exists. An interrupted time-series design evaluates a change in level or slope after an intervention, ideally with multiple pre-intervention observations and a comparison series. Both designs are vulnerable to simultaneous shocks, anticipation, and model-dependent extrapolation.

5. Confounding, Bias, and Validity

5.1 A practical causal workflow

A defensible analysis follows these steps:

1. State the intervention precisely, including timing, intensity, and eligible population.

2. Define the estimand and outcome horizon before analyzing results.

3. Draw a causal diagram and identify plausible confounders, mediators, and colliders.

4. Select a design that creates or approximates exchangeability.

5. Audit data-generating and treatment-assignment processes.

6. Conduct balance, pre-trend, placebo, sensitivity, and robustness checks.

7. Report uncertainty, limitations, subgroup effects, and deviations from protocol.

8. Translate the estimate into operational, financial, or policy consequences.

5.2 Common sources of bias

Selection bias occurs when inclusion or observation depends on variables related to treatment and outcome. Confounding occurs when a common cause of treatment and outcome is not adequately controlled. Collider bias can arise when analysis conditions on a variable influenced by both treatment and outcome. Measurement error can attenuate or distort effects, particularly when measurement differs by treatment arm. Survivorship bias results from analyzing only units that remain observable. Regression to the mean can be mistaken for treatment improvement when unusually high or low observations trigger intervention.

Sensitivity analysis asks how strong unmeasured confounding would need to be to overturn a conclusion. No sensitivity analysis proves causation, but it makes dependence on assumptions explicit. Negative-control outcomes or exposures can also reveal residual bias when they should not be affected by treatment.

6. Applied Case: A Subscription-Retention Experiment

The following is an illustrative case based on a common digital-product setting.

Background and problem

A streaming platform observes that customers who receive personalized renewal reminders have higher renewal rates. Marketing proposes sending reminders to all customers 14 days before expiry. However, existing reminders are targeted to highly engaged customers, so the observed association may reflect selection rather than effectiveness.

The decision question is: What is the causal effect of sending a renewal reminder 14 days before expiry on renewal within 30 days, and does the benefit justify communication costs and possible customer annoyance?

Design

Eligible customers are those whose subscriptions expire within the next 14 days, who have not opted out of marketing, and who have not already received a renewal reminder. Customers are randomly assigned at the account level to:

• control: no reminder during the experiment;
• treatment: one personalized email reminder.

The primary outcome is renewal within 30 days of expiry. Guardrail outcomes include unsubscribe rate, customer complaints, and support contacts. Assignment is logged before message delivery, and analysis follows intention-to-treat principles.

Suppose 10,000 customers are assigned equally. In the treatment group, 2,700 of 5,000 renew; in control, 2,500 of 5,000 renew. Thus:

\[

\hat p_1=2700/5000=0.54,

\qquad \hat p_0=2500/5000=0.50.

\]

The estimated risk difference is:

\[

\widehat{ATE}=0.54-0.50=0.04,

\]

or a 4-percentage-point increase. The relative risk is \(0.54/0.50=1.08\), indicating an 8% relative increase. The number needed to treat, when the risk difference is positive and interpreted causally, is approximately \(1/0.04=25\): about 25 reminders are required for one additional renewal, subject to sampling uncertainty.

Analysis and decision

The team should calculate a confidence interval for the risk difference, verify sample-ratio balance, inspect missing outcomes, and confirm that no control customers received the reminder through another channel. Suppose unsubscribe rates rise from 1.0% to 1.8%, while complaints remain unchanged. The business should compare the incremental profit from additional renewals with message cost and the value assigned to customer experience. A statistically positive retention effect does not automatically justify universal deployment if the intervention creates material harm or if the effect is concentrated among customers who would have renewed anyway.

A follow-up experiment could compare message timing, frequency, or content, but treatment variants should not be introduced before the primary decision is resolved unless the design explicitly supports multiple comparisons. If treatment is selectively offered after the experiment based on predicted response, the targeting model should be evaluated using out-of-sample causal validation rather than ordinary response prediction.

Learning points

The original observational association did not establish effectiveness. Randomization addressed confounding; pre-specified guardrails addressed unintended consequences; ITT estimated the effect of offering the reminder; and decision analysis required both effect magnitude and cost-benefit interpretation.

7. Product, Policy, and Governance Implications

Causal evidence supports decisions at several levels. Product teams use experiments to evaluate features, pricing, ranking systems, and communication. Operations teams evaluate staffing, process changes, and fraud controls. Public agencies use quasi-experiments to assess programs when randomization is infeasible. In each setting, external validity matters: an effect observed in early adopters, one city, or one season may not transfer to other populations.

Ethically responsible experimentation requires informed consideration of risk, privacy, fairness, and autonomy. Users should not be exposed to unreasonable harm merely because an experiment may produce knowledge. Sensitive outcomes may require stronger review, data minimization, access controls, and legally appropriate consent or notification. Fairness analysis should examine whether treatment effects differ across protected or vulnerable groups. A policy that improves an average outcome while worsening outcomes for a disadvantaged group requires explicit scrutiny.

AI systems make causal analysis more valuable but do not eliminate the need for design. Machine-learning models can assist with heterogeneity discovery, covariate balance, uplift modeling, and heterogeneous policy learning. However, flexible models can overfit, exploit post-treatment variables, or produce impressive predictions without valid causal interpretation. Double machine learning and related orthogonalization methods can estimate causal parameters in high-dimensional settings under appropriate assumptions, but they still depend on identification, overlap, and reliable measurement.

Contemporary experimentation increasingly includes sequential decision systems, adaptive allocation, federated or privacy-preserving experimentation, and platform-scale interference. These developments require specialized methods rather than routine application of a standard t-test. Reproducible analysis plans, versioned treatment definitions, audit trails, and independent review are essential in high-stakes environments.

8. Best Practices

1. Begin with a decision and a clearly defined intervention, not with an available dataset.

2. Specify the estimand, population, outcome, time horizon, and practical decision threshold.

3. Prefer randomization when feasible and ethically acceptable.

4. Randomize at the level that prevents contamination and interference.

5. Use primary metrics and guardrails selected before examining results.

6. Plan power around a meaningful effect, not merely a desired sample size.

7. Preserve intention-to-treat analysis and document protocol deviations.

8. Use only pre-treatment covariates for routine precision adjustment.

9. Treat quasi-experimental assumptions as arguments requiring evidence, not as software settings.

10. Report effect sizes and uncertainty, not p-values alone.

11. Examine heterogeneity cautiously and validate targeting decisions prospectively.

12. Communicate limitations, transportability, ethics, and operational consequences.

Challenges and Limitations

No causal design provides certainty independent of assumptions. Randomized experiments may suffer from noncompliance, short follow-up, interference, selective participation, or limited generalizability. Quasi-experiments depend more heavily on untestable assumptions and may identify only local effects. Outcomes can be delayed, multidimensional, or strategically manipulated. Cost, privacy, and ethical constraints may limit experimentation. Finally, a statistically credible effect is not equivalent to a desirable policy: distributional consequences, opportunity costs, implementation feasibility, and long-term adaptation must also be considered.

Chapter Summary

Causal analytics estimates how outcomes change under interventions by constructing credible counterfactuals. The potential-outcomes framework defines individual and average treatment effects, while causal diagrams clarify confounding, mediation, and selection. Randomized A/B tests create exchangeability through assignment, but valid inference also requires careful treatment definition, measurement, power planning, compliance management, and guardrail monitoring. ITT is generally the principal effect of assignment or offer.

When randomization is unavailable, difference-in-differences, regression discontinuity, instrumental variables, matching, synthetic controls, and interrupted time series can provide useful evidence under design-specific assumptions. Parallel trends, exclusion, continuity at a threshold, positivity, and no unmeasured confounding must be defended rather than assumed. Effect estimates should be interpreted with uncertainty, practical significance, heterogeneity, external validity, and ethical consequences in view.

For product and policy decisions, causal evidence is strongest when it is connected to a clearly specified decision, a transparent analysis plan, robust diagnostics, responsible governance, and prospective validation of deployment choices.

Key Terms

Counterfactual: The outcome that would have occurred under an alternative treatment condition.

Potential outcome: The outcome a unit would experience under a specified treatment state.

Treatment: An intervention, exposure, policy, or product condition whose effect is studied.

Control: The comparison condition against which treatment is evaluated.

Average treatment effect (ATE): The average difference between treated and untreated potential outcomes in a target population.

Average treatment effect on the treated (ATT): The average causal effect among units that received treatment.

Randomization: Assignment by a chance mechanism to create comparable groups.

Intention-to-treat (ITT): Effect estimated according to assigned treatment, regardless of compliance.

Confounding: Distortion caused by a variable related to both treatment and outcome.

Exchangeability: Comparability of treatment groups with respect to potential outcomes.

Positivity: Nonzero probability of each treatment level for relevant covariate patterns.

Consistency: Alignment between the observed outcome and the potential outcome for the received treatment.

Interference: A situation in which one unit’s treatment affects another unit’s outcome.

Difference-in-differences: A design comparing outcome changes over time between treated and comparison groups.

Regression discontinuity: A design using a treatment threshold to compare units near a cutoff.

Instrumental variable: A variable that shifts treatment but affects the outcome only through treatment under identifying assumptions.

Propensity score: The probability of treatment conditional on observed covariates.

Guardrail metric: A monitored outcome intended to detect adverse effects of an intervention.

Heterogeneous treatment effect: Variation in causal effect across units or subgroups.

External validity: The extent to which findings generalize to other populations, settings, or times.

Review Questions

Short Answer Questions

1. Why does correlation between treatment exposure and an outcome not establish causation?

2. Define the average treatment effect and explain its relationship to potential outcomes.

3. What is the purpose of a guardrail metric in an A/B test?

4. State the parallel-trends assumption in difference-in-differences analysis.

5. Why is intention-to-treat analysis often preferred in randomized experiments?

Long Answer Questions

1. Explain how an A/B test should be designed from research question through analysis and reporting.

2. Compare randomized experiments with quasi-experimental designs in terms of identification, assumptions, and generalizability.

3. Discuss how causal diagrams help distinguish confounders, mediators, and colliders.

4. Explain the assumptions and interpretation of instrumental-variable estimation, including LATE.

5. Critically evaluate the ethical and governance requirements for experimentation in digital products.

Analytical / Application Questions

1. A treatment group has a conversion rate of 12% among 4,000 users, while a control group has a rate of 10% among 4,000 users. Calculate the absolute risk difference and relative risk, and explain which business interpretation each supports.

2. A city introduces a transport policy, while a comparable city does not. The treated city’s average outcome rises from 60 to 70, and the comparison city’s outcome rises from 55 to 60. Estimate the difference-in-differences effect and identify the key assumption required for interpretation.

3. An online platform randomizes individual users to a feature, but users frequently communicate with one another. Explain why the standard no-interference assumption may fail and propose a more suitable experimental design.

Multiple Choice Questions with Answers

Question 1

Which quantity compares the average potential outcome under treatment with the average potential outcome under control?

A. Average treatment effect

B. Propensity score

C. Intracluster correlation coefficient

D. Odds ratio

Correct Answer: A. Average treatment effect

Question 2

What is the principal purpose of randomization in an A/B test?

A. To guarantee a large treatment effect

B. To make treatment assignment independent of potential outcomes in expectation

C. To eliminate measurement error

D. To ensure every participant complies with treatment

Correct Answer: B. To make treatment assignment independent of potential outcomes in expectation

Question 3

Which assumption is central to a basic difference-in-differences analysis?

A. Perfect compliance

B. No missing observations

C. Parallel trends absent treatment

D. Equal treatment effects for every unit

Correct Answer: C. Parallel trends absent treatment

Question 4

A treatment raises renewal from 0.50 to 0.54. What is the absolute risk difference?

A. 0.04

B. 0.08

C. 0.50

D. 1.04

Correct Answer: A. 0.04

Question 5

Which analysis estimates the effect according to assigned treatment, regardless of whether units complied?

A. Per-protocol analysis

B. Intention-to-treat analysis

C. As-treated analysis

D. Subgroup analysis

Correct Answer: B. Intention-to-treat analysis

Question 6

In a regression-discontinuity design, the causal estimate is usually most directly applicable to:

A. All units in the national population

B. Units near the treatment cutoff

C. Only units far above the cutoff

D. Units excluded from the assignment rule

Correct Answer: B. Units near the treatment cutoff

Question 7

Which condition is part of positivity?

A. Every unit must receive treatment

B. Treatment must have a statistically significant effect

C. Each relevant covariate pattern must have a nonzero probability of each treatment level

D. Outcomes must be normally distributed

Correct Answer: C. Each relevant covariate pattern must have a nonzero probability of each treatment level

Question 8

Why can conditioning on a post-treatment mediator be problematic when estimating a total treatment effect?

A. It may remove part of the treatment pathway

B. It guarantees perfect randomization

C. It increases the treatment sample size

D. It eliminates all confounding automatically

Correct Answer: A. It may remove part of the treatment pathway

Question 9

If 5,000 treated users have a renewal rate of 54% and 5,000 control users have a renewal rate of 50%, the approximate number needed to treat is:

A. 4

B. 8

C. 25

D. 50

Correct Answer: C. 25

Question 10

What does an instrumental-variable estimate commonly identify under the standard IV assumptions?

A. The average effect for always-treated units

B. The local average treatment effect for compliers

C. The effect of treatment on every individual

D. The predictive accuracy of the treatment model

Correct Answer: B. The local average treatment effect for compliers