Causal Inference and Evaluation Beyond Experiments

Introduction

Product teams routinely ask causal questions: Did the new onboarding flow increase activation? Did a pricing change reduce churn? Did a recommendation model increase purchases, or did high-intent users simply adopt it first? These questions concern causal effects, not merely associations.

A randomized controlled experiment is usually the strongest design for estimating causal impact because randomization makes treatment assignment independent of potential outcomes, on average. Yet product experimentation is not always feasible. A feature may be mandatory, a policy may already have been launched, treatment may be assigned by geography or eligibility, or ethical, technical, legal, and commercial constraints may prohibit randomization. In these settings, causal inference methods can help estimate impact from observational or quasi-experimental data.

These methods do not turn observational data into experimental data automatically. Their credibility depends on explicit assumptions, careful design, high-quality measurement, and sensitivity analysis. The central professional skill is therefore not selecting the most sophisticated estimator, but constructing a defensible argument about what would have happened without the intervention.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish causal effects from correlations and define potential outcomes.
2. Use causal diagrams to identify confounding, mediators, colliders, and adjustment variables.
3. Explain and apply matching, regression adjustment, difference-in-differences, synthetic control, instrumental variables, and regression discontinuity designs.
4. State the assumptions required by each method and diagnose threats to validity.
5. Interpret estimates in product-management terms without overstating causal certainty.
6. Design a practical evaluation plan when randomization is infeasible.

1. The Causal Question

Let Y(1) denote the outcome a unit would experience if it received treatment and Y(0) the outcome it would experience without treatment. A unit may be a user, account, store, market, or time-region combination. The individual treatment effect is:

The fundamental problem of causal inference is that, for the same unit, only one of these potential outcomes is observed. If a user receives a new feature, we observe Y_i(1) but not Y_i(0). The unobserved outcome is the counterfactual.

For a population, the average treatment effect (ATE) is:

The expectation is over the target population. The ATE is meaningful only when treatment and outcome are clearly defined. For example, “the effect of recommendations” is incomplete unless the treatment specifies which recommendation system, for whom, during what period, and on which outcome.

The average treatment effect on the treated (ATT) is:

where D = 1 indicates treatment. ATT answers: What was the average effect for those who actually received the intervention? ATE and ATT can differ when treatment effects vary across users.

A useful product statement is therefore: “Among eligible users exposed to the new onboarding process, weekly activation was estimated to be 3.2 percentage points higher than it would have been under the prior process.” This is more precise than saying that activation “correlated with” the launch.

2. Causal Diagrams and Identification

A directed acyclic graph (DAG) represents variables as nodes and causal relationships as directed arrows. Consider:

Figure: A prior-intent variable creates a non-causal path between feature use and the outcome.

Here, prior intent influences both feature use and the outcome. A raw comparison of users who use the feature with those who do not may therefore attribute pre-existing differences to the feature.

2.1 Confounders, mediators, and colliders

A confounder is a pre-treatment variable that affects both treatment and outcome. Adjusting for appropriate confounders can block non-causal backdoor paths. Examples include prior usage, region, device type, and account size, provided they precede treatment.

A mediator lies on the causal pathway from treatment to outcome. If a feature affects engagement, which then affects retention, engagement is a mediator. Adjusting for it estimates a direct effect rather than the total effect and can remove part of the effect of interest.

A collider is a variable caused by two other variables. Conditioning on a collider can create a spurious association. For example, both user enthusiasm and product quality may influence whether a user contacts support. Restricting analysis to support contacts can make enthusiasm and quality appear related even if they were not otherwise associated.

The backdoor criterion provides a practical rule: adjust for a set of pre-treatment variables that blocks every path from treatment to outcome that begins with an arrow entering treatment, while not adjusting for descendants of treatment. This rule depends on the causal graph being substantively credible; a DAG is not validated merely because it can be drawn.

2.2 Identification assumptions

Most observational estimators require some version of the following assumptions:

• Consistency: the observed outcome under the received treatment equals the corresponding potential outcome; treatment versions must be sufficiently well defined.
• Conditional exchangeability: after conditioning on measured covariates X, potential outcomes are independent of treatment: Y(1),Y(0) ⟂ D | X. This means no important unmeasured confounding remains within covariate strata.
• Positivity: every relevant covariate profile has a positive probability of receiving each treatment: 0 < P(D=1|X) < 1.
• Correct temporal ordering: covariates used for adjustment precede treatment and are not affected by it.

These assumptions cannot be proven from the outcome data alone. They must be defended using product knowledge, institutional context, data audits, placebo tests, and sensitivity analysis.

3. Matching and Regression Adjustment

3.1 Matching

Matching constructs a comparison group whose observed pre-treatment characteristics resemble those of treated units. Common approaches include exact matching, coarsened exact matching, nearest-neighbour matching, and propensity-score matching.

The propensity score is the probability of treatment conditional on observed covariates:

A propensity model may be estimated using logistic regression or another suitable classifier. In a simple logistic model:

Here β₀ is the intercept and β_j are coefficients; X_j are pre-treatment covariates. The propensity score is a balancing device, not a causal mechanism. A high-quality analysis evaluates balance after matching rather than relying on predictive accuracy.

A standardized mean difference for covariate X is commonly calculated as:

where s² denotes sample variance. It is dimensionless because the numerator and denominator use the same units. Smaller absolute values indicate better balance; a frequently used diagnostic target is below 0.1, though no universal threshold guarantees validity.

Matching estimates effects for the matched population. It can discard observations, become unstable when treated and untreated units have little overlap, and fail when important confounders are unmeasured. Matching on post-treatment variables is generally inappropriate for estimating total effects.

3.2 Regression adjustment

A basic outcome regression is:

where Y_i is the outcome, D_i is treatment, X_i is a vector of covariates, τ is the adjusted treatment contrast, and ε_i is the residual. In a linear model, τ is the expected difference associated with treatment for units with the same included covariates, under model and identification assumptions.

For binary outcomes, logistic regression models log odds rather than percentage-point differences. Product teams should translate results into predicted probabilities or marginal effects, because odds ratios are often misunderstood.

Regression adjustment is efficient but sensitive to functional-form errors, extrapolation, omitted confounding, and inappropriate standard errors. Flexible machine-learning models can improve prediction, but prediction alone does not establish causality. Causal machine-learning methods such as doubly robust estimators can combine treatment and outcome models, yet still require credible identification assumptions and careful validation.

4. Difference-in-Differences

Difference-in-differences (DiD) is useful when treatment is introduced for one group or region but not another, and outcomes are observed before and after implementation.

For treated group T and control group C, pre-period 0 and post-period 1, the simple estimator is:

The first difference removes time-invariant differences between groups. The second subtracts changes common to both groups.

Worked example

Suppose weekly retention is measured as a proportion:

Thus:

The estimated effect is 0.05, or five percentage points. This interpretation depends chiefly on the parallel-trends assumption: absent treatment, treated and control groups would have experienced the same average change. Pre-treatment outcome trends can provide evidence, but they cannot prove parallel trends after implementation.

A panel regression may include unit and time fixed effects:

Here α_i captures time-invariant unit characteristics, λ_t captures shocks common to all units at time t, and τ is the DiD effect. Standard errors often need clustering at the level of treatment assignment, such as market or account, because observations within a cluster may be correlated.

Threats include differential pre-trends, staggered adoption, anticipation, spillovers, changing group composition, and simultaneous interventions. With staggered treatment timing, a simple two-way fixed-effects coefficient can combine effects across cohorts in unintuitive ways, particularly when treatment effects vary over time. Event-study specifications and modern cohort-specific estimators are often preferable.

5. Synthetic Control

Synthetic control is designed for settings with one or a small number of treated aggregate units, such as a country, city, market, or platform region. It constructs a weighted combination of untreated donor units that approximates the treated unit’s pre-treatment outcomes and predictors.

Let donor units be indexed by j = 1,…,J. Weights w_j satisfy:

The synthetic control outcome after treatment is:

The estimated effect for the treated unit is:

Weights are selected to minimize pre-treatment discrepancy between the treated unit and the weighted donor combination. The method is attractive because it makes the comparison construction visible rather than hiding it in a regression coefficient.

A credible donor pool should not be affected by the intervention, should be sufficiently similar before treatment, and should not experience major idiosyncratic shocks. Inference commonly uses placebo or permutation tests: repeatedly pretend each donor unit was treated and compare post-treatment gaps. These tests are informative but not identical to conventional randomization inference.

Synthetic control becomes difficult when no donor combination reproduces pre-treatment behavior, treatment spills over into donor units, or there are many simultaneous changes. Generalized and augmented synthetic-control methods can accommodate more complex settings, but their additional flexibility does not remove the need for design justification.

6. Instrumental Variables

An instrumental variable Z changes the probability of treatment D but affects the outcome Y only through treatment. It is useful when treatment is confounded and a credible source of quasi-random variation exists.

A common two-stage least-squares procedure is:

First stage:

Second stage:

Here D̂_i is predicted treatment from the first stage. The instrument must satisfy several assumptions:

• Relevance: Z changes treatment probability; in the simple model, π₁ ≠ 0.
• Exclusion restriction: Z affects Y only through D.
• Independence: the instrument is not related to unobserved determinants of the outcome, conditional on included covariates.
• Monotonicity, for the usual local interpretation: the instrument does not make some units less likely to receive treatment while making others more likely.

Under these conditions, the estimate is often a local average treatment effect (LATE) for compliers—units whose treatment status is changed by the instrument—not necessarily the ATE for all users.

A weak instrument produces imprecise and potentially biased estimates. First-stage strength must be assessed, but no single diagnostic threshold substitutes for substantive reasoning. Product examples might include a geographic rollout rule, an operational assignment mechanism, or an eligibility threshold, provided the proposed instrument has no direct path to the outcome.

7. Regression Discontinuity

Regression discontinuity (RD) exploits a treatment rule based on whether a continuous running variable crosses a known cutoff. If users with scores just below and just above the threshold are comparable, the discontinuity in outcomes estimates a local causal effect.

The sharp RD estimand is:

where X is the running variable and c is the cutoff. The notation compares the expected outcome approaching the cutoff from the treated and untreated sides.

A local linear specification is often written:

where D_i = 1 if X_i ≥ c in a sharp design. The coefficient τ represents the discontinuity at the cutoff, not a global effect.

Important checks include continuity of pre-treatment covariates, absence of manipulation around the cutoff, sensitivity to bandwidth and polynomial specification, and sufficient observations near the threshold. A fuzzy RD applies when crossing the cutoff changes treatment probability rather than treatment deterministically; it is interpreted similarly to an instrumental-variable design.

RD is powerful but local. An effect for users barely qualifying for premium support may not apply to users far below the eligibility threshold. Product managers must therefore define the decision population before interpreting the estimate.

8. Choosing a Design

A practical selection process is:

1. Define treatment, outcome, unit, estimand, and treatment time.
2. Draw a DAG and list plausible confounders and spillovers.
3. Identify the institutional mechanism generating treatment assignment.
4. Examine data coverage, overlap, timing, missingness, and measurement changes.
5. Select the simplest design whose assumptions are credible.
6. Pre-specify primary outcomes, comparison groups, time windows, and robustness checks.
7. Report diagnostics, uncertainty, limitations, and the population to which the result applies.

Statistical uncertainty should be reported using confidence intervals or other suitable interval estimates. A confidence interval reflects sampling and model uncertainty under the analysis framework; it does not quantify bias from unmeasured confounding or a violated parallel-trends assumption.

9. Illustrative Case: Regional Launch of a Fraud-Detection Feature

The following is a fictional illustrative case.

Background and problem

A digital marketplace introduced a new fraud-detection feature in six large regions. The feature was enabled for all eligible transactions in those regions, so user-level randomization was infeasible. Management wanted to know whether the feature reduced fraudulent transactions without materially reducing legitimate purchase completion.

The primary outcome was the fraud-loss rate per 10,000 transactions. A secondary outcome was legitimate purchase completion. The unit of analysis was region-week. Eight comparable regions had not yet received the feature.

Analysis and approach

Historical data showed that treated and potential control regions had similar fraud-loss trends for 20 weeks before launch. The team first created a pre-analysis DAG. Marketing intensity, transaction mix, and seasonality could affect both treatment timing and fraud outcomes. The team therefore recorded these variables and examined whether any control region received related security changes.

A DiD model with region and week fixed effects was estimated, with standard errors clustered by region. An event-study plot was used to inspect pre-treatment coefficients. The team also conducted a placebo test by assigning a false launch date 10 weeks before the actual launch. Finally, a synthetic-control analysis was run separately for the largest treated region as a robustness check.

Outcome

The main estimate suggested a reduction of 1.8 fraud-loss events per 10,000 transactions, with a 95% confidence interval from 0.7 to 2.9. Legitimate completion fell by 0.3 percentage points, but the interval included zero. The pre-launch event-study coefficients were small and showed no systematic trend. The placebo analysis produced no comparable post-treatment discontinuity. The largest region’s synthetic-control estimate was directionally consistent.

Management approved continued deployment but required monitoring of false positives, customer complaints, and effects in smaller regions. The report explicitly stated that the estimate relied on parallel trends and could be biased if treated regions experienced security-related changes not shared by controls.

Learning points

The case illustrates that credible evaluation is a process rather than a single regression. Clear outcomes, treatment timing, a defensible comparison group, pre-trend diagnostics, placebo tests, robustness analyses, and transparent limitations together make the evidence more useful. It also illustrates that a statistically beneficial feature may still require operational safeguards when it imposes costs on legitimate users.

10. Limitations, Ethics, and Governance

Causal analysis can reproduce inequity if data reflect discriminatory historical decisions. A model may estimate that an intervention works on average while harming a protected subgroup. Analyses should therefore report heterogeneous effects where scientifically justified, audit treatment and outcome measurement across groups, and avoid using sensitive attributes casually or unlawfully.

Privacy governance is essential. Analysts should apply data minimization, access controls, retention limits, purpose limitation, and appropriate aggregation. Linking behavioral, financial, and demographic data may create re-identification risks even when individual names are removed. Legal requirements vary by jurisdiction, but ethical responsibility exceeds mere legal compliance.

Multiple testing, outcome switching, selective reporting, and repeated peeking can create misleading evidence. Analysts should pre-specify primary outcomes and distinguish confirmatory from exploratory analyses. Missing data, bot activity, account sharing, changing definitions, and selective exposure can invalidate an otherwise sophisticated method.

AI-assisted causal analysis can help discover covariates, estimate heterogeneous effects, and automate diagnostics. However, generative systems may invent causal explanations, silently use post-treatment variables, or obscure model choices. Every AI-assisted result requires human review of the DAG, treatment mechanism, data lineage, assumptions, and reproducibility. Prediction systems should not be described as causal merely because they improve forecast accuracy.

11. Best Practices for Product Managers

• Start with a precise counterfactual question rather than a preferred method.
• Prefer randomization when feasible; use quasi-experimental designs when it is not.
• Separate pre-treatment variables from variables affected by the product.
• Inspect overlap and comparison-group quality before estimating effects.
• Use visual diagnostics: outcome trends, covariate balance, event studies, and RD density plots.
• Match the estimand to the decision: ATE, ATT, LATE, or a local cutoff effect.
• Report effect sizes in product units, such as percentage points, dollars per account, or support tickets per 1,000 users.
• Include uncertainty and plausible threats to validity in executive summaries.
• Test robustness to alternative windows, controls, bandwidths, donor pools, and specifications.
• Monitor implementation fidelity, spillovers, heterogeneous effects, and unintended harms after launch.

Chapter Summary

Causal inference estimates what would have happened under an alternative treatment condition. Potential outcomes formalize this counterfactual problem, while DAGs help identify confounders, mediators, colliders, and defensible adjustment sets. Matching and regression use observed pre-treatment information but depend on exchangeability, positivity, and correct specification. Difference-in-differences compares changes across treated and control groups under the parallel-trends assumption. Synthetic control constructs a weighted donor unit for aggregate interventions. Instrumental variables exploit treatment variation generated by a valid instrument and generally identify a local effect for compliers. Regression discontinuity uses a treatment cutoff and estimates a local effect near that threshold.

No method eliminates the need for judgment. The quality of a causal claim depends on treatment definition, data integrity, institutional knowledge, diagnostics, uncertainty reporting, and transparent assumptions. For product leaders, the goal is not methodological sophistication for its own sake, but better decisions supported by evidence whose limitations are understood.

Key Terms

Average treatment effect (ATE): Average causal effect across the target population.

Average treatment effect on the treated (ATT): Average causal effect among units that received treatment.

Backdoor path: A non-causal path from treatment to outcome that begins with an arrow into treatment.

Causal diagram (DAG): Graph representing hypothesized directional causal relationships.

Collider: Variable caused by two or more variables; conditioning on it can induce bias.

Consistency: Assumption that the observed outcome corresponds to the potential outcome under the received treatment.

Confounder: Pre-treatment variable affecting both treatment and outcome.

Difference-in-differences: Estimator comparing changes over time between treated and control groups.

Exchangeability: Conditional comparability of treated and untreated units with respect to potential outcomes.

Instrumental variable: Variable that changes treatment but affects the outcome only through treatment.

Local average treatment effect (LATE): Instrumental-variable effect for compliers.

Mediator: Variable on the causal pathway between treatment and outcome.

Parallel trends: Assumption that treated and control groups would have followed the same trend absent treatment.

Positivity: Requirement that each relevant covariate profile has a positive probability of each treatment level.

Potential outcome: Outcome a unit would experience under a specified treatment condition.

Propensity score: Probability of treatment conditional on observed covariates.

Regression discontinuity: Design exploiting a treatment assignment cutoff.

Synthetic control: Weighted combination of untreated units used to approximate a treated aggregate unit.

Treatment effect heterogeneity: Variation in causal effects across people, places, or conditions.

Review Questions

5 Short Answer Questions

1. What is the fundamental problem of causal inference?
2. Distinguish a confounder from a mediator.
3. State the parallel-trends assumption in difference-in-differences.
4. What population does a local average treatment effect describe?
5. Why is positivity important for matching and regression adjustment?

5 Long Answer Questions

1. Explain how a DAG can guide covariate adjustment in a product evaluation. Include examples of a confounder, mediator, and collider.
2. Compare matching and regression adjustment, including their assumptions, diagnostics, and limitations.
3. Derive and interpret the difference-in-differences estimator using a product rollout example.
4. Explain the assumptions and interpretation of instrumental variables and regression discontinuity.
5. Develop a complete evaluation plan for a non-randomized product launch, including estimand, design, diagnostics, and governance.

3 Analytical / Application Questions

1. A feature is adopted mainly by highly engaged users. Propose a DAG and an analysis strategy to estimate its effect on retention. Justify which variables you would and would not adjust for.
2. A service is launched in four cities while eight cities remain untreated. Pre-launch trends are visibly different across groups. Assess whether DiD is credible and recommend an alternative or redesign.
3. A premium-support program is assigned to accounts with a risk score of 70 or higher. Explain how you would evaluate its effect using RD, including the running variable, cutoff, validity checks, estimand, and threats to interpretation.

Multiple Choice Questions with Answers

Question 1

Which quantity compares the average potential outcome under treatment with the average potential outcome without treatment for the target population?

A. Average treatment effect

B. First-stage coefficient

C. Propensity score

D. Standardized mean difference

Correct Answer: A. Average treatment effect

Question 2

Which variable should generally not be adjusted for when estimating the total effect of a product intervention?

A. Pre-treatment region

B. Prior usage before launch

C. A post-treatment mediator

D. Pre-treatment account size

Correct Answer: C. A post-treatment mediator

Question 3

In the worked difference-in-differences example, treated retention increased by 0.09 and control retention increased by 0.04. What is the estimated effect?

A. 0.04

B. 0.05

C. 0.09

D. 0.13

Correct Answer: B. 0.05

Question 4

What is the central identifying assumption of a standard difference-in-differences analysis?

A. Treatment must be assigned by a strict cutoff

B. The instrument must have a strong first stage

C. Treated and control groups would have had parallel trends without treatment

D. All units must receive treatment simultaneously

Correct Answer: C. Treated and control groups would have had parallel trends without treatment

Question 5

In a synthetic-control analysis, donor weights are typically constrained so that they are:

A. Negative and sum to zero

B. Non-negative and sum to one

C. Equal to the treatment probabilities

D. Larger for post-treatment observations

Correct Answer: B. Non-negative and sum to one

Question 6

Which condition is part of the exclusion restriction for an instrumental variable?

A. The instrument affects the outcome only through treatment

B. The instrument must be the outcome variable

C. Treatment must be continuous

D. All covariates must be post-treatment

Correct Answer: A. The instrument affects the outcome only through treatment

Question 7

What population is usually identified by an instrumental-variable estimate under monotonicity?

A. Always-treated units only

B. Never-treated units only

C. Compliers whose treatment changes because of the instrument

D. Every unit in the global population equally

Correct Answer: C. Compliers whose treatment changes because of the instrument

Question 8

In regression discontinuity, the running variable is:

A. The outcome measured after treatment

B. The variable determining proximity to the assignment cutoff

C. The residual from the second-stage regression

D. The synthetic-control weight

Correct Answer: B. The variable determining proximity to the assignment cutoff

Question 9

Which diagnostic is especially relevant after propensity-score matching?

A. Whether the post-treatment mediator is balanced

B. Whether pre-treatment covariates are balanced between groups

C. Whether the outcome has a zero mean

D. Whether all treatment effects are identical

Correct Answer: B. Whether pre-treatment covariates are balanced between groups

Question 10

A confidence interval around an observational causal estimate does not, by itself, measure:

A. Sampling uncertainty under the model

B. The width of the reported interval

C. Bias from an unmeasured confounder

D. The estimated treatment contrast

Correct Answer: C. Bias from an unmeasured confounder