Experiment Design and Causal Validation

Introduction

Product discovery is not simply the collection of opinions. It is the disciplined reduction of uncertainty about a product opportunity, a user problem, a proposed solution, or a business model. Experiments provide a structured way to learn by exposing a carefully selected group of people to a stimulus and observing what happens. Causal validation goes further: it asks whether the observed outcome was caused by the intervention rather than by seasonality, selection, novelty, concurrent campaigns, or random variation.

For product managers, this distinction is fundamental. A landing page may receive many visits because an advertisement was unusually attractive, not because the underlying proposition is compelling. A usability test may reveal confusion caused by an unrealistic prototype rather than by the product concept. An A/B test may show a conversion increase that disappears when the experiment ends. Sound experimentation therefore combines a clear decision problem, an explicit hypothesis, an appropriate design, meaningful metrics, ethical safeguards, and cautious interpretation.

This chapter develops a practical and analytical approach to interviews, usability tests, smoke tests, landing pages, fake-door tests, pilots, A/B tests, and quasi-experiments. It also explains sampling, metrics, guardrails, uncertainty, causal identification, and the managerial implications of imperfect evidence.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish exploratory research from causal validation.
2. Formulate testable hypotheses and define decision rules.
3. Select an experiment design appropriate to the product uncertainty and stage.
4. Choose samples, primary metrics, secondary metrics, and guardrails.
5. Explain randomisation, control groups, confounding, bias, and statistical uncertainty.
6. Interpret conversion results and simple A/B-test estimates correctly.
7. Design ethical smoke tests, fake-door tests, pilots, and quasi-experiments.
8. Communicate evidence without overstating what an experiment proves.

1. From Product Questions to Testable Hypotheses

A product question is often too broad to test directly. “Will customers like this product?” should be decomposed into more precise uncertainties:

• Problem uncertainty: Do target users experience the problem frequently and intensely enough to seek a solution?
• Solution uncertainty: Can users understand and successfully use the proposed solution?
• Value uncertainty: Does the solution create a benefit that users value?
• Behavioural uncertainty: Will users adopt, return, pay, or recommend it?
• Operational uncertainty: Can the organisation deliver the experience reliably and economically?
• Causal uncertainty: Did the intervention produce the observed change?

A useful hypothesis specifies a population, intervention, outcome, comparison, and time horizon. For example:

Among independent physiotherapists who currently schedule appointments manually, showing an automated reminder feature will increase the proportion completing a booking within seven days, compared with showing the existing scheduling workflow.

This is stronger than “users will like reminders” because it identifies who is studied, what changes, what is measured, against what baseline, and over what period.

A complete experiment brief should include:

The most useful hypotheses are falsifiable. A statement such as “the feature will improve engagement” is weak unless engagement is operationalised. A statement such as “the feature will increase seven-day activation from 30% to at least 36% among eligible new accounts” can be tested.

2. Exploratory Evidence and Causal Evidence

Discovery methods differ in the type of evidence they generate. Interviews are usually exploratory and interpretive. They help identify language, motivations, workarounds, and unmet needs, but they do not normally estimate population-level behavioural effects. Usability tests reveal whether participants can complete tasks and where they struggle, but a small usability sample cannot establish market demand. A randomised A/B test can provide stronger evidence that a treatment caused a difference, but only for the population, treatment, and outcome represented in the test.

Three questions should always be separated:

1. Descriptive: What happened?
2. Explanatory: Why might it have happened?
3. Causal: Did the intervention cause it?

For example, a new onboarding flow may be followed by higher activation. The descriptive claim is that activation increased after launch. The causal claim requires a credible counterfactual: what activation would have been for comparable users who did not receive the new flow at the same time?

The counterfactual cannot be observed for the same person simultaneously. Experimental design approximates it through randomisation, while quasi-experimental design uses naturally occurring comparisons and explicit assumptions.

3. A Portfolio of Discovery and Validation Experiments

3.1 Interviews

Interviews are appropriate when the team needs to understand experiences, meanings, constraints, decision processes, and existing alternatives. A good interview uses a discussion guide but remains conversational. Questions should focus on recent behaviour rather than hypothetical preference:

• “Tell me about the last time you solved this problem.”
• “What did you try first?”
• “What did it cost in time, money, or risk?”
• “What made you continue or abandon that approach?”

Avoid leading questions such as “Would you use an automated version of this?” Hypothetical intent is often inflated by politeness, imagination, and low commitment. Evidence is stronger when participants describe observed behaviour, provide artefacts, or make a meaningful commitment.

Sampling should be purposeful rather than merely convenient. Include variation in role, frequency of the problem, sophistication, and context. Continue interviewing until additional sessions produce little new insight, while recognising that “saturation” is a judgement rather than a mechanical guarantee. Analyse notes systematically by coding recurring problems, triggers, workarounds, consequences, and contradictory cases.

3.2 Usability tests

A usability test evaluates whether representative participants can perform realistic tasks with a product or prototype. Define:

• the target participant;
• the scenario and task;
• the success criterion;
• the observation protocol;
• the questions asked after the task; and
• the severity classification for failures.

Useful measures include task completion, critical errors, time on task, abandonment, observed confusion, and post-task confidence. Do not overinterpret a small sample as a population estimate. Its primary value is diagnostic: it identifies interaction problems and tests whether the design communicates its intended value.

Moderators should avoid rescuing participants too early. A standardised prompt such as “What would you expect to do next?” preserves comparability. Think-aloud protocols can reveal reasoning, but they may also alter task behaviour; use them when insight into cognition is more valuable than naturalistic performance.

3.3 Smoke tests and landing pages

A smoke test measures interest in a proposed value proposition before the complete product exists. A landing page may describe a product, show a demonstration, and ask visitors to join a waitlist, request a consultation, or begin a qualified sign-up. It is especially useful for testing message clarity and initial demand.

The central metric is often:

The numerator and denominator must be defined consistently. A “visitor” may mean a unique qualified visitor rather than every page view, and a desired action should be sufficiently meaningful to reflect value rather than curiosity. Traffic source, device, geography, and audience intent strongly affect results. A high conversion rate from a highly targeted advertisement does not automatically imply broad product-market demand.

The page should be honest about availability and avoid collecting personal data without a legitimate purpose. If the product is not yet available, say so clearly. A waitlist registration is evidence of interest, not evidence of retention, payment, or sustained use.

3.4 Fake-door tests

A fake-door test places an apparently available feature or product entry point in an existing experience. When a user selects it, the organisation explains that the feature is not yet available and measures interest or invites feedback. This can efficiently test discoverability and demand, but it creates a trust risk.

An ethical fake-door test should:

• avoid charging users or taking irreversible actions;
• disclose the unavailability immediately after the click;
• explain why the test is being conducted;
• provide a useful alternative or feedback route;
• avoid targeting vulnerable users with consequential expectations; and
• monitor complaints and trust indicators as guardrails.

A click means that the proposition or entry point attracted attention. It does not prove that users would complete the full workflow or value the finished feature.

3.5 Pilots

A pilot introduces a limited version of a product or process to a restricted population, often with substantial human support. Pilots are useful for learning about workflow fit, implementation effort, operational failure, and early value. They are not automatically controlled experiments. Enthusiastic pilot participants may be unusually motivated, and intensive support may not scale.

A strong pilot specifies eligibility, duration, service level, support boundaries, success criteria, data collection, and exit conditions. It should distinguish product effects from founder effort. If a product works only because a team manually intervenes every day, that operational dependency is part of the finding, not an inconvenience to ignore.

3.6 A/B tests

An A/B test randomly assigns eligible units—usually users, accounts, sessions, stores, or geographic regions—to a control condition or treatment condition. Randomisation aims to make the groups comparable in expectation, so differences in outcomes can be attributed to the treatment, subject to implementation quality and statistical uncertainty.

The unit of randomisation must match the interference structure. If users influence one another, individual randomisation may cause contamination. For example, a social feature may require cluster or geographic randomisation. Repeated observations from one user should not be treated as independent users without appropriate analysis.

An A/B test requires a pre-specified primary outcome. Testing many outcomes and selecting the most favourable one increases the risk of false positive conclusions. Secondary outcomes are valuable for diagnosis, but decision-making should remain anchored to the primary metric and guardrails.

4. Metrics, Guardrails, and the Measurement Plan

A metric is not merely a number; it is an operational definition of what the organisation values. A useful metric specification identifies the event, unit, time window, eligibility rule, exclusions, source system, and known data-quality risks.

4.1 Metric categories

A leading metric may respond quickly but be weakly connected to durable value. A lagging metric may be strategically important but slow to observe. Product managers should avoid replacing the real outcome with a more convenient proxy without stating the limitation.

4.2 Effect measures

For a binary outcome, let p_T be the proportion with the outcome in treatment and p_C the corresponding proportion in control.

These quantities answer different questions. If treatment conversion is 12% and control conversion is 10%, the absolute difference is 2 percentage points, the relative risk is 1.2, and relative uplift is 20%. Relative uplift can sound impressive when the baseline is small; absolute change is often more useful for capacity and financial planning.

For a continuous outcome, such as task time or revenue per user, the difference in means is commonly reported. Means can be distorted by extreme values, so medians, trimmed means, quantiles, or robust analyses may also be appropriate. The choice should be made before inspecting results where possible.

Guardrails should be connected to plausible harms. A faster checkout that increases purchases but also increases payment failures, refunds, or customer complaints may not be a successful experiment. Guardrails require thresholds and monitoring plans, not merely a list of concerns.

5. Uncertainty and Statistical Interpretation

Observed differences contain both signal and noise. If treatment conversion is 12% and control conversion is 10%, the estimated difference is 2 percentage points. The estimate may not equal the effect in the broader target population because the sample is finite and because the experiment may have implementation or sampling limitations.

For sufficiently large, independent groups, an approximate standard error for the difference between two proportions is:

Here n_T and n_C are the numbers of eligible observations in treatment and control. This approximation assumes binary outcomes, independent observations, and sample sizes large enough for a normal approximation to be reasonable. Clustered assignment, repeated users, or rare outcomes require methods that account for dependence or sparsity.

An approximate 95% confidence interval for the difference is:

The 1.96 value is an approximation for a two-sided 95% interval under a normal sampling distribution. A confidence interval describes the uncertainty of the estimation procedure under repeated sampling; it is not a probability statement that a particular fixed parameter has a 95% chance of lying in this one calculated interval.

Worked example: interpreting a conversion experiment

Suppose 1,000 control users produce 100 purchases, while 1,000 treatment users produce 120 purchases.

p_C = 100 ÷ 1000 = 0.10 and p_T = 120 ÷ 1000 = 0.12. Therefore:

• absolute risk difference = 0.12 − 0.10 = 0.02, or 2 percentage points;
• relative uplift = 0.02 ÷ 0.10 = 0.20, or 20%;
• relative risk = 0.12 ÷ 0.10 = 1.20.

Using the approximate standard error:

SE = √[(0.12 × 0.88 ÷ 1000) + (0.10 × 0.90 ÷ 1000) ≈ 0.0141.

The approximate 95% interval is 0.02 ± 1.96 × 0.0141, or approximately −0.0076 to 0.0476. Because this interval includes zero, the result is not conventionally statistically distinguishable from no difference at the 5% level. The practical conclusion is not “the treatment has no effect.” Rather, the experiment is compatible with a small negative effect, no effect, or an increase of about 4.8 percentage points under the stated approximation. A decision should also consider cost, prior evidence, risk, and whether the interval is narrow enough for the business decision.

Statistical significance is not the same as practical importance. With a very large sample, a tiny effect may be statistically detectable but economically irrelevant. Conversely, an important effect may be estimated imprecisely in a small sample. Product decisions should consider the effect size, interval, expected value, implementation cost, and downside risk.

6. Sample Selection and Experimental Power

Sampling has two distinct dimensions:

1. Representativeness: Does the sample resemble the population to which the decision will be applied?
2. Precision: Is the sample large enough to estimate the effect with useful uncertainty?

Random sampling improves representativeness when feasible, but product experiments often use eligible users rather than a random sample of the entire market. In that case, the internal causal comparison may still be credible, while generalisation beyond the tested population remains limited.

Sample-size planning depends on baseline rate, minimum effect worth detecting, desired power, significance level, allocation ratio, and outcome variance. Power is the probability of detecting a specified effect if that effect truly exists, under the chosen design and analysis. It is not the probability that the hypothesis is true.

Do not repeatedly inspect an experiment and stop at the first favourable result unless the analysis plan supports sequential monitoring. Repeated unplanned looks inflate false-positive risk. Pre-registration within an organisation—recording the hypothesis, primary metric, exclusions, stopping rule, and analysis before launch—improves credibility even when no external preregistration is required.

7. Quasi-Experiments and Causal Reasoning Without Randomisation

Randomisation is not always possible. Product changes may be rolled out to one market first, a policy may affect only users above a threshold, or an external event may create a natural comparison. Quasi-experiments can be informative, but their validity depends on assumptions that must be argued rather than assumed.

Difference-in-differences

Suppose treatment and comparison groups are observed before and after an intervention. The difference-in-differences estimator is:

Y_T,after is the treatment-group outcome after the intervention, with the other symbols defined analogously. The key assumption is parallel trends: absent the intervention, the treatment and comparison groups would have changed similarly. Pre-intervention trends can provide evidence, but they cannot prove future parallel trends.

Regression discontinuity

If treatment is assigned at a known threshold—for example, eligibility for a benefit above a score—outcomes just above and below the cutoff may be compared. This estimates a local treatment effect near the threshold. It requires no manipulation around the cutoff and a sufficiently smooth relationship between the assignment variable and outcome, among other assumptions.

Interrupted time series

A time series can be analysed before and after a clearly timed intervention. The design is stronger with many pre-intervention observations, stable measurement, and no concurrent event that explains the change. Seasonality, autocorrelation, and unrelated campaigns must be addressed.

Quasi-experiments should report the identifying assumption, alternative explanations, sensitivity checks, and the limits of generalisation. A sophisticated method does not rescue a poorly justified comparison.

8. A Practical Experiment-Design Workflow

A disciplined workflow is:

1. State the decision. What will be done differently depending on the result?
2. Identify the uncertainty. Is it about the problem, solution, behaviour, operations, or causality?
3. Write the hypothesis. Specify population, intervention, outcome, comparison, and timeframe.
4. Select the least expensive credible design. Begin with interviews or usability work when the concept is immature; use controlled tests when behavioural causality matters.
5. Define the unit of analysis and assignment. Decide whether users, accounts, teams, stores, or regions are independent units.
6. Specify metrics and guardrails. Define event logic, windows, exclusions, and thresholds before launch.
7. Assess ethics and operational risk. Consider deception, privacy, accessibility, vulnerable groups, and service obligations.
8. Plan sample and duration. Account for seasonality, novelty, learning effects, and delayed outcomes.
9. Instrument and audit. Validate exposure, assignment, event logging, missing data, and treatment fidelity.
10. Analyse according to the plan. Report effect size, uncertainty, heterogeneity, and deviations.
11. Decide and document. Scale, iterate, stop, or gather more evidence; record what the result does and does not establish.

Avoid changing several elements at once when the objective is to identify a mechanism. If a new message, pricing model, interface, and fulfilment process are launched together, the overall outcome may be useful for a package decision but cannot reveal which component caused it.

9. Illustrative Case: A Reminder Feature for a Clinic Platform

The following is a fictional illustrative case.

Background

CarePath, a software platform for small clinics, observes that many clinics create appointments but fail to complete patient follow-up. The product team proposes automated SMS reminders. The team believes reminders will increase completed follow-up appointments but worries about message fatigue and opt-outs.

Problem and hypothesis

The team defines the population as newly onboarded clinics with at least 20 eligible follow-up appointments per month. The primary outcome is the proportion of eligible appointments completed within 14 days. The treatment group receives the automated reminder workflow; the control group continues with the existing process. The hypothesis is that reminders will increase completion by at least 3 percentage points without increasing opt-outs by more than 1 percentage point.

Approach

Before the A/B test, researchers interview clinic administrators and conduct usability tests of the reminder setup. These reveal that clinics use different definitions of “follow-up” and that some patients prefer email. The team standardises eligibility, adds channel preference, and tests the workflow with a small pilot to detect operational failures.

For the controlled experiment, randomisation occurs at the clinic level rather than appointment level because staff behaviour and messaging policies could otherwise spill across appointments. The primary metric, opt-out rate, delivery failure rate, and complaint rate are pre-specified. The test runs through a full four-week scheduling cycle to avoid comparing an unusually busy week with a quiet week.

Results and analysis

The treatment group shows a higher completion rate, but the estimated interval is wide because only 38 clinics participate. Several treatment clinics have delivery failures due to incorrect contact data. The team does not simply declare success. It separates product performance from data-quality failure, conducts an implementation audit, and examines whether results differ between clinics that completed setup correctly and those that did not.

The treatment appears promising among clinics with verified contact data, while the overall intention-to-treat result is smaller. The appropriate conclusion is that the reminder workflow may improve completion when implemented correctly, but the evidence is insufficient to claim a general effect across all clinics. The next step is to improve data validation and run a larger test, not to selectively market the favourable subgroup as the overall result.

Learning points

1. Qualitative work can prevent a badly specified quantitative test.
2. The clinic, rather than the appointment, was the appropriate assignment unit because of interference.
3. Guardrails exposed a cost—delivery failure—that the primary metric would have hidden.
4. Intention-to-treat analysis preserves the effect of offering the intervention; per-protocol analysis can explain implementation but is more vulnerable to selection bias.
5. A promising result still requires replication, operational scaling, and economic analysis.

10. Ethics, Governance, and Responsible Experimentation

Product experiments affect real people and therefore require governance. Consent may be explicit, implied by normal product use, or subject to legal and organisational requirements; the appropriate standard depends on the context, sensitivity of data, jurisdiction, and risk. Privacy principles include data minimisation, purpose limitation, access control, retention limits, and transparent communication.

Deception requires special caution. A fake-door test that creates a minor, immediately corrected expectation differs ethically from a test that causes financial loss, denies essential service, or manipulates vulnerable people. Experiments involving health, finance, employment, education, children, or safety may require formal review and stricter safeguards.

Fairness analysis should examine whether effects and harms differ across relevant groups. A treatment that improves the average outcome but disadvantages a protected or vulnerable group requires investigation. Do not infer sensitive characteristics unnecessarily; use legally and ethically appropriate data practices.

Governance should specify who may approve tests, what risk tiers exist, how incidents are escalated, and when an experiment must stop. An experiment log should retain the hypothesis, version, assignment logic, data definitions, analysis plan, decisions, and known limitations. Reproducibility is a managerial asset: it prevents organisational memory from being replaced by anecdotes.

11. Technology, Automation, and AI in Experimentation

Modern analytics platforms can automate assignment, event collection, dashboards, sequential monitoring, and heterogeneous-effect analysis. AI tools can help generate interview probes, classify qualitative feedback, identify anomalous data, or suggest hypotheses. These tools do not remove the need for causal reasoning. Automated pattern detection can amplify confounding, proxy bias, data leakage, and spurious correlations.

When AI generates or analyses research material, teams should verify source data, document prompts and transformations, protect confidential information, and retain human review. If an algorithm itself is the intervention, model version, training data, threshold, latency, and fallback behaviour become part of the treatment definition. A model update during an experiment can contaminate the comparison unless versioning is controlled.

Contemporary experimentation increasingly uses feature flags, holdout groups, switchback designs, and cluster randomisation to manage interference and operational constraints. These methods are valuable, but each introduces assumptions. Tool sophistication should follow decision risk; a simple, transparent test is preferable to a complex analysis that stakeholders cannot audit.

Challenges and Limitations

Experiments do not eliminate uncertainty. Common threats include:

• Selection bias: participants or exposed users differ systematically from the intended population.
• Attrition: dropout differs between conditions or is related to the outcome.
• Contamination: control users receive elements of treatment.
• Interference: one unit’s treatment changes another unit’s outcome.
• Novelty effects: initial excitement fades.
• Seasonality: timing produces temporary differences.
• Instrumentation error: events are missing, duplicated, or inconsistently defined.
• Multiple testing: many analyses increase false discoveries.
• Heterogeneous effects: averages conceal important differences by segment or context.
• External validity: an effect in one market, channel, or user group may not transfer.
• Metric gaming: teams optimise the measured proxy rather than durable value.

A credible product manager treats these as design problems, not merely statistical footnotes. Replication, sensitivity analysis, qualitative follow-up, and staged rollout can reduce uncertainty.

Best-Practice Checklist

Before launch, ask:

• Is the decision and hypothesis explicit?
• Is the intervention meaningful and sufficiently isolated?
• Is the comparison credible?
• Is the assignment unit appropriate?
• Are primary and guardrail metrics defined before results?
• Are sample, duration, and stopping rules defensible?
• Have privacy, fairness, accessibility, and deception risks been reviewed?
• Can exposure and outcomes be measured accurately?
• Is the analysis appropriate for clustering, missingness, and repeated observations?
• Will the final report distinguish evidence, assumptions, and judgement?

Chapter Summary

Experiment design converts product uncertainty into structured learning. Interviews and usability tests are primarily exploratory and diagnostic; smoke tests, landing pages, and fake-door tests assess early interest; pilots investigate operational fit; A/B tests provide stronger causal evidence through randomised comparison; and quasi-experiments estimate effects when randomisation is not feasible, subject to explicit assumptions.

A sound experiment begins with a decision, a falsifiable hypothesis, a defined population, intervention, comparison, outcome, and timeframe. It selects an appropriate assignment unit, specifies primary and guardrail metrics, plans sampling and duration, and protects participants. Results should be expressed through effect sizes and uncertainty, not through statistical significance alone. Absolute differences, relative uplift, confidence intervals, practical importance, implementation quality, and generalisability all matter.

Causal validation is ultimately a reasoning discipline. Randomisation, statistical models, dashboards, and AI tools support that discipline but cannot replace clear assumptions, ethical governance, careful measurement, and intellectually honest interpretation.

Key Terms

Hypothesis: A testable statement predicting how an intervention will affect a defined outcome.

Counterfactual: The outcome that would have occurred for the same unit under an alternative condition.

Intervention: The treatment, feature, message, policy, or experience being evaluated.

Control group: A comparison group that does not receive the focal intervention or receives the current experience.

Randomisation: Assignment by a chance mechanism intended to balance confounders in expectation.

Causal effect: The difference between potential outcomes under treatment and comparison conditions.

Confounder: A variable associated with both the intervention and outcome that can distort causal interpretation.

Primary metric: The pre-specified outcome used to answer the main experimental question.

Guardrail metric: A measure used to detect unacceptable harm or undesirable trade-offs.

Conversion rate: The proportion of eligible units completing a specified desired action.

Absolute risk difference: The treatment outcome proportion minus the control outcome proportion.

Relative uplift: The absolute difference divided by the control outcome proportion.

Confidence interval: An interval produced by a procedure designed to contain the parameter at a stated long-run frequency.

Statistical power: The probability of detecting a specified effect under a stated design if that effect exists.

Usability test: An observation of representative users performing realistic tasks with a product or prototype.

Smoke test: A low-cost test of interest in a proposed product or proposition before full development.

Fake-door test: A test using an apparently available entry point to measure interest in an unbuilt feature, with ethical disclosure.

Pilot: A limited implementation used to learn about value, workflow, and operational feasibility.

Difference-in-differences: A quasi-experimental estimator comparing changes over time between treatment and comparison groups.

External validity: The extent to which findings generalise beyond the tested population, setting, and time.

Review Questions

Short Answer Questions

1. Why is a counterfactual necessary for a causal claim?
2. Distinguish a primary metric from a guardrail metric.
3. Give one ethical requirement for a fake-door test.
4. What is the difference between absolute risk difference and relative uplift?
5. State the parallel-trends assumption in a difference-in-differences design.

Long Answer Questions

1. Compare interviews, usability tests, pilots, and A/B tests in terms of purpose, evidence, sampling, and limitations.
2. Explain how a product manager should design a landing-page smoke test while avoiding misleading conclusions about demand.
3. Discuss why randomisation at the wrong unit can invalidate an A/B test, using interference as part of your explanation.
4. Explain statistical uncertainty and practical importance using the worked conversion example in this chapter.
5. Describe an end-to-end experiment governance process covering metrics, privacy, fairness, documentation, and stopping rules.

Analytical / Application Questions

1. A treatment increases sign-up conversion from 8% to 10% but increases cancellation within 30 days from 12% to 18%. Design a decision framework using primary, secondary, and guardrail metrics.
2. A feature is rolled out to one city while a similar city remains unchanged. Propose a quasi-experimental analysis, identify its key assumptions, and list plausible threats to validity.
3. Design an experiment for a proposed AI-generated customer-support reply feature. Specify the hypothesis, assignment unit, primary outcome, guardrails, ethical risks, and interpretation plan.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes causal validation from descriptive measurement?

A. Causal validation asks whether the intervention produced the observed outcome.

B. Causal validation requires only a large sample.

C. Descriptive measurement always uses randomisation.

D. Causal validation is the same as collecting user opinions.

Correct Answer: A. Causal validation asks whether the intervention produced the observed outcome.

Question 2

A control conversion rate is 10% and a treatment conversion rate is 12%. What is the absolute risk difference?

A. 1.2 percentage points

B. 2 percentage points

C. 12 percentage points

D. 20 percentage points

Correct Answer: B. 2 percentage points

Question 3

Which is the strongest example of a guardrail metric for a new checkout flow?

A. Number of interface colours

B. Prototype completion time only

C. Payment failures and customer complaints

D. Number of team members attending the launch

Correct Answer: C. Payment failures and customer complaints

Question 4

Why might a clinic-level rather than appointment-level randomisation be appropriate?

A. It guarantees a larger sample.

B. It prevents all sampling error.

C. Clinic staff behaviour may spill across appointments.

D. It eliminates the need for a control group.

Correct Answer: C. Clinic staff behaviour may spill across appointments.

Question 5

What does a fake-door test primarily measure?

A. Completed long-term retention

B. Interest in an apparently available entry point

C. The final feature’s technical reliability

D. Lifetime customer profitability

Correct Answer: B. Interest in an apparently available entry point

Question 6

Which assumption is central to a basic difference-in-differences interpretation?

A. Treatment and comparison groups have identical sample sizes.

B. The groups would have followed parallel trends without the intervention.

C. Every user is exposed to both conditions.

D. The outcome must be binary.

Correct Answer: B. The groups would have followed parallel trends without the intervention.

Question 7

In an A/B test, what is the main purpose of pre-specifying a primary metric?

A. To guarantee a positive result

B. To prevent the need for instrumentation

C. To anchor analysis to the main decision and reduce selective reporting

D. To make the control group unnecessary

Correct Answer: C. To anchor analysis to the main decision and reduce selective reporting

Question 8

A 95% confidence interval for an estimated treatment effect includes zero. Which conclusion is most appropriate?

A. The treatment definitely has no effect.

B. The treatment definitely harms users.

C. The estimate is compatible with no effect under the stated method and assumptions.

D. The experiment proves the product should launch.

Correct Answer: C. The estimate is compatible with no effect under the stated method and assumptions.

Question 9

Which research method is primarily diagnostic rather than a population-level estimate of demand?

A. A small moderated usability test

B. A sufficiently powered randomised experiment

C. A controlled geographic rollout

D. A large-scale holdout test

Correct Answer: A. A small moderated usability test

Question 10

What is a major ethical requirement for a fake-door test?

A. Hide the unavailability until after collecting payment.

B. Disclose unavailability immediately and avoid irreversible harm.

C. Target only users who cannot complain.

D. Treat every click as proof of purchase intent.

Correct Answer: B. Disclose unavailability immediately and avoid irreversible harm.