Experimental and Causal Research in AI/ML

Introduction and Context

Artificial intelligence and machine learning systems are often evaluated as if performance were determined by a single score on a test set. In practice, however, an observed improvement may result from many causes: a better algorithm, additional training data, a changed preprocessing pipeline, a more favorable split, hyperparameter tuning, data leakage, or random variation. Experimental and causal research provides a disciplined way to determine what actually produced an observed outcome.

An experiment deliberately changes one or more factors and observes the consequences under controlled conditions. Causal research goes further: it asks whether changing a treatment would change an outcome, and under what assumptions that conclusion is justified. In AI/ML, treatments may include a model architecture, feature, training procedure, interface, recommendation policy, or deployment intervention. Outcomes may include accuracy, calibration, latency, fairness, user retention, safety incidents, or business value.

This chapter explains controlled experiments, quasi-experiments, ablation studies, benchmark design, online testing, confounding, randomization, leakage, baseline selection, multiple testing, and causal inference. The goal is not merely to obtain a higher metric, but to produce evidence that is credible, reproducible, practically relevant, and ethically defensible.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish prediction evaluation from causal evaluation.

2. Design controlled and quasi-experimental studies for AI/ML systems.

3. Select appropriate units of randomization, treatments, outcomes, and baselines.

4. Detect confounding, leakage, selection bias, and interference.

5. Design ablation studies and benchmarks that support valid conclusions.

6. Interpret online A/B tests, confidence intervals, practical significance, and multiple comparisons.

7. Explain potential outcomes, causal graphs, and common causal estimands.

8. Communicate experimental findings with sufficient methodological transparency.

1. Prediction, Association, and Causation

A predictive question asks: “How accurately can a model predict an outcome for new cases?” A causal question asks: “What would happen if an intervention were applied, compared with what would have happened without it?” These questions are related but not interchangeable.

Suppose a model predicts hospital readmission using the number of specialist consultations. A strong association may exist, but increasing consultations may not reduce readmission; patients receiving more consultations may be more seriously ill. Prediction can therefore exploit variables that are useful without being causes of the target.

A causal claim requires a comparison between potential outcomes. For an individual or unit i, let Y_i(1) denote the outcome if the unit receives treatment and Y_i(0) the outcome if it does not. The individual treatment effect is:

Y_i(1) − Y_i(0).

Only one of these potential outcomes is observed for each unit. The unobserved outcome is the fundamental problem of causal inference. Researchers therefore estimate average effects under assumptions and study designs that make treated and untreated groups comparable.

A common estimand is the average treatment effect (ATE):

ATE = E[Y(1) − Y(0)] = E[Y(1)] − E[Y(0)].

Here, E denotes the population mean, and treatment is assumed to be well-defined. The ATE is meaningful only when the treatment, outcome, population, and time horizon are clearly specified.

Association is a statistical relationship. Causation is a claim about the consequences of intervention. Randomized experiments can identify causal effects because random assignment makes treatment independent of pre-treatment characteristics in expectation. Observational studies require stronger assumptions and careful adjustment.

2. Elements of an AI/ML Experiment

A useful experimental protocol defines the following elements before data collection or analysis:

• Research question: What precise claim is being tested?
• Unit of analysis: An image, record, user, session, device, organization, or time period.
• Treatment or intervention: The factor deliberately changed.
• Control condition: The comparison procedure or status quo.
• Outcome: The primary quantity used to judge the intervention.
• Covariates: Pre-treatment variables used for description or improved precision.
• Assignment mechanism: How units are allocated to treatment conditions.
• Analysis population: The units to which the conclusion applies.
• Time horizon: When outcomes are measured.
• Analysis plan: Metrics, exclusions, statistical models, and stopping rules.

A central design principle is to distinguish primary outcomes from exploratory outcomes. For example, a fraud-detection experiment may define prevented fraud per 1,000 transactions as primary, while latency, false-positive rate, and customer complaints are secondary. Without this distinction, researchers may selectively emphasize whichever metric looks favorable.

Internal and External Validity

Internal validity concerns whether the estimated effect is credible for the studied units under the experimental conditions. Randomization, blinding where feasible, prevention of leakage, and consistent measurement improve internal validity.

External validity concerns whether the result generalizes to other populations, environments, time periods, or hardware. A model improvement on a curated benchmark may not transfer to noisy production data. External validity should be investigated through representative sampling, evaluation on shifted datasets, and replication in realistic settings rather than assumed from a large sample alone.

3. Controlled Experiments and Randomization

In a controlled experiment, researchers assign units to at least two conditions and compare outcomes. A simple parallel-group design assigns each unit to either control or treatment. Randomization can be implemented using a random number generator, a randomized assignment service, or a pre-registered allocation procedure.

Randomization serves two purposes. First, it balances measured and unmeasured pre-treatment characteristics on average. Second, it supports probability-based uncertainty calculations. Randomization does not automatically solve every problem: noncompliance, attrition, interference, implementation differences, and measurement errors can still threaten validity.

Choosing the Unit of Randomization

The randomization unit should match the way treatment operates and the way outcomes may spill over. If a recommendation policy affects a user’s future recommendations, randomizing individual page views can contaminate the control condition because the same user may experience both policies. User-level randomization may be more appropriate. If an intervention affects an entire school or hospital, cluster randomization may be necessary.

Important considerations include:

• Interference: One unit’s treatment changes another unit’s outcome.
• Cluster dependence: Units within a group have correlated outcomes.
• Repeated exposure: The same user or device appears many times.
• Carryover: Treatment effects persist after assignment changes.
• Compliance: Assigned treatment is not always received.

When observations are correlated, treating every event as independent can make uncertainty appear smaller than it is. Analyses should account for clustering, commonly through cluster-level summaries, mixed-effects models, or cluster-robust standard errors, depending on the design and estimand.

Blocking and Stratification

If an important pre-treatment variable strongly predicts the outcome, researchers may stratify or block randomization. For example, users can be randomized separately within geographic region or prior activity level. This improves balance and may increase precision. The blocking variables must be measured before treatment and defined independently of future outcomes.

Sample Size and Power

Power is the probability of detecting an effect of a specified size under a chosen design and significance threshold. Planning requires a meaningful minimum detectable effect, expected variability or event rate, sample size, and desired power. A statistically significant result is not guaranteed by a large sample, and a non-significant result does not prove that the effect is zero. It may indicate insufficient precision.

For a difference in means, a simplified planning relationship is often expressed as:

n per group ≈ 2(z_(1−α/2) + z_(1−β))²σ² / δ².

Here, n is the required sample size in each equally sized group, σ is the assumed common standard deviation, δ is the smallest effect worth detecting, α is the two-sided Type I error level, β is the Type II error probability, and z_p is the pth standard-normal quantile. This approximation assumes independent observations, equal variances, approximately normal sampling behavior, and a simple two-group comparison. Real studies may require inflation for clustering, attrition, repeated testing, or unequal allocation.

4. Baselines and Fair Comparisons

A baseline is the reference against which a new method is evaluated. Appropriate baselines are essential because a complex method can appear effective merely because the comparison is weak.

Useful baseline categories include:

1. No-treatment or status-quo baseline: The system currently used in practice.

2. Simple heuristic: A rule such as majority class, most frequent value, or last observation carried forward.

3. Classical statistical model: For example, linear regression or logistic regression.

4. Published or official benchmark baseline: A documented implementation with known settings.

5. A strong contemporary competitor: A method that a practitioner would reasonably choose.

6. Oracle or upper-bound reference: Used cautiously to show theoretical limits, not as a deployable comparator.

Comparisons should hold data, preprocessing, compute budget, tuning budget, and evaluation protocol constant unless the research question explicitly concerns those differences. A model tuned extensively against a test set has an unfair advantage over an untuned baseline. Hyperparameter selection should use training and validation data, leaving the test set untouched until final evaluation.

Metrics must match the decision. Accuracy can be misleading with class imbalance. Precision, recall, F1, area under the precision–recall curve, calibration, expected cost, latency, energy use, and subgroup performance may all be relevant. A model should not be declared superior because it improves one metric while causing unacceptable deterioration in another.

5. Data Leakage and Other Threats to Validity

Data leakage occurs when information unavailable at prediction time, or information derived from the evaluation set, influences training or model selection. Leakage can produce impressive but non-reproducible results.

Common forms include:

• Randomly splitting near-duplicate records across train and test sets.
• Computing normalization statistics using the full dataset before splitting.
• Using future information in time-dependent prediction.
• Allowing a patient, user, or device to appear in multiple partitions when the target is person-specific.
• Selecting features after inspecting test performance.
• Repeatedly tuning on a public benchmark.
• Embedding labels or post-outcome administrative codes in input features.

Prevention requires defining the prediction moment, splitting by the relevant entity or time, fitting preprocessing only on training data, versioning datasets, and documenting all transformations. A data audit should ask: “Could this variable be known at the exact time the prediction would be made?”

Other threats include:

• Confounding: A common cause influences both treatment and outcome.
• Selection bias: Inclusion in the study depends on treatment, outcome, or related variables.
• Attrition: Outcomes are missing differently across conditions.
• Measurement bias: The outcome is measured differently by treatment group.
• Regression to the mean: Extreme observations tend to be less extreme on repeat measurement.
• Novelty and Hawthorne effects: Behavior changes because users know they are being studied.
• Implementation confounding: Treatment and control differ in software version, traffic, or infrastructure as well as in the intended intervention.

6. Ablation Studies

An ablation study removes or changes components of a system to estimate their contribution. In machine learning, a full model may contain an architecture, pretraining, data augmentation, loss term, retrieval module, prompt strategy, and post-processing rule. An ablation compares the full system with variants that remove one component while holding other factors constant.

A basic one-factor-at-a-time ablation is easy to communicate but can be inefficient and misleading when components interact. If feature A helps only when feature B is present, removing A alone may suggest that A is useless. Factorial designs can study main effects and interactions more systematically. With two binary components, four conditions—neither, A only, B only, and both—allow researchers to inspect whether the combined effect exceeds what would be expected from separate effects.

Ablations should report:

• The exact component changed.
• Whether parameters and compute budgets were re-tuned.
• Results over multiple random seeds where relevant.
• Confidence intervals or dispersion measures.
• Effects on all important metrics, not only the headline score.
• Whether the ablation tests necessity, sufficiency, or merely association.

A component that improves benchmark accuracy may increase latency, memory, bias, or vulnerability. Ablation is therefore a systems analysis tool, not just a search for a higher score.

7. Benchmark Design and Reproducibility

A benchmark is a standardized collection of tasks, data, metrics, and procedures. Good benchmarks make comparisons easier, but they can also distort research when optimized repeatedly.

A credible benchmark specifies dataset provenance, inclusion criteria, train/validation/test partitions, preprocessing, permitted external data, model-selection rules, hardware, software versions, random seeds, metrics, and statistical reporting. Hidden test sets reduce direct overfitting, but benchmark maintainers must also monitor contamination and repeated submissions.

Benchmark limitations include:

• Construct validity: the metric may not represent real-world utility.
• Distribution shift: benchmark data may not resemble deployment data.
• Label noise or annotator disagreement.
• Saturation: small score differences may have little practical meaning.
• Dataset artifacts: models exploit shortcuts rather than intended capabilities.
• Selective reporting: only successful tasks or seeds are reported.

Reproducibility involves more than releasing code. Researchers should release, where legally and ethically possible, data documentation, configuration files, evaluation scripts, environment specifications, and a record of failed or excluded runs. Reproducibility and replicability are related but distinct: reproducibility usually concerns obtaining consistent results from the same materials, while replicability concerns obtaining consistent findings in a new implementation or dataset.

8. Online Experiments and A/B Testing

An online A/B test randomly assigns eligible users to a control experience or treatment experience and measures outcomes during a defined period. It is widely used for ranking, recommendation, interface, notification, and pricing systems.

A sound online test requires:

1. A stable assignment mechanism and persistent user bucketing.

2. A clearly specified primary metric.

3. Guardrail metrics such as latency, crashes, complaints, or fairness indicators.

4. Monitoring for sample-ratio mismatch, logging failures, and treatment contamination.

5. A pre-specified analysis window and stopping rule.

6. Protection against repeated users being counted as independent observations.

The difference in sample means estimates an average effect when assignment is randomized and measurement is consistent. A confidence interval describes uncertainty under the statistical model and design; it is not the probability that a particular hypothesis is true. Practical significance must also be considered. For example, a statistically detectable 0.02% increase in click-through rate may not justify engineering cost or reduced user satisfaction.

Sequentially checking significance and stopping as soon as a favorable p-value appears inflates false-positive risk. Valid approaches include fixed-horizon analysis, group-sequential designs, alpha-spending procedures, or methods explicitly designed for continuous monitoring.

9. Multiple Testing and Selective Reporting

When many hypotheses are tested, the chance of at least one false positive increases. If m independent hypotheses are each tested at level α, the probability of at least one false rejection under the global null is:

1 − (1 − α)^m.

This expression assumes independence and identical testing levels; dependence changes the exact probability, but the general warning remains. Testing ten outcomes at α = 0.05 can yield a family-wise false-positive probability of approximately 1 − 0.95^10 ≈ 0.401, not 0.05.

Researchers can control the family-wise error rate using methods such as Bonferroni or Holm adjustment. Bonferroni tests each of m hypotheses at α/m and is simple but can be conservative. When screening many features or models, controlling the false discovery rate may be more appropriate; procedures such as Benjamini–Hochberg aim to limit the expected proportion of false discoveries among rejected hypotheses.

Best practice is to pre-specify primary outcomes, identify secondary and exploratory analyses, report all tested outcomes, and distinguish confirmatory claims from hypothesis generation. Adjustments should be chosen according to the family of claims, not selected after seeing which method produces the most favorable result.

10. Quasi-Experiments and Causal Inference

A quasi-experiment estimates causal effects without full random assignment. It is useful when randomization is infeasible, unethical, expensive, or already absent from historical data.

Difference-in-Differences

Difference-in-differences compares changes over time in a treated group with changes in an untreated comparison group:

Effect = (Y_T,after − Y_T,before) − (Y_C,after − Y_C,before).

The central assumption is parallel trends: absent treatment, the treated and comparison groups would have experienced similar average changes. Researchers should examine pre-treatment trends and consider whether treatment timing, composition, or other events violate this assumption.

Regression Discontinuity

A regression-discontinuity design uses a treatment rule based on a threshold. Units just above and below the cutoff may be comparable if they cannot precisely manipulate the assignment variable. The estimated effect is local to the threshold and should not automatically be generalized to all units.

Instrumental Variables

An instrumental variable affects treatment assignment but affects the outcome only through treatment, apart from specified assumptions. The instrument must be relevant, satisfy an exclusion restriction, and generally meet independence assumptions. These conditions are difficult to verify, so instrumental-variable conclusions require strong substantive justification.

Matching and Covariate Adjustment

Matching or regression adjustment can make observed covariates more comparable. Such methods do not automatically control for unmeasured confounding. A causal interpretation requires assumptions such as conditional exchangeability, positivity, and consistency:

• Exchangeability: Conditional on measured covariates, treatment assignment is independent of potential outcomes.
• Positivity: Each relevant covariate pattern has a positive probability of receiving each treatment.
• Consistency: The observed outcome under the received treatment corresponds to the defined potential outcome.

A causal diagram, or directed acyclic graph (DAG), can help identify confounders, mediators, colliders, and inappropriate adjustment variables. Adjusting for a mediator may remove part of the effect being estimated; conditioning on a collider can create bias. Domain knowledge is essential because statistical association alone cannot determine the graph.

11. Worked Example: Evaluating an AI-Assisted Triage Model

Consider an illustrative hospital project. A new model recommends triage priority for incoming emergency-department cases. The hospital wants to know whether the model reduces the time to clinician assessment without increasing unsafe under-triage.

Background and Problem

Historical data show that time to assessment varies by shift, department congestion, patient age, and arrival mode. The model was trained on historical records. A naïve comparison of patients assigned high priority by the model with other patients would be confounded because the model’s recommendations correlate with severity and staffing conditions.

Design

Eligible cases are randomized at the patient level to either standard triage or standard triage plus the model recommendation. The randomization service operates before the clinician makes the triage decision. The primary outcome is minutes from arrival to clinician assessment. Guardrail outcomes are under-triage events, 72-hour unexpected return, clinician override rate, and subgroup performance by age, sex, language, and disability status.

Because the same patient should not receive conflicting conditions, the patient is the unit of randomization. Assignment is stratified by hospital site and shift. The analysis follows intention-to-treat: patients are analyzed according to assigned condition, even when clinicians do not follow the recommendation. A secondary per-protocol analysis may describe effects among cases where the recommendation was actually displayed, but it requires additional assumptions and can be biased by selective adherence.

Leakage and Measurement Controls

The model is frozen before the trial. Features are restricted to information available at arrival. Post-triage orders and later diagnoses are excluded. The test data are not used for threshold tuning. Outcome definitions are finalized before the main analysis, and reviewers assessing safety events are blinded to assignment where feasible.

Interpretation

Suppose the treatment group has a lower mean assessment time, but under-triage is also higher. The model cannot be declared successful solely because the primary metric improved. The decision must weigh the pre-specified safety boundary, clinical importance, uncertainty intervals, and subgroup results. If a benefit appears only on one shift, the researchers should investigate implementation and staffing interactions rather than generalize broadly.

Learning Points

The example demonstrates that causal evaluation requires a defined intervention, valid randomization, leakage prevention, clinically meaningful outcomes, guardrails, and a decision rule established before results are viewed. It also shows why a predictive model’s historical accuracy does not establish that deploying it improves care.

12. Ethical, Governance, and Professional Issues

AI/ML experiments can affect privacy, autonomy, access to services, and safety. Ethical review is necessary when experiments involve people, sensitive data, or consequential decisions. Researchers should minimize data collection, protect identities, define retention periods, and document lawful data use.

Randomization is not automatically ethical merely because it is scientifically strong. Equipoise—the presence of genuine uncertainty about which condition is better—is important for high-stakes interventions. Safety monitoring, stopping rules, adverse-event escalation, and human oversight should be established in advance.

Fairness requires examining whether treatment effects and errors differ across relevant groups. Aggregate improvement can conceal harm to a minority population. Researchers should avoid inferring protected attributes irresponsibly, consult affected communities where appropriate, and explain limitations rather than presenting fairness as a single universally sufficient metric.

Governance also includes access control, audit logs, model-version tracking, incident response, conflict-of-interest disclosure, and transparent reporting. In regulated domains, applicable requirements may include medical-device, privacy, consumer-protection, employment, or financial regulations. Compliance is not a substitute for methodological validity, and methodological validity is not a substitute for compliance.

13. Contemporary Developments and Best Practices

Modern AI research increasingly evaluates foundation models, retrieval systems, agents, and human–AI collaboration. These systems introduce new experimental concerns: prompt sensitivity, contamination of evaluation data, changing APIs, non-deterministic outputs, evaluator bias, and interaction effects between users and models.

Recommended practice includes:

• Register the question, estimand, primary outcome, and analysis plan before testing.
• Separate development, validation, and final evaluation data.
• Use strong, transparent baselines and equal tuning opportunities.
• Report uncertainty, random-seed variation, and practical as well as statistical significance.
• Audit for leakage, contamination, sample-ratio mismatch, and implementation differences.
• Use randomization when feasible; otherwise state causal assumptions explicitly.
• Treat ablations as designed experiments rather than informal feature removal.
• Report negative, null, and heterogeneous findings.
• Replicate important results across datasets, time periods, sites, or model seeds.
• Preserve a complete experiment log and version all data, code, models, and configurations.

Challenges and Limitations

No design eliminates uncertainty. Randomized experiments may be costly, short-lived, or unrepresentative. Quasi-experiments depend on assumptions that cannot be fully tested. Benchmarks can become targets rather than measurements of capability. Online tests may miss long-term harms, learning effects, network effects, or rare safety failures. Statistical significance can coexist with negligible practical value, while important effects may remain uncertain because data are limited.

Causal conclusions are always relative to a defined treatment, comparison, population, and time period. “The model works” is therefore incomplete. A stronger statement specifies what changed, for whom, compared with what, measured how, and under which assumptions.

Chapter Summary

Experimental and causal research enables AI/ML practitioners to distinguish genuine intervention effects from associations, artifacts, and random variation. A rigorous study defines its unit, treatment, control, outcomes, estimand, assignment mechanism, and analysis population. Randomization supports causal interpretation, while blocking, clustering adjustments, and power planning improve efficiency and precision.

Strong baselines, leakage prevention, fair tuning, and transparent benchmark protocols are essential for credible model comparisons. Ablation studies reveal component contributions but must consider interactions and system-level costs. Online A/B tests require persistent assignment, guardrail metrics, appropriate stopping rules, and attention to repeated users. Multiple testing must be controlled or clearly labeled as exploratory.

When randomization is unavailable, quasi-experimental methods such as difference-in-differences, regression discontinuity, matching, and instrumental variables can help, but only under explicit assumptions. Causal diagrams clarify confounding, mediation, and collider bias. Finally, ethical governance, privacy, subgroup analysis, reproducibility, and practical significance are integral parts of professional AI/ML experimentation.

Key Terms

Causal effect: The change in an outcome attributable to an intervention under a defined comparison.

Average treatment effect: The population-average difference between potential outcomes under treatment and control.

A/B test: A randomized online experiment comparing two system versions.

Ablation study: An experiment that removes or changes system components to assess their contribution.

Baseline: A reference method or condition used for comparison.

Benchmark: A standardized dataset, task, metric, and evaluation procedure.

Confounding: Distortion caused by a variable related to both treatment and outcome.

Control condition: The reference condition against which treatment is evaluated.

Data leakage: Use of information unavailable at prediction time or derived from evaluation data.

Difference-in-differences: A quasi-experimental estimator comparing before–after changes across treated and comparison groups.

Estimand: The precisely defined quantity a study seeks to estimate.

External validity: The extent to which findings generalize beyond the study setting.

Internal validity: The credibility of a conclusion for the studied units and conditions.

Potential outcome: The outcome a unit would have under a specified treatment condition.

Randomization: Assignment by a chance mechanism to create comparable groups.

Regression discontinuity: A design using treatment assignment determined by a threshold.

Statistical significance: Evidence against a specified null model under a chosen statistical procedure.

Treatment effect heterogeneity: Variation in an intervention’s effect across people, groups, or contexts.

Validity: The degree to which a study supports the interpretation or claim made from its evidence.

Review Questions

Short Answer Questions

1. Why is a predictive association not necessarily a causal relationship?

2. What is the purpose of randomization in an experiment?

3. Give two examples of data leakage in machine learning.

4. What is the role of a guardrail metric in an online experiment?

5. Why should primary and exploratory outcomes be distinguished?

Long Answer Questions

1. Explain how to design a controlled experiment for evaluating a new recommendation algorithm, including units, assignment, outcomes, baselines, and threats to validity.

2. Compare controlled experiments with quasi-experiments, including the assumptions and limitations of difference-in-differences.

3. Discuss how baseline selection, tuning budgets, and data splitting affect the credibility of AI/ML comparisons.

4. Explain ablation studies, including the limitations of one-factor-at-a-time ablations and the value of factorial designs.

5. Describe the ethical and governance requirements for evaluating an AI system in a high-stakes domain.

Analytical / Application Questions

1. A model is evaluated after normalization statistics were computed on the complete dataset before the train/test split. Identify the threat, explain why it biases evaluation, and propose a corrected pipeline.

2. An A/B test improves click-through rate but increases latency, customer complaints, and performance disparities for one subgroup. Design a decision framework using primary and guardrail metrics.

3. A policy is introduced in one region while a similar region remains untreated. Pre-treatment outcome trends differ substantially between regions. Assess whether difference-in-differences is credible and propose additional analyses or designs.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes causal inference from prediction?

A. Causal inference asks what would happen under an intervention, whereas prediction asks how accurately outcomes can be forecast.

B. Causal inference always requires a neural network, whereas prediction does not.

C. Prediction establishes causality whenever accuracy exceeds 90%.

D. Causal inference does not require an outcome variable.

Correct Answer: A. Causal inference asks what would happen under an intervention, whereas prediction asks how accurately outcomes can be forecast.

Question 2

What is the primary purpose of randomization?

A. To guarantee a statistically significant result

B. To make treatment assignment independent of pre-treatment characteristics in expectation

C. To eliminate measurement error

D. To ensure that every participant receives treatment

Correct Answer: B. To make treatment assignment independent of pre-treatment characteristics in expectation

Question 3

Which is an example of data leakage?

A. Using training-set statistics to transform validation data

B. Using information available at prediction time

C. Using future diagnoses to predict an earlier clinical decision

D. Evaluating a frozen model on a held-out test set

Correct Answer: C. Using future diagnoses to predict an earlier clinical decision

Question 4

In an ablation study, a researcher typically:

A. Adds unrelated datasets until performance increases

B. Removes or changes a component while keeping other conditions as comparable as possible

C. Tests only the final model on training data

D. Avoids reporting negative results

Correct Answer: B. Removes or changes a component while keeping other conditions as comparable as possible

Question 5

What assumption is central to a basic difference-in-differences design?

A. The treatment group must be twice as large as the control group

B. The groups must have identical outcomes before treatment

C. Absent treatment, the groups would have followed parallel trends

D. Treatment must be randomly assigned

Correct Answer: C. Absent treatment, the groups would have followed parallel trends

Question 6

If ten independent hypotheses are each tested at alpha = 0.05 without adjustment, what is the approximate probability of at least one false positive under the global null?

A. 0.005

B. 0.05

C. 0.10

D. 0.401

Correct Answer: D. 0.401

Question 7

Which practice best protects the final test set?

A. Use it repeatedly for hyperparameter tuning

B. Select features based on its results

C. Keep it untouched until the final evaluation

D. Combine it with training data after every experiment

Correct Answer: C. Keep it untouched until the final evaluation

Question 8

Why might user-level rather than page-view-level randomization be appropriate for a recommendation experiment?

A. Users always have identical preferences

B. Treatment effects may persist and spill over across a user’s page views

C. Page views cannot be logged

D. User-level experiments require no outcome measurement

Correct Answer: B. Treatment effects may persist and spill over across a user’s page views

Question 9

Which condition is part of the positivity assumption?

A. Every relevant covariate pattern has a positive probability of receiving each treatment

B. All outcomes must be normally distributed

C. Treatment effects must be identical for every person

D. The study must contain exactly two groups

Correct Answer: A. Every relevant covariate pattern has a positive probability of receiving each treatment

Question 10

A statistically significant improvement in accuracy should be interpreted as:

A. Proof that the system is safe to deploy

B. Evidence of an effect under the stated design, which must also be judged for practical importance and side effects

C. Proof that no leakage occurred

D. Evidence that all subgroups benefit equally

Correct Answer: B. Evidence of an effect under the stated design, which must also be judged for practical importance and side effects