Analysis of Variance and Experimental Design

Introduction

Scientific, product, and managerial decisions often involve comparing more than two alternatives. A machine-learning team may compare several feature-engineering pipelines, a product manager may test multiple interface designs, and a researcher may study the joint effects of model architecture and training strategy. Repeated pairwise tests are unsuitable because they inflate the probability of false discoveries. Analysis of variance (ANOVA) provides a unified framework for testing differences among means while controlling the overall Type I error rate.

ANOVA is most valuable when embedded within sound experimental design. Randomization protects against systematic bias, replication quantifies natural variation, blocking removes variation attributable to known nuisance factors, and control groups establish a meaningful counterfactual. Factorial designs additionally reveal whether factors interact—an important issue in AI and ML systems, where the effectiveness of one intervention often depends on another.

Learning Objectives

After studying this chapter, students should be able to:

1. formulate and conduct a one-way ANOVA;

2. partition total variability into treatment and error components;

3. calculate and interpret an ANOVA table and effect sizes;

4. analyze main effects and interactions in factorial experiments;

5. apply randomization, replication, blocking, and control groups;

6. design and evaluate product A/B tests;

7. plan sample sizes for one-way ANOVA and factorial interactions;

8. diagnose assumptions and select suitable response-analysis methods; and

9. recognize ethical, operational, and governance issues in experimentation.

1. Foundations of Experimental Design

An experimental unit is the smallest unit independently assigned to a treatment, such as a person, server, classroom, or manufacturing batch. A factor is an explanatory variable deliberately varied by the researcher. Its possible settings are called levels. A specific combination of factor levels is a treatment.

The measured outcome is the response variable. A response should be defined before data collection, including its unit, measurement period, inclusion rules, and handling of missing observations.

1.1 Randomization

Randomization assigns experimental units to treatments using a chance mechanism. It reduces systematic differences between treatment groups and supports causal inference by making treatment assignment independent of potential outcomes, subject to the design.

Random assignment is different from random sampling. Random sampling supports generalization to a population, whereas random assignment supports causal comparison within the experimental population.

1.2 Replication

Replication means applying each treatment independently to multiple experimental units. It allows estimation of experimental error and improves precision. Repeated measurements on the same unit are not independent replications unless the analysis explicitly models their dependence.

Pseudoreplication occurs when correlated observations are incorrectly treated as independent. For example, thousands of predictions from one trained model are not necessarily thousands of independent model-training replications.

1.3 Blocking

A block is a group of relatively similar experimental units. Treatments are randomized within blocks so that comparisons are made under comparable conditions. Typical blocking variables include location, device type, experience level, time period, and baseline risk.

Blocking is useful when a nuisance variable explains substantial response variation. Excessive blocking, however, can complicate implementation and produce sparsely populated treatment combinations.

1.4 Control Groups and Blinding

A control group represents what would happen without the intervention. Controls may receive the current product, standard process, placebo, or no treatment. The relevant causal effect is defined relative to this counterfactual.

Blinding reduces behavioral and measurement bias. In software experiments, users may be unaware of assignment, while evaluators can be blinded when scoring subjective outputs such as generated text quality.

2. One-Way Analysis of Variance

One-way ANOVA compares the means of three or more groups defined by one factor. Suppose factor level i contains n_i observations, where i=1,…,k. The model is

\[

Y_{ij}=\mu+\tau_i+\varepsilon_{ij},

\]

where \(\mu\) is the overall mean, \(\tau_i\) is the effect of level i, and \(\varepsilon_{ij}\) is random error. An identifying constraint such as \(\sum_i n_i\tau_i=0\) is required.

The hypotheses are

\[

H_0:\mu_1=\mu_2=\cdots=\mu_k

\]

against the alternative that at least one population mean differs. Rejecting \(H_0\) does not establish that every mean differs from every other mean.

2.1 Variability Decomposition

Let \(\bar Y_{i\cdot}\) be the mean of group i, \(\bar Y_{\cdot\cdot}\) the grand mean, and \(N=\sum_i n_i\). Total variability is

\[

SS_T=\sum_{i=1}^{k}\sum_{j=1}^{n_i}(Y_{ij}-\bar Y_{\cdot\cdot})^2.

\]

It decomposes as

\[

SS_T=SS_B+SS_E,

\]

where

\[

SS_B=\sum_{i=1}^{k}n_i(\bar Y_{i\cdot}-\bar Y_{\cdot\cdot})^2

\]

measures between-group variability, and

\[

SS_E=\sum_{i=1}^{k}\sum_{j=1}^{n_i}(Y_{ij}-\bar Y_{i\cdot})^2

\]

measures within-group error variability.

The degrees of freedom and mean squares are

\[

df_B=k-1,\qquad df_E=N-k,\qquad df_T=N-1,

\]

\[

MS_B=\frac{SS_B}{k-1},\qquad MS_E=\frac{SS_E}{N-k}.

\]

The test statistic is

\[

F=\frac{MS_B}{MS_E}.

\]

Under the null hypothesis and standard ANOVA assumptions, \(F\) follows an F distribution with \(k-1\) and \(N-k\) degrees of freedom.

2.2 Worked One-Way ANOVA

Three preprocessing methods produce the following validation scores:

• Method A: 2, 3, 4
• Method B: 5, 6, 7
• Method C: 8, 9, 10

The group means are 3, 6, and 9, while the grand mean is 6.

The between-group sum of squares is

\[

SS_B=3(3-6)^2+3(6-6)^2+3(9-6)^2=27+0+27=54.

\]

Within each group, deviations from the group mean are −1, 0, and 1. Therefore,

\[

SS_E=3(1^2+0^2+1^2)=6.

\]

Thus \(SS_T=54+6=60\). With \(k=3\) and \(N=9\), the ANOVA table is:

| Source | SS | df | MS | F |

|---|---:|---:|---:|---:|

| Methods | 54 | 2 | 27 | 27 |

| Error | 6 | 6 | 1 | — |

| Total | 60 | 8 | — | — |

Hence,

\[

F=\frac{27}{1}=27.

\]

For \(F(2,6)\), the upper-tail p-value is 0.001. At the 5% significance level, the equal-means hypothesis is rejected. The data provide strong evidence that at least one method has a different population mean.

2.3 Effect Size

Statistical significance does not measure practical importance. A common descriptive measure is eta squared:

\[

\eta^2=\frac{SS_B}{SS_T}.

\]

For the example,

\[

\eta^2=\frac{54}{60}=0.90.

\]

Thus 90% of the observed response variation is associated with differences among methods. Because \(\eta^2\) is upward-biased in small samples, omega squared is often preferred:

\[

\omega^2=\frac{SS_B-(k-1)MS_E}{SS_T+MS_E}.

\]

Here,

\[

\omega^2=\frac{54-2(1)}{60+1}=\frac{52}{61}\approx0.852.

\]

2.4 Post-Hoc Comparisons

After rejecting the omnibus null hypothesis, planned contrasts or multiple-comparison procedures can identify the differences. Tukey’s honestly significant difference procedure is appropriate for all pairwise comparisons under the classical equal-variance model. Dunnett’s procedure compares several treatments with one control. Bonferroni adjustment is flexible but can be conservative.

Comparisons should normally be selected before examining results. Reporting only favorable comparisons creates selective-inference bias.

3. Assumptions and Diagnostics

Classical fixed-effects ANOVA relies on:

1. Independence: errors are independent across experimental units.

2. Normality: errors within treatment combinations are approximately normally distributed.

3. Equal variances: treatment populations have a common error variance.

4. Correct specification: the model includes relevant design features such as blocks and interactions.

Independence is primarily guaranteed by design, not by a residual plot. Normality can be evaluated using residual Q–Q plots, while residual-versus-fitted plots help detect unequal variance and nonlinear patterns. Formal tests should not replace graphical and substantive assessment.

ANOVA is often robust to moderate non-normality when group sizes are balanced and observations are independent. Severe heteroscedasticity with unequal sample sizes can distort the F test. Alternatives include Welch’s ANOVA, heteroscedasticity-robust inference, transformation, permutation methods, or a generalized linear model suited to the response distribution.

For counts, proportions, survival times, and repeated measures, ordinary ANOVA may be inappropriate. Poisson or negative-binomial models, logistic regression, survival models, and mixed-effects models may better represent the response.

4. Factorial Designs and Two-Way ANOVA

A factorial design studies two or more factors simultaneously. In a complete factorial design, every level of one factor is combined with every level of the other factors.

For factor A with a levels, factor B with b levels, and r independent replicates per cell, the balanced two-way model is

\[

Y_{ijk}=\mu+\alpha_i+\beta_j+(\alpha\beta)_{ij}+\varepsilon_{ijk}.

\]

The terms represent the overall mean, the main effect of A, the main effect of B, and the A×B interaction.

An interaction exists when the effect of one factor changes across levels of another. When an interaction is scientifically important, main effects averaged over the other factor may be misleading. Cell means and simple effects should then be examined.

4.1 Balanced Two-Way ANOVA Formulas

For a balanced design, let \(\bar Y_{i\cdot\cdot}\) be the mean for level i of A, \(\bar Y_{\cdot j\cdot}\) the mean for level j of B, \(\bar Y_{ij\cdot}\) a cell mean, and \(\bar Y_{\cdot\cdot\cdot}\) the grand mean. Then

\[

SS_A=br\sum_i(\bar Y_{i\cdot\cdot}-\bar Y_{\cdot\cdot\cdot})^2,

\]

\[

SS_B=ar\sum_j(\bar Y_{\cdot j\cdot}-\bar Y_{\cdot\cdot\cdot})^2,

\]

\[

SS_{AB}=r\sum_i\sum_j(\bar Y_{ij\cdot}-\bar Y_{i\cdot\cdot}-\bar Y_{\cdot j\cdot}+\bar Y_{\cdot\cdot\cdot})^2,

\]

and

\[

SS_E=\sum_i\sum_j\sum_k(Y_{ijk}-\bar Y_{ij\cdot})^2.

\]

The respective degrees of freedom are \(a-1\), \(b-1\), \((a-1)(b-1)\), and \(ab(r-1)\).

4.2 Worked Two-Way ANOVA

Suppose two algorithm families, A1 and A2, are evaluated under two compute settings, B1 and B2. Two independent runs are observed in each cell:

| | B1 | B2 |

|---|---|---|

| A1 | 8, 10 | 12, 14 |

| A2 | 14, 16 | 22, 24 |

Cell means are 9, 13, 15, and 23. The A-level means are 11 and 19; the B-level means are 12 and 18; and the grand mean is 15.

For factor A,

\[

SS_A=(2)(2)[(11-15)^2+(19-15)^2]=4(32)=128.

\]

For factor B,

\[

SS_B=(2)(2)[(12-15)^2+(18-15)^2]=4(18)=72.

\]

The interaction deviations for cells A1B1, A1B2, A2B1, and A2B2 are respectively 1, −1, −1, and 1. Therefore,

\[

SS_{AB}=2(1^2+(-1)^2+(-1)^2+1^2)=8.

\]

Within each cell, the two observations differ from their cell mean by −1 and 1, giving a within-cell sum of squares of 2. Across four cells,

\[

SS_E=4(2)=8.

\]

The total sum of squares is \(128+72+8+8=216\).

| Source | SS | df | MS | F |

|---|---:|---:|---:|---:|

| Algorithm A | 128 | 1 | 128 | 64 |

| Compute B | 72 | 1 | 72 | 36 |

| A×B | 8 | 1 | 8 | 4 |

| Error | 8 | 4 | 2 | — |

| Total | 216 | 7 | — | — |

At \(\alpha=0.05\), the critical value \(F_{1,4}\) is approximately 7.71. The observed main-effect statistics for A and B exceed this value, whereas the interaction statistic does not. In this small experiment, there is evidence of main effects but insufficient evidence of interaction. The very small error degrees of freedom make replication and confidence intervals particularly important.

4.3 Unbalanced Factorial Data

When cell sizes differ, factor sums of squares are generally not orthogonal. Results may depend on whether sequential, adjusted, or marginal hypotheses are tested. “Type I,” “Type II,” and “Type III” sums of squares answer different questions. Researchers should state the estimand, inspect empty cells, fit the model using regression methods, and avoid treating software defaults as universal choices.

5. A/B Testing

An A/B test is a randomized controlled experiment comparing a current version, A, with a proposed version, B. Its logic is equivalent to a two-group experiment, although implementation at product scale creates additional concerns.

A rigorous A/B test specifies:

• the randomization unit, such as user, account, session, or organization;
• a primary outcome and analysis window;
• eligibility and exclusion rules;
• a minimum practically important effect;
• guardrail metrics, such as latency, complaints, or safety incidents;
• the intended analysis and stopping rule.

For a continuous response, the estimated treatment effect is

\[

\widehat\Delta=\bar Y_B-\bar Y_A,

\]

with an independent-samples standard error

\[

SE(\widehat\Delta)=\sqrt{\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}}.

\]

Welch’s method is generally suitable when variances are not assumed equal. Binary outcomes may be analyzed using risk differences, risk ratios, odds ratios, or logistic regression.

Repeatedly examining results and stopping when significance appears inflates the Type I error rate. Valid alternatives include a fixed analysis horizon, alpha-spending sequential designs, group-sequential methods, or properly specified Bayesian decision rules.

Network effects and interference violate the assumption that one unit’s treatment does not affect another unit’s outcome. In social platforms, marketplaces, and collaborative systems, cluster or network randomization may be needed.

6. Sample-Size and Power Planning

Power is the probability of rejecting the null hypothesis when a specified alternative is true. Sample-size planning should occur before data collection and should reflect practical effect sizes rather than effects selected merely because they produce convenient sample sizes.

6.1 Planning for One-Way ANOVA

For a fixed-effects one-way ANOVA with common population standard deviation \(\sigma\), Cohen’s effect size is

\[

f=\sqrt{\frac{\sum_i p_i(\mu_i-\mu)^2}{\sigma^2}},

\]

where \(p_i\\) is the planned population or allocation proportion for group i and \(\mu=\sum_i p_i\mu_i\). This formula requires a common within-group variance and a clearly specified set of group means.

Under the alternative, power is calculated from a noncentral F distribution. For the standard fixed-effects formulation, the noncentrality parameter is

\[

\lambda=Nf^2.

\]

The numerator degrees of freedom are \(k-1\), and the denominator degrees of freedom are \(N-k\). Because the critical value and denominator degrees of freedom depend on N, software normally searches iteratively for the smallest adequate sample size.

Example. Consider four equally sized treatment groups, \(\alpha=0.05\), desired power 0.80, and \(f=0.25\). A noncentral-F calculation gives a required total sample size of approximately 180, or 45 units per group. Here,

\[

\lambda=180(0.25)^2=11.25.

\]

This calculation assumes independent observations, equal variances, balanced allocation, and a fixed-effects omnibus test. If 10% attrition is expected, recruitment should be inflated rather than reducing the final analyzable sample below 180.

6.2 Planning a Factorial Interaction

Factorial studies should be powered for the effect most important to the decision. A design powered for large main effects may have inadequate power for an interaction.

In a balanced 2×2 design, define the difference-in-differences interaction as

\[

\Delta_I=(\mu_{22}-\mu_{21})-(\mu_{12}-\mu_{11}).

\]

With r independent observations per cell and common standard deviation \(\sigma\), the variance of its estimator is

\[

\operatorname{Var}(\widehat\Delta_I)=\frac{4\sigma^2}{r}.

\]

A large-sample two-sided approximation is

\[

r\approx\frac{4\sigma^2(z_{1-\alpha/2}+z_{1-\beta})^2}{\Delta_I^2}.

\]

For \(\alpha=0.05\), power 0.80, \(\sigma=10\), and a minimum important interaction of 10 units,

\[

r\approx\frac{4(10^2)(1.96+0.842)^2}{10^2}=31.4.

\]

Rounding upward gives 32 observations per cell, or 128 observations overall. An exact noncentral t or F calculation should be used for final planning, especially with small samples. With anticipated 10% attrition, recruiting \(\lceil32/0.90\rceil=36\) units per cell gives 144 recruits.

Power calculations should also account for clustering, unequal allocation, multiple primary outcomes, missing data, and expected noncompliance where relevant.

7. Response Analysis and Reporting

A defensible analysis should follow the design. Blocking variables used in randomization should normally be represented in the model. Baseline covariates can improve precision when selected independently of treatment outcomes. In product experiments, variance-reduction methods such as CUPED use pre-experiment measurements correlated with the response, provided the covariate is unaffected by treatment.

Recommended reporting includes:

1. treatment and control definitions;

2. experimental unit and randomization method;

3. sample sizes, exclusions, and missing data;

4. cell means with standard deviations or standard errors;

5. effect estimates with confidence intervals;

6. ANOVA table or model test results;

7. diagnostics and sensitivity analyses;

8. multiplicity and stopping procedures; and

9. practical implications, not only p-values.

Intent-to-treat analysis compares units according to their assigned treatment and preserves the protection offered by randomization. Per-protocol analyses may be informative but can become biased if adherence is related to outcomes.

8. Illustrative Case: Evaluating an AI Assistant and Training Program

Background

This is a fictional illustrative case. A consulting organization plans to improve analysts’ report quality. It can introduce an AI writing assistant and offer either standard or advanced training.

Problem

Management wants to know whether the assistant improves quality, whether advanced training helps, and whether the assistant is especially effective when combined with advanced training. A simple A/B test of the assistant would not separately estimate these questions.

Approach

The organization uses a 2×2 factorial design:

• AI assistant: unavailable or available;
• training: standard or advanced.

The experimental unit is an analyst. Analysts are stratified by job level and department, then randomly allocated within strata. The control condition is no assistant with standard training. Independent blinded evaluators score a standardized report on a 0–100 rubric. Security incidents and completion time are guardrail outcomes.

A pilot study suggests an error standard deviation of approximately 10 points. Management considers a 10-point interaction practically important. The interaction planning calculation requires about 32 analyzable analysts per cell. Allowing 10% attrition, 36 are recruited per cell, giving 144 recruits.

Analysis

The preregistered model contains assistant, training, their interaction, and blocking indicators. Representative adjusted cell means are:

| | Standard training | Advanced training |

|---|---:|---:|

| No assistant | 68 | 73 |

| AI assistant | 74 | 88 |

The estimated interaction is

\[

(88-73)-(74-68)=15-6=9 \text{ points}.

\]

Thus the assistant improves scores by 6 points under standard training and by 15 points under advanced training. Suppose the model-based 95% confidence interval for the interaction is 3 to 15 points. The data then support a positive interaction, although its lower confidence bound is below the original 10-point target.

Outcome

Management deploys the combined assistant-and-advanced-training package rather than the assistant alone. Deployment remains conditional on security and confidentiality guardrails. The organization also schedules monitoring to determine whether the effect persists outside the standardized assessment.

Learning Points

The factorial design evaluates two interventions efficiently and identifies synergy. Blocking improves comparability, blinding reduces scoring bias, and an interaction confidence interval communicates uncertainty better than a significance statement alone. The case also shows that statistical evidence and practical importance are distinct considerations.

9. Challenges, Ethics, and Contemporary Practice

Experiments can be statistically valid yet operationally or ethically unacceptable. Participants should not be exposed to unreasonable harm, deception, discrimination, or privacy loss. Sensitive experiments may require informed consent, institutional review, data-protection assessment, or regulatory approval.

Algorithmic experiments should examine heterogeneous effects across relevant demographic or risk groups. Subgroup analysis must be prespecified or clearly labeled exploratory because extensive subgroup searching creates false discoveries. Fairness concerns are not resolved merely because the overall average effect is positive.

Organizations must protect assignment data, prevent unauthorized exposure to experimental features, maintain audit trails, and define rollback procedures. Experiments involving medical, financial, employment, or educational decisions may be governed by sector-specific laws.

Contemporary experimentation increasingly uses automated randomization platforms, sequential monitoring, Bayesian models, multi-armed bandits, and adaptive experimentation. Bandits can reduce exposure to inferior treatments while optimizing rewards, but they do not automatically provide unbiased conventional treatment-effect estimates. Adaptive allocation, changing populations, delayed responses, and model updates require specialized analysis.

Chapter Summary

ANOVA compares mean responses by partitioning total variation into systematic factor variation and unexplained error. One-way ANOVA tests an omnibus equality-of-means hypothesis, while factorial ANOVA estimates main effects and interactions. Valid inference depends on appropriate experimental units, randomization, independent replication, suitable controls, and correct representation of blocks and response distributions.

A/B testing applies randomized experimentation to product decisions but requires prespecified metrics, valid stopping rules, and attention to interference and multiplicity. Sample-size planning for ANOVA uses the noncentral F distribution, while factorial studies should be powered for important interactions when those interactions drive decisions. Effect sizes, confidence intervals, diagnostics, guardrail metrics, and ethical governance are essential for translating statistical findings into responsible action.

Key Terms

ANOVA: A framework for comparing means by partitioning response variability.

A/B test: A randomized comparison between a control version and an alternative version.

Blocking: Grouping similar experimental units before treatment randomization.

Control group: The group representing the counterfactual or standard condition.

Effect size: A quantitative measure of the magnitude of a treatment or factor effect.

Experimental unit: The smallest unit independently assigned to a treatment.

Factor: An explanatory variable deliberately varied in an experiment.

Factorial design: A design containing combinations of levels from two or more factors.

Interaction: A condition in which the effect of one factor depends on another factor’s level.

Main effect: The average effect of one factor across levels of other factors.

Mean square: A sum of squares divided by its degrees of freedom.

Power: The probability of rejecting a false null hypothesis for a specified alternative.

Pseudoreplication: Incorrect treatment of dependent measurements as independent replicates.

Randomization: Assignment of experimental units to treatments using a chance mechanism.

Replication: Independent application of treatments to multiple experimental units.

Response variable: The measured outcome of an experiment.

Sum of squares: A measure of variability based on squared deviations.

Treatment: A particular factor level or combination of factor levels.

Review Questions

Short Answer Questions

1. Distinguish random assignment from random sampling.

2. What null hypothesis is tested by a one-way ANOVA?

3. Why is replication necessary in an experiment?

4. What does a factorial interaction mean?

5. Why can repeated significance testing invalidate an A/B test?

Long Answer Questions

1. Explain the decomposition of total variability in one-way ANOVA and derive the associated degrees of freedom.

2. Discuss the roles of randomization, replication, blocking, controls, and blinding in causal experimentation.

3. Explain how two-way ANOVA separates main effects, interactions, and random error.

4. Describe the assumptions of classical ANOVA, appropriate diagnostic methods, and alternatives when assumptions fail.

5. Develop a complete protocol for an online A/B test, including metrics, assignment, stopping, analysis, and governance.

Analytical / Application Questions

1. Four feature-selection methods are evaluated using independent model-training runs. Design an experiment that accounts for variation among datasets and specify an appropriate ANOVA model.

2. A 2×2 experiment produces cell means of 20, 26, 24, and 38 in the natural factorial order. Calculate the difference-in-differences interaction and interpret it.

3. Design a sample-size strategy for a three-factor experiment in which a two-factor interaction is the primary decision criterion. State the quantities and assumptions required for power analysis.

Multiple Choice Questions with Answers

Question 1

What does a significant one-way ANOVA F test establish?

A. Every pair of population means differs

B. At least one population mean differs from another

C. All population variances are unequal

D. The largest sample mean is unbiased

Correct Answer: B. At least one population mean differs from another

Question 2

For groups {2,3,4}, {5,6,7}, and {8,9,10}, what is the one-way ANOVA F statistic?

A. 9

B. 18

C. 27

D. 54

Correct Answer: C. 27

Question 3

In the worked two-way ANOVA, what is the F statistic for the A×B interaction?

A. 2

B. 4

C. 8

D. 36

Correct Answer: B. 4

Question 4

What does an interaction between factors A and B indicate?

A. The effect of A depends on the level of B

B. A and B have identical marginal means

C. The response variance is zero

D. Neither factor has a main effect

Correct Answer: A. The effect of A depends on the level of B

Question 5

What is the principal purpose of blocking?

A. To eliminate the need for randomization

B. To increase the number of treatments

C. To control variation from known nuisance variables

D. To guarantee normally distributed responses

Correct Answer: C. To control variation from known nuisance variables

Question 6

Which situation is an example of pseudoreplication?

A. Randomly assigning 100 users to each product version

B. Repeating treatment independently across multiple laboratories

C. Treating repeated measurements from one unit as independent units

D. Randomizing treatments separately within blocks

Correct Answer: C. Treating repeated measurements from one unit as independent units

Question 7

For four balanced groups, Cohen's f = 0.25, alpha = 0.05, and power = 0.80, the chapter's noncentral-F calculation requires approximately how many observations?

A. 45 total

B. 90 total

C. 128 total

D. 180 total

Correct Answer: D. 180 total

Question 8

For a balanced 2×2 design with sigma = 10, a target interaction of 10, alpha = 0.05, and power = 0.80, the large-sample calculation gives approximately how many analyzable observations per cell?

A. 16

B. 24

C. 32

D. 64

Correct Answer: C. 32

Question 9

In an A/B test, which choice must match the level at which treatment is independently assigned?

A. The experimental unit

B. The p-value threshold

C. The response measurement unit

D. The number of guardrail metrics

Correct Answer: A. The experimental unit

Question 10

Which method is appropriate for comparing several treatment means when equal variances are doubtful?

A. Welch's ANOVA

B. Dunnett's procedure alone

C. A paired sign test for every observation

D. Ordinary ANOVA with the error term removed

Correct Answer: A. Welch's ANOVA