Causal Research, Experiments, and Product Testing

Introduction

Market research describes markets and consumers; causal research asks what will happen if a decision changes. Product managers routinely face causal questions: Will a shorter sign-up form increase activation? Does a new price reduce demand? Will a revised onboarding flow improve retention? Does a prototype solve a meaningful customer problem better than the current product?

Causal evidence is especially valuable because correlation alone does not establish that an intervention produced an outcome. Users who adopt a feature may already be more engaged, and customers who buy a premium plan may have higher willingness to pay before seeing the price. Experiments attempt to isolate the effect of a controllable treatment from competing explanations.

This chapter explains causal reasoning, experimental design, A/B testing, concept and usability testing, prototype evaluation, test markets, statistical power, effect sizes, confounding, and external validity. The emphasis is managerial: a good study is not merely statistically significant; it produces decision-relevant evidence under ethical, operational, and commercial constraints.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish descriptive, predictive, and causal research.
2. Formulate causal hypotheses and identify treatment, outcome, mediator, moderator, and confounder variables.
3. Design A/B tests, concept tests, usability studies, prototype tests, and test markets.
4. Interpret effect sizes, uncertainty, statistical power, confidence intervals, and practical significance.
5. Diagnose threats arising from confounding, selection, attrition, interference, novelty, and poor measurement.
6. Assess internal validity, external validity, ethical risk, and organizational usefulness.
7. Translate experimental evidence into product decisions without overstating what the data prove.

1. From Association to Causation

A causal claim states that changing an intervention produces a change in an outcome, all else relevant being equal. In product research, the intervention may be a feature, message, price, design, service level, or distribution strategy.

Consider the claim: “Users who receive personalized recommendations retain longer.” This may be an association. More engaged users may be more likely to receive recommendations, creating a selection effect. A causal version is: “Assigning comparable users to receive personalized recommendations increases retention over 30 days.” The second claim specifies an intervention, a comparison, a population, and an outcome.

1.1 The counterfactual

The causal effect for one unit is the difference between its outcome under treatment and its outcome under control. Let Y_i(1) denote the outcome for unit i if treated, and Y_i(0) the outcome if not treated. The individual causal effect is:

Only one of these potential outcomes is observed for a unit, a condition known as the fundamental problem of causal inference. Randomized experiments estimate average effects by making treatment assignment independent of potential outcomes, in expectation.

The average treatment effect (ATE) is:

Here, E[·] denotes the population average. The formula applies when treatment and control are well defined and the outcome is measured consistently. In a simple randomized experiment, the difference in sample means estimates the ATE:

where Ȳ_T and Ȳ_C are the observed mean outcomes in treatment and control groups. This estimate is unbiased under valid randomization, no systematic post-assignment exclusions, and consistent measurement.

For binary outcomes such as conversion, the effect may be expressed as an absolute difference:

or a relative lift:

where p_T and p_C are treatment and control proportions. Absolute difference is often more useful for forecasting business impact, whereas relative lift can communicate proportional change. Neither should be reported without the underlying baseline.

1.2 Causal vocabulary

• Treatment or intervention: The product change deliberately assigned or introduced.
• Control: The comparison condition, often the current experience or a placebo-like alternative.
• Experimental unit: The entity assigned to treatment, such as a user, account, store, or geographic market.
• Outcome: The measured consequence, such as conversion, task success, revenue, or retention.
• Mediator: A variable through which treatment may operate. For example, a faster page may improve comprehension, which increases purchase.
• Moderator: A variable that changes the size or direction of an effect, such as device type or customer tenure.
• Confounder: A variable associated with both treatment exposure and outcome that creates a misleading causal association.
• Estimand: The precise effect being estimated, including population, treatment contrast, outcome, and time horizon.

2. Designing a Sound Experiment

A credible experiment begins before recruitment or instrumentation. The product team should write a decision statement: “If the new onboarding sequence improves completed setup by at least three percentage points without increasing support contacts, we will deploy it to new individual users.” This statement identifies the minimum useful effect, guardrails, target population, and decision rule.

2.1 Specify the causal question

A strong causal question includes:

1. Population: Which users or markets are relevant?
2. Treatment: What exactly changes?
3. Control: What is the realistic alternative?
4. Outcome: What behavior or perception is measured?
5. Time horizon: When should the effect appear?
6. Decision threshold: What magnitude justifies action?
7. Guardrails: What harms or trade-offs must not worsen?

Avoid vague outcomes such as “engagement.” Define an operational measure, such as weekly active use, completion of a core workflow, or seven-day retained activation.

2.2 Randomization and control

Random assignment distributes both observed and unobserved characteristics across conditions, reducing confounding on average. Randomization can occur at the user, account, session, store, or geographic-market level. The unit must match the way treatment operates. If all members of an account see the same price, randomizing individual employees may create contamination; account-level assignment is more appropriate.

The control condition should represent the decision alternative. A “no change” control is useful for measuring incremental impact. A head-to-head control is appropriate when choosing between two designs. A placebo or attention control may be needed when expectations themselves influence responses.

2.3 Blocking, stratification, and clustering

Stratification assigns treatment and control within important subgroups, such as new and existing users. It improves balance and can increase precision when subgroups differ substantially. Blocking by geography may be necessary when legal rules, infrastructure, or seasonality vary across regions.

Cluster randomization assigns groups rather than individuals. It reduces contamination but usually requires more observations because units within a cluster are correlated. The intracluster correlation coefficient (ICC), denoted ρ, describes similarity within clusters. A common design-effect approximation is:

where m is average cluster size. Required sample size is approximately multiplied by DE relative to independent individual assignment. This approximation assumes roughly equal cluster sizes and a common ICC; highly unequal clusters require more careful planning.

2.4 Pre-registration and analysis plans

Before examining outcomes, document the hypothesis, primary outcome, analysis population, sample-size target, stopping rule, exclusions, subgroup analyses, and treatment of missing data. Pre-registration does not guarantee quality, but it limits hindsight-driven hypotheses and selective reporting.

A primary outcome should be selected because it represents product value, not because it is likely to produce significance. Secondary and exploratory measures may be useful, but they should be labelled accordingly.

3. A/B Testing in Digital Products

An A/B test randomly assigns traffic to version A and version B and compares outcomes. A/B testing is powerful when assignment is stable, exposure is controlled, and outcomes can be measured over an appropriate period.

3.1 Basic workflow

1. Translate the product decision into a testable hypothesis.
2. Identify the assignment unit and randomization mechanism.
3. Implement treatment and control with persistent assignment.
4. Validate exposure, logging, eligibility, and data quality.
5. Estimate the required sample size and duration.
6. Monitor technical and safety guardrails without repeatedly changing the stopping rule.
7. Analyse the pre-specified primary outcome and uncertainty.
8. Examine heterogeneity cautiously and replicate important findings.
9. Make a decision based on effect size, risk, cost, and strategic fit.

3.2 Effect size and confidence interval

Suppose a control experience converts 10.0% of eligible users and a treatment converts 11.5%. The absolute effect is 1.5 percentage points. The relative lift is:

If the estimated effect is 1.5 percentage points with a 95% confidence interval from 0.3 to 2.7 percentage points, the data are compatible with a small positive effect and a larger positive effect. The interval does not mean there is a 95% probability that the fixed effect lies inside it; under repeated sampling, a method producing 95% intervals would capture the true parameter approximately 95% of the time.

For a difference in independent means, a simple standard error is:

where s_T² and s_C² are sample variances and n_T and n_C are sample sizes. A large-sample 95% interval is approximately estimate ± 1.96 × standard error when normal approximation conditions are reasonable. Binary outcomes often use a proportion-specific standard error or a suitable regression model.

Statistical significance is not the same as practical importance. With millions of observations, a tiny effect may be statistically distinguishable from zero but economically irrelevant. Conversely, a meaningful effect may be estimated imprecisely in a small test.

3.3 Power and sample size

Statistical power is the probability of detecting a specified effect under a chosen significance threshold, assuming the effect exists. It depends on effect size, sample size, outcome variability, allocation ratio, and the significance level.

For a rough two-group comparison of means with equal group sizes, the required observations per group can be approximated by:

Here, σ is the assumed common standard deviation, δ is the minimum detectable difference, α is the two-sided Type I error rate, 1−β is power, and z_q is the q-th standard-normal quantile. This formula assumes independent observations, equal variance, equal allocation, and a continuous outcome. Binary outcomes require a proportion-based calculation or software using the anticipated baseline rate.

Power planning should use a minimum practically important effect, not an optimistic effect copied from an earlier test. If multiple variants, outcomes, or repeated looks are planned, the analysis must account for multiplicity or use a sequential-testing design.

3.4 Sequential monitoring and experimentation risks

Repeatedly checking results and stopping when significance appears can inflate false-positive risk. Solutions include a fixed horizon, alpha-spending or sequential methods, and clearly defined stopping rules. “Peeking” is not automatically invalid, but unplanned peeking followed by an ordinary fixed-sample p-value is misleading.

Other common risks include:

• Novelty effects: Users respond to a new experience temporarily.
• Maturation: Users change naturally during the test period.
• Seasonality: Weekends, holidays, or campaigns alter behavior.
• Interference: One user’s treatment affects another user’s outcome.
• Carryover: A previous condition influences later behavior.
• Sample-ratio mismatch: Actual assignment proportions differ unexpectedly.
• Instrumentation failure: Events are missing or defined differently across variants.
• Survivorship bias: Analysis includes only users who remain active or complete an intermediate step.

A statistically significant result cannot repair a broken assignment mechanism or invalid measurement system.

4. Concept Testing and Demand Evidence

Concept tests evaluate an idea before substantial development. They are appropriate for comparing positioning, identifying perceived benefits, assessing comprehension, and screening alternatives. They are not reliable proof of actual purchase behavior by themselves.

4.1 Concept-test design

A concept stimulus should communicate the target user, problem, proposed solution, principal benefit, and relevant constraints. Keep competing concepts comparable in length, visual quality, and information. Randomize presentation order to reduce order effects.

Useful measures include:

• Problem importance and frequency
• Concept comprehension
• Perceived differentiation
• Relevance and credibility
• Preference among alternatives
• Purchase or trial intent
• Expected usage occasion
• Open-ended concerns and improvement suggestions

Intent measures are subject to hypothetical bias: respondents may express interest without paying, switching, waiting, or changing habits. Stronger evidence combines stated responses with behavioural proxies such as willingness to join a waitlist, choose a trial, provide a deposit where ethically appropriate, or complete a realistic task.

A concept test should distinguish “people like the idea” from “the idea solves a high-priority problem.” A compelling concept that addresses an infrequent problem may be less valuable than a less exciting concept addressing a recurrent, costly problem.

5. Usability Studies and Prototype Testing

Usability research examines whether representative users can learn and complete meaningful tasks with a product or prototype. It answers questions different from those answered by an A/B test. A usability study explains where and why users struggle; an A/B test estimates the causal effect of a released alternative at scale.

5.1 Task-based usability study

Define realistic tasks rather than asking participants whether they like the interface. A task should include a goal and sufficient context but avoid revealing the correct path. Observe success, errors, time, hesitation, requests for help, and confidence. Think-aloud protocols can expose mental models, but they may alter task performance; use them when diagnostic insight is more important than naturalistic timing.

For a binary task-success outcome, the observed success rate is:

where x is the number of successful attempts and n is the number of valid attempts. With small samples, avoid treating a percentage as precise population evidence. A five-person study can reveal severe usability failures, but it cannot estimate a population success rate with the precision of a large randomized study.

5.2 Prototype fidelity

• Low-fidelity prototypes are fast and useful for information architecture, problem framing, and early concept exploration.
• Mid-fidelity prototypes support interaction flows, content, and navigation decisions.
• High-fidelity prototypes approximate visual and interactive details and are useful for testing realistic workflows, but they can cause participants to overlook operational limitations.

The prototype should be matched to the uncertainty. Do not build high fidelity to answer a question that a sketch could answer. Conversely, do not infer performance of a complex real-time service from a static mock-up.

5.3 Measuring usability

A practical usability scorecard can combine task success, critical errors, completion time, satisfaction, and observed workarounds. Define success criteria before testing. For example: at least 85% of target users complete the primary task without moderator help, no critical error occurs, and median completion time is below four minutes.

Moderators should use neutral prompts, avoid teaching the interface, and record deviations consistently. Recruitment should include novices and experienced users when both are part of the target population.

6. Test Markets and Field Experiments

A test market introduces a product, price, promotion, or channel change in selected geographic or organizational markets. It provides stronger evidence about commercial behaviour than a survey, while preserving more operational realism than a laboratory or online experiment.

A randomized geographic test may assign matched regions to treatment and control. Outcomes can include sales, repeat purchase, distribution, margin, customer complaints, and competitor response. Because markets differ before treatment, analysts often use pre-period data and difference-in-differences models.

A basic difference-in-differences estimand is:

The treatment change is compared with the control change. The central assumption is parallel trends: absent treatment, treatment and control markets would have followed similar average trends. Pre-treatment trends can provide evidence about this assumption, but they cannot prove it. Other concerns include spillovers across market boundaries, concurrent promotions, small numbers of clusters, and limited statistical power.

Test markets are expensive and vulnerable to competitive learning. Competitors may copy the intervention before the organization scales it. A staged rollout, geo-experiment, or synthetic-control design may sometimes provide a better balance between learning and exposure.

7. Confounding, Validity, and Generalization

7.1 Confounding and selection

Confounding occurs when a third factor influences both exposure and outcome. For example, customers who opt into a premium onboarding session may be more motivated and may also retain longer. Comparing them with non-participants confounds the session with motivation.

Randomization addresses confounding before treatment. In observational settings, matching, regression adjustment, instrumental variables, natural experiments, or difference-in-differences may help, but each depends on assumptions. Statistical adjustment cannot control for an unmeasured confounder merely because many variables were included in a model.

Selection bias can arise during recruitment, treatment uptake, follow-up, or analysis. The intention-to-treat effect compares users according to assignment, preserving the benefit of randomization. A per-protocol or treatment-on-the-treated effect may be operationally interesting but is more vulnerable to post-assignment differences.

7.2 Internal and external validity

Internal validity concerns whether the observed difference in the study is caused by the intervention rather than bias, confounding, measurement error, or chance. External validity concerns whether the result generalizes to other users, contexts, times, channels, and implementations.

A tightly controlled experiment may have high internal validity but limited realism. A field test may generalize better but face interference, imperfect compliance, and competing events. Generalization should be argued, not assumed. Ask whether the mechanism, user population, treatment implementation, outcome definition, and environmental conditions are comparable.

Treatment-effect heterogeneity is important. An average positive effect may conceal harm to a subgroup. Estimate subgroup effects when they are theoretically motivated and pre-specified, and interpret them cautiously because subgroup analyses can be noisy and generate false discoveries.

8. Ethics, Governance, and Responsible Experimentation

Product experiments involve real people and can affect access, money, safety, dignity, and privacy. The organization should establish review proportional to risk. Informed consent may be required for research participation, especially when collecting sensitive data or exposing participants to material risk. Ordinary product experimentation is not automatically exempt from ethical responsibility.

Core principles include:

• Minimize foreseeable harm and avoid discriminatory treatment.
• Do not withhold essential safety, accessibility, or legally required services.
• Collect only necessary data and protect it through access controls and retention limits.
• Explain material changes clearly when user expectations or rights are affected.
• Monitor guardrail metrics, including complaints, refunds, safety incidents, and disparities.
• Maintain an audit trail of hypotheses, assignments, versions, analyses, and decisions.
• Avoid dark patterns that exploit cognitive bias to obtain consent or payment.

Automated experimentation and AI-generated variants create additional risks. Models can optimize clicks while degrading trust, accessibility, or long-term retention. AI-generated copy should be checked for factual accuracy, bias, copyright concerns, privacy leakage, and inappropriate personalization. The experiment owner remains accountable for the intervention and its consequences.

9. Integrated Case: An Illustrative Onboarding Experiment

Background

A fictional subscription software company, FlowDesk, observes that many newly registered teams do not complete their first project. Product managers propose a guided onboarding checklist with sample data and contextual prompts.

Problem and hypothesis

The decision question is whether the checklist increases seven-day activation without increasing support demand. The hypothesis is: “Among newly registered small-business teams, assignment to the guided checklist increases the proportion completing a first project within seven days.” The minimum useful effect is set at three percentage points. Guardrails are support contacts per activated team, cancellation within 30 days, and accessibility-related complaints.

Design

The account, rather than the individual user, is the experimental unit because all team members should see the same onboarding experience. Eligible accounts are randomly assigned 50:50 to the existing onboarding flow or the checklist. Assignment is persistent. The primary outcome is account-level completion of a first project within seven days. The analysis follows intention to treat.

Before launch, the team checks that project-completion events, assignment flags, and support-contact events are recorded consistently. It defines a two-week recruitment period and a seven-day outcome window for each account, avoiding a decision based on early peeking.

Results

Suppose 10,000 accounts are assigned to each condition. Control activation is 20.0% and treatment activation is 22.4%. The estimated absolute effect is:

Thus, the checklist increases activation by 2.4 percentage points, or 12% relative to the control baseline. The result is below the pre-specified three-point minimum, so the team should not automatically deploy it. It should examine the confidence interval, implementation cost, and guardrails. If support contacts increase materially, the net value may be negative even if activation rises.

The team also finds a larger apparent effect for accounts invited by partners than for self-serve accounts. Because this subgroup analysis was exploratory, it is treated as a hypothesis for replication rather than as definitive proof. Interviews reveal that the checklist’s sample data confuse users who already have an established workflow. The next iteration may offer a choice between sample data and importing an existing project.

Outcome and learning points

The experiment prevented a premature full rollout and identified a promising design direction. It also demonstrates why a positive metric is not enough: the effect missed the decision threshold, subgroup evidence was uncertain, and qualitative research explained the mechanism and a usability problem.

10. Best-Practice Decision Framework

A product manager can evaluate an experiment using five questions:

1. Causal credibility: Was assignment or comparison credible? Were exposure and measurement valid?
2. Magnitude: Is the effect large enough to matter financially, strategically, or experientially?
3. Precision: How wide is the interval, and could the result plausibly include no meaningful effect?
4. Trade-offs: Did guardrails, equity, cost, or operational load worsen?
5. Transferability: Does the tested population and context represent the intended launch?

A useful decision matrix is:

Challenges and Limitations

No experiment eliminates uncertainty. Small samples produce imprecise estimates; large samples can produce statistically significant trivial effects. Product interventions may evolve during testing, violating treatment consistency. Users may adapt, competitors may respond, and long-term effects may differ from short-term outcomes. Network products create interference, while privacy restrictions may limit tracking. Operational teams may also selectively implement a successful treatment, making the scaled version different from the tested version.

The most serious limitation is often a mismatch between the estimand and the decision. A test of click-through rate may not answer whether a feature improves customer value. A short usability session may not answer whether customers retain after repeated use. A concept preference may not answer whether a product earns revenue. Every study should therefore state what it can establish, what it cannot establish, and what subsequent evidence is needed.

Chapter Summary

Causal research evaluates whether an intervention changes an outcome, rather than merely identifying an association. Random assignment, an appropriate control, valid measurement, pre-specified outcomes, and disciplined analysis provide the foundation for credible experiments. A/B tests estimate incremental effects at scale; concept tests explore appeal and comprehension; usability studies diagnose task difficulties; prototype tests reduce design uncertainty; and test markets assess commercial behaviour in realistic settings.

Effect size, confidence interval, statistical power, and practical significance must be interpreted together. Confounding, selection, interference, novelty, attrition, instrumentation errors, and multiple testing can undermine conclusions. Internal validity supports causal credibility, while external validity determines how responsibly findings can be generalized. Ethical experimentation requires proportional review, privacy protection, non-discrimination, safety monitoring, and accountability for automated or AI-assisted interventions. The strongest product decisions combine quantitative experiments with qualitative explanation, business thresholds, guardrails, and replication.

Key Terms

Causal effect: Change in an outcome attributable to an intervention.

Counterfactual: The outcome that would have occurred under an alternative treatment condition.

Treatment: The intervention assigned or introduced in a study.

Control: The comparison condition against which treatment is evaluated.

Experimental unit: Entity assigned to treatment or control.

Average treatment effect: Population-average difference between treatment and control potential outcomes.

Confounder: Variable associated with both treatment exposure and outcome.

Mediator: Variable through which an intervention produces an outcome.

Moderator: Variable that changes the size or direction of an effect.

A/B test: Randomized comparison of two product or marketing variants.

Effect size: Magnitude of an observed or estimated difference.

Statistical power: Probability of detecting a specified effect under stated assumptions.

Confidence interval: Range produced by a statistical procedure intended to capture a population parameter at a stated long-run rate.

Practical significance: Business or user importance of an effect.

Intention-to-treat: Analysis based on original assignment, regardless of treatment uptake.

Internal validity: Credibility that the study intervention caused the observed difference.

External validity: Extent to which findings generalize beyond the study context.

Test market: Field experiment conducted in selected geographic or organizational markets.

Interference: Situation in which one unit’s treatment affects another unit’s outcome.

Guardrail metric: Outcome monitored to detect harmful or unacceptable trade-offs.

Review Questions

Short Answer Questions

1. Why does correlation between feature adoption and retention not necessarily establish causation?
2. Define treatment, control, experimental unit, and outcome.
3. What is the difference between an absolute effect and a relative lift?
4. Why can a usability study and an A/B test provide different kinds of evidence?
5. What is the purpose of a guardrail metric?

Long Answer Questions

1. Explain how randomization improves causal inference and discuss situations in which cluster randomization is preferable.
2. Discuss statistical power, minimum practically important effect, and the danger of interpreting statistical significance as business importance.
3. Compare concept testing, prototype testing, usability studies, and A/B testing in terms of purpose, timing, evidence, and limitations.
4. Explain internal validity and external validity, using a product experiment to illustrate the trade-off between them.
5. Describe the ethical and governance responsibilities involved in experimentation on digital products.

Analytical / Application Questions

1. A treatment increases conversion from 8% to 9%. Calculate the absolute percentage-point effect and relative lift, then explain what additional evidence is needed before launch.
2. A company randomizes individual employees even though the feature is visible to every employee in the same account. Identify the design problem and recommend a better experimental unit.
3. A geographic test shows sales increased more in treatment regions, but treatment regions were already growing faster before launch. Explain why the causal conclusion is weak and propose an improved analysis or design.

Multiple Choice Questions with Answers

Question 1

Which condition most directly supports a causal interpretation in an A/B test?

A. Treatment users report higher satisfaction after choosing the feature themselves

B. Users are randomly assigned to treatment and control with valid outcome measurement

C. The treatment group is larger than the control group

D. The treatment has a higher click-through rate than an earlier campaign

Correct Answer: B. Users are randomly assigned to treatment and control with valid outcome measurement

Question 2

A control conversion rate is 10% and a treatment conversion rate is 12%. What is the relative lift?

A. 2%

B. 10%

C. 20%

D. 22%

Correct Answer: C. 20%

Question 3

In an experiment, what is a confounder?

A. A variable caused only by the treatment

B. A variable associated with both treatment exposure and the outcome

C. The primary outcome selected before analysis

D. A random measurement fluctuation

Correct Answer: B. A variable associated with both treatment exposure and the outcome

Question 4

Which study is most appropriate for discovering why users cannot complete a checkout task?

A. A task-based usability study

B. A national sales forecast

C. A post-launch revenue report

D. A market-share calculation

Correct Answer: A. A task-based usability study

Question 5

What does statistical power describe?

A. The probability of detecting a specified effect when it exists

B. The probability that the treatment is morally acceptable

C. The percentage of users assigned to treatment

D. The size of the confidence interval only

Correct Answer: A. The probability of detecting a specified effect when it exists

Question 6

If users within the same account all see the same treatment, what is usually the appropriate randomization unit?

A. A page view

B. An individual click

C. The account

D. A survey question

Correct Answer: C. The account

Question 7

Which statement best describes external validity?

A. Whether the treatment was implemented exactly as designed

B. Whether the observed result can generalize to other relevant contexts

C. Whether a p-value is below a threshold

D. Whether all participants completed the survey

Correct Answer: B. Whether the observed result can generalize to other relevant contexts

Question 8

In a difference-in-differences analysis, the key identifying assumption is commonly that treatment and control groups would have had similar trends without treatment. What is this called?

A. Stable-unit assumption

B. Parallel trends assumption

C. Random-effects assumption

D. Placebo assumption

Correct Answer: B. Parallel trends assumption

Question 9

Why should a product team pre-specify a primary outcome?

A. To guarantee a positive result

B. To prevent all measurement error

C. To reduce selective reporting and hindsight-driven analysis

D. To eliminate the need for a control group

Correct Answer: C. To reduce selective reporting and hindsight-driven analysis

Question 10

A statistically significant effect is extremely small and has no meaningful financial or user impact. How should it be interpreted?

A. It is automatically a successful product change

B. It demonstrates high external validity

C. It may lack practical significance despite statistical significance

D. It proves the null hypothesis is true

Correct Answer: C. It may lack practical significance despite statistical significance