Experimentation, Testing and Innovation Analytics

Introduction

Entrepreneurial ideas are not validated by enthusiasm, persuasive presentations or internal consensus. They are strengthened—or rejected—through disciplined experimentation. Experimentation converts uncertain assumptions into testable propositions, exposes entrepreneurs to real user behaviour, and creates evidence for decisions about investment, redesign, pivoting or termination.

In design thinking, experimentation follows problem framing and ideation, but it is not merely a final-stage activity. Entrepreneurs should test assumptions throughout the innovation process. Early experiments may investigate whether a problem is important, whether a proposed solution is understandable, whether users will attempt a desired behaviour, and whether a viable business model can be developed.

This chapter examines usability tests, concept tests, landing-page experiments and minimum viable products (MVPs). It also develops the analytical foundations required to define hypotheses, select metrics, establish learning criteria, interpret qualitative and quantitative evidence, analyse cohorts, conduct A/B tests and assess experiment validity. The central principle is that an experiment is valuable only when it produces trustworthy learning that changes a decision.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Distinguish assumptions, hypotheses, tests, metrics and learning criteria.
2. Design usability, concept, landing-page and MVP experiments.
3. Select behavioural and attitudinal measures appropriate to an innovation question.
4. Interpret qualitative evidence, conversion rates, cohorts and A/B-test results.
5. Identify threats to internal, external, construct and statistical validity.
6. Make ethically responsible decisions from incomplete and ambiguous evidence.

1. Experimentation as a Decision System

An experiment is a structured intervention designed to generate evidence about an uncertain proposition. It is not simply an activity involving users. A useful experiment links five elements:

An assumption is a belief about customers, context, behaviour, technology, operations or economics. A hypothesis is an assumption expressed in a form that can be observed or measured. A prediction is usually more specific than a hypothesis and identifies the expected direction or magnitude of an outcome.

A strong hypothesis has four characteristics:

• Specificity: it identifies a population, intervention and outcome.
• Falsifiability: evidence could show that it is wrong.
• Relevance: it concerns a decision that matters.
• Measurability: the outcome can be observed with reasonable reliability.

A weak statement is: “Customers will love our service.” A stronger statement is: “Among office workers who purchase breakfast at least twice weekly, at least 25% of qualified landing-page visitors will request early access when shown a pre-order service.” The second statement defines a population, behaviour and threshold.

Experimentation follows a learning loop:

Assumption → Hypothesis → Test design → Evidence collection
     ↑                                            ↓
Decision ← Learning criterion ← Analysis and interpretation

The loop should not be confused with a demand to run experiments endlessly. The purpose is decision efficiency: reduce important uncertainty at an acceptable cost and risk.

2. From Assumptions to Testable Hypotheses

An assumption map ranks beliefs according to importance and uncertainty. High-importance, high-uncertainty assumptions deserve early testing. Examples include:

• Desirability: Do customers experience a meaningful problem?
• Usability: Can users understand and complete the proposed task?
• Feasibility: Can the solution be delivered technically and operationally?
• Viability: Will revenue, cost and retention support a sustainable model?
• Ethical acceptability: Does the solution avoid unacceptable harm, exclusion or deception?

A hypothesis should specify the unit of analysis. The unit may be a person, session, account, transaction, firm or cohort. Confusing units produces misleading conclusions. For example, ten purchases by one enthusiastic customer do not demonstrate adoption by ten customers.

Metrics and operational definitions

A metric is a defined measure, not merely a number. “Engagement” is ambiguous until translated into an event such as completing a lesson, returning within seven days or sharing a document. An operational definition specifies:

• the event or state being measured;
• the population included;
• the observation period;
• the data source;
• exclusions and treatment of missing data.

Metrics may be:

• Leading indicators, which provide early evidence, such as task completion or activation;
• Lagging indicators, which reflect later outcomes, such as retention, revenue or referral;
• Behavioural metrics, based on observed actions;
• Attitudinal metrics, based on reported perceptions or intentions;
• Guardrail metrics, which identify undesirable effects, such as complaints, errors or cancellation.

A single metric is rarely sufficient. A balanced metric set normally includes a primary metric, diagnostic metrics and guardrails. For a digital service, the primary metric might be successful activation; diagnostic metrics may include page load, registration completion and first-use time; guardrails may include support contacts and privacy complaints.

Conversion and retention measures

If x users complete a target action among n eligible users, the observed conversion rate is:

Here, p̂ is the sample conversion rate, x is the number of conversions and n is the number of eligible observations. The denominator must be defined consistently. Counting all page views in one condition and unique qualified visitors in another invalidates comparison.

For a cohort initially containing N₀ users, if N_t remain active at time t, a simple retention rate is:

This measure assumes that “active” is clearly defined and that the cohort membership and observation window are stable. Retention should not be inferred from cumulative logins, which may count repeated activity by a shrinking group.

3. Usability Testing

Usability testing evaluates whether intended users can understand and use a product or prototype to accomplish realistic tasks. It is especially appropriate when the key uncertainty concerns comprehension, navigation, workflow, error recovery or perceived effort.

Designing a usability test

A professional usability protocol normally contains:

1. Research question: What usability uncertainty is being examined?
2. Participant criteria: Which users have relevant experience or context?
3. Task scenario: What realistic goal should participants attempt?
4. Success definition: What counts as completion, partial completion or failure?
5. Observation plan: What behaviours, errors, questions and time points will be recorded?
6. Interview prompts: What will be asked after the task without leading the participant?
7. Ethical procedure: How will consent, privacy and withdrawal be handled?

Tasks should describe a goal rather than provide a sequence of interface instructions. “You have arrived at the airport and need to change your booking to tomorrow” is more diagnostic than “Click the settings icon and select change booking.”

A facilitator should avoid rescuing participants too quickly. If a participant asks, “What should I click?”, the facilitator can respond, “What would you expect to do next?” The purpose is not to embarrass users but to reveal where the design fails to communicate.

Qualitative evidence

Useful qualitative observations include hesitation, repeated scanning, incorrect interpretations, workarounds, emotional reactions and spontaneous language. Analysis should distinguish:

• Observation: “Four participants searched for the cancellation option under payment settings.”
• Interpretation: “The information architecture may not match users’ mental models.”
• Decision implication: “Test a revised navigation structure before adding new features.”

Small usability studies are not normally intended to estimate population conversion rates. Their strength is diagnostic depth. A recurring failure across carefully selected participants can reveal a design problem even when the number of participants is small; conversely, a small successful demonstration cannot establish broad market demand.

A coding matrix can improve rigour. Each observation is assigned a category such as navigation, terminology, trust, error recovery or accessibility. Researchers should retain disconfirming evidence rather than reporting only quotations that support the preferred design.

4. Concept Testing

Concept testing evaluates a proposed value proposition before substantial development. The concept may be expressed through a storyboard, service description, mock-up, prototype or short demonstration.

Concept tests can investigate:

• problem recognition and importance;
• perceived usefulness;
• comprehension of the offering;
• differentiation from alternatives;
• credibility and trust;
• willingness to try or pay;
• concerns, barriers and unintended interpretations.

Concept tests are vulnerable to hypothetical bias. People may state that they would purchase a product without making the sacrifice required in real life. Therefore, stated intention should be combined with behavioural signals such as joining a waitlist, selecting a preferred plan, completing a realistic task or making a refundable reservation.

A concept-test interview should separate reactions to the problem from reactions to the proposed solution. Questions such as “How do you currently handle this situation?” often reveal existing alternatives more reliably than “Would you use our product?” The latter invites politeness and speculation.

Concept comparisons must be designed carefully. Showing one concept first can create primacy effects; showing all concepts to every participant can create fatigue and comparison behaviour that would not occur in the market. Randomising presentation order and recording the context of evaluation can reduce these effects.

5. Landing-Page Experiments

A landing page is a rapid experiment for testing a value proposition, audience-message fit and an intended call to action. It is not proof of a complete business model. A visitor may click because of curiosity, novelty or an incentive, while long-term retention remains unknown.

A credible landing-page experiment defines:

• the target audience and acquisition source;
• the message and offer;
• the call to action;
• the traffic eligibility rule;
• the conversion event;
• the experiment period;
• the learning threshold and next decision.

For example, a page may promise “same-day repair pickup for small retailers” and invite qualified visitors to request a pilot appointment. The experiment should not count accidental clicks or duplicate submissions as independent evidence.

Funnel analysis

A funnel decomposes a journey into stages. If n_i users enter stage i and n_i+1 proceed to the next stage, the stage conversion rate is:

The overall conversion from the first to final stage is:

where n₀ is the number entering the funnel and n_k is the number completing the final action. These formulas assume that the stages are ordered, the denominator is known and observations are not double-counted. Funnel analysis identifies where loss occurs but does not by itself explain why.

Worked example: landing-page interpretation

An experiment attracts 800 qualified visitors. Of these, 160 begin the request form and 48 submit it.

The visitor-to-form-start rate is:

The form-completion rate among starters is:

The overall visitor-to-request rate is:

The evidence suggests two distinct questions: whether the page motivates action and whether the form creates friction. It would be inappropriate to conclude that the value proposition is weak solely from the overall rate without examining traffic quality, message comprehension and form usability.

6. Minimum Viable Products

An MVP is the smallest ethically acceptable product or service configuration that enables a team to test a critical assumption with real or realistically committed users. “Minimum” refers to the learning objective, not merely to low cost. “Viable” means that users can receive a meaningful outcome; it does not mean that the venture is already profitable.

An MVP may be:

• a concierge service delivered manually;
• a Wizard-of-Oz interface in which humans perform hidden operations;
• a no-code prototype;
• a limited geographic or customer segment launch;
• a paid pilot;
• a single-use workflow rather than a full product platform.

An MVP should not deliberately expose users to unsafe conditions, misleading claims, inadequate privacy protection or avoidable service failure. Manual operation is acceptable when it is disclosed appropriately and does not create material risk.

The MVP learning plan should identify:

1. the critical assumption;
2. the narrowest suitable user segment;
3. the minimum experience required to deliver value;
4. the behaviour that will be observed;
5. the time required to observe it;
6. the criterion for continuation, redesign or stopping.

A common error is to build a feature-complete product before testing whether the underlying problem matters. Another is to treat any usage as validation, without examining repeat use, willingness to pay, operational cost or user outcomes.

7. A/B Testing and Quantitative Inference

An A/B test randomly assigns eligible units to a control condition A or treatment condition B. The treatment may be a different headline, onboarding sequence, price presentation or workflow. Random assignment helps balance observed and unobserved characteristics in expectation, so outcome differences can be more plausibly attributed to the treatment.

Let p_A and p_B denote the true conversion probabilities under conditions A and B. The absolute treatment effect is:

The relative lift, when p_A ≠ 0, is:

The absolute difference is usually more decision-relevant for capacity and revenue; relative lift can exaggerate the practical importance of a small baseline change.

For observed conversion rates p̂_A and p̂_B, the observed difference is p̂_B − p̂_A. A rough standard error for the difference between two independent proportions is:

where n_A and n_B are sample sizes. This approximation assumes independent observations, binary outcomes, sufficiently large samples for the normal approximation, and a pre-specified comparison. For small counts or extreme proportions, exact or specialised methods may be preferable.

A 95% approximate confidence interval for the difference is:

The interval expresses sampling uncertainty under the model; it does not correct for biased traffic, repeated peeking, multiple unplanned comparisons or a flawed outcome definition.

Statistical significance and practical significance

A result can be statistically distinguishable from zero but too small to justify implementation. Conversely, a meaningful improvement may fail to reach conventional statistical significance when the sample is small. Decisions should consider:

• effect size and confidence interval;
• cost and reversibility of implementation;
• expected impact on strategic goals;
• guardrail outcomes;
• quality and representativeness of traffic;
• duration and novelty effects.

Teams should specify a minimum practically important effect before testing. They should also avoid stopping the test whenever a desired pattern appears. Repeatedly checking results and stopping early increases false-positive risk unless the analysis plan accounts for sequential monitoring.

Worked A/B example

Suppose version A receives 1,000 visitors and 100 convert; version B receives 1,000 visitors and 120 convert.

The absolute difference is 0.02, or 2 percentage points. The relative lift is:

The approximate standard error is:

The approximate 95% interval is:

Thus the interval is approximately −0.0074 to 0.0474, or −0.74 to 4.74 percentage points. Under these assumptions, the evidence is compatible with a small negative effect as well as a positive effect. The responsible conclusion is not “B definitely wins,” but “the test does not yet establish a precise positive effect.”

8. Cohort Analysis and Behaviour Over Time

A cohort is a group sharing a defined starting condition, such as signup week, acquisition channel, institution or first purchase month. Cohort analysis prevents aggregate metrics from concealing changes in user quality or retention.

For example, overall monthly active users may rise because acquisition has increased, while each new cohort retains fewer users. A cohort table can display the proportion of each cohort returning in weeks 1, 2, 4 and 8. Interpretation requires attention to:

• cohort size and statistical noise;
• calendar effects and seasonality;
• acquisition source;
• product changes introduced during observation;
• censoring, where newer cohorts have not had enough time to mature;
• survivorship bias, where only active users are analysed.

Cohorts should be defined before inspecting results when possible. Otherwise, analysts may select a segmentation that produces an attractive story by chance.

9. Validity, Bias and Data Quality

A persuasive experiment can still be invalid. Four forms of validity are particularly important.

Internal validity

Internal validity concerns whether the observed outcome difference was caused by the intervention rather than another factor. Threats include non-random assignment, unequal exposure periods, contamination between groups, instrumentation changes and concurrent campaigns.

External validity

External validity concerns generalisation beyond the tested sample, context or time. A test with early adopters may not predict behaviour among mainstream customers. A result during a promotional event may not apply during ordinary demand.

Construct validity

Construct validity asks whether the metric actually represents the concept of interest. Clicks may not represent value; stated willingness to pay may not represent actual payment; time on page may indicate confusion rather than engagement.

Statistical conclusion validity

This concerns whether the data and analysis support the stated inference. Problems include low power, excessive variance, multiple testing, dependence among observations and inappropriate statistical models.

Other common biases include:

• Selection bias: participants differ systematically from the target population.
• Response bias: participants provide socially desirable answers.
• Novelty effect: initial interest fades after exposure becomes routine.
• Hawthorne effect: behaviour changes because users know they are observed.
• Confirmation bias: researchers notice evidence supporting their preferred idea.
• Survivorship bias: analysis excludes people who stopped using the product.
• Simpson’s paradox: aggregate trends reverse when data are separated into meaningful subgroups.

A validity review should be completed before declaring a learning outcome. The question is not simply “What happened?” but “What alternative explanations remain plausible?”

10. Evidence Synthesis and Decision Rules

No single evidence type answers every entrepreneurial question. Qualitative evidence explains meaning and friction; behavioural experiments reveal action; financial analysis examines economic consequences; operational pilots reveal delivery constraints.

A useful evidence matrix records:

A decision rule should be written before results are interpreted. For example: “Continue to a paid pilot if at least 20 qualified firms complete the workflow, at least 40% repeat within 30 days, no critical service failures occur, and median delivery cost remains below the allowable threshold.” Such rules reduce post hoc rationalisation.

Decision options include:

• Persevere: evidence supports the current direction.
• Iterate: the problem appears meaningful but the solution or experience requires change.
• Pivot: a different segment, problem, channel or business model is better supported.
• Pause: evidence is insufficient or operational constraints prevent a fair test.
• Stop: the critical assumption is contradicted or the risk is unacceptable.

11. Applied Case: FreshRoute Pilot

The following is a fictional illustrative case.

Background

FreshRoute is a proposed subscription service delivering prepared breakfasts to employees in small business districts. The founding team believes that commuters want healthier food but avoid existing options because queues are long and menus are unpredictable.

Problem and assumptions

The team identifies three high-risk assumptions:

1. Office workers experience the problem frequently.
2. They will pre-order breakfast before commuting.
3. The team can deliver orders within a 15-minute collection window at acceptable cost.

Approach

First, the team conducts twelve contextual interviews near transport stations. Participants describe queueing as inconvenient, but only seven report buying breakfast at least twice per week. This distinction leads the team to define a narrower target segment: workers who regularly purchase breakfast and travel through one station.

Next, six participants complete a prototype ordering task. Four misunderstand the difference between “pickup time” and “delivery time.” The team redesigns the interface and retests it with four new participants. All four complete the revised task without assistance, although two request clearer allergen information.

The team then runs a landing-page test for two weeks. Qualified visitors are invited to reserve a discounted pilot slot. Of 600 qualified visitors, 150 start the form and 45 complete it. The overall conversion is 7.5%, while form-start conversion is 25% and completion among starters is 30%.

Rather than treating 45 reservations as proof of demand, the team launches a concierge MVP for 30 customers. Orders are prepared manually, allergen information is displayed, and customers receive a clear statement that the pilot is limited and operationally assisted. Twenty-four customers complete a second order in the following week. However, average delivery cost is too high because orders are geographically dispersed.

Analysis and outcome

The evidence supports a meaningful problem and willingness to try the service, but it challenges the original operating model. Repeat ordering is encouraging, while delivery economics are weak. The team therefore pivots from individual delivery to scheduled collection at two office hubs. A new pilot is designed to test whether concentration reduces cost without reducing repeat purchase.

Learning points

• Interviews refined the target segment rather than proving demand.
• Usability testing exposed a specific terminology problem.
• The landing page measured response to an offer, not sustainable adoption.
• The concierge MVP tested repeat behaviour and operational cost simultaneously.
• A pivot was based on an assumption that failed—delivery economics—not on a general judgement that the idea was a failure.

12. Ethics, Governance and Responsible Experimentation

Entrepreneurs have a duty to protect participants and customers, especially when testing products involving finance, health, employment, education, children or personal data. Responsible experimentation includes informed consent where appropriate, data minimisation, secure storage, transparent withdrawal procedures and clear communication of risks.

Deceptive experiments require particular caution. Concealing a minor interface detail may differ ethically from misrepresenting price, safety, medical benefit or availability. Personalisation systems can produce discriminatory outcomes if treatment exposure or conversion is uneven across protected groups. Analysts should examine guardrail metrics by relevant segments when lawful and appropriate.

Privacy regulations may impose requirements concerning lawful processing, purpose limitation, access, deletion, profiling and automated decision-making. Legal compliance is a minimum standard, not a substitute for ethical judgement. Teams should ask whether participants would reasonably understand what is being tested and whether the potential benefit justifies the burden or risk.

AI can assist experiment design by generating alternative copy, clustering interview transcripts or detecting anomalous patterns. However, AI-generated interpretations may reproduce bias, hallucinate themes or expose confidential data. Human researchers should verify coding decisions, protect sensitive information and retain an auditable record of data transformations.

13. Contemporary Developments and Best Practice

Modern experimentation increasingly combines product analytics, feature flags, remote research, behavioural segmentation and automated dashboards. These tools improve speed but can also encourage metric fixation and excessive testing. High-performing innovation teams therefore use a written experiment brief containing:

• decision context and critical assumption;
• hypothesis and predicted direction;
• target population and sampling rule;
• primary, diagnostic and guardrail metrics;
• randomisation or comparison method;
• minimum practically important effect;
• duration and stopping rule;
• ethical and privacy review;
• analysis plan and decision options.

Best practice is to test the riskiest assumption first, use the smallest credible experiment, measure real behaviour where feasible, triangulate qualitative and quantitative evidence, inspect disconfirming cases, document null results and distinguish learning from proof. The goal is not to produce certainty. It is to make better decisions than would be possible through intuition alone.

Chapter Summary

Experimentation is a disciplined system for reducing entrepreneurial uncertainty. It begins with assumptions and converts them into falsifiable hypotheses linked to defined metrics and decision criteria. Usability tests diagnose comprehension and interaction problems; concept tests examine reactions to proposed value; landing-page experiments evaluate audience-message fit; and MVPs expose real users to the smallest viable value-delivery system.

Conversion rates, funnel measures and cohort retention provide useful quantitative evidence when denominators, observation periods and units of analysis are clearly defined. A/B tests can estimate comparative effects under random assignment, but statistical significance must be distinguished from practical importance. Qualitative evidence explains why behaviour occurs, while behavioural and operational evidence tests what users actually do.

Validity threats—including selection, novelty, instrumentation, construct and statistical problems—must be considered before drawing conclusions. Ethical experimentation requires privacy protection, truthful communication, risk management and attention to discriminatory effects. The most valuable experiment is not the one that confirms the founders’ preferred idea, but the one that produces trustworthy learning and improves a consequential decision.

Key Terms

Assumption: A belief about a customer, context, solution, operation or business model that may be uncertain.

Hypothesis: A falsifiable proposition linking conditions to an observable outcome.

Experiment: A structured test designed to generate evidence for a decision.

Metric: An operationally defined measure of an outcome or behaviour.

Learning criterion: A pre-specified result that determines a decision.

Usability testing: Evaluation of whether users can accomplish realistic tasks with a product or prototype.

Concept testing: Assessment of a proposed offering before substantial development.

Landing-page experiment: A test using a web page to examine response to a proposition and call to action.

Minimum viable product: The smallest ethically acceptable offering that tests a critical assumption while delivering meaningful value.

Conversion rate: The proportion of eligible observations completing a defined action.

Funnel: A sequence of stages through which users progress toward a target outcome.

Cohort: A group sharing a defined starting condition or time period.

Retention: Continued activity or return by members of an initial cohort.

A/B test: A randomised comparison between control and treatment conditions.

Absolute treatment effect: The difference between outcome probabilities in two conditions.

Relative lift: The treatment effect expressed relative to the control outcome.

Guardrail metric: A measure used to detect harmful or undesirable effects.

Internal validity: The extent to which an observed effect can be attributed to the tested intervention.

External validity: The extent to which findings generalise to other settings or populations.

Construct validity: The extent to which a measure represents the concept it is intended to measure.

Review Questions

Short Answer Questions

1. What distinguishes an assumption from a testable hypothesis?
2. Why should a landing-page experiment define its denominator carefully?
3. What is the diagnostic purpose of usability testing?
4. Explain the difference between a leading indicator and a lagging indicator.
5. Why can an MVP be manually operated?

Long Answer Questions

1. Explain how an entrepreneur can move from an assumption map to a complete experiment brief.
2. Compare usability testing, concept testing, landing-page experiments and MVPs in terms of their purposes and limitations.
3. Explain absolute treatment effect, relative lift and practical significance in A/B testing.
4. Discuss the major threats to internal, external, construct and statistical conclusion validity.
5. Explain how qualitative evidence and cohort analysis can be combined to evaluate a new digital service.

Analytical / Application Questions

1. A landing page receives 1,200 qualified visitors, 240 form starts and 60 completed requests. Calculate the three relevant conversion rates and identify the most important follow-up question.
2. Design an experiment for a proposed subscription service. State the critical assumption, hypothesis, target population, primary metric, guardrail metric, learning criterion and decision options.
3. An A/B test shows a 3-percentage-point increase in conversion, but customer complaints and cancellation rates also rise. Analyse whether the treatment should be adopted and explain what additional evidence is required.

Multiple Choice Questions with Answers

Question 1

Which statement is the strongest testable hypothesis?

A. Customers will love the application.

B. The product has an excellent market opportunity.

C. At least 25% of qualified visitors will request a pilot after viewing the service page.

D. The founding team believes the problem is important.

Correct Answer: C. At least 25% of qualified visitors will request a pilot after viewing the service page.

Question 2

What is the main purpose of usability testing?

A. To estimate total market size

B. To diagnose whether users can understand and complete realistic tasks

C. To calculate long-term profitability

D. To prove that a business model will scale

Correct Answer: B. To diagnose whether users can understand and complete realistic tasks

Question 3

A page has 800 qualified visitors and 48 completed requests. What is the visitor-to-request conversion rate?

A. 3%

B. 6%

C. 16.7%

D. 60%

Correct Answer: B. 6%

Question 4

Which is a guardrail metric?

A. A primary activation event

B. A new feature name

C. The number of design meetings

D. Customer complaints following a treatment change

Correct Answer: D. Customer complaints following a treatment change

Question 5

What is the principal benefit of random assignment in an A/B test?

A. It guarantees a positive result.

B. It eliminates the need for a sufficiently large sample.

C. It helps balance other characteristics between conditions in expectation.

D. It ensures that all users behave identically.

Correct Answer: C. It helps balance other characteristics between conditions in expectation.

Question 6

A cohort is best defined as:

A. All users who have ever visited a website

B. A group sharing a defined starting condition or time period

C. Only users who converted immediately

D. A random collection of unrelated observations

Correct Answer: B. A group sharing a defined starting condition or time period

Question 7

Which finding most directly challenges construct validity?

A. The sample was recruited from one city.

B. The test ended during a holiday period.

C. The metric called engagement counts page views that may reflect confusion.

D. The treatment group was exposed for fewer days.

Correct Answer: C. The metric called engagement counts page views that may reflect confusion.

Question 8

In the FreshRoute case, what was the main reason for pivoting?

A. No participant ever reordered.

B. The problem was found to be completely imaginary.

C. The delivery economics were too weak under the original operating model.

D. The landing page received no traffic.

Correct Answer: C. The delivery economics were too weak under the original operating model.

Question 9

If version A converts at 10% and version B at 12%, the absolute treatment effect is:

A. 0.2 percentage points

B. 2 percentage points

C. 12 percentage points

D. 20 percentage points

Correct Answer: B. 2 percentage points

Question 10

Which practice best reduces confirmation bias in experimentation?

A. Reporting only positive quotations

B. Changing the hypothesis after seeing every result

C. Recording disconfirming evidence and defining decision criteria in advance

D. Using only founder opinions

Correct Answer: C. Recording disconfirming evidence and defining decision criteria in advance