Experiment Design and Evidence-Based Validation

Introduction

Customer discovery becomes strategically valuable only when conversations and observations are converted into disciplined learning. An entrepreneur rarely needs absolute certainty before acting; rather, the entrepreneur needs sufficiently credible evidence to decide whether to persevere, modify the proposition, or abandon it. Experiment design provides the bridge between an uncertain business idea and an evidence-based decision.

An experiment is a deliberate, limited test of a proposition under specified conditions. In a lean-startup context, the proposition may concern a customer problem, a target segment, a desired behaviour, a solution, a price, or a distribution channel. The central discipline is to state what must be true, identify observable evidence, define the standard for success in advance, and specify the action that follows each possible result.

This chapter develops practical methods for designing and interpreting experiments, including smoke tests, usability tests, pricing tests, A/B tests, and sequential experimentation. It also examines validity, causality, ethical conduct, and the dangers of confusing activity with evidence.

Learning Objectives

By the end of this chapter, students should be able to:

1. Formulate testable customer and business hypotheses.
2. Create experiment cards containing a hypothesis, method, metric, threshold, and decision rule.
3. Select appropriate smoke, usability, pricing, A/B, and sequential tests.
4. Distinguish descriptive evidence from causal evidence.
5. Evaluate threats to internal, external, construct, and statistical conclusion validity.
6. Apply experimental results to product, pricing, and venture decisions.
7. Design ethically responsible experiments involving customers and data.

1. From Assumptions to Testable Hypotheses

A startup model contains assumptions: beliefs that have not yet been sufficiently demonstrated. Common assumptions include:

• a defined group experiences a significant problem;
• the problem occurs frequently enough to matter;
• customers will adopt a proposed solution;
• a reachable acquisition channel exists;
• customers will pay a sustainable price; and
• the venture can deliver the promised value economically.

A useful hypothesis is specific, observable, and falsifiable. The statement “students need better financial tools” is too broad. A stronger version is: “Among postgraduate students who manage irregular income, at least 30% will request a demonstration of a budgeting service after viewing a problem-focused landing page.”

A hypothesis normally contains four elements:

1. Population: the people or organisations being studied.
2. Condition or intervention: what participants see, use, or experience.
3. Expected behaviour or outcome: what will be observed.
4. Threshold and time frame: how much evidence is enough and when it will be measured.

It is useful to separate a problem hypothesis from a solution hypothesis. A problem hypothesis asks whether the difficulty exists and matters. A solution hypothesis asks whether a proposed intervention changes behaviour or creates value. Testing a solution before establishing the problem often produces misleading enthusiasm.

Evidence levels

Evidence can be classified by its proximity to actual value creation:

The appropriate evidence depends on the risk being tested. A landing-page click may be adequate for testing message resonance, but not for validating long-term retention or profitability.

2. The Experiment Card

An experiment card is a compact decision document. It prevents teams from changing the success criterion after seeing an attractive result.

Core fields

A strong card also records the date, version of the proposition, recruitment source, incentives, exclusions, and data owner.

Example experiment card

Assumption: Independent tutors will pay for automated lesson-summary software.

Hypothesis: Among qualified tutors who complete a product demonstration, at least 20% will start a paid pilot at £15 per month within seven days.

Method: Recruit tutors through two professional communities. Show a functional prototype, ask participants to process one lesson, and present the paid-pilot offer. Record eligible participants, pilot starts, and reasons for refusal.

Primary metric: Paid-pilot conversion rate.

Threshold: At least 20% of eligible participants start a pilot, with at least 20 participants observed.

Decision rule: If the threshold is met and no serious usability or trust problem appears, continue to a retention test. If conversion is 10–19%, revise the value proposition and repeat. If conversion is below 10%, investigate the problem, segment, or offer before building further.

The threshold is not a universal law. It is a managerial commitment based on expected economics, opportunity cost, and the credibility of the test. A high-priced enterprise product may require fewer but stronger commitments, whereas a consumer product may require a larger behavioural sample.

3. Metrics, Rates, and Uncertainty

A proportion is often used to summarise binary behaviour, such as signup or purchase:

Here, x is the number of positive outcomes, n is the number of eligible observations, and p̂ is the observed proportion. Both x and n must be defined consistently. For example, do not count page visitors in the denominator if the hypothesis concerns qualified visitors who reached the offer page.

A simple conversion funnel may be described as:

The unit is a proportion or percentage. Funnel-stage rates should not automatically be multiplied unless the stages refer to the same cohort and the required conditional probabilities are known. If 40% of visitors view a demo and 25% of demo viewers purchase, the overall purchase rate is 0.40 × 0.25 = 0.10, or 10%, under the assumption that the second rate is conditional on viewing the demo.

Observed results are estimates, not exact properties of a market. Random variation is especially important with small samples. A useful approximate standard error for a sample proportion is:

This approximation is most appropriate when observations are independent and the expected numbers of successes and failures are not very small. For small samples, exact or Wilson confidence intervals are preferable to relying on the simple normal approximation. A confidence interval expresses uncertainty under a specified statistical model; it does not guarantee that the true market value lies within the interval.

For managerial experimentation, the practical question is often not “Is the result statistically significant?” but “Is the result large enough and credible enough to justify the next investment?” Statistical significance can be absent when a practically important effect is measured with too little data, and it can be present for an effect too small to matter commercially.

4. Smoke Tests: Testing Interest Before Building

A smoke test presents a credible offer before the complete product exists. Examples include a landing page, waitlist, call-to-action button, concierge offer, or pre-order page. Its purpose is to measure interest in a proposition, not to prove that the final product works.

A well-designed smoke test should:

1. target a clearly defined segment;
2. describe a concrete problem and proposed benefit;
3. use a real acquisition source rather than only friends;
4. measure a meaningful action, such as booking a call or requesting a paid pilot;
5. disclose the product’s stage honestly; and
6. define the denominator and stopping rule beforehand.

A click is usually weak evidence because it is low-cost and reversible. A qualified booking, deposit, signed letter of intent, or paid pilot is more informative. However, even payment can be misleading if the offer is refundable, heavily discounted, or made under unusual personal influence.

Smoke tests can also test messages. Two pages may present the same underlying offer using different problem descriptions. If traffic is allocated credibly and the pages differ only in the message, the test can compare message response. It cannot, by itself, establish product satisfaction or retention.

Ethical boundary

A smoke test must not falsely claim that a product is available, misrepresent prices, collect payment without fulfilment arrangements, or conceal material limitations. If a button says “Reserve a pilot,” the team must explain what reservation means and how participant data will be used.

5. Usability Tests: Learning from Interaction

Usability testing examines whether intended users can understand and complete important tasks. It is particularly useful for discovering friction, confusing language, missing information, and errors in a prototype or service process.

A usability protocol should include:

• a realistic participant profile;
• task scenarios rather than instructions that reveal the answer;
• a neutral moderator script;
• observation of behaviour, not merely opinions;
• completion, error, time, and assistance measures; and
• a debrief that separates observed difficulty from participant suggestions.

A task-completion rate can be expressed as:

The metric is meaningful only when “completed” is defined in advance. Completion without assistance is a stricter criterion than completion after moderator intervention. Time-on-task is informative but should not be interpreted alone: a slower user may be more careful, and a fast user may make hidden errors.

Moderators should avoid leading questions such as “Was the payment button easy to find?” Better prompts include “What would you do next?” and “Please describe what you are looking for.” Silence is often valuable because excessive assistance contaminates the observation.

Usability testing is commonly diagnostic rather than statistically representative. Five or ten participants may reveal severe interface problems, but they cannot establish population-level conversion rates. Repeated rounds with revised prototypes are usually more useful than one large test of an immature design.

6. Pricing Experiments

Pricing tests investigate willingness to pay, price sensitivity, packaging, and perceived value. Stated willingness to pay is weaker than revealed behaviour. Therefore, pricing evidence should progress from discussion to realistic offers, paid pilots, deposits, or renewal decisions where ethically and operationally feasible.

Possible methods include:

• Van Westendorp-style questioning, which explores perceived cheap, expensive, and unacceptable prices. It is useful for exploratory perception but relies on stated responses.
• Monadic offer tests, where different participants receive different prices or packages. This avoids asking one person to compare every option, although samples must be comparable.
• Conjoint or discrete-choice studies, which estimate preferences among combinations of attributes and prices. These require careful design and are model-dependent.
• Real transaction tests, such as paid pilots or renewal offers. These provide stronger behavioural evidence but may take longer and create customer obligations.

Revenue is not maximised by selecting the highest price in isolation. A simplified revenue relationship is:

If price changes conversion, the comparison must use the resulting number of buyers. For example, an offer priced at £20 with 30 buyers generates £600, whereas an offer priced at £30 with 18 buyers generates £540, before costs and retention are considered. The second price may nevertheless be preferable if it attracts higher-retention customers or lowers service costs.

Pricing experiments should specify whether tax, discounts, refunds, payment fees, and recurring renewal are included. They should also avoid discriminatory or deceptive pricing practices. Participants should understand what they will receive and whether the price is experimental or introductory.

7. A/B Testing and Causal Inference

An A/B test randomly assigns eligible units—such as users, visits, stores, or organisations—to version A or version B. The objective is to estimate the causal effect of changing one treatment while holding other conditions as constant as possible.

Let p_A and p_B denote outcome proportions in groups A and B. The observed absolute difference is:

The relative lift, when p_A is non-zero, is:

For example, if A converts at 8% and B at 10%, the absolute improvement is 2 percentage points, while relative lift is (0.10 − 0.08) ÷ 0.08 = 0.25, or 25%.

Randomisation supports causal interpretation because, in expectation, observed and unobserved characteristics are balanced between groups. It does not repair poor implementation. If users can encounter both versions, if assignment changes during the test, or if one version is shown at a different time or to a different traffic source, confounding may remain.

A credible A/B test requires:

• one clearly specified primary outcome;
• random assignment at the correct unit;
• stable exposure and tracking;
• an adequate observation period covering normal cycles;
• no mid-test decision based on repeated unplanned significance checks;
• treatment fidelity; and
• attention to adverse effects and important subgroups.

The unit of randomisation matters. If treatment is assigned to an organisation but analysis treats every employee as independent, precision will be overstated because employees within one organisation may behave similarly. This is called clustering. In such cases, analysis should account for the organisation-level assignment.

A/B tests can establish a causal effect of the tested change under the tested conditions. They do not automatically explain why the effect occurred, whether it persists, or whether it generalises to a different population or season.

8. Sequential Experimentation

Sequential experimentation treats learning as a series of linked tests rather than one giant launch. A sensible sequence moves from cheap, high-uncertainty questions to costly, outcome-based questions:

Problem evidence → message smoke test → prototype usability → paid pilot → retention → scalable acquisition

Each stage should have a gate. For example, a team may refuse to invest in engineering until a target segment demonstrates repeated problem behaviour; refuse to scale acquisition until a paid pilot produces satisfactory activation; and refuse to expand until retention and unit economics are credible.

Sequential design is efficient because it reduces irreversible commitment. It also creates a cumulative evidence record. However, it can become circular if the team repeatedly changes the hypothesis after every disappointing result without recording what was learned. A revised hypothesis is legitimate when the evidence changes the theory, but it should be explicitly documented.

A practical evidence log records:

• original hypothesis and prior expectation;
• experiment version and date;
• sample and recruitment source;
• observed result;
• deviations from protocol;
• alternative explanations;
• confidence and remaining uncertainty; and
• next decision.

The next experiment should target the most consequential remaining uncertainty, not merely the easiest metric to improve.

9. Validity, Bias, and Causality

Four forms of validity are especially relevant.

Construct validity asks whether the measure represents the concept. A click may not represent genuine purchase intent; self-reported satisfaction may not represent continued use.

Internal validity asks whether the observed difference was caused by the intervention rather than by selection, history, measurement changes, attrition, or other confounders.

External validity asks whether findings generalise to other customers, contexts, channels, or time periods.

Statistical conclusion validity concerns whether the analysis supports the stated inference, including issues of sample size, dependence, variance, and multiple comparisons.

Common biases include:

• selection bias: participants differ systematically from the target market;
• survivorship bias: only successful users are analysed;
• novelty effects: initial excitement fades;
• demand effects: participants infer the desired answer;
• confirmation bias: researchers selectively notice supportive evidence;
• instrumentation bias: tracking changes between conditions; and
• attrition bias: dropout differs across groups.

Causal claims require more than correlation. If users who receive a tutorial retain better, the tutorial may be responsible—but motivated users may also be more likely to complete it. Random assignment, a credible comparison group, pre-specified analysis, and consistent exposure strengthen causal inference. When randomisation is impossible, quasi-experimental methods such as before-and-after comparisons, matched controls, interrupted time series, or difference-in-differences may help, but they require stronger assumptions and careful interpretation.

10. Applied Case: LedgerLite for Freelancers

This is an illustrative case.

Background

LedgerLite is a proposed mobile service that helps freelance designers reserve money for taxes and monitor irregular income. The founding team believed that freelancers experienced anxiety because tax obligations were difficult to estimate.

Problem

Interviews produced enthusiastic statements but little behavioural commitment. The team risked building a full accounting platform without knowing whether tax planning was the urgent problem.

Analysis and approach

The team created three experiment cards.

Experiment 1: problem smoke test. A landing page invited freelance designers to calculate a monthly tax reserve. The threshold was 25% of qualified visitors completing the calculator and 8% booking a planning call. Traffic came from two freelancer newsletters, not personal contacts. Results were 31% calculator completion and 11% call bookings, supporting further investigation but not product validation.

Experiment 2: usability test. Twelve freelancers used a clickable prototype to import income, estimate a reserve, and schedule a transfer. The team defined successful completion as completing all three tasks without moderator assistance. Seven participants completed all tasks; five misunderstood whether the displayed reserve was a recommendation or a legal tax calculation. The team changed the language, added a prominent disclaimer, and simplified the transfer flow.

Experiment 3: paid pilot. A revised prototype offered a four-week pilot for £12. Of 30 eligible participants, nine paid and activated the service. The observed activation rate was:

The pre-specified threshold was 25%, so the team proceeded to measure repeated weekly use. However, it did not claim product-market fit: participants came from newsletters, the pilot was short, and the offer included personal onboarding.

A subsequent sequential test compared two onboarding messages. Random assignment produced 52% week-two usage under Message A and 61% under Message B. The absolute difference was 9 percentage points. The team checked that assignment and tracking were stable, then repeated the test with a new cohort before making the message permanent.

Outcome

The evidence supported a narrower proposition: freelancers valued tax-reserve guidance when it was simple, transparent, and accompanied by human reassurance. It did not support building a broad accounting suite. The team chose to focus on reserve estimation and recurring reminders, while postponing invoicing and expense management.

Learning points

1. Interest in a problem did not establish willingness to pay.
2. Usability testing uncovered a trust and interpretation problem that conversion data alone would not explain.
3. A paid pilot generated stronger evidence than a survey, but its short duration limited conclusions about retention.
4. Sequential experiments reduced development risk and converted uncertainty into explicit decisions.
5. Ethical disclaimers were part of product validity, not merely legal decoration.

11. Governance, Ethics, and Responsible Experimentation

Experiments involving people require respect for autonomy, privacy, fairness, and safety. Teams should collect only necessary data, explain material risks, protect personally identifiable information, and establish retention and deletion practices. Consent should be meaningful, especially where financial, health, employment, or educational data are involved.

Deception requires exceptional caution. A misleading interface that induces a purchase or makes cancellation difficult may increase a short-term metric while damaging trust and creating regulatory exposure. Dark patterns, undisclosed price discrimination, fabricated scarcity, and unauthorised experimentation on vulnerable populations are unacceptable.

Relevant legal requirements vary by jurisdiction but may include privacy law, consumer-protection rules, electronic marketing regulations, accessibility obligations, and sector-specific requirements. An experiment should be reviewed for compliance before launch, not after a complaint.

AI can assist with experiment ideation, segmentation, transcript coding, and anomaly detection. It should not replace human judgement about consent, fairness, causal interpretation, or high-impact decisions. AI-generated summaries require validation because automated coding can reproduce sampling and language biases. Teams should document model use, data provenance, access controls, and escalation procedures.

12. Best Practices for Professional Experimentation

• State the decision before collecting data.
• Test the riskiest assumption, not the most convenient one.
• Use behaviour and commitment where feasible.
• Define the denominator, threshold, time frame, and stopping rule in advance.
• Keep the intervention minimal so learning is interpretable.
• Separate diagnostic research from confirmatory measurement.
• Randomise when causal attribution matters.
• Avoid changing several important variables simultaneously.
• Examine negative outcomes and unintended effects.
• Record null results and protocol deviations.
• Distinguish statistical evidence from commercial significance.
• Repeat important findings across cohorts or contexts.
• Stop when evidence is sufficient for the decision, not merely when data are interesting.

Chapter Summary

Experiment design converts entrepreneurial assumptions into observable tests and explicit decisions. An experiment card links a falsifiable hypothesis to a target population, method, metric, threshold, and decision rule. Smoke tests are useful for testing interest and messaging before building; usability tests reveal interaction and comprehension problems; pricing tests investigate willingness to pay and value perception; and A/B tests provide stronger causal evidence when randomisation, stable measurement, and appropriate analysis are present. Sequential experimentation reduces risk by moving from inexpensive problem and message tests toward paid pilots, retention, and scalable growth.

Evidence must be interpreted in relation to validity. Clicks, opinions, and short-term activation are not equivalent to sustained customer value. Selection, novelty, demand, attrition, and measurement biases can distort results. Ethical experimentation requires transparency, privacy protection, fairness, accessibility, and responsible handling of AI and customer data. The professional entrepreneur does not ask only whether a metric increased; they ask what was tested, what alternative explanations remain, and what decision the evidence justifies.

Key Terms

Hypothesis: A falsifiable statement predicting an observable outcome.

Assumption: A belief in the venture model that has not yet been adequately demonstrated.

Experiment card: A structured record of a hypothesis, test method, metric, threshold, and decision rule.

Smoke test: A test of market interest using an offer or representation before the complete product exists.

Usability test: Observation of users attempting realistic tasks with a product or prototype.

Pricing experiment: A test of price, packaging, willingness to pay, or price response.

A/B test: A randomised comparison of two versions of an intervention.

Sequential experimentation: A series of linked tests in which each result informs the next decision.

Conversion rate: The proportion of eligible entrants completing a specified action.

Absolute difference: The subtraction of one outcome rate from another.

Relative lift: The difference between two rates divided by the baseline rate.

Randomisation: Allocation by chance to reduce systematic differences between groups.

Causality: A relationship in which an intervention produces a change in an outcome.

Construct validity: The extent to which a measure represents the intended concept.

Internal validity: The credibility of a causal conclusion within the tested setting.

External validity: The extent to which findings generalise beyond the test setting.

Selection bias: Distortion caused by systematic differences in who enters or remains in a test.

Statistical conclusion validity: The adequacy of evidence and analysis for the stated statistical inference.

Decision rule: A pre-specified action associated with possible experiment outcomes.

Dark pattern: A deceptive interface design intended to manipulate user behaviour.

Review Questions

Short Answer Questions

1. What are the four essential elements of a testable hypothesis?
2. Why is a paid pilot generally stronger evidence than a landing-page click?
3. Define internal validity and give one threat to it.
4. What is the difference between absolute difference and relative lift?
5. Why should experiment thresholds be defined before results are observed?

Long Answer Questions

1. Explain how an experiment card supports evidence-based entrepreneurial decision-making.
2. Compare smoke tests, usability tests, pricing tests, and A/B tests in terms of purpose and evidential strength.
3. Discuss the conditions required for an A/B test to support a causal claim.
4. Explain how sequential experimentation can reduce venture risk while also creating decision discipline.
5. Evaluate the ethical risks of experimentation involving customer data, pricing, and AI-supported analysis.

Analytical / Application Questions

1. A landing page receives 500 qualified visitors. Forty start a paid pilot. Calculate the conversion rate and propose two possible interpretations, including one limitation.
2. A team observes higher retention among users who completed an onboarding tutorial. Design a stronger experiment to determine whether the tutorial caused the difference.
3. Design an experiment card for a proposed digital service. Include the assumption, hypothesis, target participants, method, primary metric, threshold, decision rules, and major validity risks.

Multiple Choice Questions with Answers

Question 1

Which statement is the strongest testable hypothesis?

A. Customers want better financial services.

B. Users will probably like the application.

C. At least 25% of qualified visitors will book a product demonstration within seven days of seeing the offer.

D. The market for the product is very large.

Correct Answer: C. At least 25% of qualified visitors will book a product demonstration within seven days of seeing the offer.

Question 2

What does the denominator in a conversion rate represent?

A. All people who have ever heard of the venture

B. Eligible entrants exposed to the specified opportunity

C. Only people who completed the desired action

D. The venture’s total addressable market

Correct Answer: B. Eligible entrants exposed to the specified opportunity

Question 3

A smoke test primarily measures:

A. The final product’s long-term retention

B. Interest in a proposition before the complete product exists

C. The legal compliance of a product

D. The causal effect of a randomised intervention

Correct Answer: B. Interest in a proposition before the complete product exists

Question 4

Which measure generally provides stronger evidence of willingness to pay?

A. A participant saying the idea is attractive

B. A social-media like

C. A paid pilot under a clearly stated offer

D. A researcher’s opinion

Correct Answer: C. A paid pilot under a clearly stated offer

Question 5

If version A converts at 8% and version B at 10%, what is the absolute difference?

A. 2 percentage points

B. 8 percentage points

C. 20 percentage points

D. 25 percentage points

Correct Answer: A. 2 percentage points

Question 6

Why is randomisation used in an A/B test?

A. To guarantee a large sample

B. To eliminate all measurement error

C. To reduce systematic differences between treatment groups

D. To ensure every participant sees both versions

Correct Answer: C. To reduce systematic differences between treatment groups

Question 7

Which is primarily a threat to construct validity?

A. A click being used as a measure of genuine purchase intent

B. Random assignment to two versions

C. Testing across two calendar weeks

D. Recording the sample size before launch

Correct Answer: A. A click being used as a measure of genuine purchase intent

Question 8

What is a major purpose of sequential experimentation?

A. To avoid documenting failed tests

B. To move from lower-cost uncertainty reduction to higher-cost validation

C. To change the success criterion after every result

D. To replace customer behaviour with expert opinion

Correct Answer: B. To move from lower-cost uncertainty reduction to higher-cost validation

Question 9

A usability test should preferably ask participants to:

A. Follow instructions that reveal the intended interface path

B. Complete realistic tasks while the researcher observes behaviour

C. Rate the product before using it

D. Suggest every possible feature before attempting a task

Correct Answer: B. Complete realistic tasks while the researcher observes behaviour

Question 10

Which practice is ethically problematic?

A. Disclosing that a prototype is still under development

B. Collecting only data necessary for the stated test

C. Using a deceptive cancellation flow to improve retention metrics

D. Recording protocol deviations

Correct Answer: C. Using a deceptive cancellation flow to improve retention metrics