Hypothesis Testing and Statistical Decision-Making

Introduction

Business analytics rarely concerns description alone. Managers must decide whether a new marketing campaign improves conversion, whether a supplier meets a quality target, whether two customer segments differ, or whether an observed change is likely to be more than random sampling variation. Hypothesis testing provides a disciplined framework for making such decisions from sample data.

A statistical test does not prove that a claim is true or false with certainty. Instead, it evaluates how compatible the observed evidence is with a specified null model. The result must then be interpreted in relation to the business context, the consequences of error, data quality, and the size of the effect.

This chapter develops the logic of hypothesis testing, tests for means and proportions, p-values, confidence intervals, Type I and Type II errors, statistical power, and practical significance. It also emphasizes responsible statistical decision-making: a statistically significant result is not automatically important, useful, or causal.

Learning Objectives

By the end of this chapter, you should be able to:

1. Formulate null and alternative hypotheses for business questions.
2. Distinguish one-sided from two-sided tests.
3. Apply one-sample and two-sample tests for means and proportions.
4. Interpret p-values and significance levels correctly.
5. Relate hypothesis tests to confidence intervals.
6. Explain Type I error, Type II error, statistical power, and practical significance.
7. Identify assumptions, limitations, and ethical risks in statistical decision-making.
8. Communicate test results accurately to technical and non-technical audiences.

1. The Logic of Statistical Hypothesis Testing

A population is the complete set of units of interest, such as all customers who received an offer. A sample is the subset observed by the analyst. A population parameter, such as the population mean μ or population proportion p, is usually unknown. A sample statistic, such as the sample mean x̄ or sample proportion p̂, is calculated from observed data.

Hypothesis testing begins with a claim about a population parameter. The null hypothesis, written H₀, represents a benchmark, status quo, or no-effect position. The alternative hypothesis, written H₁ or Hₐ, represents the effect or difference that the analysis seeks evidence for.

For example, suppose a delivery company claims that its average delivery time is 48 hours. A test might be:

• H₀: μ = 48 hours
• Hₐ: μ ≠ 48 hours

The null hypothesis is not necessarily the analyst's preferred belief. It is the reference model against which the data are evaluated. The analyst calculates a test statistic measuring the distance between the observed sample result and the null value, relative to expected sampling variability.

A general standardized test statistic has the form:

The standard error estimates the typical sampling variation of the statistic. A large absolute test statistic indicates that the sample result is relatively far from what the null hypothesis predicts.

One-sided and two-sided alternatives

The alternative hypothesis determines the direction of the test:

A two-sided test is appropriate when deviations in either direction matter. A one-sided test is appropriate only when the direction is specified in advance and an effect in the opposite direction would not support the decision claim. Choosing a one-sided test after inspecting the data is a form of biased analysis.

2. Significance Levels, Critical Regions, and p-Values

The significance level, denoted α, is the maximum probability of rejecting H₀ when H₀ is true that the analyst is willing to tolerate. Common choices are 0.05 and 0.01. The value should be selected before examining the test result and should reflect the consequences of a false positive.

The p-value is the probability, assuming H₀ is true, of obtaining a result at least as extreme as the observed result in the direction specified by Hₐ. It is not the probability that H₀ is true, and it is not the probability that the result occurred by chance in an informal sense.

The decision rule is:

• If p-value ≤ α, reject H₀.
• If p-value > α, fail to reject H₀.

“Fail to reject” is preferable to “accept H₀.” A non-significant result may reflect no meaningful effect, insufficient sample size, high variability, poor measurement, or an effect smaller than the study could detect.

For a two-sided test using a standard normal reference distribution, the critical values at α = 0.05 are approximately −1.96 and +1.96. The central 95% of the reference distribution lies between these values. For a right-tailed test at α = 0.05, the critical value is approximately 1.645.

3. Tests for One Population Mean

3.1 One-sample z-test

When the population standard deviation σ is known, a one-sample z-test for a population mean uses:

Here, x̄ is the sample mean, μ₀ is the null value, σ is the known population standard deviation, and n is the sample size. The formula applies when observations are independent and the sampling distribution of x̄ is normal or approximately normal. Normality may arise from a normally distributed population or, for sufficiently large samples, from the Central Limit Theorem.

3.2 One-sample t-test

In practice, σ is usually unknown and is estimated by the sample standard deviation s. The one-sample t-statistic is:

The statistic follows a t-distribution with n − 1 degrees of freedom under the null hypothesis, assuming independent observations and an approximately normal population. The t-distribution has heavier tails than the standard normal distribution, especially for small samples, reflecting additional uncertainty from estimating σ.

Worked example: average service time

A call centre historically reports an average service time of 6 minutes. A random sample of 25 calls has mean x̄ = 6.4 minutes and sample standard deviation s = 1.0 minute. Test whether the current mean differs from 6 minutes at α = 0.05.

Hypotheses:

• H₀: μ = 6
• Hₐ: μ ≠ 6

The test statistic is:

Degrees of freedom are 25 − 1 = 24. For a two-sided test, t = 2.00 with 24 degrees of freedom gives a p-value of approximately 0.057. Since 0.057 > 0.05, the analyst fails to reject H₀ at the 5% level. The evidence is suggestive of an increase, but it does not meet the selected significance threshold. This conclusion does not establish that the mean is exactly 6 minutes.

Paired observations

Sometimes each observation has a natural partner: before-and-after measurements on the same customer, machine, or employee. In that situation, the analysis should use differences dᵢ rather than treating the two measurements as independent. The paired t-test is:

Here d̄ is the mean of the paired differences, sd is their sample standard deviation, and μd₀ is usually 0. Pairing can reduce variability because each unit acts as its own control, but the differences should be approximately normally distributed, particularly for small samples.

4. Comparing Two Population Means

4.1 Independent samples

Suppose two independent groups have sample means x̄₁ and x̄₂, standard deviations s₁ and s₂, and sample sizes n₁ and n₂. To test H₀: μ₁ − μ₂ = Δ₀, the general unequal-variance, or Welch, test uses:

Usually Δ₀ = 0. Welch's test is generally preferred because it does not require equal population variances. Its degrees of freedom are estimated using the Welch–Satterthwaite formula:

The formula is applicable to independent observations and approximately normal group means. With large samples, the method is often reasonably robust to moderate non-normality.

Worked example: two delivery methods

An organisation compares delivery times under two methods. Method A has n₁ = 40, x̄₁ = 3.8 days, s₁ = 0.8 days. Method B has n₂ = 35, x̄₂ = 4.2 days, s₂ = 0.9 days. Test H₀: μ₁ − μ₂ = 0 against Hₐ: μ₁ − μ₂ ≠ 0.

The observed difference is −0.4 days. The standard error is:

Thus:

The Welch degrees of freedom are approximately 68. A two-sided p-value is approximately 0.047. At α = 0.05, the null hypothesis is rejected. The estimated mean delivery time for Method A is lower by 0.4 days, although the practical importance should be assessed against cost, reliability, and customer requirements.

4.2 Pooled-variance t-test

A pooled test assumes the two populations have equal variances. Its use is less common when that assumption is doubtful. Analysts should not automatically perform a variance pre-test and then select a method mechanically; Welch's procedure is a strong default for independent means.

5. Tests for One and Two Proportions

A population proportion p describes the fraction of units with a binary characteristic, such as conversion, default, defect, or response. The sample proportion is:

where x is the number of successes and n is the sample size.

5.1 One-proportion z-test

To test H₀: p = p₀, the large-sample z-statistic is:

The standard error uses p₀, not p̂, because the test evaluates variation under the null hypothesis. The normal approximation is appropriate when np₀ and n(1 − p₀) are sufficiently large; many introductory applications use a minimum of about 5 or 10 expected observations in each category. When counts are small, exact binomial methods are safer.

Worked example: conversion rate

A website historically converts 8% of visitors. In a sample of 250 visitors, 28 convert, so p̂ = 28 ÷ 250 = 0.112. Test whether the conversion rate has increased at α = 0.05.

• H₀: p = 0.08
• Hₐ: p > 0.08

The one-sided p-value is approximately 0.031. Since 0.031 < 0.05, the null hypothesis is rejected. The sample provides evidence of an increase. The estimated increase is 3.2 percentage points, but the organisation should also consider implementation cost and whether the sample represents ordinary traffic.

5.2 Comparing two proportions

For two independent samples, the null hypothesis H₀: p₁ − p₂ = 0 uses the pooled proportion:

The test statistic is:

The pooled estimate is appropriate for testing equality because the null hypothesis assumes a common proportion. For confidence intervals estimating p₁ − p₂, an unpooled standard error is normally used. Small counts may require exact or simulation-based methods.

6. Confidence Intervals and Hypothesis Tests

A confidence interval provides a range of parameter values compatible with the data and a specified confidence procedure. For a mean with unknown σ, a two-sided interval is:

where t* is the appropriate critical value with n − 1 degrees of freedom. For a difference between independent means, the interval is:

A 95% confidence interval is constructed by a method that captures the true parameter in 95% of repeated samples under the model. It does not mean there is a 95% probability that the fixed parameter lies in this particular interval, according to the frequentist interpretation.

For a two-sided test at α = 0.05, rejecting H₀: parameter = null value is equivalent to a 95% confidence interval that excludes the null value. Confidence intervals are often more informative than p-values because they show direction, estimated magnitude, and precision.

7. Type I Error, Type II Error, and Statistical Power

A decision can be classified by whether H₀ is true and whether it is rejected:

A Type I error is a false positive. Its probability is controlled by α under the assumptions of the test. A Type II error is a false negative, with probability β for a specified alternative. Statistical power is:

Power is the probability of rejecting H₀ when a particular alternative is true. It is not a fixed property of a test; it depends on effect size, sample size, variability, significance level, and whether the test is one- or two-sided.

Power increases when the true effect is larger, the sample is larger, measurement variability is smaller, or α is increased. These changes involve trade-offs. Increasing α reduces the chance of missing an effect but increases the false-positive risk.

A prospective power analysis can determine the sample size needed to detect a practically important effect with a target power, commonly 80% or 90%. Analysts should define the minimum important effect before collecting data rather than designing a study to detect any statistically significant difference.

8. Statistical Versus Practical Significance

Statistical significance concerns evidence against a null model. Practical significance concerns whether the effect is large enough to matter in operations, finance, policy, or customer experience.

With a very large sample, a trivial improvement may produce p < 0.001. With a small sample, a valuable improvement may fail to reach p < 0.05. A responsible report therefore includes:

• the estimated effect;
• a confidence interval;
• the p-value and test used;
• sample size and relevant assumptions;
• business costs and benefits;
• possible risks and unintended consequences.

For example, reducing average handling time by 0.02 minutes may be statistically significant across millions of calls but irrelevant if service quality declines. Conversely, a 10% reduction in machine downtime may be strategically important even when a small pilot lacks sufficient power.

Effect-size measures can support interpretation. For a mean difference, the raw difference in business units is often most useful. A standardized difference such as Cohen's d is:

where sp is a suitable pooled standard deviation. Standardized measures assist comparison across studies but do not replace domain-specific interpretation.

9. A Professional Testing Workflow

A robust workflow is:

1. Translate the business question into a population parameter.
2. Define the decision-relevant effect and the null value.
3. State H₀ and Hₐ before inspecting the outcome.
4. Select α and identify the consequences of errors.
5. Choose the test according to data type, design, and assumptions.
6. Examine data quality, independence, outliers, missingness, and sampling bias.
7. Calculate the test statistic, p-value, and confidence interval.
8. Assess effect size and practical significance.
9. Conduct sensitivity or robustness checks where appropriate.
10. Communicate the conclusion without overstating certainty or causation.

The test must match the design. For example, a randomised A/B test can support a causal comparison under proper implementation, whereas an observational comparison may be confounded by customer type, season, or channel.

10. Illustrative Case: Evaluating a Customer-Retention Offer

Background

A subscription business introduces a retention offer for customers identified as likely to cancel. Management wants to know whether the offer increases renewal rates enough to justify its cost.

Problem

Among 600 customers receiving the offer, 378 renew. Among 600 comparable customers not receiving the offer, 342 renew. The renewal rates are 0.630 and 0.570, respectively. The estimated difference is 0.060, or 6 percentage points.

Analysis

The hypotheses are:

• H₀: p₁ − p₂ = 0
• Hₐ: p₁ − p₂ > 0

The pooled proportion is:

The standard error is:

Therefore:

The one-sided p-value is approximately 0.017. At α = 0.05, the result is statistically significant. A 95% confidence interval for the difference, using an unpooled standard error, is approximately 0.006 to 0.114. Thus, the data are compatible with an improvement ranging from about 0.6 to 11.4 percentage points.

Approach and decision

Management should compare the expected additional retained revenue with offer cost, contact cost, and possible effects on customers who would have renewed without an offer. If the offer costs $20 per contacted customer and the contribution from a retained customer is $300, the programme may be financially attractive, but the calculation must account for incremental rather than total renewals. Random assignment is important because targeted customers may differ systematically from the comparison group.

Outcome and learning points

The sample supports an association between the offer and higher renewal. If the design was randomised and implementation was sound, a causal interpretation is more credible. The result is not sufficient by itself to justify indefinite rollout. Management should examine subgroup effects, customer fairness, long-term retention, and whether discounts create expectations that reduce future margins.

11. Assumptions, Limitations, and Ethical Issues

Assumptions and data problems

Common assumptions include independent observations, an appropriate sampling design, valid measurement, and a correctly specified reference distribution. Violations can distort p-values and confidence intervals. Repeated observations from the same customer, for example, may make a nominal sample appear larger than the effective sample.

Analysts should inspect missing data, unusual values, inconsistent definitions, and selection mechanisms. Non-response and convenience sampling may create bias that no test can repair. Statistical significance cannot compensate for a non-representative sample.

Multiple testing and selective reporting

Testing many outcomes, segments, or time periods increases the chance of at least one false positive. If m independent tests are each conducted at α = 0.05, the probability of at least one false rejection is:

The independence assumption is simplifying; nevertheless, the principle is important. Pre-specification, replication, and procedures such as Bonferroni or false-discovery-rate control can reduce misleading findings.

Ethical governance

Analysts should document hypotheses, exclusions, transformations, stopping rules, and decision criteria. They should avoid p-hacking, HARKing (presenting an unexpected result as if predicted), and selective publication. Automated analytics systems require monitoring for biased data, unequal error rates, privacy violations, and inappropriate use of sensitive attributes.

A p-value should never be used as the sole basis for decisions affecting employment, credit, healthcare, or access to essential services. Human review, transparency, proportionality, and applicable laws or organisational policies are essential.

12. Technology and Contemporary Practice

Statistical software can calculate tests, confidence intervals, power analyses, bootstrap intervals, and diagnostic plots quickly. However, software does not choose the correct design or interpret practical consequences. Analysts should record the software, method, version, data preparation, and assumptions so results are reproducible.

Modern analytics increasingly uses resampling, randomisation tests, Bayesian methods, and sequential experimentation. These approaches can be valuable when normal approximations are poor or when decision-makers need probability statements about parameters. They still require explicit assumptions and careful governance. Machine-learning models may also be evaluated with hypothesis tests, but dependence among observations and repeated model selection can invalidate simple textbook procedures.

Chapter Summary

Hypothesis testing evaluates sample evidence against a null model. The null and alternative hypotheses must reflect the business question, population parameter, and direction of interest. One-sample and two-sample t-tests are used for means when population variability is estimated; z procedures are used for proportions under suitable large-sample conditions and for means when σ is known. Paired data require analysis of within-unit differences.

The significance level controls the tolerated Type I error rate, while the p-value measures the compatibility of the observed result with H₀. A p-value is not the probability that H₀ is true. Confidence intervals complement tests by showing plausible effect sizes and precision. Type II error represents failure to detect a specified effect, and power equals 1 − β.

Statistical significance must be distinguished from practical significance. Sound decisions consider effect size, uncertainty, cost, risk, sampling design, data quality, multiple testing, and ethical consequences. Statistical software supports calculation, but professional judgement remains essential.

Key Terms

Hypothesis test: A procedure for evaluating evidence about a population claim.

Null hypothesis (H₀): The benchmark claim tested by the analysis.

Alternative hypothesis (Hₐ): The effect or difference for which evidence is sought.

Population parameter: A numerical characteristic of a population.

Sample statistic: A numerical characteristic calculated from a sample.

Significance level (α): The tolerated probability of a Type I error under the testing procedure.

p-value: The probability, assuming H₀, of a result at least as extreme as the observed result.

Test statistic: A standardized measure of departure from the null value.

Standard error: An estimate of the sampling variability of a statistic.

Critical value: A boundary separating rejection and non-rejection regions.

One-sided test: A test with an alternative specifying one direction.

Two-sided test: A test allowing departures in either direction.

Type I error: Rejecting a true null hypothesis.

Type II error: Failing to reject a false null hypothesis.

Statistical power: The probability of detecting a specified alternative; 1 − β.

Confidence interval: An interval produced by a procedure designed to capture a parameter at a stated long-run rate.

Practical significance: The real-world importance of an observed effect.

Welch t-test: A two-sample mean test that does not require equal variances.

Pooled proportion: The combined success proportion used to test equality of two proportions.

Multiple testing: Conducting several statistical tests, increasing false-positive risk.

Review Questions

Short Answer Questions

1. What is the difference between a null hypothesis and an alternative hypothesis?
2. Why is “fail to reject H₀” preferable to “accept H₀”?
3. Define a p-value in statistical terms.
4. What is the difference between a Type I error and a Type II error?
5. Why can statistical significance differ from practical significance?

Long Answer Questions

1. Explain the complete workflow for conducting and communicating a hypothesis test in a business analytics project.
2. Compare one-sample t-tests, independent two-sample t-tests, and paired t-tests, including when each should be used.
3. Explain how confidence intervals complement p-values in managerial decision-making.
4. Discuss the factors that affect statistical power and explain why prospective power analysis is useful.
5. Evaluate the assumptions and ethical risks that should be considered before using a hypothesis test to support an operational decision.

Analytical / Application Questions

1. A retailer claims that its average order value is $75. A random sample of 36 orders has x̄ = $78 and s = $12. Formulate a two-sided test, calculate the t-statistic, and explain what additional information is needed to make the final decision.
2. An email campaign produces 96 purchases from 800 recipients, while a control campaign produces 78 purchases from 800 recipients. Formulate and carry out an appropriate two-proportion test at α = 0.05, then discuss whether the result is practically important.
3. A dashboard reports p = 0.003 for a small improvement found after testing 40 customer segments and five outcome measures. Critically assess the evidence, identify the main statistical risk, and recommend a more reliable decision process.

Multiple Choice Questions with Answers

Question 1

Which statement correctly describes a p-value?

A. The probability that the null hypothesis is true

B. The probability of observing a result at least as extreme as the sample result, assuming the null hypothesis is true

C. The probability that the alternative hypothesis is true

D. The probability that the sample was selected without bias

Correct Answer: B. The probability of observing a result at least as extreme as the sample result, assuming the null hypothesis is true

Question 2

A two-sided test at α = 0.05 rejects the null hypothesis when the p-value is:

A. Greater than 0.95

B. Equal to 0.50

C. Less than or equal to 0.05

D. Less than or equal to 0.50

Correct Answer: C. Less than or equal to 0.05

Question 3

Which test is generally appropriate for comparing two independent means when population variances may differ?

A. Welch two-sample t-test

B. Paired t-test

C. One-proportion z-test

D. Chi-square goodness-of-fit test

Correct Answer: A. Welch two-sample t-test

Question 4

If a study has Type II error probability β = 0.20, its power is:

A. 0.20

B. 0.50

C. 0.80

D. 1.20

Correct Answer: C. 0.80

Question 5

For a one-sample t-test with n = 25 observations, the degrees of freedom are:

A. 24

B. 25

C. 26

D. 12.5

Correct Answer: A. 24

Question 6

In a one-proportion z-test, the standard error under H₀: p = p₀ uses:

A. p̂ only

B. p₀(1 − p₀) ÷ n under the square root

C. s ÷ √n

D. The difference between two sample means

Correct Answer: B. p₀(1 − p₀) ÷ n under the square root

Question 7

A statistically significant result with a very tiny effect may be:

A. Practically unimportant despite strong statistical evidence

B. Proof of causation in every study design

C. Evidence that the null hypothesis is certainly false

D. Automatically suitable for implementation

Correct Answer: A. Practically unimportant despite strong statistical evidence

Question 8

Which situation most clearly calls for a paired analysis?

A. Comparing randomly selected customers from two unrelated regions

B. Comparing two independent supplier samples

C. Comparing employee productivity before and after training for the same employees

D. Comparing two unrelated proportions

Correct Answer: C. Comparing employee productivity before and after training for the same employees

Question 9

The main purpose of selecting a significance level before examining results is to:

A. Guarantee a large effect

B. Control the planned false-positive tolerance and reduce selective decision-making

C. Increase the sample mean

D. Eliminate Type II errors

Correct Answer: B. Control the planned false-positive tolerance and reduce selective decision-making

Question 10

If many independent tests are each conducted at α = 0.05, what generally happens to the chance of at least one false positive?

A. It decreases to zero

B. It remains exactly 0.05

C. It increases as the number of tests increases

D. It becomes equal to statistical power

Correct Answer: C. It increases as the number of tests increases