Hypothesis Testing for Business Decisions

Introduction

Business analysts frequently use samples to make decisions about larger populations. A retailer may want to know whether a new promotion increases average spending; a bank may test whether a revised process reduces loan-processing time; and a manufacturer may investigate whether two suppliers produce components with different defect rates. In each case, the analyst must distinguish genuine evidence from random sample variation.

Hypothesis testing provides a structured method for making such decisions. It begins with a claim about a population, uses sample data to measure the strength of evidence against that claim, and ends with a conclusion stated in business terms. The method does not prove that a claim is true or false with certainty. Instead, it quantifies how compatible the observed data are with a specified assumption.

This chapter develops one-sample and two-sample tests, p-values, significance levels, effect sizes, statistical power, and Type I and Type II errors. It also emphasizes assumptions, interpretation, practical significance, and responsible use in business analytics.

Learning Objectives

By the end of this chapter, you should be able to:

1. Formulate null and alternative hypotheses for business problems.
2. Select and apply appropriate one-sample and two-sample tests.
3. Interpret test statistics, p-values, confidence intervals, and significance levels.
4. Explain Type I and Type II errors and the role of statistical power.
5. Distinguish statistical significance from practical or managerial significance.
6. Calculate and interpret common effect sizes.
7. Communicate conclusions without overstating what the data demonstrate.

1. The Logic of Hypothesis Testing

A statistical hypothesis is a claim about a population parameter, such as a population mean μ, proportion p, or difference between two means. The analyst begins with two competing statements:

• Null hypothesis, H₀: the baseline claim, often representing no change, no difference, or a specified benchmark.
• Alternative hypothesis, H₁ or Hₐ: the claim supported if the evidence is sufficiently inconsistent with H₀.

For example, suppose a call centre historically resolves calls in an average of 8 minutes. Management tests a new system believed to reduce the mean time:

• H₀: μ = 8 minutes
• Hₐ: μ < 8 minutes

The alternative is one-sided because only improvement in the direction of shorter calls is relevant. If management is concerned about any change, the alternative would be two-sided: Hₐ: μ ≠ 8.

The hypotheses concern population parameters, not individual observations and not sample statistics. A sample mean x̄ is evidence used to evaluate a claim about μ.

One-sided and two-sided tests

A right-tailed test has Hₐ: parameter > benchmark. A left-tailed test has Hₐ: parameter < benchmark. A two-tailed test has Hₐ: parameter ≠ benchmark. The direction must be selected before examining the results; changing from two-sided to one-sided after seeing the data inflates the risk of a misleading result.

The testing workflow

1. Define the business question and population.
2. Identify the parameter of interest.
3. State H₀ and Hₐ before analysing results.
4. Select a significance level α, commonly 0.05.
5. Check design and distributional assumptions.
6. Calculate a test statistic and p-value, or use a confidence interval.
7. Reject H₀ if p ≤ α; otherwise, fail to reject H₀.
8. Report the estimated effect, uncertainty, practical meaning, and limitations.

“Fail to reject” is preferable to “accept H₀.” A non-significant result means that the sample does not provide sufficiently strong evidence against H₀; it does not prove that H₀ is true.

2. Significance Levels, p-Values, and Test Statistics

The significance level, α, is the maximum probability of a Type I error that the analyst is willing to tolerate under the testing procedure. If α = 0.05, the procedure is designed to reject a true H₀ approximately 5% of the time in repeated samples, assuming the model and assumptions are correct.

A test statistic measures how far the sample result is from the null value relative to its sampling variability. A general form is:

A p-value is the probability, assuming H₀ is true, of observing a result at least as extreme as the one obtained, in the direction specified by Hₐ. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an informal sense.

For a two-sided test, extreme results occur in both tails. For a one-sided test, extremeness is evaluated in only the specified direction.

Decision rule

• If p ≤ α, reject H₀.
• If p > α, fail to reject H₀.

A p-value of 0.03 does not mean that an effect has a 97% probability of being real. It means that, under H₀, results at least this extreme would occur with probability 0.03 according to the model.

Confidence intervals and tests

For a two-sided test at significance level α, a corresponding 100(1 − α)% confidence interval excludes the null value if and only if the test rejects H₀, subject to the same model and calculation. A confidence interval is often more useful because it reports plausible effect sizes rather than only a decision.

3. One-Sample Tests for a Mean

A one-sample mean test evaluates whether a population mean differs from a benchmark μ₀. When the population standard deviation is unknown, the standard test is the one-sample t-test:

Here, x̄ is the sample mean, μ₀ is the null mean, s is the sample standard deviation, and n is the sample size. The statistic follows a t distribution with n − 1 degrees of freedom when observations are independent and the population is approximately normal, especially for small samples.

If the population standard deviation σ is known, a z-test may be used:

In most business applications σ is unknown, so the t-test is more appropriate. With a large sample, the t distribution becomes close to the standard normal distribution, but the t-test remains valid when its assumptions are satisfied.

Assumptions

• Observations are independent, or the sampling design appropriately accounts for dependence.
• The variable is quantitative.
• For small samples, the population distribution is reasonably close to normal and has no severe outliers.
• The sample is representative of the population to which the conclusion will be applied.

Worked example: average delivery time

A logistics company claims that its mean delivery time is 48 hours. A random sample of 25 deliveries has x̄ = 46.2 hours and s = 5 hours. Test whether the mean delivery time is less than 48 hours at α = 0.05.

H₀: μ = 48; Hₐ: μ < 48.

The degrees of freedom are 24. The one-tailed p-value for t = −1.80 with 24 degrees of freedom is approximately 0.042. Since 0.042 < 0.05, reject H₀. The evidence supports the conclusion that average delivery time is below 48 hours. The estimated improvement is 1.8 hours, but its practical importance should also be assessed against service targets and operating costs.

4. One-Sample Tests for a Proportion

A one-sample proportion test evaluates a population proportion p against p₀. For a sufficiently large sample, the z statistic is:

Here, p̂ = x ÷ n is the sample proportion, x is the number of successes, n is the sample size, and p₀ is the null proportion. The normal approximation is generally appropriate when np₀ and n(1 − p₀) are both sufficiently large; otherwise, an exact binomial test is preferable.

Worked example: customer retention

A subscription business historically retains 80% of customers for at least six months. In a random sample of 200 customers using a new onboarding process, 170 are retained. Test whether retention has increased at α = 0.05.

H₀: p = 0.80; Hₐ: p > 0.80.

The sample proportion is p̂ = 170 ÷ 200 = 0.85. Therefore:

The one-tailed p-value is approximately 0.038. Because 0.038 < 0.05, reject H₀. The sample provides evidence of improved retention. The estimated increase is 5 percentage points, which should be translated into expected retained customers and financial value.

5. Two-Sample Tests

Two-sample tests compare outcomes from two populations or conditions. The groups may be independent, such as customers assigned to two different branches, or paired, such as the same stores measured before and after a change.

Independent two-sample means

Let μ₁ and μ₂ be the population means for groups 1 and 2. The null hypothesis commonly states μ₁ − μ₂ = 0. The recommended general method is Welch’s t-test, which does not require equal population variances:

Here, x̄₁ and x̄₂ are sample means, s₁ and s₂ are sample standard deviations, n₁ and n₂ are sample sizes, and Δ₀ is the null difference, usually 0. Welch’s degrees of freedom are estimated from the sample variances and sizes by software or statistical tables.

Assumptions include independent groups, quantitative outcomes, and approximately normal sampling behaviour or sufficiently large samples. Independence is a design issue: increasing sample size does not repair data gathered from strongly dependent observations.

Paired-sample mean test

When each observation in one condition is naturally matched to an observation in another condition, calculate differences dᵢ = xᵢ,after − xᵢ,before and test the mean difference μ_d:

Here, d̄ is the mean paired difference, s_d is its standard deviation, and μd₀ is usually 0. Pairing can reduce variability caused by stable differences between subjects, stores, or employees.

Two proportions

For independent groups, the null hypothesis p₁ = p₂ uses the pooled estimate:

and the test statistic is:

The pooled formula applies to the null test of equal proportions. For confidence intervals estimating p₁ − p₂, an unpooled standard error is normally used. If counts are small, Fisher’s exact test or another exact method may be appropriate.

6. Effect Size and Practical Significance

Statistical significance depends on effect size, variability, and sample size. A very small effect can be statistically significant in a huge sample, while a meaningful effect can fail to reach significance in a small or noisy sample. Analysts must therefore report both statistical and practical significance.

For a difference between means, the raw effect is:

A common standardized effect size is Cohen’s d:

For independent groups under an equal-variance framework, the pooled standard deviation is:

Cohen’s d is dimensionless. Rough conventions are 0.2 small, 0.5 medium, and 0.8 large, but domain-specific benchmarks are preferable. For proportions, useful measures include the percentage-point difference p̂₁ − p̂₂, relative risk p̂₁ ÷ p̂₂, and odds ratio. These measures answer different business questions and should not be treated as interchangeable.

7. Type I Error, Type II Error, and Power

A Type I error occurs when a true H₀ is rejected. Its probability is controlled by α. In business, this could mean adopting an ineffective marketing campaign or declaring a supplier superior when no real difference exists.

A Type II error occurs when a false H₀ is not rejected. Its probability is β for a specified true alternative. The power of a test is:

Power is the probability of detecting a specified effect when that effect truly exists. Power increases with larger samples, larger effects, lower variability, and higher α. It generally decreases when a two-sided test is used instead of a one-sided test, all else equal.

Power analysis before data collection can determine a required sample size. A responsible design specifies:

• α, often 0.05;
• desired power, often 0.80 or 0.90;
• the smallest practically important effect;
• expected variability or baseline proportion;
• one-sided or two-sided testing;
• operational constraints and attrition.

A non-significant result from a low-powered study is weak evidence. Analysts should examine the confidence interval: if it includes both meaningful benefit and meaningful harm, more data may be required.

8. A Detailed Applied Case: Online Checkout Redesign

This is an illustrative case. A retailer redesigns its checkout page and wants to determine whether the redesign increases completed purchases. During a controlled experiment, 5,000 visitors see the existing page and 5,000 see the redesigned page. The existing page produces 400 purchases; the redesign produces 440.

Background and problem

The business question is whether the redesign improves conversion sufficiently to justify implementation. The outcome is binary, so a two-proportion test is appropriate.

Hypotheses

Let p₁ be the conversion rate for the redesigned page and p₂ the rate for the existing page:

• H₀: p₁ − p₂ = 0
• Hₐ: p₁ − p₂ > 0

The test is one-sided because the decision criterion is improvement, although a two-sided test may be preferable if unexpected harm is also important.

Analysis

The observed rates are:

The observed increase is 0.008, or 0.8 percentage points. The pooled rate is 840 ÷ 10,000 = 0.084. The standard error under H₀ is:

Thus:

The one-sided p-value is approximately 0.075. At α = 0.05, the result is not statistically significant, so the analyst fails to reject H₀.

Interpretation and decision

The data estimate a positive effect, but the evidence is not strong enough under the chosen 5% threshold to conclude that the redesign improves conversion. This does not establish that the redesign has no benefit. The experiment may have insufficient power to detect an increase of only 0.8 percentage points. Management should examine a confidence interval, assess the monetary value of the estimated increase, consider longer testing or a larger sample, and verify random assignment.

Potential confounders include device type, traffic source, seasonality, technical outages, repeated visitors, and unequal exposure time. If the redesign increases purchases but also raises refunds or customer-support contacts, conversion alone is an incomplete success measure.

Learning points

1. The correct test follows the outcome type and experimental design.
2. A positive estimate is not automatically statistically significant.
3. Business decisions require effect size, uncertainty, costs, and risks.
4. A/B tests require valid randomisation, stable implementation, and pre-specified analysis.
5. Multiple outcomes or repeated interim testing can increase false-positive risk.

9. Assumptions, Limitations, and Responsible Practice

Data and design limitations

A mathematically correct test cannot compensate for biased sampling, poor measurement, nonresponse, confounding, or dependence. Analysts should ask whether the sample represents the target population and whether assignment to groups was genuinely random. Observational comparisons often require regression, matching, stratification, or causal-design methods rather than a simple two-sample test.

Multiple testing

Testing many outcomes, customer segments, or time periods increases the probability of at least one false positive. Pre-specification, correction procedures such as Bonferroni or false discovery rate control, and transparent reporting help manage this risk.

Non-normality and outliers

The t-test is often robust for moderate or large samples, but severe skewness, influential outliers, or clustered observations can make simple procedures misleading. Transformations, robust methods, bootstrap intervals, nonparametric tests, or hierarchical models may be better choices. The method must match the data-generating process.

Ethical and governance issues

Business testing may affect prices, access to services, credit decisions, employment opportunities, or customer privacy. Analysts should document hypotheses, sampling, exclusions, stopping rules, software settings, and limitations. Sensitive experiments require legal, ethical, and data-protection review. Statistical significance must never be used to conceal discriminatory impact or justify decisions that lack substantive fairness.

Technology and AI

Statistical software can calculate p-values, confidence intervals, and power rapidly, but automated output does not establish valid hypotheses or causal interpretation. AI-assisted analytics should be checked for data leakage, inappropriate subgroup analysis, hidden multiple testing, and unsupported claims. Reproducible scripts, version-controlled data preparation, and independent review improve reliability.

10. Best-Practice Reporting Template

A professional report should state:

• the business question and target population;
• the outcome, parameter, and hypotheses;
• the design, sample sizes, and inclusion rules;
• the test, assumptions, and significance level;
• the estimate, test statistic, p-value, and confidence interval;
• the raw and standardized effect size where useful;
• the decision and its operational meaning;
• possible Type I and Type II consequences;
• limitations, ethical considerations, and recommended action.

For example: “The redesigned page increased observed conversion by 0.8 percentage points, from 8.0% to 8.8%. The one-sided two-proportion test gave z = 1.44 and p ≈ 0.075; therefore, at α = 0.05 we did not reject equal conversion rates. The estimate is commercially relevant but uncertain, so additional evidence is recommended before full deployment.”

Chapter Summary

Hypothesis testing is a disciplined framework for using sample evidence in business decisions. Analysts formulate a null hypothesis representing a benchmark and an alternative representing the question of interest. One-sample t-tests assess means, proportion tests assess rates, and two-sample procedures compare independent or paired groups. Test statistics standardize the distance between an observed estimate and the null value, while p-values measure compatibility with H₀ under the assumed model.

The significance level controls the planned Type I error rate, whereas Type II error concerns failure to detect a real effect. Power, equal to 1 − β, depends on sample size, variability, effect size, and α. Statistical significance must be interpreted alongside confidence intervals, raw effects, standardized effect sizes, costs, risks, and practical importance. Valid conclusions require suitable design, representative data, appropriate assumptions, transparent reporting, and ethical consideration of consequences.

Key Terms

Null hypothesis: Baseline claim tested using sample evidence.

Alternative hypothesis: Claim supported when evidence is sufficiently inconsistent with H₀.

Parameter: Numerical characteristic of a population, such as μ or p.

Statistic: Numerical quantity calculated from a sample.

Significance level (α): Maximum planned probability of a Type I error.

p-value: Probability, assuming H₀, of a result at least as extreme as observed.

Test statistic: Standardized measure of distance from the null value.

One-sided test: Test with an alternative specifying one direction.

Two-sided test: Test for a difference in either direction.

One-sample t-test: Test of a population mean against a benchmark when σ is unknown.

Welch’s t-test: Two-mean test that does not require equal variances.

Paired test: Test based on differences within matched observations.

Effect size: Magnitude of an observed or population difference.

Cohen’s d: Standardized difference between two means.

Type I error: Rejecting a true null hypothesis.

Type II error: Failing to reject a false null hypothesis.

Statistical power: Probability of detecting a specified true effect.

Practical significance: Business or operational importance of an effect.

Confidence interval: Range of values expressing sampling uncertainty under a stated method.

Review Questions

Short Answer Questions

1. What is the difference between a null hypothesis and an alternative hypothesis?
2. What does a p-value represent, and what does it not represent?
3. Why is “fail to reject H₀” preferred to “accept H₀”?
4. State the meaning of a Type I error and a Type II error.
5. Identify two factors that increase statistical power.

Long Answer Questions

1. Explain the complete hypothesis-testing workflow for a business problem involving a population mean.
2. Compare independent two-sample, paired-sample, and one-sample mean tests, including when each is appropriate.
3. Explain why statistical significance and practical significance may lead to different business conclusions.
4. Discuss the assumptions required for a two-proportion test and describe alternatives when sample counts are small.
5. Explain how multiple testing, biased sampling, and non-independence can undermine business conclusions.

Analytical / Application Questions

1. A company claims that its average service time is 12 minutes. A random sample of 36 services has mean 11.4 minutes and standard deviation 2.4 minutes. Formulate a two-sided test, calculate the t statistic, and explain what additional information is required to make a final decision.
2. An email campaign is sent to 1,000 randomly selected customers, producing 130 purchases. The historical purchase rate is 10%. Formulate a one-sided proportion test and explain whether the observed increase should automatically justify expanding the campaign.
3. A/B testing shows a statistically significant 0.3 percentage-point increase in conversion after a redesign. Explain what information management should review before implementation, including effect size, confidence interval, power, costs, and possible unintended outcomes.

Multiple Choice Questions with Answers

Question 1

Which statement correctly describes a p-value?

A. The probability that the null hypothesis is true

B. The probability of observing a result at least as extreme, assuming the null hypothesis is true

C. The probability that the alternative hypothesis is false

D. The proportion of observations that support the alternative hypothesis

Correct Answer: B. The probability of observing a result at least as extreme, assuming the null hypothesis is true

Question 2

A business tests Hₐ: μ < 50. What type of test is this?

A. Right-tailed

B. Two-tailed

C. Left-tailed

D. Paired-only

Correct Answer: C. Left-tailed

Question 3

A sample has x̄ = 46.2, μ₀ = 48, s = 5, and n = 25. What is the one-sample t statistic?

A. −1.80

B. −0.36

C. 1.80

D. −9.00

Correct Answer: A. −1.80

Question 4

If a test has β = 0.20, what is its statistical power?

A. 0.02

B. 0.20

C. 0.ว่า80

D. 1.20

Correct Answer: C. 0.ว่า80

Question 5

Which situation most directly calls for a paired-sample test?

A. Comparing two unrelated customer groups

B. Comparing two suppliers using different products

C. Comparing the same stores before and after a policy change

D. Comparing a sample mean with a national benchmark

Correct Answer: C. Comparing the same stores before and after a policy change

Question 6

What does α = 0.05 represent in a correctly specified testing procedure?

A. A 5% probability that the alternative hypothesis is true

B. A planned 5% probability of a Type I error

C. A 95% probability that the sample is representative

D. A 5% probability of a Type II error

Correct Answer: B. A planned 5% probability of a Type I error

Question 7

Why should effect size be reported with a p-value?

A. A p-value measures the financial value of an effect

B. Effect size determines whether the null hypothesis is true

C. Statistical significance does not indicate whether an effect is practically important

D. Effect size removes all sampling error

Correct Answer: C. Statistical significance does not indicate whether an effect is practically important

Question 8

Which test is generally suitable for comparing two independent means without assuming equal variances?

A. Welch’s t-test

B. A paired t-test

C. A one-sample proportion z-test

D. A chi-square goodness-of-fit test only

Correct Answer: A. Welch’s t-test

Question 9

If p = 0.08 and α = 0.05, what is the formal decision?

A. Reject H₀

B. Accept H₁ with certainty

C. Fail to reject H₀

D. Set α equal to 0.08 after the test

Correct Answer: C. Fail to reject H₀

Question 10

Which action generally increases statistical power, holding other factors constant?

A. Reducing the sample size

B. Increasing measurement variability

C. Increasing the sample size

D. Changing a two-sided test to a one-sided test after seeing the data

Correct Answer: C. Increasing the sample size