Hypothesis Testing and Evidence-Based Decisions

Introduction

Finance is conducted under uncertainty. Analysts compare investment returns, assess whether a trading strategy adds value, evaluate credit-risk models, test market relationships, and determine whether observed performance reflects a genuine economic effect or random variation. Hypothesis testing provides a disciplined framework for making such inferences from samples.

A statistical test does not prove that a proposition is true. Rather, it measures how compatible observed data are with a specified null hypothesis, given a model and its assumptions. Sound financial decisions therefore require more than reporting a p-value. The analyst must also consider effect size, statistical power, economic significance, model risk, data quality, and the costs of incorrect decisions.

Learning Objectives

By the end of this chapter, the reader should be able to:

• formulate null and alternative hypotheses for financial questions;
• conduct and interpret parametric and non-parametric tests for means, variances, and relationships;
• explain p-values, confidence intervals, effect sizes, statistical power, and Type I and Type II errors;
• distinguish statistical significance from economic significance;
• select tests consistently with measurement scales, sampling design, and distributional assumptions;
• evaluate evidence critically in investment, risk-management, and corporate-finance applications.

1. The Logic of Statistical Hypothesis Testing

A population parameter is an unknown characteristic such as a mean return μ, variance σ², correlation ρ, or regression coefficient β. A sample provides an estimate of that parameter. Hypothesis testing asks whether the difference between an observed estimate and a benchmark is sufficiently large relative to sampling variability.

The null hypothesis, H₀, represents a benchmark or absence of the claimed effect. The alternative hypothesis, H₁ or Hₐ, represents the research claim. For example, an analyst testing whether a fund outperforms a benchmark may specify:

• H₀: μₚ − μᵦ = 0
• Hₐ: μₚ − μᵦ > 0

A test statistic converts the sample result into a standardised measure. In general:

The sampling distribution of the test statistic is derived under H₀. The analyst then calculates a p-value or compares the statistic with a critical value.

One-tailed and two-tailed tests

A one-tailed test is appropriate when deviations in only one direction are relevant and the direction was specified before examining the data. A two-tailed test considers departures in either direction. For example, testing whether a portfolio has a non-zero abnormal return is two-tailed; testing whether it exceeds zero is one-tailed.

Changing a two-tailed test into a one-tailed test after seeing the results inflates the apparent evidence and is a form of selective analysis. The direction and significance level should be determined before testing.

Significance level and decision errors

The significance level α is the maximum probability of rejecting H₀ when H₀ is true. Common values are 0.10, 0.05, and 0.01. A rejection rule based on α = 0.05 tolerates a 5% long-run Type I error rate under the stated testing procedure.

A Type I error is a false positive: rejecting a true H₀. A Type II error is a false negative: failing to reject a false H₀. Statistical power is the probability of detecting a specified effect when that effect exists:

Here β is the probability of a Type II error. Power depends on sample size, effect size, variability, significance level, and whether the test is one- or two-tailed. A non-significant result may indicate no meaningful effect, insufficient power, noisy data, or a poorly specified test.

2. p-Values, Confidence Intervals, and Effect Sizes

A p-value is the probability, assuming H₀ is true and the test assumptions hold, of obtaining a result at least as extreme as the observed result in the relevant direction. It is not the probability that H₀ is true, and it is not the probability that the result occurred by chance in an unrestricted sense.

If p ≤ α, the conventional decision is to reject H₀. If p > α, the correct wording is “fail to reject H₀,” not “accept H₀.” Evidence may be insufficient without proving the null proposition.

A confidence interval provides a range of parameter values compatible with the data and procedure. A 95% confidence interval does not mean that there is a 95% probability that the fixed parameter lies in this particular interval. In repeated sampling, 95% of intervals constructed by the same valid method would contain the parameter.

For a sample mean with known population standard deviation σ, a 100(1 − α)% confidence interval is:

where x̄ is the sample mean, n is sample size, z_{α/2} is the standard-normal critical value, and σ is the population standard deviation. When σ is unknown, it is replaced by the sample standard deviation s and the t critical value:

The t interval assumes independent observations from a population that is approximately normal, particularly when n is small. For large samples, the central limit theorem often makes the mean approximately normal, but dependence, heavy tails, and structural breaks remain important concerns in finance.

Effect size

Statistical significance is affected by sample size. Effect size describes the magnitude of an effect independently, or partly independently, of sample size. For a one-sample mean, Cohen’s d is:

where μ₀ is the benchmark mean and s is the sample standard deviation. For two independent groups, a standardised mean difference can use a pooled standard deviation when equal-variance assumptions are defensible. For a correlation, the effect size is the correlation coefficient itself; for regression, it may be β, an economically interpretable change in the dependent variable, or a partial R².

Effect-size interpretation is context-dependent. A statistically significant monthly alpha of 0.02% may be economically irrelevant after transaction costs, while a statistically significant increase in default probability of 0.5 percentage points may be material for a large loan portfolio.

3. Parametric Tests for Means

Parametric tests specify a distributional model using parameters such as μ and σ². They are often more powerful than non-parametric alternatives when assumptions are reasonable.

One-sample t-test

To test H₀: μ = μ₀ when σ is unknown, use:

Under H₀, the statistic follows a t distribution with n − 1 degrees of freedom if observations are independent and normally distributed. The test is commonly applied to mean abnormal returns, average forecast errors, or mean changes in operating ratios.

Worked example: mean abnormal return

An analyst observes 25 monthly abnormal returns with sample mean 0.80% and sample standard deviation 2.00%. Test H₀: μ = 0 against Hₐ: μ ≠ 0 at the 5% level.

The standard error is:

Therefore:

The degrees of freedom are 24. The two-tailed 5% critical value is approximately 2.064. Since |2.00| < 2.064, the analyst fails to reject H₀ at the 5% level. The estimate is positive, but the evidence is slightly insufficient for conventional two-tailed significance. A 95% confidence interval is approximately 0.80% ± 2.064 × 0.40%, or −0.026% to 1.626% per month. The interval includes zero.

Paired t-test

A paired test applies when each observation in one condition is naturally matched with an observation in another condition—for example, portfolio returns before and after a policy change measured in the same months. Let dᵢ be the difference for pair i. The test is performed on the mean difference:

where d̄ is the average difference and s_d is the standard deviation of differences. Pairing can reduce noise, but it does not automatically establish causality.

Two-sample tests

For independent groups, the classical pooled-variance t-test assumes equal population variances. In practice, Welch’s t-test is usually preferable because it does not require equal variances. For means x̄₁ and x̄₂:

where s₁² and s₂² are sample variances and n₁ and n₂ are sample sizes. Welch’s degrees of freedom are estimated using the Welch–Satterthwaite formula:

The observations must be independent within and between groups. If returns overlap over time, standard errors may be understated and a simple t-test can be misleading.

4. Tests for Variances

Volatility and risk dispersion are central to finance. Testing a variance requires particular care because financial returns often exhibit skewness, fat tails, volatility clustering, and time dependence.

Chi-square test for one variance

If observations are independent and normally distributed, testing H₀: σ² = σ₀² uses:

Under H₀, χ² follows a chi-square distribution with n − 1 degrees of freedom. The test is sensitive to departures from normality, so its textbook validity is often weak for daily financial returns.

F-test for two variances

For two independent normal samples, testing equality of variances uses:

Under H₀: σ₁² = σ₂², F follows an F distribution with n₁ − 1 and n₂ − 1 degrees of freedom. The result is highly sensitive to non-normality. Levene’s test or the Brown–Forsythe test is generally more robust for comparing group dispersions.

Practical interpretation

A statistically significant variance difference does not necessarily imply a better or worse investment. Higher volatility may accompany higher expected return, while lower measured volatility may result from illiquidity, stale pricing, or smoothing. Risk should be assessed with appropriate measures such as downside deviation, expected shortfall, maximum drawdown, and stress losses, not variance alone.

5. Non-Parametric Tests

Non-parametric procedures make fewer assumptions about the population distribution. They are useful for ordinal data, outliers, small samples, and strongly non-normal observations. Their apparent robustness does not remove the need for independent observations and an appropriate sampling design.

Wilcoxon signed-rank test

For paired observations, calculate non-zero differences, rank their absolute values, and compare the sums of positive and negative ranks. It tests a distributional location condition and is often described as a test of median differences only under suitable symmetry assumptions. The sign test is less powerful but requires fewer assumptions because it uses only the direction of differences.

Mann–Whitney U test

For two independent samples, rank all observations together and compare rank sums. It is not universally a test of means or medians. Under similar distributional shapes, it can be interpreted as evidence that one population tends to produce larger observations than the other. Ties and unequal distributional shapes require cautious interpretation.

Spearman rank correlation

Spearman’s coefficient measures monotonic association using ranks. With no ties, it can be calculated as:

where dᵢ is the difference between the two ranks for observation i and n is the number of paired observations. It detects monotonic, not necessarily linear, association. It does not establish causation.

6. Tests for Relationships

Pearson correlation

The sample Pearson correlation is:

It ranges from −1 to +1 and measures linear association. Testing H₀: ρ = 0 under bivariate normality uses:

with n − 2 degrees of freedom. Outliers can substantially alter r, and correlation does not imply causation.

Regression coefficients

In a simple regression model:

where yᵢ is the dependent variable, xᵢ is the explanatory variable, α is the intercept, β is the slope, and εᵢ is the error term. A test of H₀: β = 0 uses:

where β̂ is the estimated slope, β₀ is the hypothesised value—often zero—and SE(β̂) is its standard error. Classical inference assumes correctly specified conditional mean, independent errors, and suitable treatment of heteroskedasticity and serial correlation. In financial time series, heteroskedasticity- and autocorrelation-consistent standard errors, clustered errors, or time-series models may be necessary.

A statistically significant coefficient may reflect omitted variables, reverse causality, common trends, data mining, or a risk premium rather than a causal effect. Economic interpretation requires units: if x is a 1-percentage-point change in a factor and β̂ = 0.30, the predicted change in y is 0.30 percentage points, conditional on the model.

7. Statistical Significance versus Economic Significance

Statistical significance asks whether an effect is difficult to reconcile with H₀ under the model. Economic significance asks whether the effect is large enough to matter after costs, risks, constraints, and alternatives.

A useful decision framework is:

1. Magnitude: How large is the estimated effect and its plausible range?
2. Precision: How wide is the confidence interval?
3. Costs: Are brokerage, market impact, taxes, financing, compliance, and implementation costs included?
4. Risk: Does the strategy increase drawdown, tail exposure, liquidity risk, or model risk?
5. Scalability: Does the effect survive realistic capital and capacity constraints?
6. Stability: Does it persist across time, markets, regimes, and reasonable specifications?
7. Opportunity cost: Is the effect superior to an available benchmark?

For an investment strategy, net economic value can be represented conceptually as:

This is a decision framework rather than a universal accounting identity; each term must be defined consistently. A p-value cannot substitute for these calculations.

8. Multiple Testing, Power, and Research Integrity

Testing many hypotheses increases the probability of at least one false positive. If m independent tests are each conducted at level α, the probability of at least one false rejection is:

For α = 0.05 and m = 20, this probability is approximately 64.2% under independence. Financial tests are often dependent, so this expression is illustrative rather than exact.

Controls include family-wise error procedures such as Bonferroni adjustment, false-discovery-rate procedures such as Benjamini–Hochberg, pre-registration, out-of-sample validation, and economic filters. Bonferroni uses an individual threshold α/m, which controls family-wise error conservatively.

Power analysis should be conducted before data collection when possible. It requires a minimum economically meaningful effect, an estimate of variability, a significance level, and a desired power. A very large sample can make trivial effects significant; a small sample can fail to detect important effects. Reporting only significant results creates publication and selection bias.

9. A Professional Testing Workflow

A defensible analysis follows these steps:

1. Define the decision and the population of interest.
2. Specify the estimand, such as mean excess return, variance difference, or factor loading.
3. State H₀ and Hₐ before inspecting outcome results.
4. Select α and determine whether the test is one- or two-tailed.
5. Inspect data quality, missing values, outliers, dependence, and measurement definitions.
6. Select a test consistent with the design and assumptions.
7. Calculate the estimate, standard error, test statistic, p-value, confidence interval, and effect size.
8. Conduct robustness checks, including alternative specifications and appropriate non-parametric or resampling methods.
9. Translate the result into economic value, risk, and implementation consequences.
10. Document all tests, exclusions, transformations, and decisions.

Assumptions should be diagnosed rather than asserted. Histograms, Q–Q plots, residual plots, autocorrelation checks, and sensitivity analyses are useful, but formal tests of assumptions should not be followed mechanically. A normality test can reject normality in a huge sample even when the t-test is practically robust, while failing to reject normality in a small sample does not prove normality.

10. Illustrative Case: Evaluating a Factor-Screening Strategy

Background

A fictional asset-management firm, Meridian Capital, screens equities for a quality factor and claims that the strategy generates positive monthly abnormal returns relative to its benchmark.

Problem

The investment committee must decide whether to allocate capital. The research team has 60 monthly observations. The mean abnormal return is 0.45% per month, the standard deviation is 1.80%, and the estimated average monthly transaction and implementation cost is 0.25%.

Analysis

The standard error of the mean is:

Testing H₀: μ = 0 gives:

With 59 degrees of freedom, this is generally not significant at the 5% two-tailed level. The gross estimated return is positive, but the evidence is not sufficiently precise for a conventional two-tailed claim of non-zero alpha.

More importantly, estimated net return is:

The net effect is smaller and has not been tested with independently estimated costs. The committee also discovers that the back-test rebalanced on closing prices, whereas actual execution would occur later; this creates implementation risk. Returns exhibit volatility clustering, so ordinary standard errors may be unreliable. The team therefore repeats the analysis using realistic execution prices, heteroskedasticity- and autocorrelation-consistent errors, a non-parametric signed-rank check, and a holdout period.

Approach and outcome

In the holdout period, the net return is smaller and statistically indistinguishable from zero. The committee does not conclude that the factor never works. Instead, it concludes that available evidence does not justify a full allocation. A small pilot allocation with predefined monitoring criteria may be considered, provided capacity, turnover, drawdown, and model drift are controlled.

Learning points

• A positive sample mean is not automatically statistically reliable.
• Costs can change an economically attractive gross result into a marginal net result.
• Back-test design and execution assumptions affect validity.
• Out-of-sample evidence is essential for reducing data-mining risk.
• Statistical and investment decisions should be separated but connected: inference informs judgment, rather than replacing it.

11. Limitations, Governance, and Contemporary Practice

Hypothesis tests inherit the weaknesses of their data and design. Non-random samples limit generalisation. Survivorship bias excludes failed securities. Look-ahead bias uses information unavailable at the decision date. Structural breaks make historical parameters unstable. Data revisions, vendor errors, and inconsistent corporate-action treatment can create artificial evidence.

Governance requires reproducible code, versioned data, an audit trail, clear ownership of assumptions, and independent review for material decisions. Analysts should disclose all primary tests, not only favourable specifications. Regulators and internal model-risk functions commonly expect validation, documentation, sensitivity analysis, and controls against misuse.

Software can calculate exact or simulated p-values, bootstrap confidence intervals, robust standard errors, and multiple-testing adjustments. AI-assisted analytics can help identify patterns or generate code, but it can also encourage undisclosed specification searching and reproduce hidden data errors. Human review must verify data provenance, formulas, assumptions, and economic interpretation. Automated significance labels should never be treated as investment recommendations.

Bootstrap methods are increasingly useful when the sampling distribution is difficult to derive. A basic bootstrap repeatedly resamples observations with replacement and uses the resulting statistic distribution to estimate uncertainty. However, ordinary resampling is inappropriate for dependent time series unless dependence is addressed through methods such as block bootstrap. Resampling does not repair selection bias or omitted-variable problems.

Chapter Summary

Hypothesis testing provides a formal method for evaluating financial claims under uncertainty. It begins with a clearly specified null and alternative hypothesis, a test statistic, an appropriate reference distribution, and a decision rule. The p-value measures compatibility with H₀; it does not measure the probability that H₀ is true. Confidence intervals express estimation uncertainty, while effect sizes describe magnitude.

Parametric tests for means and variances can be powerful when assumptions are credible. Welch, paired, and one-sample t-tests address different designs. Variance tests are especially sensitive to non-normality. Non-parametric procedures such as Wilcoxon, Mann–Whitney, Kruskal–Wallis, Friedman, Spearman, and sign tests can provide useful robustness, but they also have specific interpretations and assumptions.

Type I and Type II errors represent different decision risks, and power is the probability of detecting a specified effect. Multiple testing, data mining, dependence, and model misspecification can produce misleading evidence. Finally, statistical significance must be distinguished from economic significance by considering costs, risk, scalability, stability, and opportunity cost. Professional evidence-based decisions integrate statistical results with economic reasoning and transparent governance.

Key Terms

Hypothesis testing: A procedure for assessing evidence about a population claim.

Null hypothesis: The benchmark proposition tested against the data.

Alternative hypothesis: The proposition representing a departure from the null.

p-value: The probability, under H₀ and the test assumptions, of a result at least as extreme as observed.

Significance level: The pre-specified maximum Type I error probability.

Type I error: Rejecting a true null hypothesis.

Type II error: Failing to reject a false null hypothesis.

Statistical power: The probability of detecting a specified effect when it exists.

Confidence interval: An interval produced by a procedure designed to contain the parameter at a stated long-run rate.

Effect size: A measure of the magnitude of an observed or estimated effect.

Parametric test: A test based on a specified distributional model and parameters.

Non-parametric test: A test requiring fewer distributional assumptions.

Standard error: An estimate of the sampling variability of an estimator.

Welch’s t-test: A two-sample mean test that does not assume equal variances.

Correlation: A standardised measure of association between two variables.

Economic significance: The practical or financial importance of an effect.

Multiple testing: Conducting several statistical tests within a research exercise.

Robustness check: An analysis examining whether conclusions survive reasonable alternative methods.

Data-mining bias: Spurious evidence created by searching many specifications or patterns.

Statistical significance: Evidence against a null hypothesis according to a chosen statistical rule.

Review Questions

Short Answer Questions

1. What does a p-value measure, and what does it not measure?
2. Distinguish Type I error from Type II error.
3. Why is “fail to reject H₀” preferable to “accept H₀”?
4. State two assumptions relevant to a one-sample t-test.
5. Why can a statistically significant result lack economic significance?

Long Answer Questions

1. Explain the logic of hypothesis testing and discuss the roles of H₀, Hₐ, α, the test statistic, p-value, and confidence interval.
2. Compare the one-sample, paired-sample, and Welch two-sample t-tests, including their applications and assumptions.
3. Explain why variance tests are sensitive to non-normality and discuss robust alternatives.
4. Compare Pearson correlation and Spearman rank correlation in a financial research context.
5. Discuss how multiple testing and data mining can distort conclusions in empirical asset management.

Analytical / Application Questions

1. A portfolio has 36 monthly abnormal returns with mean 0.60% and standard deviation 1.50%. Test the null of zero mean using a two-tailed 5% t-test, calculate the test statistic, and explain what additional information is needed for a complete decision.
2. Two funds have equal average returns, but Fund A has substantially higher volatility and liquidity risk. Explain why a test of equal means alone is insufficient for an investment decision.
3. Design a research protocol for evaluating a trading strategy that addresses costs, multiple testing, out-of-sample validation, statistical power, dependence, and economic significance.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a p-value?

A. The probability that the null hypothesis is true

B. The probability of observing a result at least as extreme as the sample result, assuming the null hypothesis is true

C. The probability that the alternative hypothesis is false

D. The proportion of observations supporting the alternative hypothesis

Correct Answer: B. The probability of observing a result at least as extreme as the sample result, assuming the null hypothesis is true

Question 2

A Type I error occurs when an analyst:

A. Fails to reject a false null hypothesis

B. Rejects a true null hypothesis

C. Uses a sample that is too large

D. Reports an economically insignificant effect

Correct Answer: B. Rejects a true null hypothesis

Question 3

Which test is generally appropriate for comparing means of two independent groups without assuming equal variances?

A. Pooled-variance t-test only

B. Welch’s t-test

C. Paired t-test

D. Sign test only

Correct Answer: B. Welch’s t-test

Question 4

If a sample has mean 0.8%, standard deviation 2.0%, and n = 25, what is the one-sample t-statistic for testing a zero mean?

A. 0.40

B. 1.25

C. 2.00

D. 2.50

Correct Answer: C. 2.00

Question 5

Statistical power is equal to:

A. α

B. β

C. 1 − α

D. 1 − β

Correct Answer: D. 1 − β

Question 6

Which method is a non-parametric alternative commonly used for paired observations?

A. Wilcoxon signed-rank test

B. F-test

C. One-sample z-test

D. Pearson variance test

Correct Answer: A. Wilcoxon signed-rank test

Question 7

A statistically significant investment return may lack economic significance when:

A. The p-value is below 0.05

B. The confidence interval excludes zero

C. Transaction and implementation costs absorb most of the gross return

D. The sample size is reported

Correct Answer: C. Transaction and implementation costs absorb most of the gross return

Question 8

What is a major risk of conducting many unadjusted hypothesis tests?

A. The sample variance must become zero

B. The probability of at least one false positive increases

C. Statistical power always becomes zero

D. All confidence intervals become narrower for valid reasons

Correct Answer: B. The probability of at least one false positive increases

Question 9

Spearman rank correlation is primarily designed to measure:

A. Equality of two population variances

B. A monotonic association based on ranks

C. The difference between paired means

D. The probability of a Type II error

Correct Answer: B. A monotonic association based on ranks

Question 10

For a two-tailed 5% test with 24 degrees of freedom, a calculated t-statistic of 2.00 is approximately:

A. Significant because it exceeds 1.00

B. Significant because every positive t-statistic rejects the null

C. Not significant at the 5% level because it is below the approximate critical value 2.064

D. Invalid because t-statistics cannot be positive

Correct Answer: C. Not significant at the 5% level because it is below the approximate critical value 2.064