Statistical Estimation and Hypothesis Testing

Introduction

Statistical estimation and hypothesis testing provide the principal tools for making defensible decisions from sample data. In business analytics, analysts rarely observe an entire population: a retailer may study a sample of customers, a bank may examine a subset of transactions, and a manufacturer may test selected units. Statistical inference uses probability to quantify what can reasonably be concluded about the wider population.

Estimation addresses questions such as “What is the likely average customer spend?” Hypothesis testing addresses questions such as “Is the observed increase in spend sufficiently large to rule out random sampling variation?” These methods are not substitutes for managerial judgement. They structure evidence, quantify uncertainty, and clarify the assumptions behind a decision.

Learning Objectives

By the end of this chapter, you should be able to:

• explain sampling distributions and the central limit theorem;
• calculate and interpret point and interval estimates;
• formulate null and alternative hypotheses;
• distinguish significance, practical importance, p-values, effect sizes, and statistical power;
• select and interpret t-tests, chi-square tests, ANOVA, and suitable non-parametric tests;
• assess assumptions, limitations, and ethical implications of statistical conclusions;
• communicate analytical findings accurately to technical and managerial audiences.

1. Populations, Samples and Sampling Distributions

A population is the complete set of units relevant to a study. A sample is a subset selected from that population. A parameter describes a population, whereas a statistic describes a sample. Common parameters include the population mean μ, variance σ², proportion p, and correlation ρ. Their sample counterparts are the mean x̄, variance s², proportion p̂, and correlation r.

A statistic varies from sample to sample. Its probability distribution over all possible samples of a fixed size is its sampling distribution. Understanding this distribution is central to estimating uncertainty.

For a simple random sample of n independent observations with population mean μ and variance σ², the sample mean has:

Here E(x̄) is the expected sample mean and SE(x̄) is its standard error. If σ is unknown, it is commonly estimated by s, giving SE(x̄) = s ÷ √n. The observations should be independent, and the sampling design should support the intended population inference.

The standard error measures sampling variability; it is not the same as the standard deviation of individual observations. Increasing n reduces the standard error at a rate proportional to 1 ÷ √n. Thus, quadrupling the sample size approximately halves the standard error.

The Central Limit Theorem

The central limit theorem (CLT) states that, under broad conditions, the sampling distribution of the mean becomes approximately normal as n increases, even when the population distribution is not normal. The approximation is more reliable for larger samples and less skewed populations. There is no universal sample-size threshold: severe skewness, outliers, dependence, or heavy tails may require larger samples or robust methods.

When the population is normal, the sampling distribution of x̄ is normal for every n. When σ is unknown and the population is normal, the standardized mean follows a t distribution with n − 1 degrees of freedom:

The t distribution has heavier tails than the standard normal distribution, especially with small degrees of freedom. As degrees of freedom increase, it approaches the standard normal distribution.

2. Point and Interval Estimation

A point estimate is a single best estimate of a parameter. For example, x̄ estimates μ and p̂ estimates p. A point estimate alone does not express sampling uncertainty.

An interval estimate gives a range of plausible parameter values at a stated confidence level. A general confidence interval has the form:

Confidence Interval for a Mean

If σ is known, or if a sufficiently large sample justifies a normal approximation, a confidence interval for μ is:

If σ is unknown and the population is reasonably normal, use:

Here α = 1 − confidence level, z is a standard-normal critical value, and t is a t critical value with n − 1 degrees of freedom. For a 95% interval, α = 0.05. The familiar z critical value is approximately 1.96, but a t critical value is used when σ is estimated, particularly for small samples.

A 95% confidence interval does not mean that there is a 95% probability that a fixed parameter lies in this particular interval. In repeated sampling, the method produces intervals containing the true parameter in approximately 95% of samples, assuming the model and sampling process are appropriate.

Confidence Interval for a Proportion

For a sufficiently large sample, an approximate interval for a population proportion is:

The approximation is less reliable when expected successes or failures are small. Exact or improved small-sample methods may then be preferable. The estimate should also be bounded between 0 and 1.

Worked Example: Estimating Average Delivery Time

A random sample of 64 deliveries has mean time 42 minutes and sample standard deviation 8 minutes. A 95% approximate confidence interval using z = 1.96 is:

Standard error = 8 ÷ √64 = 1 minute.

Margin of error = 1.96 × 1 = 1.96 minutes.

Confidence interval = 42 ± 1.96, or (40.04, 43.96) minutes.

The appropriate interpretation is that the procedure gives intervals containing the population mean in approximately 95% of repeated samples. It is reasonable to report that the estimated average delivery time is 42 minutes, with a 95% confidence interval from 40.04 to 43.96 minutes, subject to sampling and model assumptions.

Sample Size for a Mean

If a desired margin of error E is specified and σ is available or reasonably estimated, the approximate required sample size is:

The result is rounded upward. For proportions, when no prior estimate exists, p = 0.5 is conservative because it produces the largest variance p(1 − p):

These formulas assume an appropriate random sample and generally do not account for finite-population corrections, clustering, non-response, or design effects.

3. Hypothesis Testing

Hypothesis testing evaluates evidence about a population claim. The null hypothesis, H₀, represents a benchmark such as no difference or no association. The alternative hypothesis, H₁ or Hₐ, represents the effect or relationship of interest.

For example:

• H₀: μ = 50 and Hₐ: μ ≠ 50 is two-sided;
• H₀: μ = 50 and Hₐ: μ > 50 is upper-tailed;
• H₀: μ = 50 and Hₐ: μ < 50 is lower-tailed.

The significance level α is the probability of rejecting H₀ when H₀ is true: a Type I error. A Type II error occurs when H₀ is not rejected even though H₁ is true. The power of a test is 1 − β, the probability of detecting a specified effect under specified conditions.

A typical testing process is:

1. define the population and parameter;
2. specify H₀ and Hₐ before inspecting outcomes;
3. select α, often 0.05;
4. choose a test consistent with the design and assumptions;
5. calculate the test statistic and p-value;
6. report effect size and confidence interval;
7. make a substantive conclusion in context.

p-Values

A p-value is the probability, assuming H₀ is true, of observing a result at least as inconsistent with H₀ as the result obtained, according to the chosen test statistic. A small p-value indicates incompatibility with H₀; it does not prove that H₀ is false, measure the probability that H₀ is true, or measure the size or importance of an effect.

If p ≤ α, reject H₀ under the stated decision rule. If p > α, do not reject H₀. “Fail to reject” is preferable to “accept,” because a non-significant result may reflect low power or imprecise data rather than evidence of no effect.

Statistical significance depends on effect size, sample size, variability, and α. A trivial effect may be statistically significant in a very large sample, whereas a practically important effect may fail to reach significance in a small sample.

Effect Sizes

Effect sizes quantify practical magnitude independently, or partly independently, of sample size. For a difference between two means, Cohen’s d is:

where s_p is the pooled standard deviation under the equal-variance formulation. For independent groups with sample sizes n₁ and n₂:

Rules such as d = 0.2, 0.5, and 0.8 as small, medium, and large are rough conventions, not universal truths. Domain consequences and costs should guide interpretation. Other effect sizes include risk difference, relative risk, odds ratio, η² for ANOVA, and Cramér’s V for categorical association.

Statistical Power and Multiple Testing

Power increases with larger samples, larger true effects, lower variability, and a higher α. Power analysis should be conducted before data collection where possible, using a meaningful minimum effect, desired power, α, and an appropriate design. Post hoc power calculations based only on an observed p-value add little information; confidence intervals and precision are usually more informative.

Testing many hypotheses increases the chance of at least one false positive. Analysts should pre-specify primary outcomes, control the family-wise error rate when necessary, or use false-discovery-rate procedures for exploratory high-dimensional work. Selective reporting and repeated testing until significance is found undermine validity.

4. t-Tests

One-Sample t-Test

A one-sample t-test compares a sample mean with a hypothesized value μ₀:

Assumptions include independent observations, a quantitative outcome, and approximate normality of the population for small samples. With larger samples, the CLT may support robustness, but outliers remain influential.

Independent-Samples t-Test

This test compares means from two independent groups. Welch’s test is generally preferred because it does not assume equal population variances. Its statistic is:

The degrees of freedom are approximated by the Welch–Satterthwaite formula:

The groups must be independent. Equal group sizes are not required.

Paired-Samples t-Test

For matched observations, calculate each difference dᵢ = afterᵢ − beforeᵢ and test whether the mean difference μ_d equals zero:

Pairing controls for stable individual differences, but the differences—not necessarily the raw scores—should be approximately normal for small samples.

5. Chi-Square Tests for Categorical Data

The chi-square goodness-of-fit test evaluates whether observed category counts agree with specified expected proportions. The test of independence evaluates whether two categorical variables are associated.

The general statistic is:

Oᵢ is an observed count and Eᵢ is an expected count. For an r × c contingency table, expected count in cell (i,j) is:

For independence, degrees of freedom are (r − 1)(c − 1). The approximation assumes independent observations and sufficiently large expected counts. Sparse tables may require combining substantively similar categories or using Fisher’s exact test, especially in a 2 × 2 table.

A significant chi-square test indicates association, not causation. Standardized residuals can identify cells contributing to the result, while Cramér’s V describes association strength:

N is the total sample size.

6. ANOVA

Analysis of variance (ANOVA) compares means across three or more groups. It partitions total variability into between-group and within-group components:

The F statistic is:

where MS = SS ÷ df. For k groups and N observations, df_between = k − 1 and df_within = N − k. The null hypothesis is that all group means are equal.

A significant omnibus F test shows that at least one mean differs, but not which means differ. Planned contrasts or corrected post hoc comparisons, such as Tukey’s method, should follow. Repeated uncorrected pairwise tests inflate Type I error.

Classical one-way ANOVA assumes independent observations, approximately normal errors within groups, and reasonably similar variances. It is often fairly robust to moderate non-normality when groups are balanced, but severe unequal variances and unequal group sizes can be problematic. Welch ANOVA addresses unequal variances. Effect sizes include:

η² is the proportion of observed total variance attributable to group membership in the sample; adjusted measures such as ω² may provide less biased population estimates.

7. Non-Parametric Alternatives

Non-parametric methods are useful when quantitative data are ordinal, markedly skewed, contain influential outliers, or violate assumptions that cannot be remedied. They often test distributions or ranks rather than means, so interpretation must match the procedure.

The Mann–Whitney test is not automatically a test of means or medians. It evaluates rank distributions and can be interpreted as a location comparison only under additional distributional similarity assumptions. Non-parametric tests are not assumption-free: independence, correct pairing, measurement quality, and appropriate sampling remain important.

8. Selecting a Test and Diagnosing Assumptions

Test selection should follow the research question and design, not convenience. First identify the outcome type: quantitative, ordinal, binary, nominal, or count. Then identify whether observations are independent, paired, repeated, clustered, or longitudinal. Finally assess distributional assumptions, outliers, variance structure, and sample size.

Useful diagnostics include histograms, boxplots, residual plots, group-wise summaries, and domain knowledge. Formal normality tests can be over-sensitive in large samples and underpowered in small samples; they should not replace graphical and substantive assessment. Transformations, robust procedures, bootstrap intervals, generalized linear models, or resampling may be preferable to mechanically deleting observations.

Confidence intervals should accompany p-values. A result such as “the estimated difference is 4.2 units, 95% CI [1.0, 7.4], p = .01” communicates direction, precision, and evidence more effectively than “significant.” Report the analysis population, missing-data treatment, exclusions, test direction, effect size, confidence interval, and assumptions.

Applied Case: Evaluating a Customer-Service Training Programme

Background. A telecommunications firm introduces training intended to reduce average call-handling time without lowering customer satisfaction. The analytics team samples 40 trained agents and 40 untrained agents. Satisfaction is measured on a 0–100 scale, and handling time is measured in minutes.

Problem. Management wants to know whether the programme improves satisfaction and reduces handling time. Agents were not randomly assigned, so pre-existing differences between groups are possible.

Analysis. The team first examines distributions, missing records, outliers, and group sizes. Because the groups are independent and variances differ somewhat, Welch’s independent-samples t-test is selected. The analysts report mean differences and 95% confidence intervals, not only p-values. They also calculate Cohen’s d to describe practical magnitude. If satisfaction is strongly ordinal or highly skewed, a Mann–Whitney analysis is used as a sensitivity analysis rather than as an automatic replacement.

Suppose the trained group’s mean satisfaction is 86 and the comparison group’s mean is 82, producing an estimated difference of 4 points. The confidence interval is [1, 7] points and p = .01. The evidence is compatible with a positive association between training and satisfaction. However, because the interval excludes zero, this statistical result does not establish that training caused the improvement. If the satisfaction scale’s minimum important difference is 8 points, the estimated 4-point improvement may be statistically significant but operationally modest.

For handling time, suppose the estimated difference is −0.8 minutes, with a 95% confidence interval [−1.5, −0.1]. This suggests shorter calls, but the business should also examine repeat calls, resolution rate, employee workload, and customer complaints. A reduction in call time may be harmful if it reflects premature termination.

Approach and solution. A stronger evaluation would measure agents before and after implementation, use matched or randomized assignment where feasible, adjust for workload and tenure, and predefine primary outcomes. A difference-in-differences design or regression model could improve causal interpretation, although each requires additional assumptions.

Outcome and learning points. Statistical significance should be interpreted with effect size, confidence intervals, operational thresholds, and study design. Confounding, selective outcomes, and measurement changes can mislead decision-makers. The final recommendation should distinguish what the data demonstrate from what remains uncertain.

9. Ethics, Governance and Contemporary Practice

Statistical analysis has ethical consequences when results affect employees, customers, credit applicants, or patients. Analysts should protect confidentiality, document data provenance, disclose exclusions, avoid misleading visualizations, and report uncertainty. Subgroup analyses should examine whether an intervention benefits or harms different populations, while respecting privacy and avoiding unsupported claims.

Multiple testing, p-hacking, HARKing (hypothesizing after results are known), and publication bias can produce apparently persuasive but unreliable findings. Pre-registration, analysis plans, reproducible code, versioned data, and transparent reporting improve credibility. Where personal data are used, governance should address lawful purpose, minimization, access controls, retention, and applicable privacy regulation.

Software can calculate tests instantly, but it cannot determine whether sampling was biased, a variable was measured validly, or a statistically significant result matters. AI-assisted analytics should therefore be treated as decision support. Analysts must verify generated code, formulas, assumptions, and interpretations; preserve an audit trail; and ensure that automated recommendations do not conceal discrimination or uncertainty.

Best-Practice Checklist

• Define the population, estimand, outcome, and sampling process.
• State hypotheses before examining inferential results.
• Match the test to measurement scale and study design.
• Inspect distributions, missingness, outliers, and dependence.
• Prefer confidence intervals and effect sizes alongside p-values.
• Adjust for multiple comparisons when appropriate.
• Report practical relevance and limitations.
• Avoid causal language for observational comparisons.
• Preserve reproducible records of data, code, and decisions.
• Communicate uncertainty honestly to decision-makers.

Chapter Summary

Sampling distributions explain how statistics vary across samples, while the central limit theorem supports approximate normal inference for many sample means. Point estimates summarize likely parameter values; confidence intervals add information about precision and should be interpreted as properties of a repeated-sampling procedure. Hypothesis tests compare data with a null model using a test statistic and p-value, but statistical significance does not establish practical importance or causation. Effect sizes, confidence intervals, and power provide essential context.

One-sample, independent, and paired t-tests address different mean-comparison designs. Chi-square tests analyse categorical counts and associations. ANOVA compares several means by partitioning variability, with post hoc procedures required when the omnibus test is significant. Non-parametric methods are valuable for ordinal, skewed, outlier-prone, or otherwise unsuitable data, but they retain important assumptions. Sound practice combines statistical technique with careful design, transparent reporting, ethical governance, and domain judgement.

Key Terms

Population: The complete set of units relevant to an investigation.

Sample: A subset of a population used to obtain evidence.

Parameter: A numerical characteristic of a population.

Statistic: A numerical characteristic calculated from a sample.

Sampling distribution: The probability distribution of a statistic over repeated samples.

Standard error: The standard deviation of a statistic’s sampling distribution.

Central limit theorem: A result stating that many sample-mean distributions become approximately normal as sample size increases.

Point estimate: A single-value estimate of a population parameter.

Confidence interval: An interval produced by a method designed to contain a parameter at a stated long-run frequency.

Null hypothesis: A benchmark claim evaluated by a statistical test.

Alternative hypothesis: The claim representing an effect or relationship of interest.

p-value: The probability, under the null model, of data at least as inconsistent as those observed.

Type I error: Rejecting a true null hypothesis.

Type II error: Failing to reject a false null hypothesis.

Statistical power: The probability of detecting a specified effect under specified conditions.

Effect size: A measure of the magnitude or practical importance of an effect.

Welch’s t-test: A two-group mean comparison that does not require equal variances.

Chi-square test: A test based on differences between observed and expected categorical counts.

ANOVA: A method that compares several means through between-group and within-group variation.

Non-parametric test: A procedure relying less directly on a specified population distribution, often using ranks.

Review Questions

Short Answer Questions

1. Distinguish a parameter from a statistic and give one example of each.
2. What does a standard error measure, and how does it change when sample size increases?
3. Why is a p-value not the probability that the null hypothesis is true?
4. State one condition under which Fisher’s exact test may be preferred to a chi-square approximation.
5. Why should an ANOVA result be followed by suitable post hoc comparisons when significant?

Long Answer Questions

1. Explain the central limit theorem and discuss conditions that affect the quality of its approximation.
2. Explain how confidence intervals, p-values, effect sizes, and statistical power complement one another.
3. Compare one-sample, independent-samples, and paired-samples t-tests, including their designs and assumptions.
4. Describe the logic of a chi-square test of independence and explain how Cramér’s V adds information.
5. Discuss the assumptions, interpretation, and limitations of one-way ANOVA and non-parametric alternatives.

Analytical / Application Questions

1. A sample of 25 invoices has mean processing time 18.4 minutes and standard deviation 4.0 minutes. Explain which interval procedure is appropriate for estimating the population mean, identify its assumptions, and describe how you would report the result.
2. A company compares satisfaction across four service channels and obtains a significant ANOVA result. Design an analysis plan that includes diagnostics, effect size, post hoc comparisons, and managerial interpretation.
3. A dashboard reports p < .001 for a 0.3-point improvement based on 100,000 observations. Evaluate the claim that the improvement is important, specifying the additional evidence and context required.

Multiple Choice Questions with Answers

Question 1

Which quantity describes the expected variability of a sample mean across repeated samples?

A. Population variance

B. Standard error

C. Effect size

D. p-value

Correct Answer: B. Standard error

Question 2

A 95% confidence interval is best interpreted as which statement?

A. There is a 95% probability that the fixed parameter is inside this calculated interval.

B. The null hypothesis has a 95% probability of being true.

C. The procedure produces intervals containing the parameter in about 95% of repeated samples.

D. 95% of observations lie inside the interval.

Correct Answer: C. The procedure produces intervals containing the parameter in about 95% of repeated samples.

Question 3

Which test is generally appropriate for comparing two independent means when population variances may differ?

A. Welch’s t-test

B. Paired t-test

C. Friedman test

D. One-sample sign test

Correct Answer: A. Welch’s t-test

Question 4

If p = 0.03 and α = 0.05, what is the conventional decision?

A. Accept the null hypothesis as proven true

B. Reject the null hypothesis

C. Increase the sample size automatically

D. Conclude that the effect is necessarily large

Correct Answer: B. Reject the null hypothesis

Question 5

In a chi-square test of independence, expected cell counts are calculated from:

A. The difference between the two largest observations

B. The sample mean and standard deviation

C. Row totals, column totals, and the grand total

D. The treatment-group means

Correct Answer: C. Row totals, column totals, and the grand total

Question 6

What does a significant omnibus ANOVA F test establish?

A. Every group mean differs from every other group mean

B. At least one group mean differs

C. The largest group has the highest population mean

D. The data are normally distributed

Correct Answer: B. At least one group mean differs

Question 7

Which procedure is a common non-parametric alternative for three or more independent groups?

A. Kruskal–Wallis test

B. Paired t-test

C. One-sample z-test

D. Pearson residual test

Correct Answer: A. Kruskal–Wallis test

Question 8

A Type I error occurs when an analyst:

A. Fails to reject a false null hypothesis

B. Rejects a true null hypothesis

C. Reports an effect size without a p-value

D. Uses a confidence interval

Correct Answer: B. Rejects a true null hypothesis

Question 9

If sample size is quadrupled, the standard error of a mean approximately:

A. Doubles

B. Quadruples

C. Halves

D. Remains unchanged

Correct Answer: C. Halves

Question 10

Which statement about statistical significance is most accurate?

A. It always implies practical importance.

B. It proves that an observed association is causal.

C. It depends partly on sample size and should be interpreted with effect size and context.

D. It is identical to statistical power.

Correct Answer: C. It depends partly on sample size and should be interpreted with effect size and context.