Sampling, Estimation and Hypothesis Testing

Introduction

Business decisions are usually made about populations—customers, employees, transactions, firms, or markets—while data are collected from samples. Sampling and statistical inference provide the logic for moving from observed sample evidence to defensible conclusions about a wider population. This logic is central to market research, quality management, auditing, people analytics, financial analysis, operations, and evidence-based strategy.

Inference is never simply a matter of calculating a p-value. A rigorous analysis requires a well-defined population, an appropriate sampling design, an estimator with known properties, an uncertainty interval, a clearly stated hypothesis, and an assessment of both statistical and practical significance. Poor sampling can produce precise estimates of the wrong quantity; inappropriate tests can produce apparently persuasive but invalid conclusions.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define populations, sampling frames, parameters, statistics, and sources of sampling error.
2. Select and evaluate probability and non-probability sampling designs.
3. Explain sampling distributions, the central limit theorem, standard error, bias, and consistency.
4. Construct and interpret confidence intervals for means, proportions, and differences.
5. Formulate and perform common parametric and nonparametric hypothesis tests.
6. Distinguish statistical significance, effect size, practical significance, and statistical power.
7. Communicate inferential results responsibly in managerial and research settings.

1. The Logic of Statistical Inference

A population is the complete set of units relevant to a question. A target population is the population to which the researcher intends to generalize; an accessible population is the population from which observations can actually be obtained. A sampling frame is the operational list or mechanism used to identify population units. The frame may omit eligible units, include ineligible units, or contain duplicates.

A parameter is a numerical characteristic of a population, such as the population mean μ, variance σ², or proportion p. A statistic is calculated from a sample, such as the sample mean x̄, sample variance s², or sample proportion p̂. An estimator is a rule for using sample data to estimate a parameter; an estimate is the numerical result produced by that rule.

Sampling error is the random difference between a sample statistic and its population parameter. It is distinct from nonsampling error, which includes nonresponse, measurement error, coverage error, interviewer effects, data processing mistakes, and specification error. Increasing sample size generally reduces sampling error, but it does not automatically correct systematic nonsampling error.

1.1 Probability and non-probability sampling

In probability sampling, every eligible population unit has a known, non-zero probability of selection. This permits design-based estimates of sampling uncertainty. In non-probability sampling, selection probabilities are unknown; examples include convenience, judgment, quota, and snowball sampling. Such designs may be useful for exploratory research or hard-to-reach populations, but conventional margins of error may not have their usual interpretation.

1.2 Sample size for a proportion

For a simple random sample intended to estimate a population proportion with approximate margin of error E at confidence level 1 − α, the planning formula is:

Here, n₀ is the required sample size before finite-population correction, z_{α/2} is the standard-normal critical value, E is the maximum desired absolute error, and p is a planning value for the proportion. If no credible prior estimate exists, p = 0.50 is conservative because it produces the largest required sample size. The formula assumes a probability sample, independence or an equivalent design, and a sufficiently large population relative to n₀.

If the population size N is not large, use:

This finite-population correction is relevant when sampling without replacement constitutes a substantial fraction of the population. A design effect, DEFF, should also be considered for cluster sampling or other complex designs; approximately, the effective sample size is n_eff = n/DEFF.

1.3 Sample size for a mean

When a planning standard deviation σ is available, a two-sided confidence interval for a mean with margin of error E requires approximately:

If σ is unknown, researchers may use a pilot study or a defensible prior estimate. Rounding must be upward because a fractional observation cannot be collected and rounding downward may violate the precision requirement.

2. Sampling Distributions and Estimator Quality

A sampling distribution is the probability distribution of a statistic over repeated samples drawn under the same design. It is a theoretical device: a researcher normally observes only one sample, but the repeated-sampling model describes the uncertainty attached to that observation.

For a simple random sample of n independent observations with population mean μ and variance σ²:

In practice σ is usually unknown, so the estimated standard error is s/√n. The mean is unbiased, and its variance decreases as sample size increases. The standard error decreases at the rate 1/√n: quadrupling the sample size approximately halves the standard error.

For a sample proportion p̂ under independent Bernoulli sampling:

Because p is unknown, p̂ is commonly substituted when estimating the standard error.

2.1 Central limit theorem

The central limit theorem states, under broadly applicable regularity conditions, that the standardized sample mean approaches a standard normal distribution as n becomes sufficiently large, even when the population distribution is not normal. The quality of the approximation depends on sample size, skewness, tail behavior, and dependence. It is not a universal license to treat every small sample as normal.

If the population itself is normal, x̄ is normally distributed for every n. If σ is unknown and the population is normal, the standardized statistic follows a t distribution with n − 1 degrees of freedom:

The t distribution has heavier tails than the standard normal distribution, reflecting uncertainty from estimating σ. As degrees of freedom increase, it approaches the standard normal distribution.

2.2 Bias, variance, consistency, and efficiency

An estimator θ̂ is unbiased for θ if E(θ̂) = θ. Bias is:

An estimator may be unbiased but highly variable, or slightly biased but substantially more precise. Mean squared error combines both considerations:

An estimator is consistent if it converges in probability to the true parameter as sample size increases. Efficiency compares the variance of estimators, usually among unbiased estimators under the same conditions. These properties help evaluate estimators, but real-world data quality and sampling design remain decisive.

3. Estimation and Confidence Intervals

A point estimate gives one best numerical estimate; an interval estimate reports a range of plausible parameter values together with a confidence level. A general approximate interval has the form:

A 95% confidence interval is constructed by a procedure that captures the fixed population parameter in approximately 95% of repeated samples, assuming the model and sampling procedure are appropriate. It is not technically correct to say that there is a 95% probability that the fixed parameter lies in this particular interval under the frequentist interpretation.

3.1 Mean with known or large-sample standard deviation

For a mean with known σ, or in situations where a normal approximation is justified:

When σ is unknown and the population is plausibly normal, use:

The t interval is especially important for small samples. Its validity depends on independence and approximate normality of the underlying observations; extreme outliers can seriously distort it.

3.2 Worked example: confidence interval for a mean

A firm surveys 25 business customers. Mean satisfaction is 72 points and sample standard deviation is 10 points. Assuming independent observations and an approximately normal population, a 95% interval uses t_{0.025,24} ≈ 2.064.

The estimated population mean satisfaction is therefore between approximately 67.9 and 76.1 points under the stated assumptions. The interval describes sampling uncertainty; it does not address nonresponse or systematic measurement error.

3.3 Confidence interval for a proportion

A commonly used large-sample Wald interval is:

It can perform poorly with small samples or proportions near 0 or 1. Wilson or exact binomial intervals are often preferable in those circumstances. The normal approximation is commonly assessed using adequate expected counts, but no single rule replaces substantive judgment.

For two independent proportions, an interval for p₁ − p₂ should use an unpooled standard error for estimation:

4. Hypothesis Testing

Hypothesis testing formalizes a decision between a null hypothesis H₀ and an alternative hypothesis H₁. H₀ commonly represents no difference, no association, or a specified benchmark. H₁ represents the research claim. Tests may be two-sided, H₁: θ ≠ θ₀, or one-sided, H₁: θ > θ₀ or θ < θ₀. A one-sided test should be justified before examining the results; choosing direction after seeing the data inflates the false-positive risk.

The significance level α is the maximum long-run probability of rejecting a true H₀ under the test procedure. A Type I error is rejecting a true H₀. A Type II error is failing to reject a false H₀; its probability is β, and power is 1 − β.

A p-value is the probability, calculated under H₀ and the specified test procedure, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an unrestricted sense. Statistical significance means that the result is relatively incompatible with H₀ under the model; it does not establish importance or causality.

4.1 A disciplined testing workflow

1. Define the target parameter and research question.
2. State H₀ and H₁ before inspecting inferential results.
3. Choose α and determine whether the test is one- or two-sided.
4. Select a test consistent with the design, measurement scale, and assumptions.
5. Examine data quality, outliers, dependence, missingness, and distributional features.
6. Calculate the test statistic and p-value, preferably with a confidence interval.
7. Report effect size and assess practical significance.
8. Make a decision proportional to the evidence and communicate limitations.

4.2 One-sample and two-sample t tests

For a one-sample mean tested against μ₀:

For two independent groups with equal population variances, the pooled-variance statistic is:

where:

The equal-variance assumption should not be imposed automatically. Welch’s t test, which does not require equal variances, is often the safer default for two independent means. For paired observations, calculate within-pair differences dᵢ and test the mean difference:

Pairing is appropriate for before-and-after measurements on the same units or matched units; treating paired observations as independent wastes information and can produce incorrect standard errors.

4.3 Tests for proportions and categorical association

For one proportion under H₀: p = p₀, a large-sample z statistic is:

For a contingency table, the Pearson chi-square statistic is:

Here Oᵢⱼ is an observed cell count and Eᵢⱼ is the expected count under independence. For an r × c table, degrees of freedom are (r − 1)(c − 1). Expected counts are calculated from row and column totals. Sparse tables may require Fisher’s exact test or category combination based on substantive reasoning, not merely computational convenience.

4.4 Analysis of variance

One-way ANOVA tests whether k group means are all equal. It partitions total variation into between-group and within-group components:

where MS_B is between-group mean square and MS_W is within-group mean square. Under H₀ and suitable assumptions, F follows an F distribution with k − 1 and N − k degrees of freedom. A significant omnibus test indicates that at least one mean differs, not which means differ. Planned contrasts or multiplicity-adjusted post-hoc comparisons are needed for detailed conclusions.

Important assumptions include independent observations, approximately normal residuals, and reasonably similar variances. ANOVA is often robust to moderate nonnormality when group sizes are balanced, but severe heteroscedasticity or unequal sizes can be consequential.

5. Nonparametric Methods

Nonparametric tests are useful when measurement scales are ordinal, distributions are strongly nonnormal, outliers dominate, or sample sizes are small. They do not mean “assumption-free.” They still require appropriate independence, sampling, and measurement conditions.

The Mann–Whitney test evaluates whether ranks differ between two independent groups. It is often described as a test of distributions or stochastic dominance; it is not automatically a test of mean differences or medians unless additional distributional assumptions hold. The Wilcoxon signed-rank test uses the ranks of nonzero paired differences and assumes a suitably symmetric distribution of differences for a location interpretation. The Kruskal–Wallis and Friedman tests are omnibus procedures and require follow-up comparisons when significant.

6. Effect Size, Practical Significance, and Power

A p-value is sensitive to sample size. A trivial effect may be statistically significant in a very large sample, while a meaningful effect may fail to reach significance in a small sample. Therefore, report the estimated effect, confidence interval, and a contextually meaningful benchmark.

For a standardized difference between two means, Cohen’s d is commonly defined as:

where s_p is the pooled standard deviation. For paired data, a standardized mean difference based on the standard deviation of differences may be used, but the exact convention should be stated. For a two-group comparison, an approximate relationship between t and d is:

when t is based on the corresponding independent-groups comparison. For ANOVA, eta squared is:

where SS_B is between-group sum of squares and SS_T is total sum of squares. Omega squared is often preferred as a less biased population-effect estimate.

For a 2 × 2 table, the odds ratio is:

An odds ratio of 1 indicates equal odds; interpretation must distinguish odds from probabilities, especially when outcomes are common.

Power depends on α, sample size, effect magnitude, variability, and the test design. A prospective power analysis can determine sample size for a target effect and desired power. A post hoc claim that an observed nonsignificant result “proves no effect” is inappropriate; confidence intervals and equivalence or noninferiority tests are better suited to claims of practical similarity.

7. Illustrative Case: Evaluating a Sales Training Programme

This is a fictional illustrative case.

Background and problem

A retail company introduces a consultative-selling programme. Management wants to know whether average monthly sales per representative increased, while avoiding the mistake of treating normal sales variation as programme success.

Analysis and approach

Twenty-four representatives are measured for one month before and one month after training. Because each representative provides a matched pair, the primary analysis uses the 24 within-person differences. The company defines a practically meaningful improvement as at least $500 per month before examining the results. The analyst inspects a difference plot, checks missing records and extreme observations, and documents that no representatives changed territories during the study.

The mean improvement is $620, with a standard deviation of differences of $1,200. The standard error is:

Testing H₀: μ_d = 0 with a paired t test gives:

The two-sided p-value is approximately 0.019. A 95% confidence interval uses t_{0.025,23} ≈ 2.069:

Interpretation and outcome

The evidence is consistent with a positive average change, and the interval excludes zero. However, the lower confidence limit is below the company’s $500 practical threshold. Thus, the data support improvement relative to zero, but they do not establish with 95% confidence that the average improvement reaches the managerial target. The study also lacks a randomized control group; seasonal demand, concurrent promotions, or experience effects could explain part of the change.

Learning points

The paired design improves precision by controlling for stable differences among representatives. Pre-specifying a practical threshold improves decision quality. Statistical significance does not establish causal attribution or guarantee a commercially worthwhile effect. A stronger next study would randomize representatives or stores to training and control conditions, measure a longer baseline, and include costs, retention, and customer outcomes.

8. Assumptions, Bias, and Research Governance

The validity of inference depends on design and assumptions. Independence can be violated by repeated measurements, teams, households, stores, or time-series dependence. Ignoring clustering usually makes standard errors too small. Missing data may be harmless if missing completely at random, but informative nonresponse can bias both estimates and tests. Outliers should be investigated rather than automatically deleted.

Researchers should distinguish exploratory from confirmatory analysis, register hypotheses where appropriate, report all relevant outcomes, and avoid selectively presenting favorable tests. Multiple testing increases the chance of false positives; corrections such as Bonferroni or false discovery rate procedures may be appropriate, depending on the inferential goal. Statistical significance should never be used to conceal weak measurement, questionable sampling, or a practically negligible effect.

Business data also raise privacy and governance concerns. Sampling files should use the minimum necessary personal information, access should be controlled, and results should be reported in ways that prevent re-identification. Automated or AI-supported analysis can assist with sample-size planning, missing-data diagnostics, simulation, and reproducible code, but it cannot repair a biased sampling frame or decide whether a business effect is ethically acceptable. Analysts should preserve an audit trail of data transformations, model choices, exclusions, and software versions.

9. Best Practices for Professional Reporting

A professional report should state the population, frame, sampling design, response rate, field dates, weighting, exclusions, and known limitations. For every major result, report the estimate, uncertainty interval, test specification if relevant, p-value, effect size, and practical benchmark. Use plain language: “The estimated difference was $620, 95% CI [$113, $1,127]” is more informative than “p = .019.”

Avoid saying that a result is “proved,” that a nonsignificant result demonstrates no difference, or that a confidence interval contains 95% probability for a fixed parameter. Use sensitivity analyses when conclusions could depend on an outlier, variance assumption, missing-data treatment, or choice between parametric and nonparametric methods.

Chapter Summary

Sampling connects a target population to the observations used for analysis. Probability designs support defensible estimates of sampling uncertainty, whereas convenience and other non-probability designs require cautious generalization. Sampling distributions explain why estimators vary, and the standard error quantifies that variation. The central limit theorem supports normal approximations under appropriate conditions, while the t distribution accounts for estimating a population standard deviation.

Confidence intervals communicate both an estimate and its uncertainty. Hypothesis tests provide a structured comparison with a null model, but p-values do not measure effect magnitude, practical value, or the probability that a hypothesis is true. Parametric tests can be efficient when their assumptions are reasonable; nonparametric procedures are useful for ordinal, skewed, or otherwise unsuitable data but still have assumptions. Sound professional inference combines sampling design, diagnostics, effect sizes, confidence intervals, power, ethical governance, and substantive judgment.

Key Terms

Population: The complete set of units relevant to a study.

Target population: The population to which the researcher intends to generalize.

Sampling frame: The operational list or process used to identify eligible units.

Parameter: A numerical characteristic of a population.

Statistic: A numerical quantity calculated from sample data.

Estimator: A rule for estimating a population parameter.

Sampling error: Random discrepancy between a sample statistic and its population parameter.

Nonsampling error: Error arising from coverage, nonresponse, measurement, processing, or specification.

Sampling distribution: The distribution of a statistic over repeated samples.

Standard error: The standard deviation of a statistic’s sampling distribution, estimated when necessary.

Bias: Systematic difference between an estimator’s expected value and the target parameter.

Confidence interval: An interval produced by a procedure designed to capture a parameter at a stated long-run rate.

Null hypothesis: A formal statement representing a benchmark or no-effect condition.

Alternative hypothesis: The competing statement supported when evidence is inconsistent with the null.

p-value: The probability, under the null model, of results at least as extreme as those observed.

Type I error: Rejection of a true null hypothesis.

Statistical power: The probability of rejecting a false null hypothesis under specified conditions.

Effect size: A measure of the magnitude of a difference, association, or relationship.

Practical significance: The real-world importance of an effect in its decision context.

Design effect: The inflation or adjustment in variance caused by a complex sampling design.

Review Questions

Short Answer Questions

1. Distinguish a parameter from a statistic and provide one business example of each.
2. Why does quadrupling the sample size approximately halve the standard error of a mean?
3. What is the difference between a Type I error and a Type II error?
4. Why can a statistically significant result fail to be practically significant?
5. State two assumptions relevant to a paired t test.

Long Answer Questions

1. Compare stratified, cluster, and multistage sampling, including their applications and principal risks.
2. Explain the central limit theorem and discuss why it does not justify automatic normal approximation for every small sample.
3. Describe how to construct and interpret a confidence interval for a population mean when the population standard deviation is unknown.
4. Compare parametric and nonparametric tests, explaining when a manager or analyst might prefer each.
5. Explain why professional statistical reporting should include effect size and confidence intervals in addition to p-values.

Analytical / Application Questions

1. A service manager surveys 100 customers, of whom 62 report satisfaction. Construct an approximate 95% confidence interval for the satisfaction proportion and explain whether the normal approximation appears reasonable.
2. A company compares employee productivity before and after software training for the same 18 employees. Specify the appropriate primary test, its null hypothesis, two assumptions, and one practical-significance criterion that management should define before analysis.
3. A marketing analyst tests 20 campaign outcomes and obtains three p-values below 0.05. Explain the multiple-testing concern and recommend an analysis and reporting strategy that controls misleading claims.

Multiple Choice Questions with Answers

Question 1

Which quantity is a population parameter?

A. The mean satisfaction score calculated from 50 surveyed customers

B. The proportion of all eligible customers who renew their contracts

C. The median calculated from a selected sample

D. The standard deviation of observed survey responses

Correct Answer: B. The proportion of all eligible customers who renew their contracts

Question 2

If the sample size for estimating a mean is multiplied by four, what approximately happens to the standard error, all else equal?

A. It doubles

B. It is unchanged

C. It is halved

D. It becomes one-fourth as large

Correct Answer: C. It is halved

Question 3

Which sampling design divides a population into subgroups and samples within each subgroup?

A. Cluster sampling

B. Convenience sampling

C. Stratified sampling

D. Snowball sampling

Correct Answer: C. Stratified sampling

Question 4

A 95% confidence interval is best interpreted as which statement?

A. There is a 95% probability that the fixed parameter lies in this particular interval

B. The procedure captures the fixed parameter in approximately 95% of repeated samples

C. 95% of observations lie inside the interval

D. The null hypothesis has a 5% probability of being true

Correct Answer: B. The procedure captures the fixed parameter in approximately 95% of repeated samples

Question 5

For a small sample mean with unknown population standard deviation and an approximately normal population, which reference distribution is generally appropriate?

A. Uniform distribution

B. Chi-square distribution

C. Student’s t distribution

D. Exponential distribution

Correct Answer: C. Student’s t distribution

Question 6

What does a p-value represent?

A. The probability that the null hypothesis is true

B. The probability that the research hypothesis is false

C. The probability, under the null model, of a result at least as extreme as observed

D. The probability that the sample was selected without bias

Correct Answer: C. The probability, under the null model, of a result at least as extreme as observed

Question 7

Which test is designed for two paired measurements, such as employee productivity before and after training?

A. Paired t test

B. Independent-samples chi-square test

C. One-way ANOVA for unrelated groups

D. Mann–Whitney test for independent groups

Correct Answer: A. Paired t test

Question 8

A statistically significant result has a very small effect size. What is the most appropriate conclusion?

A. The result must be strategically important

B. The null hypothesis has been proven false in every population

C. The effect is statistically detectable but may have little practical importance

D. The study must have used a biased sample

Correct Answer: C. The effect is statistically detectable but may have little practical importance

Question 9

Which method is especially appropriate for comparing more than two independent groups when strong nonnormality or ordinal measurement is a concern?

A. Kruskal–Wallis test

B. Paired t test

C. One-sample z test

D. Pearson correlation test

Correct Answer: A. Kruskal–Wallis test

Question 10

What is a Type I error?

A. Failing to reject a false null hypothesis

B. Rejecting a true null hypothesis

C. Using a sample that is larger than necessary

D. Reporting an effect size without a p-value

Correct Answer: B. Rejecting a true null hypothesis