Sampling, Estimation and Statistical Inference

Introduction

Managerial decisions are commonly made with incomplete information. A leader may want to estimate customer satisfaction, evaluate employee engagement, compare branch performance, test whether a marketing intervention increases sales, or determine whether a process has become less variable. In most situations, measuring every member of the relevant population is costly, slow, or impossible. Statistical inference provides a disciplined way to use sample evidence to make statements about a wider population.

This chapter develops the logic of sampling, estimation, confidence intervals, hypothesis testing, analysis of variance, and non-parametric methods. It also emphasizes issues that are especially important in leadership and managerial economics: sampling bias, practical significance, statistical power, data quality, ethical data governance, and the limits of generalization.

Statistical inference does not eliminate uncertainty. Rather, it measures and communicates uncertainty so that decisions are more transparent and defensible.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish populations, samples, parameters, and statistics.
2. Select an appropriate probability or non-probability sampling design.
3. Explain sampling distributions and the central limit theorem.
4. Construct and interpret confidence intervals for means and proportions.
5. Formulate and evaluate hypotheses using parametric and non-parametric tests.
6. Interpret p-values, effect sizes, statistical power, and practical significance.
7. Use one-way analysis of variance and understand its assumptions.
8. Recognize bias, measurement limitations, multiple testing, and ethical risks.
9. Translate statistical results into responsible managerial recommendations.

1. The logic of statistical inference

A population is the complete set of units about which a conclusion is intended. It might consist of all customers in a market, all employees in an organization, or all transactions processed during a year. A sample is a subset observed for analysis. A parameter is a numerical characteristic of a population, while a statistic is the corresponding quantity calculated from a sample.

For example, the population mean customer expenditure is a parameter, often denoted by μ. The sample mean is a statistic, denoted by x̄. A sample proportion p̂ estimates the population proportion p.

The inferential process is:

Population → sampling design → observed sample → statistic → uncertainty assessment → population conclusion.

A sound analysis requires alignment between the target population, the sampling frame, the sampling method, the measurement instrument, and the intended decision. A sophisticated test cannot repair a fundamentally biased sample.

Sampling error and non-sampling error

Sampling error is the natural difference between a sample statistic and its population parameter arising because only part of the population is observed. It can be quantified under probability sampling.

Non-sampling error includes coverage error, non-response, interviewer effects, inaccurate measurement, data processing errors, and deliberate or accidental misclassification. Increasing sample size reduces sampling error but does not necessarily reduce non-sampling error.

For example, surveying 10,000 customers through an app may still produce biased results if older customers rarely use the app. The problem is coverage, not sample size.

2. Sampling designs

Probability sampling

In probability sampling, every population unit has a known, non-zero probability of selection. This supports design-based estimation and defensible generalization.

In stratified sampling, proportional allocation selects sample sizes according to stratum population shares. Disproportionate allocation may deliberately oversample a small but strategically important group. Weighted analysis is then often required.

In cluster sampling, the relevant design effect is commonly greater than one because observations within clusters are correlated. Treating clustered observations as independent can understate standard errors and produce excessive false positives.

Non-probability sampling

Convenience, purposive, quota, snowball, and voluntary-response sampling are non-probability designs. They may be useful in exploratory research, qualitative studies, pilot work, or hard-to-reach populations. However, selection probabilities are unknown, so conventional margins of error do not automatically have their usual interpretation.

A quota sample can resemble the population on observed characteristics while remaining biased on unobserved characteristics. For example, a customer panel matched by age and region may still differ in loyalty, income, or digital engagement.

Sample size planning

For estimating a population mean when the population standard deviation σ is known or a credible planning value is available, the approximate sample size for a two-sided confidence interval with margin of error E is:

Here, n is the required sample size, z_{α/2} is the standard-normal critical value, σ is the population standard deviation or planning estimate, and E is the desired half-width of the interval. The formula assumes a random sample, independent observations, and an approximately normal sampling distribution.

For estimating a population proportion with margin of error E:

where p is the anticipated population proportion. If p is unknown, p = 0.50 is conservative because it maximizes p(1 − p). For a finite population of size N, a finite-population correction may reduce the required sample size:

This correction is relevant when the sample is a substantial fraction of the population and selection is without replacement.

3. Sampling distributions and the central limit theorem

A sampling distribution is the probability distribution of a statistic over repeated samples of the same size from the same population. The standard error measures the typical sampling variation of a statistic.

For a sample mean from independent observations with population standard deviation σ:

If σ is unknown, it is commonly estimated by the sample standard deviation s:

For a sample proportion:

In practice, p is replaced by p̂ for estimation.

The central limit theorem states, approximately, that the sampling distribution of the sample mean approaches a normal distribution as n increases, under broadly applicable conditions such as independent observations and finite variance. It does not state that the raw data become normal, nor does it guarantee reliable inference for highly dependent, heavily contaminated, or extremely skewed data at small sample sizes.

The standard error is not the same as the standard deviation of individual observations. Standard deviation describes variation among observations; standard error describes variation among repeated estimates.

4. Point estimation and interval estimation

A point estimate is a single value used to estimate a parameter. Examples include x̄, p̂, and the difference between two sample means. A good estimator is often evaluated by:

• Unbiasedness: its expected value equals the parameter.
• Consistency: it approaches the parameter as sample size increases.
• Efficiency: it has relatively small variance among competing estimators.
• Robustness: it is not excessively affected by unusual observations or assumption violations.

A confidence interval provides a range of plausible values for a population parameter, together with a confidence level describing the long-run performance of the method.

Confidence interval for a mean

When σ is known, or when a normal approximation is justified:

When σ is unknown and the population is reasonably normal, especially for small samples, use the t distribution:

The symbols t_{α/2, n−1} represent the critical value of the t distribution with n − 1 degrees of freedom.

Confidence interval for a proportion

For a sufficiently large sample, a commonly used Wald interval is:

The approximation can be poor when n is small or p̂ is near zero or one. Wilson or exact binomial intervals are preferable in such situations.

Worked example: estimating average service time

A random sample of 64 service calls has mean duration 12.4 minutes and sample standard deviation 4.8 minutes. A 95% t-based interval is approximately:

Using t ≈ 2.000, the margin of error is 2.000 × 0.6 = 1.2 minutes. The interval is therefore 11.2 to 13.6 minutes. A suitable interpretation is: using this sampling procedure, intervals constructed in this way would capture the true population mean in approximately 95% of repeated samples. It is not technically correct to say that there is a 95% probability that the fixed parameter lies in this particular calculated interval.

5. Hypothesis testing

Hypothesis testing begins with a claim about a population parameter. The null hypothesis, H₀, usually represents no difference, no effect, or a specified benchmark. The alternative hypothesis, H₁ or Hₐ, represents the effect or difference of interest.

A test statistic measures how far the sample result is from the null value relative to its standard error. For a one-sample mean with known σ:

When σ is estimated:

The p-value is the probability, assuming H₀ is true, of obtaining a result at least as extreme as the observed result in the specified direction. A small p-value indicates incompatibility with H₀; it does not measure the probability that H₀ is true, nor does it measure the size or importance of the effect.

A decision rule may compare the p-value with a significance level α, commonly 0.05. If p ≤ α, the analyst rejects H₀; otherwise, the analyst fails to reject H₀. “Fail to reject” is not equivalent to proving H₀ true.

Errors and power

A Type I error rejects a true null hypothesis. Its long-run probability is controlled by α under the stated assumptions. A Type II error fails to reject a false null hypothesis. Its probability is β, and statistical power is:

Power increases with larger effects, larger samples, lower variability, and a higher α, although increasing α also increases Type I error risk. A non-significant result may reflect a genuinely negligible effect, insufficient power, poor measurement, or excessive variability.

One-sided and two-sided tests

A two-sided test detects departures in either direction. A one-sided test is appropriate only when the opposite direction is irrelevant or cannot support the decision, and the direction must be specified before examining results. Switching from two-sided to one-sided after seeing the data is a form of selective analysis.

Effect size and practical significance

Statistical significance should be accompanied by an effect estimate and uncertainty interval. For comparing a mean with a benchmark, the raw difference is often more useful to managers than the p-value alone. A standardized effect size such as Cohen’s d is:

where s_p is a pooled standard deviation under assumptions of comparable variances. Effect-size interpretation is contextual: a small standardized effect may be financially important at large scale, while a statistically significant effect may be operationally trivial.

6. Comparing groups

Independent two-sample comparisons

To compare two independent group means, the preferred general method is often Welch’s t-test, which does not require equal population variances. The estimated difference is x̄₁ − x̄₂, and its standard error is:

The degrees of freedom are estimated using the Welch–Satterthwaite approximation. A paired t-test should be used instead when each observation in one condition is naturally matched with an observation in the other condition, such as employee productivity before and after training.

For paired data, calculate individual differences dᵢ and test the mean difference d̄ using:

where μ_{d,0} is usually zero.

Categorical outcomes

For categorical variables, a chi-square test of independence evaluates whether two categorical variables are associated. Expected counts under independence are:

The Pearson chi-square statistic is:

where O_{ij} and E_{ij} are observed and expected counts. Small expected counts can make the approximation unreliable; exact methods or category consolidation may be appropriate. Association does not establish causation.

7. Analysis of variance

Analysis of variance (ANOVA) compares the means of three or more groups. Its null hypothesis is that all population means are equal. ANOVA partitions total variation into variation between group means and variation within groups.

For k groups and total sample size N:

where SS_T is total sum of squares, SS_B is between-group sum of squares, and SS_W is within-group sum of squares. The mean squares are:

The F statistic is:

Under the null hypothesis and standard assumptions, F follows an F distribution with k − 1 and N − k degrees of freedom.

ANOVA assumes independent observations, approximately normal residuals within groups, and reasonably similar variances. Independence is primarily a design issue and cannot be repaired by a normality test. If the overall ANOVA is significant, post hoc comparisons such as Tukey’s method can identify which groups differ while controlling familywise error.

A useful effect-size measure is eta squared:

It describes the proportion of observed total variation attributable to group membership in the sample, although interpretation depends on design and context.

Repeated measures and factorial designs

Repeated-measures ANOVA is used when the same units are observed under several conditions or times. It accounts for within-unit dependence and requires additional assumptions, such as sphericity in standard formulations. Factorial ANOVA examines two or more categorical factors and can test main effects and interactions. An interaction means that the effect of one factor depends on the level of another; it should not be reduced to separate main-effect conclusions alone.

8. Non-parametric methods

Non-parametric methods are useful when outcome distributions are strongly skewed, ordinal, contaminated by outliers, or unsuitable for a parametric model. They are not assumption-free; they have their own requirements concerning independence, measurement scale, and distributional shape.

The Mann–Whitney test evaluates whether ranks differ between two independent groups. It is often described as a test of medians, but that interpretation requires similarly shaped distributions. The Kruskal–Wallis test compares rank distributions across several groups; a significant result requires appropriate post hoc procedures to locate differences. Non-parametric tests may have lower power than valid parametric tests when parametric assumptions are well satisfied.

9. Bias, dependence and data limitations

Important threats to inference include:

• Selection bias: inclusion probabilities differ systematically.
• Non-response bias: respondents differ from non-respondents.
• Survivorship bias: analysis excludes failed, closed, or departed units.
• Recall and social-desirability bias: respondents misreport behavior or attitudes.
• Measurement error: the instrument does not accurately capture the construct.
• Confounding: a third variable explains an observed association.
• Simpson’s paradox: an aggregate relationship reverses after stratification.
• P-hacking: repeated analyses are conducted until a desired p-value appears.
• Multiple testing: many tests increase the chance of at least one false positive.
• Dependence: repeated observations, teams, branches, or customers violate independence.

When many hypotheses are tested, analysts may control the familywise error rate using procedures such as Bonferroni or Holm adjustments, or control the false discovery rate when exploratory discovery is the priority. The correction should match the decision context rather than being applied mechanically.

Missing data also matter. Complete-case analysis is valid only under restrictive conditions. Analysts should investigate missingness patterns, distinguish missing completely at random from more structured mechanisms, and report the method used. Imputation can reduce bias but introduces model dependence and should preserve uncertainty rather than treating imputed values as observed facts.

10. A disciplined workflow for managerial inference

1. Define the decision and target population.
2. Specify the estimand: mean, proportion, difference, association, or causal effect.
3. Develop the sampling frame and select an appropriate design.
4. Pilot the measurement instrument and assess reliability and validity.
5. Determine sample size using a meaningful effect and acceptable precision.
6. Inspect data quality, missingness, outliers, dependence, and coding.
7. Visualize distributions before selecting a test.
8. State hypotheses and analysis rules before examining inferential results where feasible.
9. Report estimates, confidence intervals, p-values, effect sizes, sample sizes, and assumptions.
10. Conduct sensitivity or robustness analyses.
11. Translate results into costs, benefits, risks, and operational consequences.
12. Document limitations and avoid claims beyond the design.

A confidence interval often communicates managerial uncertainty better than a binary significant/non-significant label. Leaders should ask: How large is the effect? How precise is the estimate? Is the effect economically meaningful? Does the sample represent the population? What alternative explanations remain?

11. Illustrative case: evaluating a leadership development programme

Background

A fictional company, Meridian Services, introduces a six-week leadership programme for frontline supervisors. Senior management wants to know whether the programme improves employee engagement and reduces voluntary turnover intentions.

Problem

The programme is offered first to supervisors who volunteer. After six months, engagement scores are higher among participating teams. Management must determine whether the programme caused the improvement or whether motivated supervisors were already more likely to volunteer.

Analysis

The evaluation team defines the target population as all frontline supervisors in the organization. It measures engagement using a validated multi-item scale and records team-level outcomes. Because employees are nested within supervisors, treating every employee response as independent would understate uncertainty. The team therefore uses supervisor-clustered standard errors or a multilevel model.

A stronger design randomly assigns eligible supervisors to immediate training or a wait-list control group. Baseline engagement is recorded before training, and outcomes are measured at baseline and follow-up. The primary estimand is the difference in mean change between groups:

where T denotes the training group, C the control group, and Y the average engagement outcome. This approach relies on a plausible parallel-trends assumption: absent the programme, the groups would have changed similarly.

Approach and solution

The team preregisters the primary outcome, includes baseline adjustment, reports a confidence interval for the treatment effect, and conducts sensitivity analyses for attrition. It also reports standardized effect size and the practical cost of implementation. Qualitative interviews are used to understand mechanisms but are not treated as a substitute for the controlled comparison.

Outcome

The estimated improvement is positive but modest. The confidence interval excludes zero, yet the lower bound is close to the organization’s minimum worthwhile effect. Management therefore expands the programme cautiously, monitors implementation fidelity, and evaluates whether benefits differ by supervisor tenure or team size. It does not claim that the programme will reduce turnover without sufficient follow-up evidence.

Learning points

The case illustrates that statistical significance is only one part of evidence. Volunteer selection, clustered data, attrition, baseline imbalance, effect size, cost, and implementation quality all influence the managerial decision. A credible design begins with the causal question, not with a preferred statistical test.

12. Ethics, governance and contemporary practice

Statistical analysis carries ethical obligations. Participants should understand how data will be used, and sensitive data should be minimized, secured, and retained only as long as necessary. Leaders should avoid using uncertain estimates to justify punitive decisions about individuals or small groups. Reporting should disclose exclusions, missing data, stopping rules, subgroup analyses, and conflicts of interest.

AI and automated analytics can assist with sampling-frame construction, anomaly detection, simulation, and power analysis. They can also amplify historical bias, obscure assumptions, create leakage between training and evaluation data, or produce overconfident predictions. Human review, audit trails, reproducible code, access controls, and subgroup performance checks are essential. Automated recommendations should not be treated as causal evidence merely because they are statistically predictive.

Contemporary practice increasingly emphasizes estimation, preregistration, reproducibility, open materials where appropriate, robust standard errors, Bayesian sensitivity analysis, and decision-relevant effect sizes. These developments complement rather than replace careful sampling and substantive judgment.

Chapter Summary

Sampling makes it possible to learn about large populations from manageable amounts of data, but inference is credible only when the target population, sampling frame, design, measurement process, and analysis are aligned. Probability sampling supports defensible generalization, whereas non-probability sampling requires caution about selection and coverage bias.

Sampling distributions and standard errors quantify uncertainty. Point estimates summarize evidence, while confidence intervals communicate precision. Hypothesis tests evaluate compatibility with a null model, but p-values do not measure effect size, importance, or the probability that a hypothesis is true. Statistical power, Type I and Type II errors, and multiple testing must be considered.

Welch’s t-test, paired methods, chi-square tests, ANOVA, and non-parametric procedures address different research designs and assumptions. ANOVA partitions variation into between-group and within-group components, while non-parametric methods provide useful alternatives for ordinal, skewed, or outlier-prone data.

Responsible managerial inference requires attention to bias, dependence, missing data, practical significance, ethics, and the limits of causal claims. Leaders should report estimates, intervals, assumptions, effect sizes, and limitations rather than relying on significant/non-significant labels alone.

Key Terms

Population: Complete set of units about which inference is intended.

Sample: Subset of a population observed for analysis.

Parameter: Numerical characteristic of a population.

Statistic: Numerical quantity calculated from a sample.

Sampling frame: Operational list or procedure identifying population units.

Sampling error: Random difference between a sample statistic and its population parameter.

Selection bias: Systematic distortion caused by unequal or inappropriate inclusion.

Standard error: Standard deviation of a statistic’s sampling distribution.

Central limit theorem: Result describing the approximate normality of sample means under suitable conditions.

Point estimate: Single-number estimate of a population parameter.

Confidence interval: Interval produced by a procedure designed to capture a parameter at a stated long-run rate.

Null hypothesis: Default claim evaluated by a statistical test.

p-value: Probability, under the null model, of data at least as extreme as those observed.

Type I error: Rejection of a true null hypothesis.

Type II error: Failure to reject a false null hypothesis.

Statistical power: Probability of detecting a specified effect when it exists.

Effect size: Quantitative magnitude of a difference or relationship.

ANOVA: Method partitioning variation to compare several group means.

Non-parametric method: Procedure relying less heavily on distributional assumptions.

Confounding: Distortion of an association by a related third variable.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.
2. Why does increasing sample size not necessarily eliminate non-response bias?
3. What does a 95% confidence interval mean in repeated sampling?
4. Differentiate a Type I error from a Type II error.
5. Why should effect size accompany a p-value?

Long Answer Questions

1. Compare stratified, cluster, and multistage sampling, including their managerial applications and limitations.
2. Explain the central limit theorem and its relevance to confidence intervals for means.
3. Describe the assumptions, calculations, and interpretation of one-way ANOVA.
4. Compare parametric and non-parametric methods for comparing two or more groups.
5. Develop a complete statistical inference workflow for evaluating an employee development programme.

Analytical / Application Questions

1. A company surveys only customers who use its mobile application and generalizes the results to all customers. Identify the likely sources of bias and propose a redesigned sampling plan.
2. An intervention produces a statistically significant 0.4-point increase on a 100-point satisfaction scale, with a narrow confidence interval. Explain what additional information a leader needs before expanding the intervention.
3. Three branches report different average sales after a training programme, but employees are nested within branches and branch sizes differ. Recommend an analysis strategy and explain how dependence, effect size, and post hoc comparisons should be handled.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a parameter from a statistic?

A. A parameter describes a sample, whereas a statistic describes a population.

B. A parameter describes a population, whereas a statistic is calculated from a sample.

C. A parameter is always larger than a statistic.

D. A statistic can be calculated only from qualitative data.

Correct Answer: B. A parameter describes a population, whereas a statistic is calculated from a sample.

Question 2

Which sampling design deliberately divides a population into subgroups and samples within each subgroup?

A. Cluster sampling

B. Convenience sampling

C. Stratified sampling

D. Snowball sampling

Correct Answer: C. Stratified sampling

Question 3

A sample has n = 64 and s = 4.8. The estimated standard error of the mean is:

A. 0.60

B. 1.20

C. 4.80

D. 38.40

Correct Answer: A. 0.60

Question 4

A p-value is best interpreted as:

A. The probability that the null hypothesis is true

B. The probability that the alternative hypothesis is true

C. The probability, under the null model, of results at least as extreme as observed

D. The size of the practical effect

Correct Answer: C. The probability, under the null model, of results at least as extreme as observed

Question 5

Which change generally increases statistical power, all else equal?

A. Reducing the sample size

B. Increasing measurement variability

C. Reducing the effect size

D. Increasing the sample size

Correct Answer: D. Increasing the sample size

Question 6

In one-way ANOVA, the F statistic is calculated as:

A. MS within divided by MS between

B. MS between divided by MS within

C. SS total divided by sample size

D. The sample mean divided by its standard deviation

Correct Answer: B. MS between divided by MS within

Question 7

Which method is commonly used to compare three or more independent groups when strong distributional assumptions are unsuitable?

A. Kruskal–Wallis test

B. Paired t-test

C. Pearson correlation only

D. Wilcoxon signed-rank test for paired data

Correct Answer: A. Kruskal–Wallis test

Question 8

If a study observes the same employees before and after training, which comparison is generally appropriate?

A. Independent-samples test treating observations as unrelated

B. Paired analysis based on within-employee differences

C. Cluster sampling test only

D. A test of two unrelated population proportions

Correct Answer: B. Paired analysis based on within-employee differences

Question 9

What is the main concern when many hypotheses are tested without adjustment?

A. The sample mean must equal the population mean

B. The probability of at least one false positive increases

C. Statistical power becomes exactly zero

D. All confidence intervals become wider automatically

Correct Answer: B. The probability of at least one false positive increases

Question 10

Which statement about statistical significance is most accurate?

A. A significant result always has managerial importance.

B. A non-significant result proves that there is no effect.

C. Significance should be considered alongside effect size, precision, design, and practical consequences.

D. A p-value directly measures the financial value of an intervention.

Correct Answer: C. Significance should be considered alongside effect size, precision, design, and practical consequences.