Statistical Inference and Evidence-Based Conclusions

Introduction

Data analysis is not complete when a table, graph, or descriptive statistic has been produced. In professional practice, analysts usually want to make statements beyond the observed dataset: whether a model improves performance, whether a treatment changes outcomes, whether two groups differ, or whether an observed association is likely to reflect a broader pattern. Statistical inference provides a disciplined framework for making such conclusions while explicitly recognising uncertainty.

Inference is especially important in artificial intelligence and machine learning. A model may perform better on one test sample because of random variation. A feature may appear associated with a target because of confounding or data leakage. A fairness difference may be estimated imprecisely when a protected group is small. Statistical inference helps analysts distinguish the strength of observed evidence from the size and practical importance of an effect.

This chapter develops the concepts of populations, samples, sampling distributions, confidence intervals, statistical significance, effect size, common hypothesis tests, and the main threats to valid interpretation. The central principle is that a statistical result is evidence, not automatic proof. Sound conclusions require appropriate design, valid assumptions, transparent reporting, and awareness of what the data cannot establish.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish a population, sample, parameter, and statistic;
• explain sampling variability and sampling distributions;
• construct and interpret confidence intervals;
• distinguish statistical significance from practical significance;
• calculate and interpret common effect sizes;
• select and interpret common hypothesis tests;
• explain Type I error, Type II error, power, p-values, and multiple testing;
• recognise sampling bias, measurement bias, confounding, dependence, and data leakage;
• distinguish association from causation;
• communicate evidence-based conclusions responsibly in data-analysis and AI projects.

1. Populations, samples, parameters, and statistics

A population is the complete set of units about which a study seeks to draw conclusions. It may be concrete, such as all loan applications received by a bank during a year, or conceptual, such as all future images that a computer-vision system may process under specified conditions.

A sample is the subset of units actually observed. A parameter is a numerical characteristic of the population, such as the population mean μ or population proportion p. A statistic is a numerical quantity calculated from a sample, such as the sample mean x̄ or sample proportion p̂. Because parameters are usually unknown, statistics are used to estimate them.

A sample can support generalisation only when it represents the target population sufficiently well. Random sampling is one important method, but representativeness also depends on coverage, response rates, measurement quality, and the time and conditions under which observations are collected.

In machine learning, a similar distinction exists between a training, validation, and test sample and the broader population of future cases. A test score estimates generalisation performance; it is not necessarily the exact performance the system will achieve after deployment.

2. Sampling distributions and the central limit idea

If repeated samples of the same size were drawn from a population, each sample would produce a possibly different statistic. The probability distribution of that statistic is its sampling distribution. It describes sampling variability, not the distribution of individual observations.

For an independent random sample of size n from a population with mean μ and variance σ², the sample mean satisfies:

Here, E(x̄) is the expected sample mean, Var(x̄) is its variance, and SE(x̄) is its standard error. If σ is unknown, it is commonly estimated by the sample standard deviation s, giving SE(x̄) = s/√n. The formula assumes independent observations from the relevant population and is most directly applicable to simple random sampling. For sampling without replacement from a finite population, a finite-population correction may be needed when the sample is a substantial fraction of the population.

The central limit theorem states, broadly, that under suitable conditions the sampling distribution of the mean becomes approximately normal as sample size increases, even when individual observations are not normally distributed. The theorem does not say that every small sample is normal, nor does it remove the effects of severe outliers, dependence, or biased sampling.

For a sample proportion p̂ from independent Bernoulli observations, the approximate standard error is:

This normal approximation is generally more reliable when the expected numbers of successes and failures are sufficiently large. For small samples or proportions near 0 or 1, exact or alternative interval methods may be preferable.

Larger samples usually reduce standard error at a rate proportional to 1/√n. Thus, quadrupling the sample size approximately halves the standard error, all else equal. Larger samples do not automatically eliminate systematic bias.

3. Confidence intervals

A confidence interval combines a point estimate with a margin of error. A general form is:

For a mean with unknown population standard deviation, a commonly used interval is:

where x̄ is the sample mean, s is the sample standard deviation, n is the sample size, and t* is the critical value from the t distribution with n − 1 degrees of freedom. This interval is appropriate when observations are independent and the mean is reasonably modelled by the t-based procedure. With large samples, the t critical value approaches the standard normal critical value.

For a 95% confidence interval, the long-run procedure is designed so that, under repeated valid sampling, approximately 95% of intervals constructed in this way contain the true parameter. It is not technically correct to say that there is a 95% probability that a fixed parameter lies inside one already-calculated frequentist interval. In practical communication, however, it is common to say that the analyst is 95% confident because the method has 95% long-run coverage under its assumptions.

Worked example: mean performance

Suppose the accuracies of 16 independently evaluated models have mean x̄ = 82%, with sample standard deviation s = 4 percentage points. For a 95% interval, the t critical value with 15 degrees of freedom is approximately 2.131.

The estimated mean accuracy is 82%, and the interval expresses uncertainty in the population mean under the sampling and modelling assumptions. It does not imply that 95% of individual future model accuracies lie between these limits.

For a difference between two independent means, an important practical form is:

This is the Welch interval, which does not require the two population variances to be equal. Here, subscripts 1 and 2 identify the groups. For paired observations, the analysis should instead be performed on within-pair differences.

Confidence intervals are often more informative than p-values because they show the direction, plausible magnitude, and precision of an effect. A narrow interval indicates greater precision; it does not by itself establish that the estimate is unbiased.

4. Hypothesis testing and statistical significance

A hypothesis test compares observed data with a specified null model. The null hypothesis, H₀, commonly represents no difference, no association, or a particular reference value. The alternative hypothesis, H₁ or Hₐ, represents the effect or relationship of interest.

A test statistic measures how far the observed result is from what H₀ predicts, using an appropriate standard-error scale. The p-value is the probability, assuming H₀ and the test assumptions are true, of obtaining a result at least as extreme as the observed result in the direction defined by the test.

A p-value is not:

• the probability that H₀ is true;
• the probability that the result occurred purely by chance;
• a measure of effect size;
• a guarantee that the finding will replicate;
• evidence that the result is practically important.

A preselected significance level α, often 0.05, defines a decision rule. If p ≤ α, the analyst rejects H₀ under that rule; otherwise, the analyst does not reject H₀. “Fail to reject” is not the same as proving H₀ true.

Two errors are central. A Type I error rejects a true null hypothesis, with probability controlled approximately by α under the assumptions. A Type II error fails to detect a false null hypothesis. Statistical power is the probability of detecting a specified effect under specified conditions:

where β is the Type II error probability. Power depends on effect size, sample size, variability, significance level, and the alternative hypothesis. A non-significant result may reflect no meaningful effect, inadequate power, imprecise measurement, or a poorly chosen analysis.

5. Effect size and practical significance

Statistical significance is influenced strongly by sample size. A very small effect can produce a small p-value in a huge dataset, while a meaningful effect may be non-significant in a small dataset. Therefore, reports should include an effect estimate and confidence interval as well as a p-value.

For two independent groups, Cohen’s d is a standardised mean difference:

where the pooled standard deviation is:

This pooled form assumes comparable population variances and independent groups. When those assumptions are inappropriate, an unstandardised difference or another standardised measure may be preferable. Conventional labels such as 0.2, 0.5, and 0.8 for small, medium, and large effects are rough guidelines, not universal standards.

For proportions, a risk difference is p₁ − p₂, a relative risk is p₁/p₂ when p₂ is non-zero, and an odds ratio is:

These measures answer different questions and should not be treated as interchangeable. In rare-outcome settings, odds ratios may approximate relative risks, but that approximation can fail when outcomes are common.

A useful professional question is not merely “Is the result significant?” but “Is the estimated effect large enough to matter for users, cost, safety, fairness, or operational decisions?” Practical significance must be evaluated in context.

6. Common hypothesis tests

The correct test depends on the outcome type, study design, sampling process, independence structure, and assumptions. A test selected only because it is familiar can produce misleading evidence.

One-sample mean test

A one-sample t test compares a sample mean with a reference value μ₀:

Under H₀ and suitable independence and distributional conditions, this statistic follows a t distribution with n − 1 degrees of freedom. The test may be two-sided, testing any difference, or one-sided, testing a pre-specified direction. A one-sided test should not be selected after seeing the data merely to obtain a smaller p-value.

Two independent means

To compare two independent groups, Welch’s t test is generally preferred when equal variances cannot reasonably be assumed. It tests a difference in means and should be accompanied by a confidence interval and an effect estimate. Independence means that one observation does not provide information about another in a way ignored by the analysis. Repeated measurements from the same person or data points from the same device violate simple independence.

Paired mean comparison

When each unit is measured twice, or when observations are deliberately matched, define differences dᵢ = xᵢ,after − xᵢ,before. A paired t test analyses the mean difference:

where d̄ is the mean difference and s_d is its sample standard deviation. Pairing can improve precision because between-unit variation is removed, but the pairs must be meaningfully formed.

Categorical data: chi-square test

A chi-square test of independence examines whether two categorical variables are associated. For each cell, the expected count under independence is:

The test statistic is:

where Oᵢⱼ and Eᵢⱼ are observed and expected counts. For an r × c table, the degrees of freedom are (r − 1)(c − 1). The approximation is unreliable when expected counts are very small; Fisher’s exact test or a carefully justified alternative may be appropriate for small tables.

A significant chi-square result indicates evidence of association, not causation. Effect measures such as risk difference, relative risk, odds ratio, or Cramér’s V should be considered.

Correlation

Pearson’s correlation measures linear association between paired quantitative variables:

Its value lies between −1 and +1. Correlation can be distorted by outliers, nonlinear patterns, restricted ranges, and confounding. A scatterplot should accompany interpretation. Spearman’s rank correlation may be useful for monotonic relationships or ordinal data, but it also does not establish causation.

Non-parametric alternatives

When assumptions for mean-based tests are seriously inappropriate, analysts may use procedures such as the Mann–Whitney test for independent groups or the Wilcoxon signed-rank test for paired data. These tests are not simply “tests of medians” in every situation; their interpretation depends on distributional shape and the quantity being tested. Resampling methods, permutation tests, and bootstrap intervals can provide flexible alternatives when justified by the design.

7. Bias, confounding, and dependence

Sampling bias occurs when the observed sample systematically differs from the target population. Examples include undercoverage, voluntary-response bias, and non-response bias. Increasing the sample size does not correct a biased sampling process.

Measurement bias arises when variables are measured inaccurately or differently across groups. In AI projects, labels may reflect human disagreement, historical prejudice, or different measurement quality across populations. Misclassification can weaken, reverse, or create associations.

Confounding occurs when a third variable is related to both an exposure and an outcome and creates a misleading association. For example, an analysis may find that users of an advanced analytics tool have higher productivity. Prior technical training could influence both tool adoption and productivity, confounding the comparison.

Confounding can be addressed through randomisation, restriction, matching, stratification, regression adjustment, or causal methods, but each requires assumptions. Adjustment is not automatically beneficial: controlling for a mediator or collider can introduce bias. A causal diagram or domain-informed analysis plan can help identify which variables should be measured and adjusted.

Dependence occurs when observations are related, such as repeated measurements, students within schools, transactions from the same customer, or images from the same subject. Treating dependent observations as independent often underestimates standard errors and produces excessive statistical significance. Clustered, mixed-effects, time-series, or repeated-measures methods may be required.

In machine learning, data leakage is a related threat. Leakage occurs when information unavailable at prediction time enters training or evaluation. It can produce apparently excellent performance that fails in deployment. Splitting data at the correct unit and time level is essential; for example, all records from one patient may need to remain in one partition.

8. Multiple testing and researcher degrees of freedom

If many hypotheses are tested, some small p-values are likely to occur by chance even when all null hypotheses are true. With m independent tests, the probability of at least one false positive at level α is:

For α = 0.05 and m = 20, this probability is approximately 1 − 0.95²⁰ ≈ 0.642, under independence. Tests are often not independent, but the example demonstrates the issue.

The family-wise error rate is the probability of at least one Type I error in a family of tests. Bonferroni adjustment uses α/m as the per-test threshold and is simple but conservative. The false discovery rate controls the expected proportion of false discoveries among rejected hypotheses; procedures such as Benjamini–Hochberg are often useful when many exploratory tests are conducted.

Analysts should pre-specify primary outcomes, report the number of tests performed, distinguish confirmatory from exploratory analyses, and avoid changing hypotheses after inspecting results. Selective reporting, repeated analysis choices, and stopping data collection when significance appears can inflate false-positive findings.

9. From association to causal conclusions

An association means that variables vary together in the observed data. A causal claim says that changing one variable would change an outcome, usually under a defined intervention and target population. Causal interpretation requires more than a low p-value.

Randomised experiments support causal inference because random assignment tends, in expectation, to balance measured and unmeasured confounders. Even experiments require attention to non-compliance, attrition, interference, treatment fidelity, and generalisability.

Observational studies can contribute valuable evidence, but causal interpretation requires explicit assumptions about confounding, measurement, selection, timing, and interference. Temporal order matters: a proposed cause must precede its effect. Reverse causation and common causes must be considered.

A useful evidence-based conclusion identifies:

1. the population and data source;
2. the estimand or quantity being estimated;
3. the design and analysis method;
4. the estimate, uncertainty interval, and practical meaning;
5. important assumptions and possible biases;
6. whether the result supports association, prediction, or causation.

A predictive model can be accurate without identifying causes. Conversely, a causal factor may be a poor predictor if it is difficult to measure or if other variables explain most individual variation.

10. Case study: evaluating an AI-assisted support tool

This is an illustrative case. A customer-service organisation introduces an AI-assisted response tool. Managers compare the average resolution time of 40 agents using the tool with that of 40 agents using the existing process. The tool group has a mean resolution time of 18.4 minutes, while the comparison group has a mean of 20.1 minutes. The observed difference is −1.7 minutes.

Problem

Management wants to know whether the tool improves efficiency and whether it should be deployed widely.

Analysis

A simple independent-groups comparison may be inappropriate if agents self-select into tool use. More experienced agents may be more willing to adopt it, and complex cases may be assigned differently. These factors could confound the comparison.

The analyst first defines the estimand: the average difference in resolution time that would result if comparable agents handled comparable cases under the two processes. The data are checked for repeated cases, extreme durations, missing values, and differences in case complexity. If agents handle many cases, the observations are clustered by agent; a model that accounts for agent-level dependence is needed.

Suppose a confidence interval for the adjusted mean difference is (−3.0, −0.4) minutes and the p-value is 0.012. The result is statistically significant at the 0.05 level, and the interval suggests a plausible reduction of 0.4 to 3.0 minutes. However, statistical significance alone does not establish that the tool caused the reduction. The analysis may still be affected by unmeasured complexity or adoption bias.

The organisation should also examine customer satisfaction, error rates, escalation rates, workload, and performance across language groups. A faster response that increases errors would not necessarily be beneficial. If many outcomes and subgroups were tested, the report should disclose the testing plan and account for multiplicity.

Approach and solution

A stronger evaluation would randomise comparable cases or agents to tool and control conditions, define primary and secondary outcomes in advance, and measure both efficiency and quality. A stepped-wedge or crossover design may be considered if operational constraints make individual randomisation difficult. Evaluation data should be separated from model-development data to prevent leakage.

Outcome and learning points

The illustrative evidence supports an association between tool use and shorter resolution time, with an estimated effect that may be operationally meaningful. It does not, by itself, prove a causal effect or justify universal deployment. A responsible decision would combine the interval estimate with study design, quality outcomes, fairness checks, implementation cost, and a follow-up controlled evaluation.

11. Best practices for professional reporting

A high-quality inference report should:

• describe the target population, sampling frame, inclusion criteria, and missing-data treatment;
• define outcomes and comparisons before analysis when possible;
• report sample sizes, estimates, confidence intervals, and appropriate effect sizes;
• state the test used and its assumptions;
• show diagnostic plots or checks for outliers, nonlinearity, dependence, and model fit;
• distinguish exploratory findings from confirmatory findings;
• report all relevant tests and subgroup analyses, not only favourable results;
• explain practical importance in domain units;
• avoid causal language when the design supports only association;
• document data provenance, versioning, code, and reproducibility procedures;
• assess privacy, fairness, and possible harms when results affect people.

Statistical software can calculate a test statistic and p-value, but it cannot decide whether the sample represents the population, whether the outcome was measured validly, or whether a causal claim is justified. Human and domain judgement remain essential.

Challenges and limitations

Inference is limited by the quality of the data-generating process. A narrow confidence interval around a biased estimate can create false confidence. A well-designed study can still be underpowered or suffer from missing data. Statistical assumptions are approximations, and robustness depends on the context.

In deployed AI systems, populations may drift over time. A confidence interval based on historical data may not describe future performance if user behaviour, sensors, policies, or prevalence changes. Fairness comparisons can also be unstable when subgroup samples are small. Privacy-preserving procedures may reduce available detail, while repeated monitoring creates additional multiple-testing issues.

Ethically, analysts should avoid presenting uncertain findings as facts, should disclose limitations that could affect decisions, and should consider who bears the risks of false positives and false negatives. In high-impact settings such as healthcare, employment, credit, and education, statistical evidence should be combined with governance, legal requirements, human oversight, and impact assessment.

Chapter Summary

Statistical inference uses sample data to learn about populations while accounting for uncertainty. Parameters describe populations; statistics describe samples. Sampling distributions explain why estimates vary, and standard errors quantify typical sampling variation. Confidence intervals combine estimates with uncertainty and should be interpreted through their repeated-sampling coverage properties.

Hypothesis tests compare data with a null model. A p-value measures the extremeness of data under that model; it is not the probability that the null hypothesis is true and is not a measure of practical importance. Effect sizes and confidence intervals are necessary for evaluating magnitude and precision. Common procedures include one-sample, independent-groups, paired, chi-square, correlation, resampling, and non-parametric methods, each with design and assumption requirements.

Valid conclusions require attention to sampling bias, measurement bias, confounding, dependence, data leakage, and multiple testing. Association does not establish causation, and prediction does not automatically identify causal mechanisms. Responsible analysts report the population, estimand, design, estimate, uncertainty, assumptions, limitations, and practical implications. In AI applications, inference must also address data drift, fairness, privacy, reproducibility, and the consequences of error.

Key Terms

Population: The complete target set about which conclusions are intended.

Sample: The observed subset drawn or obtained from a population.

Parameter: A numerical characteristic of a population.

Statistic: A numerical quantity calculated from a sample.

Sampling distribution: The distribution of a statistic across repeated samples.

Standard error: An estimate of the sampling variability of a statistic.

Confidence interval: An interval produced by a procedure designed to cover a population parameter at a stated long-run rate.

Null hypothesis: A reference claim used in hypothesis testing.

Alternative hypothesis: A claim representing a difference, association, or effect of interest.

p-value: The probability, under the null model and assumptions, of data at least as extreme as those observed.

Statistical significance: A decision or result judged unlikely under a chosen null-model threshold.

Effect size: A measure of the magnitude of a difference or relationship.

Power: The probability of detecting a specified effect under specified conditions.

Type I error: Rejecting a true null hypothesis.

Type II error: Failing to reject a false null hypothesis.

Confounding: Distortion of an exposure–outcome relationship by another variable.

Sampling bias: Systematic difference between the observed sample and target population.

Data leakage: Use of information in model development or evaluation that would not be available at prediction time.

Family-wise error rate: The probability of at least one false rejection in a family of tests.

Causal inference: Reasoning about the effect of an intervention or exposure on an outcome.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.
2. What does a standard error measure?
3. Why is a p-value not the probability that the null hypothesis is true?
4. State one difference between statistical significance and practical significance.
5. Give two examples of confounding or sampling bias in an AI project.

Long Answer Questions

1. Explain sampling distributions and the central limit theorem, including the conditions under which the standard-error formula for a sample mean is appropriate.
2. Describe how to construct and interpret a confidence interval for a mean difference.
3. Compare independent-groups, paired, chi-square, and correlation procedures, including when each may be appropriate.
4. Explain Type I error, Type II error, power, and the effect of sample size on inference.
5. Discuss why an observed association in observational data does not necessarily support a causal conclusion.

Analytical / Application Questions

1. An AI classifier has an accuracy of 91% on a test sample of 100 cases. Identify at least four additional questions that must be answered before generalising this result to deployment.
2. A company tests 30 possible features for association with customer churn and reports only the three significant results. Explain the inferential problem and propose a more responsible analysis strategy.
3. An evaluation finds that an AI tool reduces processing time but increases error rates for one demographic group. Design an evidence-based evaluation and recommendation, including effect measures, uncertainty, fairness considerations, and limitations.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a sampling distribution?

A. The distribution of all observations in the population

B. The distribution of a statistic across repeated samples

C. The distribution of measurement errors in one sample

D. The distribution of population parameters across studies

Correct Answer: B. The distribution of a statistic across repeated samples

Question 2

For an independent sample mean with known population standard deviation σ and sample size n, which is the standard error?

A. σ × n

B. σ/n

C. σ/√n

D. √n/σ

Correct Answer: C. σ/√n

Question 3

What does a 95% frequentist confidence interval mean?

A. There is a 95% probability that the fixed parameter lies in this calculated interval

B. 95% of individual observations lie within the interval

C. The procedure produces intervals containing the parameter in approximately 95% of repeated valid samples

D. The null hypothesis has a 95% probability of being true

Correct Answer: C. The procedure produces intervals containing the parameter in approximately 95% of repeated valid samples

Question 4

A study reports p = 0.003. Which interpretation is correct?

A. There is a 0.3% probability that the null hypothesis is true

B. Assuming the null model and assumptions, data at least this extreme have probability 0.003

C. The effect is necessarily large and practically important

D. The finding will replicate with probability 0.997

Correct Answer: B. Assuming the null model and assumptions, data at least this extreme have probability 0.003

Question 5

Which measure is most directly a standardised difference between two independent group means?

A. Cohen’s d

B. Pearson’s r

C. Relative risk

D. Cramér’s V

Correct Answer: A. Cohen’s d

Question 6

Which situation most clearly calls for a paired analysis?

A. Comparing two unrelated groups of customers

B. Comparing two categorical variables in a contingency table

C. Comparing the same participants before and after an intervention

D. Comparing two independent population proportions

Correct Answer: C. Comparing the same participants before and after an intervention

Question 7

What is a major consequence of treating clustered observations as independent?

A. Standard errors may be underestimated

B. All effect sizes become exactly zero

C. The sample mean becomes undefined

D. Confounding is automatically removed

Correct Answer: A. Standard errors may be underestimated

Question 8

Why is multiple testing a concern?

A. Testing more hypotheses guarantees that every effect is real

B. Testing many hypotheses increases the chance of at least one false positive

C. Multiple testing makes confidence intervals impossible to calculate

D. Multiple testing eliminates sampling variability

Correct Answer: B. Testing many hypotheses increases the chance of at least one false positive

Question 9

Which statement best distinguishes association from causation?

A. A statistically significant association always proves causation

B. Causation can be established from correlation alone

C. A causal claim concerns the outcome change under an intervention, requiring stronger design or assumptions

D. Association and causation are identical when the sample is large

Correct Answer: C. A causal claim concerns the outcome change under an intervention, requiring stronger design or assumptions

Question 10

What is data leakage in machine learning evaluation?

A. Random variation caused by a small test set

B. Using information during development or evaluation that would not be available at prediction time

C. Removing duplicate records before model training

D. Using a confidence interval to report model performance

Correct Answer: B. Using information during development or evaluation that would not be available at prediction time