4. Statistical Reasoning and Inference

Introduction: From Observed Data to Defensible Knowledge

Data science rarely deals with complete certainty. A dataset is usually a partial view of a larger process: a sample of customers, a set of recorded transactions, a collection of sensor readings, or the outcomes observed from a limited experiment. Statistical reasoning provides the concepts and methods required to move from these observations to statements about a broader population while making uncertainty explicit.

Descriptive statistics summarise what has been observed. Inferential statistics go further: they use probability models and sampling principles to estimate unknown quantities, evaluate claims, compare alternatives, and support decisions. This distinction is central to artificial intelligence and machine learning. A model may achieve an accuracy of 91 percent on a test set, but the important questions are whether that estimate is reliable, how it would change on new data, whether the improvement over another model is meaningful, and whether the result is practically useful.

Statistical inference does not eliminate uncertainty. Instead, it quantifies and manages it. It also makes visible several risks that can otherwise remain hidden: selection bias, measurement error, confounding, overfitting, data leakage, multiple testing, and the tendency to confuse statistical significance with practical importance. For professionals working with data, statistical reasoning is therefore not merely a collection of formulas. It is a disciplined way of asking what the data support, how strongly they support it, and what limitations remain.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Explain the roles of probability, random variables, probability distributions, sampling, and statistical inference.

2. Distinguish populations, samples, parameters, and statistics.

3. Describe common discrete and continuous probability distributions and identify suitable applications.

4. Explain sampling variability, the law of large numbers, and the central limit theorem.

5. Construct and interpret point estimates and confidence intervals.

6. Formulate and evaluate null and alternative hypotheses using appropriate statistical tests.

7. Interpret p-values, statistical power, effect sizes, and practical significance correctly.

8. Identify sources of bias, uncertainty, error, and invalid inference in data projects.

9. Apply statistical reasoning to machine learning, experimentation, business, and organisational decisions.

10. Evaluate ethical and professional responsibilities associated with statistical claims.

Probability as a Language for Uncertainty

Probability provides a formal language for reasoning about uncertain events. An event is a set of possible outcomes, while a probability assigns a value between 0 and 1 to represent the event’s plausibility. Probabilities are governed by three basic principles: they are non-negative, the probability of the entire sample space is 1, and probabilities of mutually exclusive events add together.

For events A and B, the probability that at least one occurs is expressed as P(A or B). Conditional probability, written P(A|B), describes the probability of A given that B has occurred. This concept is essential in data science because information changes uncertainty. For example, the probability that a transaction is fraudulent may be different after observing its location, amount, and device characteristics.

The multiplication rule is written as P(A and B) = P(A|B)P(B). If A and B are independent, knowledge of one does not change the probability of the other, and P(A|B) = P(A). Bayes’ theorem follows from these relationships:

P(A|B) = [P(B|A)P(A)] / P(B).

Bayesian reasoning is particularly valuable when a rare event is being detected. A highly accurate test can still produce many false positives if the event is uncommon. The prior probability, the evidence provided by the data, and the resulting posterior probability must be considered together.

A random variable assigns numerical values to outcomes. A discrete random variable takes countable values, such as the number of system failures in a day. A continuous random variable can take values over an interval, such as response time or temperature. A probability mass function describes probabilities for discrete variables, while a probability density function describes the relative concentration of probability for continuous variables. For a continuous variable, the probability of one exact value is generally zero; probabilities are obtained from areas over intervals.

The expected value of a random variable is its long-run average, and the variance measures its average squared deviation from that expected value. The standard deviation, the square root of variance, is expressed in the same units as the observations and is often easier to interpret. These quantities describe a distribution but do not fully determine its shape.

Probability Distributions Used in Data Science

A probability distribution is a model of how values or outcomes are expected to occur. Selecting a distribution requires substantive knowledge about the process, not merely a desire to fit a convenient formula.

The Bernoulli distribution describes one trial with two possible outcomes, often labelled success and failure. If the probability of success is p, the random variable has expected value p and variance p(1-p). It is appropriate for a single binary outcome, such as whether a user clicks an advertisement.

The binomial distribution describes the number of successes in a fixed number n of independent Bernoulli trials with a common success probability p. It is useful for questions such as how many of 100 independently selected users complete a purchase. Its assumptions become questionable when trials are dependent, probabilities differ substantially, or sampling is performed without replacement from a small population.

The Poisson distribution models counts of events occurring in a fixed interval when events occur independently at an approximately constant average rate. It is often used as a starting model for support requests per hour or failures per kilometre. Its mean and variance are both equal to its rate parameter. If observed variance is much greater than the mean, a Poisson model may be inadequate; this condition is called overdispersion.

The normal distribution is continuous, symmetric, and characterised by a mean and standard deviation. Its bell-shaped form is important because sums and averages of many independent contributions often approximate normality under broad conditions. However, not every variable is normally distributed, and assuming normality for heavily skewed income, waiting-time, or count data can distort inference.

The exponential distribution is commonly used to model waiting times between events in a Poisson process. It has a useful memoryless property, although real-world waiting times may not satisfy the assumptions required for this model.

The t-distribution resembles the normal distribution but has heavier tails. It is especially useful when estimating a mean from a small sample and the population standard deviation is unknown. As sample size increases, the t-distribution approaches the normal distribution.

In modern analytics, distributions are also used as components of probabilistic models. Logistic regression models the probability of a binary outcome, while generalized linear models allow different outcome distributions and link functions. Bayesian models specify prior distributions and update them with observed data. In machine learning, predictive distributions can communicate uncertainty rather than returning only a single prediction.

Sampling, Representativeness, and the Data-Generating Process

A population is the complete set of units about which a claim is intended to apply. A sample is the subset actually observed. A parameter is a numerical characteristic of the population, such as the population mean μ. A statistic is a quantity calculated from the sample, such as the sample mean x̄. Parameters are usually unknown; statistics are observable and are used to learn about parameters.

Simple random sampling gives every unit an equal and known chance of selection. Stratified sampling divides a population into relevant subgroups and samples within each subgroup, which can improve representation of small but important groups. Cluster sampling selects naturally occurring groups, such as schools or geographic regions, and may reduce cost, although observations within clusters are often correlated. Systematic sampling selects units at a regular interval after a suitable random start.

Convenience sampling, voluntary response, and uncontrolled web data may be easy to obtain but can produce serious selection bias. Increasing the sample size does not automatically correct such bias. A very large sample that systematically excludes certain people can provide a highly precise estimate of the wrong target.

Sampling variability is the natural difference between a sample statistic and the population parameter caused by random selection. It differs from sampling bias, which is a systematic tendency for the sampling process to overrepresent or underrepresent certain values. Nonresponse bias occurs when selected units do not participate and respondents differ meaningfully from nonrespondents. Measurement bias arises when the data collection instrument or procedure consistently records values inaccurately.

The law of large numbers states that, under appropriate conditions, sample averages tend to approach the population mean as the sample size increases. The central limit theorem states, broadly, that the distribution of a suitably standardised sample mean approaches a normal distribution as sample size grows, even when the underlying observations are not themselves normal. The theorem supports many confidence intervals and hypothesis tests, but its use depends on assumptions such as independence and an adequately large effective sample size.

The standard error measures the typical sampling variability of a statistic. For a sample mean based on independent observations with population standard deviation σ, the standard error is σ/√n. In practice, σ is usually replaced by the sample standard deviation. The square-root relationship has an important practical implication: reducing standard error by half generally requires approximately four times as many observations.

Estimation: Quantifying What Is Unknown

A point estimate is a single value used to estimate a population parameter. The sample mean estimates a population mean, the sample proportion estimates a population proportion, and a sample difference in means estimates a difference between population means. A good estimator is often evaluated through bias, variance, consistency, and efficiency.

An estimator is unbiased if its expected value equals the parameter it estimates. Unbiasedness alone does not guarantee usefulness. An estimator with slightly more bias but substantially lower variance may have lower overall mean squared error. Consistency means that the estimator approaches the true parameter as sample size increases under suitable conditions.

A confidence interval combines a point estimate with a margin of error. A common form is:

estimate ± critical value × standard error.

For a mean under suitable conditions, a 95 percent interval may be calculated using a t critical value. For a proportion, a normal approximation may be used when the sample contains sufficient expected successes and failures, although other intervals can perform better, particularly for small samples or proportions near 0 or 1.

The correct interpretation of a 95 percent confidence interval is procedural: if the same sampling process were repeated many times and an interval were constructed each time using the same method, approximately 95 percent of those intervals would contain the fixed population parameter. It is not strictly correct to say that there is a 95 percent probability that the parameter lies in one already-computed frequentist interval. In practice, researchers often use less formal language, but the distinction matters for rigorous reasoning.

Confidence intervals communicate both direction and precision. A narrow interval indicates high precision under the model and sampling design; it does not necessarily indicate absence of bias. A wide interval may reflect a small sample, high variability, or a costly-to-measure outcome. The desired width should be considered during study planning through sample-size and power calculations.

Prediction intervals are different from confidence intervals. A confidence interval estimates an average or other population parameter, whereas a prediction interval describes the likely range for a future individual observation. Prediction intervals are generally wider because they include both uncertainty in the estimated mean and natural variation among individual outcomes.

Hypothesis Testing as Structured Evidence Assessment

Hypothesis testing evaluates whether observed data are inconsistent with a specified null model. The null hypothesis, H₀, commonly represents no difference, no association, or a parameter equal to a reference value. The alternative hypothesis, H₁ or Hₐ, represents the effect or relationship of interest. Before examining the results, the analyst specifies the test, significance level, direction of interest, and relevant assumptions.

A test statistic measures how far the observed result is from what would be expected if H₀ were true, usually in standard-error units. The p-value is the probability, assuming H₀ and the test assumptions are true, of observing a result at least as extreme as the one obtained. It is not the probability that H₀ is true, nor is it the probability that the result occurred by chance in an unrestricted sense.

If the p-value is less than a preselected significance level α, the result is described as statistically significant and H₀ is rejected. Otherwise, the correct conclusion is generally that there is insufficient evidence to reject H₀, not that H₀ has been proven true.

A Type I error occurs when a true null hypothesis is rejected. Its long-run probability is controlled by α under the assumptions of the test. A Type II error occurs when a false null hypothesis is not rejected. Statistical power, 1 − β, is the probability of detecting a specified effect under a specified alternative. Power increases with larger samples, larger true effects, lower variability, and a less stringent significance level, although each choice has consequences.

One-sided tests are appropriate only when effects in the opposite direction would not be treated as relevant evidence and when the direction was specified in advance. Changing from a two-sided to a one-sided test after seeing the data improperly alters the evidence threshold.

Common procedures include one-sample and two-sample t-tests for means, tests for proportions, chi-square tests for categorical association, analysis of variance for comparing several means, and regression-based tests for coefficients. The choice depends on the research question, outcome type, design, dependence structure, and assumptions. Nonparametric methods or resampling methods may be preferable when distributional assumptions are doubtful, although they also require careful interpretation.

Statistical Significance, Effect Size, and Practical Importance

Statistical significance answers a narrow question about compatibility with a null model. It does not tell us whether an effect is large, useful, safe, or worth implementing. With a very large sample, a trivial effect may produce a very small p-value. With a small sample, a meaningful effect may fail to reach significance because the estimate is imprecise.

Effect size describes the magnitude of a difference or relationship. Examples include a mean difference, percentage-point change, risk ratio, odds ratio, correlation, or standardised mean difference. An effect should be interpreted in its substantive context. A 0.2 percentage-point reduction in error may be commercially valuable in a high-volume system but irrelevant in a low-stakes application; the same numerical change can have different consequences in different settings.

Confidence intervals help integrate statistical and practical reasoning. If an interval excludes zero, it may support statistical significance at the corresponding level, but its entire range should be evaluated against a minimum practically important effect. Equivalence tests and non-inferiority tests can be more appropriate than ordinary null-hypothesis tests when the aim is to establish that a difference is smaller than an acceptable margin.

Multiple comparisons create another danger. If many hypotheses are tested, some small p-values are likely to appear by chance even when all null hypotheses are true. Analysts may control the family-wise error rate using procedures such as Bonferroni adjustment or control the false discovery rate using methods such as Benjamini–Hochberg. Pre-registration, clear primary outcomes, and transparent reporting reduce selective emphasis on favourable results.

Bias, Confounding, and the Limits of Inference

Bias is systematic error that shifts estimates away from the target quantity. Confounding occurs when an observed association between an exposure and outcome is partly or wholly explained by another variable related to both. For example, a relationship between training participation and employee performance may be confounded by prior experience if experienced employees are more likely to receive training and also perform better.

Randomisation helps balance both observed and unobserved confounders, on average, and supports causal interpretation. In observational studies, analysts may use stratification, matching, regression adjustment, inverse-probability weighting, or causal diagrams, but these methods depend on assumptions. Association alone does not establish causation.

Data leakage is a particularly important issue in machine learning. It occurs when information unavailable at prediction time, or information derived from the target, enters the training process. Leakage can produce impressive validation results that collapse in deployment. The distinction between exploratory analysis and confirmatory analysis also matters: repeatedly exploring data and testing hypotheses on the same evidence can make nominal p-values misleading.

Uncertainty may arise from sampling, measurement, missing data, model specification, parameter estimation, and future changes in the data-generating process. A reported result should therefore identify the target population, data collection process, time period, assumptions, and relevant sources of uncertainty.

Statistical Reasoning in AI and Organisational Practice

In supervised learning, a test-set metric is an estimate of generalisation performance. Confidence intervals or bootstrap intervals can quantify uncertainty in accuracy, precision, recall, F1 score, calibration, or expected loss. Resampling must respect the data structure: observations from the same patient, customer, device, or time period should not be treated as independent if dependence is present.

Model comparison requires more than comparing two point estimates. Paired evaluation on the same cases can improve efficiency, while repeated cross-validation can reveal variability across splits. For time-dependent data, random shuffling may be inappropriate because it allows future information to influence the evaluation. A temporal holdout better reflects deployment.

A/B testing applies statistical inference to controlled changes such as a new interface, recommendation algorithm, or pricing message. Analysts should define the primary outcome, randomisation unit, experiment duration, minimum detectable effect, stopping rule, and guardrail metrics before the experiment begins. Peeking at results and stopping immediately after a favourable p-value can inflate the false-positive rate unless sequential methods are used.

In organisations, statistical evidence supports resource allocation, quality control, risk assessment, forecasting, policy evaluation, and operational monitoring. Managers should ask not only whether a result is significant but also whether it is robust, actionable, equitable, and aligned with strategic objectives. A statistically persuasive result can still be unsuitable for implementation if it imposes unacceptable costs, creates legal exposure, or harms a protected group.

An Illustrative Case: Evaluating a Fraud-Detection Model

Background

A financial services organisation develops a machine-learning model to identify potentially fraudulent card transactions. The existing rule-based system has a fraud detection rate of 82 percent among confirmed fraud cases. The new model is evaluated on a holdout sample collected during a later month.

Situation and Problem

The new model detects 85 percent of confirmed fraud cases. A project team claims that the model is superior and recommends immediate deployment. However, the improvement is based on a finite sample, and the new system also flags more legitimate transactions. The organisation must determine whether the improvement is reliable and worthwhile.

Analysis

The relevant estimand must first be defined. Is the organisation estimating the difference in recall on transactions from the next month, or performance across all future customers and fraud types? The sampling and time period affect generalisability. Because transactions from the same account may be correlated, treating every transaction as independent could make uncertainty appear smaller than it really is.

The team calculates the difference in recall and a confidence interval using a paired evaluation of the same labelled cases. It also reports precision, false-positive rate, expected financial loss, review workload, and performance across customer segments. The point estimate favours the new model, but the interval is wide and includes a small improvement. Thus, statistical evidence for superiority is not strong enough for an unconditional claim.

The business impact is then assessed. Suppose the model prevents additional fraud losses but creates a substantial number of unnecessary manual reviews. A cost-sensitive analysis shows that performance is beneficial only if review capacity is expanded or thresholds are adjusted. Segment analysis also identifies a higher false-positive rate for a particular customer group, requiring investigation of data quality and fairness implications.

Approach and Solution

The organisation conducts a prospective, time-based pilot with pre-specified success criteria. It uses a holdout period not used for threshold tuning, monitors both fraud loss and customer disruption, and evaluates confidence intervals for key metrics. Thresholds are selected using business costs rather than recall alone. The model is deployed with human review and a rollback plan, while drift monitoring is established for changes in transaction patterns.

Outcome

The pilot indicates a modest but credible reduction in fraud loss, although the original recall improvement is smaller than the initial estimate. After threshold adjustment and reviewer training, the system produces a positive operational outcome. Deployment is approved with continuing monitoring rather than being treated as a one-time statistical decision.

Learning Points

The case illustrates that a point estimate is not a complete result. Statistical uncertainty, dependence, measurement quality, practical costs, subgroup effects, and changing data conditions all influence whether a model should be adopted. It also demonstrates that statistical significance and organisational value are related but distinct questions.

Common Errors and Professional Safeguards

Several mistakes recur in statistical work. Analysts may interpret a p-value as the probability that a hypothesis is true, report confidence intervals as probabilities about fixed parameters, or treat non-significance as proof of no effect. They may also ignore missing data, use a convenience sample while claiming population-wide conclusions, run many tests without adjustment, or select the analysis after observing the outcome.

Other errors include confusing correlation with causation, assuming that a large dataset is automatically representative, using inappropriate standard errors for clustered observations, and relying on a normal approximation for severely skewed or sparse data. In machine learning, leakage and repeated tuning against the test set are especially damaging.

Best practice begins with a clear estimand: specify the population, outcome, comparison, time horizon, and interpretation intended. Analysts should plan sample size and power, document inclusion and exclusion rules, inspect data quality, visualise distributions, check assumptions, report effect sizes with uncertainty intervals, and distinguish confirmatory from exploratory findings. Sensitivity analyses should examine how conclusions change under plausible alternative assumptions. Reproducible code, versioned data, transparent reporting, and independent review strengthen credibility.

Ethics, Governance, and Responsible Inference

Statistical claims can affect access to credit, healthcare, employment, education, and public services. Ethical practice requires attention to consent, privacy, fairness, and the consequences of errors. An apparently accurate model may still be unacceptable if its errors are concentrated among vulnerable groups or if the data reflect historical discrimination.

Governance processes should define who is responsible for approving analyses, auditing models, documenting assumptions, and responding to harmful outcomes. Analysts should communicate uncertainty rather than presenting estimates as facts. Suppressing unfavourable outcomes, changing hypotheses after seeing results without disclosure, or presenting a technically significant result as practically decisive undermines professional integrity.

Privacy-preserving techniques, access controls, data minimisation, and careful disclosure are relevant when statistical analysis uses personal data. Statistical significance cannot legitimise unlawful or unethical data use. Responsible inference combines mathematical validity with awareness of social context and institutional accountability.

Emerging Directions

Statistical inference is increasingly integrated with automated machine learning, causal inference, probabilistic programming, and uncertainty-aware artificial intelligence. Conformal prediction provides prediction sets or intervals with distribution-free coverage guarantees under specified exchangeability conditions. Bayesian hierarchical models allow information to be partially pooled across groups, often improving estimation for small subpopulations while retaining group-level differences.

Researchers are also developing methods for adaptive experimentation, online learning, distribution shift, and selective prediction. These developments do not remove the need for foundational statistical reasoning. Instead, they make it more important: advanced methods still require clearly defined targets, credible assumptions, valid evaluation designs, and careful communication of uncertainty.

Chapter Summary

Statistical reasoning enables data scientists to move from observed samples to cautious, evidence-based conclusions about broader populations and future outcomes. Probability formalises uncertainty, while distributions model the behaviour of random variables. Sampling determines whether conclusions are representative, and sampling variability explains why estimates differ across samples. Point estimates provide a concise summary, whereas confidence intervals communicate precision and uncertainty.

Hypothesis tests evaluate data against a null model, but p-values do not measure the probability that a hypothesis is true. Type I and Type II errors, power, effect sizes, and confidence intervals must be considered together. Statistical significance is not equivalent to practical importance. Bias, confounding, measurement error, dependence, multiple testing, and data leakage can invalidate apparently strong findings. In AI and organisational applications, rigorous evaluation requires attention to generalisation, cost, fairness, operational constraints, and changing data. Responsible statistical practice is therefore both a technical and an ethical discipline.

Key Terms

Probability: A numerical representation of the likelihood of an event.

Conditional probability: The probability of an event given that another event has occurred.

Random variable: A variable whose numerical value depends on the outcome of a random process.

Probability distribution: A mathematical description of the possible values and probabilities of a random variable.

Population: The complete set of units about which an inference is intended.

Sample: A subset of a population observed for analysis.

Parameter: A numerical characteristic of a population.

Statistic: A numerical quantity calculated from sample data.

Sampling variability: Natural variation in a statistic across repeated samples.

Standard error: The typical sampling variability of an estimator.

Point estimate: A single sample-based value used to estimate a population parameter.

Confidence interval: An interval produced by a procedure designed to capture a population parameter at a stated long-run rate.

Null hypothesis: A specified claim used as a reference model in hypothesis testing.

p-value: The probability, assuming the null model and assumptions, of observing a result at least as extreme as the observed result.

Type I error: Rejection of a true null hypothesis.

Type II error: Failure to reject a false null hypothesis.

Statistical power: The probability of detecting a specified effect under a specified alternative condition.

Effect size: A quantitative measure of the magnitude of a difference or relationship.

Bias: Systematic deviation of an estimate or process from the target quantity.

Confounding: Distortion of an observed relationship by another variable related to the exposure and outcome.

Review Questions

Short Answer Questions

1. Distinguish a population parameter from a sample statistic.

2. What is the difference between sampling variability and sampling bias?

3. Explain the correct interpretation of a p-value.

4. Why can a statistically significant result be practically unimportant?

5. What is the purpose of a confidence interval?

Long Answer Questions

1. Explain the central limit theorem and discuss why it is important for statistical inference.

2. Compare the binomial, Poisson, normal, and t-distributions, including suitable applications and assumptions.

3. Describe the process of hypothesis testing, including null and alternative hypotheses, significance level, test statistic, p-value, and decision errors.

4. Discuss how selection bias, confounding, measurement error, and data leakage can affect conclusions in data science.

5. Explain how statistical inference should be adapted when evaluating a machine-learning model for organisational deployment.

Analytical / Application Questions

1. A model improves accuracy from 90.0 percent to 90.8 percent in a very large test set, but the improvement increases computational cost substantially. What additional statistical and practical evidence should be examined before deployment?

2. An online experiment produces a p-value of 0.03 after analysts checked results every few hours and stopped when significance appeared. Analyse why the reported evidence may be unreliable and propose a better design.

3. A fraud model has higher overall recall but a much higher false-positive rate for one customer group. Explain how you would investigate the result statistically, operationally, and ethically.

22. Multiple Choice Questions with Answers

Question 1

Which statement best describes a p-value?

A. The probability that the null hypothesis is true

B. The probability that the observed result occurred entirely by chance

C. The probability, assuming the null hypothesis, of obtaining a result at least as extreme as observed

D. The probability that the alternative hypothesis is practically important

Correct Answer: C. The probability, assuming the null hypothesis, of obtaining a result at least as extreme as observed

Question 2

A very large sample produces a statistically significant difference of 0.1 percentage points. What is the most appropriate interpretation?

A. The difference must be practically important

B. The difference may be statistically detectable but should be evaluated for practical importance

C. The null hypothesis has been proven false in every population

D. The result is invalid because large samples cannot be used for testing

Correct Answer: B. The difference may be statistically detectable but should be evaluated for practical importance

Question 3

Which situation most directly illustrates sampling bias?

A. Two random samples produce slightly different means

B. A survey excludes people without internet access while claiming to represent all residents

C. A sample standard deviation is used instead of a population standard deviation

D. A confidence interval is wider than expected

Correct Answer: B. A survey excludes people without internet access while claiming to represent all residents

Question 4

What does a 95 percent frequentist confidence interval mean?

A. There is a 95 percent probability that the fixed parameter lies in this specific interval

B. The parameter has a 95 percent probability of changing into the interval

C. About 95 percent of intervals from repeated use of the method would contain the parameter

D. The sample contains 95 percent of the population

Correct Answer: C. About 95 percent of intervals from repeated use of the method would contain the parameter

Question 5

Which condition is most likely to make a simple Poisson model inappropriate?

A. Events are counted over a fixed interval

B. The average event rate is approximately constant

C. The observed variance is substantially greater than the observed mean

D. The outcome is a count rather than a measurement

Correct Answer: C. The observed variance is substantially greater than the observed mean

Question 6

A Type II error occurs when an analyst:

A. Rejects a true null hypothesis

B. Fails to reject a false null hypothesis

C. Uses a one-sided test instead of a two-sided test

D. Reports an effect size with a confidence interval

Correct Answer: B. Fails to reject a false null hypothesis

Question 7

Why is randomisation valuable in a controlled experiment?

A. It guarantees a statistically significant result

B. It eliminates all measurement error

C. It tends to balance confounding factors across groups

D. It makes the sample representative of every population

Correct Answer: C. It tends to balance confounding factors across groups

Question 8

A machine-learning evaluation uses future transaction information during training. This is primarily an example of:

A. Stratified sampling

B. Data leakage

C. Statistical power

D. Practical significance

Correct Answer: B. Data leakage

Question 9

Which change generally reduces the standard error of a sample mean?

A. Increasing sample size while retaining independence

B. Introducing selection bias

C. Increasing measurement variability

D. Testing more hypotheses

Correct Answer: A. Increasing sample size while retaining independence

Question 10

An organisation compares many treatments and reports only the one with p < 0.05. What statistical problem is most relevant?

A. The law of large numbers

B. Multiple-comparison error and selective reporting

C. The memoryless property

D. Underfitting caused by a small effect size

Correct Answer: B. Multiple-comparison error and selective reporting