Statistics and Probability for Data Science

Introduction

Data science turns observations into evidence for decisions. Statistics provides the language for describing data, quantifying uncertainty, evaluating claims, and building predictive relationships. Probability supplies a formal model for uncertainty: it allows a data scientist to reason about events that have not yet been observed and to connect samples with populations.

These ideas are central to machine learning. A classification model estimates probabilities, a regression model describes conditional relationships, cross-validation uses repeated sampling, and an A/B test evaluates whether an observed difference is likely to reflect a real intervention rather than random variation. Sound statistical practice also requires distinguishing statistical significance from practical importance. A very small effect may be statistically significant in a huge dataset but irrelevant to a business decision; a valuable effect may fail to achieve significance in a small or noisy sample.

This chapter develops a practical foundation for descriptive statistics, probability distributions, sampling, confidence intervals, hypothesis testing, correlation, regression, Bayesian reasoning, and experimental design. Python is used as an implementation context, but the emphasis is on interpretation and responsible reasoning rather than memorising library commands.

Learning objectives

By the end of this chapter, you should be able to:

1. Summarise data using appropriate numerical and graphical techniques.
2. Explain probability, conditional probability, independence, and Bayes’ theorem.
3. Distinguish common discrete and continuous probability distributions.
4. Explain sampling variability, bias, standard error, and the central limit theorem.
5. Construct and interpret confidence intervals and hypothesis tests.
6. Interpret correlation and simple linear regression without confusing association with causation.
7. Apply Bayesian reasoning to diagnostic and classification problems.
8. Design experiments that support credible causal conclusions.
9. Distinguish statistical significance, effect size, uncertainty, and practical importance.
10. Recognise ethical, methodological, and computational limitations in statistical analysis.

1. Data, variables, and measurement

A population is the complete set of units about which a question is asked. A sample is the subset actually observed. A parameter is a numerical property of a population, such as the population mean μ. A statistic is a quantity calculated from a sample, such as the sample mean x̄.

Variables may be:

• Categorical, such as payment method or diagnosis.
• Discrete, taking countable values, such as number of defects.
• Continuous, taking values on an interval, such as temperature or response time.
• Ordinal, where categories have order but unequal differences, such as satisfaction ratings.

Measurement scale affects the valid analysis. A mean is meaningful for income and temperature measured in degrees Celsius, but usually not for unordered labels such as department names. Missing values, duplicated records, inconsistent units, outliers, and data-entry errors should be investigated before modelling. Removing observations solely because they are inconvenient can introduce bias.

2. Descriptive statistics

Descriptive statistics organise observed data; they do not automatically generalise beyond the data collected.

2.1 Centre and spread

For observations x₁, x₂, …, xₙ, the arithmetic mean is:

Here, n is the number of observations. The mean uses every value and is sensitive to extreme observations. The median is the middle ordered observation, or the average of the two middle observations when n is even. The mode is the most frequent value and is particularly useful for categorical variables.

The range is maximum minus minimum. The sample variance is:

The sample standard deviation is s = √s². The denominator n − 1, called the degrees-of-freedom correction, gives an unbiased estimator of population variance under independent random sampling. If the data represent the entire population rather than a sample, population variance uses denominator n and is conventionally written σ².

The interquartile range is IQR = Q₃ − Q₁, where Q₁ and Q₃ are the first and third quartiles. It describes the middle 50% of observations and is less sensitive to outliers than the standard deviation. The coefficient of variation is:

It is dimensionless and can compare relative variability only when the mean is meaningful and not near zero, usually for ratio-scale measurements with a positive mean.

Worked example: robust and non-robust summaries

Suppose five delivery times, in minutes, are 8, 9, 10, 11, and 32. The mean is 14 minutes, while the median is 10 minutes. The value 32 may represent an exceptional delay or a data error. The mean accurately describes the arithmetic average of these five observations, but the median better represents a typical delivery if the operational question concerns the usual customer experience. A responsible analyst investigates the cause of 32 rather than automatically deleting it.

2.2 Distribution shape and visualisation

A histogram displays the distribution of a quantitative variable using intervals. A box plot shows the median, quartiles, and potential extreme values. A scatter plot shows the relationship between two quantitative variables. Bar charts are appropriate for category counts, while line charts are often appropriate for ordered time observations.

Skewness describes asymmetry. A right-skewed income distribution often has a mean above its median. Bimodality may indicate two subpopulations, such as two customer segments. Visualisation should include axis labels, units, sample size where relevant, and honest scales. Truncated axes can exaggerate differences, whereas inappropriate bin widths can conceal structure.

3. Probability and random variables

A probability model assigns values from 0 to 1 to events. For an event A, P(A) is its probability, P(Aᶜ) = 1 − P(A) is the probability that A does not occur, and P(A ∪ B) is the probability that at least one of A or B occurs. The addition rule is:

The subtraction prevents the overlap from being counted twice. Conditional probability is:

Events A and B are independent when P(A | B) = P(A), equivalently P(A ∩ B) = P(A)P(B). Mutually exclusive events cannot occur together, so P(A ∩ B) = 0; except in a degenerate case, mutually exclusive events are not independent.

A random variable assigns numerical values to outcomes. Its expected value is a probability-weighted average:

for a discrete variable. Variance is E[(X − μ)²], where μ = E(X), and measures dispersion around the expected value. A continuous variable is described by a density; probabilities are areas under the density, and the probability of any exact single value is ordinarily zero.

4. Probability distributions

A distribution is a model, not a guarantee that observed data will match it exactly. Its suitability depends on the data-generating process and assumptions.

For a binomial random variable X with n independent trials and success probability p:

Its mean is np and variance is np(1 − p). The binomial model is inappropriate when trials are dependent or when p changes materially between trials.

A Poisson model has probability:

where λ is the expected count for the specified interval. A key Poisson property is that mean and variance are both λ. Overdispersion—variance substantially greater than the mean—may indicate heterogeneity or dependence and require another model.

A normal random variable has density:

The standardised value, or z-score, is z = (x − μ)/σ. The normal model is symmetric and continuous; it should not be applied mechanically to strongly skewed or bounded data.

5. Sampling and the central limit theorem

Sampling error is variation caused by observing a sample rather than the entire population. Sampling bias is systematic error caused by the way observations are selected. Voluntary-response surveys, undercoverage, nonresponse, and convenience samples can all produce biased estimates even when the sample is large.

Random sampling gives units a known chance of selection. Stratified sampling samples within important subgroups and can improve precision. Cluster sampling selects groups such as schools or branches and is often cheaper, but observations within clusters may be correlated. A sampling frame should be examined for exclusions and duplication.

The standard error of the sample mean, under independent observations with finite variance, is:

Since σ is usually unknown, it is estimated by s, giving s/√n. The formula shows that quadrupling sample size approximately halves standard error. It does not remove systematic bias.

The central limit theorem states, under suitable conditions such as independent observations and finite variance, that the sampling distribution of the mean becomes approximately normal as n increases, even when individual observations are not normal. The approximation may require a larger n for highly skewed or heavy-tailed populations. Dependence and extreme observations can invalidate simple applications.

6. Confidence intervals

A confidence interval estimates a population quantity while representing sampling uncertainty. For a population mean with unknown σ, a common interval is:

Here tₐ/₂,n−1 is the critical value from the t distribution, α is the total error probability, and n − 1 is the degrees of freedom. A 95% confidence procedure produces intervals that contain the true parameter in approximately 95% of repeated samples, assuming the model and sampling procedure are appropriate. It is not technically correct to say that the fixed parameter has a 95% probability of lying inside this already computed interval in the frequentist interpretation.

For a large-sample proportion p̂, an approximate interval is p̂ ± zₐ/₂√[p̂(1 − p̂)/n]. This approximation can be poor for small samples or proportions near 0 or 1; Wilson or exact methods may be preferable.

Worked example: a mean interval

A random sample of 25 response times has x̄ = 42 seconds and s = 10 seconds. Using t₀.₀₂₅,₂₄ ≈ 2.064, the standard error is 10/√25 = 2 seconds. The margin of error is 2.064 × 2 = 4.128 seconds. The 95% interval is approximately (37.872, 46.128) seconds. This supports plausible values for the population mean under the stated assumptions; it does not describe the range of individual response times.

7. Hypothesis testing and statistical significance

A hypothesis test begins with a null hypothesis H₀ and an alternative hypothesis H₁. The null commonly represents no difference or a specified parameter value. A test statistic measures how far the observed result is from what H₀ predicts. The p-value is the probability, assuming H₀ and the test assumptions are true, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an unrestricted sense.

A significance level α is chosen before examining results. If p ≤ α, the analyst rejects H₀; otherwise, the analyst fails to reject H₀. Failure to reject is not proof that H₀ is true. A Type I error rejects a true H₀; a Type II error fails to reject a false H₀. Statistical power is P(reject H₀ | H₁ is true), and tends to increase with effect size, sample size, and measurement precision.

For comparing two independent means, a Welch t-test is often preferable to the equal-variance version when group variances may differ. For paired measurements, differences within each pair should be analysed. Categorical outcomes may be assessed using chi-square methods, but expected-count conditions and study design matter. Multiple testing increases the chance of false positives; corrections such as family-wise error control or false discovery rate procedures may be needed.

Statistical significance versus practical importance

A result should be reported with an effect estimate, confidence interval, sample size, and relevant units—not only a p-value. For example, a new interface that reduces average task time by 0.2 seconds may produce p < 0.001 in millions of trials but have no meaningful operational benefit. Conversely, a 15% reduction in hospital readmissions may be practically important even when a small pilot yields p = 0.08. Decisions should consider a minimally important effect, cost, risk, uncertainty, and consequences of error.

8. Correlation and regression

The Pearson correlation coefficient measures the strength and direction of a linear association:

It lies between −1 and 1. Correlation is dimensionless and symmetric: correlation of X with Y equals correlation of Y with X. It is sensitive to outliers and can be misleading for nonlinear relationships, restricted ranges, or mixtures of groups. Correlation does not establish causation; confounding, reverse causation, and selection effects are possible.

Simple linear regression models the conditional mean of Y given X:

Here β₀ is the intercept, β₁ is the expected change in Y for a one-unit increase in X, and εᵢ is an error term. Ordinary least squares estimates coefficients by minimising the sum of squared residuals:

The coefficient of determination is R² = 1 − SSE/SST, where SST = ∑(yᵢ − ȳ)². In a model with an intercept, R² is the proportion of observed variation around ȳ explained by the fitted linear relationship in the sample. It is not a causal measure and does not guarantee good predictions outside the observed range.

Regression inference commonly assumes linearity in the conditional mean, independent errors, constant error variance, and approximately normal errors for small-sample interval tests. Diagnostics include residual-versus-fitted plots, leverage, influence, and out-of-sample validation. Prediction intervals for individual future observations are wider than confidence intervals for the mean response because they include individual-level noise.

9. Bayesian reasoning

Bayesian analysis updates prior beliefs using observed evidence. Bayes’ theorem is:

H is a hypothesis and E is evidence. P(H) is the prior probability, P(E | H) is the likelihood, P(E) is the evidence probability, and P(H | E) is the posterior probability.

A diagnostic example illustrates base-rate effects. Suppose disease prevalence is P(D) = 0.01, sensitivity P(+ | D) = 0.90, and false-positive probability P(+ | Dᶜ) = 0.05. Then:

P(+) = 0.90(0.01) + 0.05(0.99) = 0.0585.

Therefore:

P(D | +) = 0.90(0.01)/0.0585 ≈ 0.154.

Thus, a positive result implies about a 15.4% probability of disease under this model, not 90%. Bayesian reasoning is valuable in medical diagnosis, fraud detection, spam filtering, and model updating. Priors should be justified transparently, and sensitivity analysis should examine whether conclusions change under plausible alternative priors.

10. Experimental design and causal reasoning

An observational study records exposure and outcome without assigning the exposure. An experiment assigns an intervention and can support causal inference when well designed. Randomisation balances measured and unmeasured factors on average. A control group provides a comparison, blinding can reduce expectation effects, and preregistration reduces outcome switching and selective reporting.

A basic parallel A/B test randomly assigns users to control or treatment. The primary outcome, analysis population, sample-size rationale, stopping rule, and duration should be specified before looking at results. Randomisation must occur at the level where interference is plausible: if users influence one another, individual randomisation may underestimate uncertainty and cluster randomisation may be needed.

Important threats include confounding, selection bias, attrition, contamination, spillover, seasonality, novelty effects, and measurement changes. Blocking or stratification can improve balance; factorial designs study multiple interventions and interactions; crossover designs can be efficient when treatment effects are reversible and carryover is controlled.

Ethical design requires informed consent where applicable, minimisation of risk, privacy protection, equitable selection, and independent review for regulated research. In product experimentation, withholding a beneficial service may create ethical concerns, especially in safety-critical settings. Personal data should be collected only for legitimate purposes, protected through access controls and minimisation, and analysed in accordance with applicable law and organisational governance.

11. Python workflow for statistical analysis

A reproducible Python workflow generally includes:

1. Define the question, target population, estimand, and practical decision threshold.
2. Inspect data types, missingness, units, duplicates, and extreme values.
3. Visualise distributions and relationships before selecting a model.
4. Calculate descriptive summaries and uncertainty intervals.
5. Fit an appropriate statistical model and check assumptions.
6. Validate predictions or generalisation using a suitable train/test or resampling strategy.
7. Report effect sizes, intervals, limitations, and practical implications.
8. Record code, package versions, data transformations, and random seeds where relevant.

Python tools such as pandas support data preparation, NumPy supports numerical operations, SciPy provides probability distributions and tests, and statsmodels supports interpretable statistical models. Scikit-learn is valuable for predictive modelling and cross-validation. Library output does not replace design reasoning: a perfectly executed test on biased data remains biased.

Applied case: evaluating a recommendation-system change

Background

This illustrative case concerns an online learning platform. The product team proposes a recommendation algorithm intended to increase weekly completed lessons without reducing learner satisfaction.

Problem

The team compares 50,000 users exposed to the new algorithm with 50,000 users who saw the existing system. The treatment group completes an average of 4.20 lessons per week and the control group 4.05. The estimated difference is 0.15 lessons, with a 95% confidence interval from 0.08 to 0.22 and p < 0.001.

Analysis

The result is statistically significant, and the interval excludes zero. However, analysts first verify that assignment was random, exposure was delivered as intended, users were not duplicated, and the outcome definition was fixed before analysis. They inspect balance across prior activity, device, geography, and subscription status. They also examine retention and satisfaction, not merely the primary outcome.

Suppose satisfaction declines by 0.04 points on a five-point scale, with a confidence interval from −0.07 to −0.01. The algorithm may increase activity while harming experience. The team should estimate the expected business value of 0.15 additional lessons, compare it with infrastructure and support costs, and establish an acceptable satisfaction guardrail. Subgroup analysis should be treated cautiously unless prespecified, because many subgroup comparisons can produce chance findings.

Approach and outcome

The product team retains the randomised design, repeats the experiment across a full seasonal cycle, and introduces a minimum satisfaction threshold. It also conducts a fairness audit to check whether recommendations systematically reduce access to advanced content for particular learner groups. Deployment proceeds only after the benefit remains practically meaningful and no unacceptable subgroup harm is detected.

Learning points

The case demonstrates that a low p-value is not a deployment decision. Causal credibility depends on randomisation and implementation quality; practical importance depends on effect magnitude, cost, user experience, and risk. Multiple outcomes, fairness, privacy, and long-term effects belong in the evaluation plan.

Contemporary developments and best practice

Modern data science increasingly uses resampling, Bayesian models, causal graphs, hierarchical models, and robust methods for non-normal data. Bootstrap methods repeatedly resample observed units to approximate the sampling distribution, but they cannot repair selection bias or dependence. Hierarchical models are useful when observations are nested, such as students within schools, because they can represent group-level variation rather than pretending all observations are independent.

Machine-learning systems create additional statistical risks: data leakage, distribution shift, feedback loops, calibration failure, and unfair error rates. A model may have high overall accuracy but poor performance for a minority group. Probabilities should be calibrated when they are used for risk decisions, and thresholds should reflect asymmetric costs. Monitoring after deployment is essential because populations and behaviour change.

Best practice is to define the estimand, preserve a clear audit trail, examine data-generating mechanisms, use robust visual and numerical diagnostics, pre-specify confirmatory analyses, separate exploratory from confirmatory work, quantify uncertainty, and communicate limitations plainly. Statistical literacy includes knowing when not to make a strong claim.

Challenges and limitations

No statistical method eliminates uncertainty. Small samples produce imprecise estimates; large samples can make trivial effects significant. Nonrandom samples limit generalisation. Missing-not-at-random data can bias complete-case analyses. Measurement error attenuates or distorts relationships. Model assumptions are approximations, and model selection can create optimistic results if validation is not independent. Correlation and prediction do not establish causal mechanisms. Finally, ethical concerns cannot be solved by a p-value: privacy, discrimination, consent, and social consequences require governance and human judgment.

Chapter Summary

Statistics for data science combines description, probability, inference, modelling, and decision-making. Means and standard deviations summarise centre and spread, while medians and IQRs are often more robust to outliers. Probability rules formalise uncertainty, and distributions such as binomial, Poisson, normal, and t provide useful models under stated conditions. Sampling variability is quantified by standard error; the central limit theorem explains why many sample means are approximately normal.

Confidence intervals communicate estimate precision, while hypothesis tests assess compatibility with a null model. P-values do not measure the probability that a hypothesis is true, and statistical significance must be interpreted alongside effect size, uncertainty, cost, and practical importance. Correlation and regression describe association and prediction but do not by themselves establish causality. Bayes’ theorem demonstrates how prior prevalence affects posterior evidence.

Credible experiments require randomisation, appropriate controls, clear outcomes, attention to power and multiple testing, and ethical safeguards. In professional data science, reproducibility, fairness, privacy, diagnostic checking, and monitoring are as important as selecting a statistical function in Python.

Key Terms

Population: The complete set of units of interest.

Sample: A subset observed from a population.

Parameter: A numerical characteristic of a population.

Statistic: A numerical quantity calculated from a sample.

Mean: The arithmetic average of observations.

Median: The middle ordered observation.

Variance: Average squared deviation from a mean, with the denominator determined by population or sample context.

Standard deviation: The square root of variance, expressed in the original units.

Interquartile range: The difference between the third and first quartiles.

Conditional probability: Probability of an event given that another event has occurred.

Independence: A relationship in which occurrence of one event does not change the probability of another.

Random variable: A numerical mapping from outcomes of a random process.

Sampling error: Random variation arising from observing a sample.

Standard error: The standard deviation of a statistic’s sampling distribution, estimated in practice.

Confidence interval: An interval produced by a procedure designed to capture a population parameter at a stated long-run rate.

P-value: Probability, under the null model, of a result at least as extreme as observed.

Effect size: A quantitative measure of the magnitude of a difference or relationship.

Correlation: A standardised measure of linear association.

Regression: A model describing a conditional relationship between variables.

Confounding: Distortion of an exposure–outcome relationship by a third factor.

Review Questions

Short Answer Questions

1. Why might the median be preferred to the mean for a strongly right-skewed distribution?
2. Distinguish a population parameter from a sample statistic.
3. What does a 95% confidence interval mean in the frequentist interpretation?
4. Define a Type I error and a Type II error.
5. Why does a positive diagnostic test not necessarily imply a high probability of disease?

Long Answer Questions

1. Explain the central limit theorem, its practical value, and conditions that can make its approximation unreliable.
2. Compare the binomial, Poisson, normal, and t distributions, including a suitable application and key assumption for each.
3. Explain why statistical significance and practical importance can lead to different conclusions.
4. Describe the assumptions, interpretation, and diagnostic process for a simple linear regression model.
5. Explain how randomisation, control groups, blinding, and preregistration improve the credibility of an experiment.

Analytical / Application Questions

1. A sample of 36 machines has mean output 128 units and standard deviation 12 units. Explain how you would construct a 95% confidence interval for the population mean, identify the critical distribution, and interpret the result.
2. A fraud detector flags 8% of transactions. Among fraudulent transactions it flags 90%, while among legitimate transactions it falsely flags 4%. Calculate the probability that a flagged transaction is fraudulent and explain the base-rate implication.
3. An A/B test reports a statistically significant 0.1% increase in conversion, but the change introduces additional annual operating costs larger than the estimated revenue gain. Recommend a decision framework using confidence intervals, effect size, uncertainty, and practical importance rather than p-value alone.

Multiple Choice Questions with Answers

Question 1

Which measure is generally most resistant to a single extremely large observation?

A. Arithmetic mean

B. Median

C. Variance

D. Range

Correct Answer: B. Median

Question 2

If events A and B are independent, which relationship is correct?

A. P(A ∩ B) = 0

B. P(A | B) = 1 − P(A)

C. P(A ∩ B) = P(A)P(B)

D. P(A ∪ B) = P(A) − P(B)

Correct Answer: C. P(A ∩ B) = P(A)P(B)

Question 3

For a binomial random variable with n trials and success probability p, what is its expected value?

A. n + p

B. np

C. n/p

D. p(1 − p)

Correct Answer: B. np

Question 4

A sample has mean 42, standard deviation 10, and size 25. What is the estimated standard error of the mean?

A. 0.4

B. 2

C. 10

D. 50

Correct Answer: B. 2

Question 5

What does a p-value represent?

A. The probability that the null hypothesis is true

B. The probability that the observed data contain no bias

C. The probability, under the null model, of a result at least as extreme as observed

D. The probability that the alternative hypothesis is practically important

Correct Answer: C. The probability, under the null model, of a result at least as extreme as observed

Question 6

Which statement about correlation is correct?

A. A strong correlation proves that one variable causes the other

B. Correlation measures only the units of the two variables

C. Correlation is unaffected by outliers

D. Correlation describes the strength and direction of linear association

Correct Answer: D. Correlation describes the strength and direction of linear association

Question 7

In Bayes’ theorem, P(H | E) denotes which quantity?

A. The prior probability of H

B. The likelihood of E given H

C. The posterior probability of H given E

D. The marginal probability of H and E added together

Correct Answer: C. The posterior probability of H given E

Question 8

Which feature most directly supports causal inference in a well-designed experiment?

A. A very large observational dataset

B. Random assignment to intervention groups

C. A high correlation between exposure and outcome

D. A statistically significant descriptive difference

Correct Answer: B. Random assignment to intervention groups

Question 9

A 95% confidence interval for a mean difference is (0.08, 0.22) lessons. Which interpretation is best?

A. Every individual differs by between 0.08 and 0.22 lessons

B. The null hypothesis has exactly a 95% probability of being false

C. The interval procedure provides plausible values for the population mean difference under its assumptions

D. The sample difference must equal 0.15 in every future sample

Correct Answer: C. The interval procedure provides plausible values for the population mean difference under its assumptions

Question 10

Why can a statistically significant result lack practical importance?

A. Statistical tests cannot use numerical data

B. A very large sample can detect a tiny effect that has little operational value

C. Confidence intervals always exclude useful effects

D. Practical importance is determined only by the p-value

Correct Answer: B. A very large sample can detect a tiny effect that has little operational value