11. Bayesian Statistics and Probabilistic Modelling

Introduction and Context

Statistical inference is concerned with learning about uncertain quantities from data. In frequentist statistics, unknown parameters are treated as fixed but estimated through procedures whose long-run properties are studied. Bayesian statistics takes a different perspective: uncertainty about parameters is represented explicitly through probability distributions. Data then update that uncertainty. This makes Bayesian reasoning particularly suitable for data science, where analysts routinely combine historical information, expert knowledge, streaming observations, missing information, and complex predictive tasks.

Bayesian modelling is not merely a method for calculating an alternative confidence interval. It is a complete framework for specifying assumptions, learning from evidence, making predictions, and taking decisions under uncertainty. A Bayesian model has three central components: a prior distribution, a likelihood, and a posterior distribution. The prior describes knowledge or uncertainty before observing current data; the likelihood describes how the data arise under possible parameter values; and the posterior combines both sources of information.

The chapter develops these ideas from conjugate models to hierarchical models and modern computational methods. It also considers credible intervals, posterior prediction, Bayesian decision theory, Markov chain Monte Carlo (MCMC), variational inference, model criticism, and professional responsibilities associated with prior specification and probabilistic decision-making.

Learning Objectives

After completing this chapter, the learner should be able to:

1. Explain the logic of Bayesian inference and distinguish priors, likelihoods, posteriors, and posterior predictive distributions.

2. Construct and interpret Bayesian models for common data-generating processes.

3. Derive and use conjugate posterior distributions for binomial, Poisson, and normal models.

4. Distinguish credible intervals from confidence intervals and interpret them correctly.

5. Formulate hierarchical models and explain partial pooling.

6. Apply Bayesian decision theory using loss functions and posterior expected loss.

7. Explain the principles and diagnostics of MCMC and variational inference.

8. Evaluate prior sensitivity, computational convergence, predictive performance, fairness, and governance risks.

The Bayesian Framework: From Assumptions to Updated Beliefs

Let \(\theta\) denote an unknown parameter and \(y\) the observed data. Bayes’ theorem is

\[

p(\theta\mid y)=\frac{p(y\mid\theta)p(\theta)}{p(y)}.

\]

Here, \(p(\theta)\) is the prior distribution, \(p(y\mid\theta)\) is the likelihood, \(p(\theta\mid y)\) is the posterior distribution, and

\[

p(y)=\int p(y\mid\theta)p(\theta)d\theta

\]

is the marginal likelihood or evidence. In practice, the denominator often acts only as a normalising constant:

\[

p(\theta\mid y)\propto p(y\mid\theta)p(\theta).

\]

The prior is not necessarily subjective in a casual sense. It may represent previous experiments, domain knowledge, physical constraints, historical data, or a deliberately weakly informative assumption. A prior should be defensible, transparent, and assessed for sensitivity. A likelihood is a probability model for the observed data conditional on parameters. It is not a probability distribution over parameters. The posterior is the updated distribution after observing data.

A Bayesian model therefore has two layers. The data model describes observations, such as \(y_i\sim\text{Bernoulli}(p)\), while the parameter model describes uncertainty, such as \(p\sim\text{Beta}(\alpha,\beta)\). This separation is one reason Bayesian models can be extended naturally to multilevel and latent-variable settings.

Prior specification

A prior may be informative, weakly informative, or approximately non-informative. An informative prior meaningfully influences inference because credible prior knowledge exists. A weakly informative prior does not attempt to be neutral; instead, it rules out implausible values while allowing the data to dominate. For example, a prior that places almost all probability on realistic regression coefficients can improve computation and protect against extreme estimates.

A prior should be examined on the scale relevant to the scientific question. In regression, a normal prior on a coefficient may imply a very different distribution of predicted probabilities depending on the intercept and covariate scale. Prior predictive checking asks: if parameters are drawn from the prior and data are simulated from the model, do the simulated outcomes look plausible? This check should occur before fitting the model.

Likelihood and data-generating assumptions

The likelihood encodes assumptions about independence, sampling, measurement, missingness, and distributional form. If repeated observations are correlated but an independent likelihood is used, posterior uncertainty may be too small. If outliers are likely but a normal likelihood is imposed, parameter estimates and predictions can be distorted. Bayesian analysis does not remove the need for careful study design and appropriate probability models.

Posterior interpretation

A posterior probability has a direct probability interpretation conditional on the model and observed data. For example, \(P(\theta>0\mid y)=0.97\) means that, under the specified model, the posterior probability that \(\theta\) is positive is 0.97. This is different from saying that a fixed parameter has a 97% chance of being positive in a long-run repeated-sampling sense.

Conjugate Models: Exact Updating in Simple Settings

A prior is conjugate to a likelihood when the posterior belongs to the same distributional family as the prior. Conjugacy is valuable pedagogically and computationally because it provides transparent updating, although modern Bayesian modelling is not restricted to conjugate distributions.

Beta–binomial model

Suppose \(y\) successes are observed in \(n\) Bernoulli trials, with success probability \(p\):

\[

y\mid p\sim\text{Binomial}(n,p),\qquad p\sim\text{Beta}(\alpha,\beta).

\]

The posterior is

\[

p\mid y\sim\text{Beta}(\alpha+y,\beta+n-y).

\]

The prior mean is \(\alpha/(\alpha+\beta)\), and \(\alpha+\beta\) can be interpreted as a prior sample-size or concentration parameter, though this interpretation must be used carefully. The posterior mean is

\[

E[p\mid y]=\frac{\alpha+y}{\alpha+\beta+n}.

\]

For example, a prior \(\text{Beta}(2,2)\) and data of 7 successes in 10 trials produce a posterior \(\text{Beta}(9,5)\). The data shift the prior toward larger success probabilities without assigning probability one to any estimate.

Gamma–Poisson model

For count data, suppose

\[

y_i\mid\lambda\sim\text{Poisson}(\lambda),\qquad \lambda\sim\text{Gamma}(a,b),

\]

where \(b\) is a rate parameter. If \(S=\sum_i y_i\) and there are \(n\) observations, then

\[

\lambda\mid y\sim\text{Gamma}(a+S,b+n).

\]

The model is useful for event counts such as incidents per exposure period, provided that the Poisson assumptions are reasonable. Excess variation relative to the Poisson distribution may require a negative-binomial or hierarchical model.

Normal models

When observations are normally distributed with known variance, a normal prior for the mean is conjugate. Let

\[

y_i\mid\mu\sim N(\mu,\sigma^2),\qquad \mu\sim N(\mu_0,\tau_0^2).

\]

The posterior mean is a precision-weighted average of the prior mean and sample mean:

\[

\mu_n=\frac{\mu_0/\tau_0^2+n\bar y/\sigma^2}{1/\tau_0^2+n/\sigma^2}.

\]

Precision is the reciprocal of variance. A source with smaller variance receives greater weight. When both mean and variance are unknown, normal-inverse-gamma or related priors can provide conjugate forms, but robust likelihoods and hierarchical structures are often more appropriate in applied work.

Credible Intervals and Posterior Prediction

A 95% credible interval is an interval containing 95% posterior probability. It may be an equal-tail interval, with 2.5% posterior probability in each tail, or a highest posterior density interval, whose points have relatively high posterior density. For skewed or multimodal posteriors, these choices can differ substantially.

A credible interval is conditional on the model, prior, and observed data. It should not be interpreted as a confidence interval. A 95% confidence procedure has a long-run coverage property over repeated samples; it does not, in the standard frequentist interpretation, assign a 95% probability to the fixed parameter after the data are observed.

Parameter uncertainty and future-outcome uncertainty are different. The posterior predictive distribution is

\[

p(\tilde y\mid y)=\int p(\tilde y\mid\theta)p(\theta\mid y)d\theta.

\]

It incorporates both uncertainty about parameters and the inherent randomness of a future observation. For a Bernoulli process, one may draw a parameter from the posterior and then draw a future success or failure. For regression, posterior predictive simulation produces a distribution of plausible outcomes for new cases. Predictive intervals are usually wider than intervals for a conditional mean because they include irreducible observation noise.

Posterior predictive checks compare replicated data generated from the fitted model with observed data. Useful summaries may include means, variances, extreme values, zero counts, group differences, or temporal patterns. A model can have reasonable parameter estimates yet fail to reproduce important features of the data.

Hierarchical and Multilevel Modelling

Many data sets contain observations nested within groups: customers within regions, patients within hospitals, or transactions within accounts. A complete-pooling model assumes one common parameter, while a no-pooling model estimates every group separately. Hierarchical modelling provides partial pooling by allowing group-specific parameters to vary around a population distribution.

For group \(j\), a simple normal model is

\[

y_{ij}\sim N(\theta_j,\sigma^2),\qquad \theta_j\sim N(\mu,\tau^2).

\]

The group means are therefore related but not identical. Groups with little data are shrunk toward the population mean, while groups with substantial evidence retain more individual information. This shrinkage is not an arbitrary correction. It follows from modelling the fact that groups are exchangeable draws from a common population.

In a logistic multilevel model, for example,

\[

\text{logit}(p_{ij})=\alpha_j+x_{ij}^{T}\beta,\qquad \alpha_j\sim N(\alpha_0,\tau_\alpha^2).

\]

Hierarchical models improve estimation for small groups and allow group-level variation to be quantified. However, the assumed population distribution and exchangeability require substantive justification. A group that is structurally different from all others should not automatically be partially pooled with them.

Bayesian Decision Theory

Inference is often a step toward action. Bayesian decision theory combines a posterior distribution with a loss function. Let \(a\) be an action and \(\theta\) the unknown state. The posterior expected loss is

\[

\rho(a\mid y)=E[L(a,\theta)\mid y].

\]

The Bayes action minimises posterior expected loss. Under squared-error loss, the Bayes action is the posterior mean. Under absolute-error loss, it is a posterior median. Under a 0–1 classification loss, it is often the class with the highest posterior probability. Different losses can produce different decisions from the same posterior.

Decision analysis makes consequences explicit. In fraud detection, false positives may inconvenience many legitimate customers, whereas false negatives may create financial loss. A threshold chosen only to maximise accuracy ignores these asymmetric costs. Bayesian decision theory can also incorporate resource constraints, utility, safety requirements, and the value of additional information.

Computational Bayesian Inference

Exact posterior calculations are available only for some models. Complex models may contain nonlinear predictors, latent variables, nonconjugate priors, missing data, or thousands of parameters. Computational methods approximate posterior distributions.

Markov chain Monte Carlo

MCMC constructs a Markov chain whose stationary distribution is the target posterior. The Metropolis–Hastings algorithm proposes a new state and accepts it with a probability determined by the posterior density and proposal mechanism. Gibbs sampling draws each parameter from its conditional posterior when those conditional distributions are available. Hamiltonian Monte Carlo (HMC) uses gradients to explore continuous, correlated parameter spaces efficiently and is implemented in systems such as Stan.

An MCMC analysis should use multiple dispersed chains and assess convergence. Important diagnostics include trace plots, effective sample size, autocorrelation, and the split \(\hat R\) statistic. A value of \(\hat R\) close to 1 is necessary but not sufficient evidence of convergence. Divergent transitions, low effective sample sizes, excessive autocorrelation, and chains trapped in separate regions indicate problems with geometry, identifiability, or model specification.

Warm-up draws used for adaptation are normally discarded. Posterior summaries should be based on sufficiently many post-warm-up draws. Increasing the number of draws does not repair a fundamentally misspecified model or a chain that has not explored the posterior adequately.

Variational inference

Variational inference (VI) turns posterior approximation into an optimisation problem. A tractable family \(q(\theta)\) is selected and fitted to be close to the posterior, often by minimising Kullback–Leibler divergence. VI is usually faster and more scalable than MCMC, making it attractive for large data sets and iterative machine-learning workflows.

Its speed comes with risks. A restrictive approximation may underestimate posterior variance, miss multimodality, or distort dependence among parameters. Consequently, VI results should be validated through simulation, predictive checks, sensitivity analysis, and, where feasible, comparison with MCMC on a representative subset.

A Practical Bayesian Workflow

A defensible workflow begins with a clearly stated estimand: the quantity about which inference or prediction is required. The analyst then specifies the data-generating process, selects scales and transformations, assigns priors, and performs prior predictive checks. The model is fitted using exact computation, MCMC, VI, or another justified method. Computational diagnostics are examined before substantive interpretation.

The next stage is posterior and posterior predictive checking. Analysts should inspect whether the model reproduces relevant patterns, whether conclusions are sensitive to reasonable priors, and whether predictions are calibrated on appropriate validation data. Model comparison may use predictive criteria such as expected log predictive density, but it should not be reduced to selecting a model solely because it has the largest numerical score. Simpler models, interpretability, operational constraints, and fairness may matter as well.

Common mistakes include treating the prior as an afterthought, interpreting the likelihood as a posterior, reporting only posterior means, ignoring skewness, confusing parameter intervals with predictive intervals, failing to check convergence, and using a convenient conjugate model despite clear overdispersion or dependence. Another mistake is allowing observed outcomes to influence prior construction in a way that double-counts the data.

Applications in Data Science and Organisations

Bayesian methods are useful in A/B testing, risk analysis, demand forecasting, medical diagnosis, recommendation systems, credit assessment, reliability engineering, anomaly detection, and sequential experimentation. They support continuous updating as new evidence arrives and provide uncertainty distributions rather than single scores.

In product experimentation, a posterior distribution for a conversion-rate difference can answer questions such as the probability that a new design improves conversion by at least a practically meaningful amount. In forecasting, posterior predictive distributions can support inventory decisions under uncertainty. In reliability analysis, prior engineering knowledge can be combined with sparse failure data. In healthcare, hierarchical models can estimate hospital-level performance while avoiding unstable rankings based on small samples.

These applications require organisational discipline. Data provenance, versioned model specifications, documented priors, reproducible computation, monitoring for drift, and clear ownership of decisions are as important as mathematical correctness.

Illustrative Case Study: Bayesian Reliability Assessment

Background

A manufacturer monitors failures of a component used in industrial equipment. Historical engineering tests suggest that the monthly failure rate is usually low, but the current component design is new and only limited field data are available.

Situation and problem

During the first 20 component-months of observation, two failures occur. Management must decide whether to continue deployment, commission additional testing, or redesign the component. A raw failure rate of 10% may be unstable because it is based on little evidence.

Analysis and approach

Let \(p\) be the probability of failure per component-month. The reliability team uses a weakly informative prior \(p\sim\text{Beta}(1,19)\), whose mean is 0.05 but which allows considerable uncertainty. With two failures in 20 observations, the posterior is

\[

p\mid y\sim\text{Beta}(3,37).

\]

The posterior mean is \(3/40=0.075\). The decision should not rely on this mean alone. The team calculates the posterior probability that the failure rate exceeds the safety threshold of 0.10 and simulates the number of failures expected in the next 100 component-months. It also repeats the analysis under alternative plausible priors to determine whether the recommendation is prior-sensitive.

Posterior predictive checks are conducted by simulating failure counts under the fitted model. Engineers also assess whether failures are independent. If failures cluster by production batch or operating temperature, a simple binomial model is inadequate and a hierarchical or covariate-dependent model is required.

Decision and outcome

Suppose the posterior probability of exceeding 0.10 is material but not decisive, while the predicted upper-tail number of failures is unacceptable under the organisation’s safety policy. The appropriate response is not to declare the component safe because the posterior mean is below 0.10. Management commissions targeted testing under high-stress conditions and temporarily limits deployment. The additional evidence is then incorporated into an updated model.

Learning points

The case illustrates the distinction between an estimate and a distribution of uncertainty, the role of prior predictive reasoning, the importance of loss and safety thresholds, and the need to examine whether the likelihood reflects the physical failure process. Bayesian analysis supports a decision; it does not replace engineering judgement or regulatory review.

Limitations, Ethics, and Governance

Bayesian results are conditional on the model and prior. A poorly chosen prior can dominate when data are sparse, while a supposedly weak prior can still be informative on an unfamiliar scale. Likelihood misspecification, unmeasured confounding, selection bias, informative missingness, and measurement error remain serious threats.

Ethically, analysts must disclose priors that materially affect decisions and should not choose priors merely to obtain a desired result. Automated decisions based on posterior probabilities may reproduce historical discrimination if the data reflect unequal access or biased measurement. A well-calibrated model can still be unfair if errors are distributed unequally across groups. Sensitive applications require subgroup evaluation, human oversight, appeal mechanisms, privacy protection, and documentation of intended use.

In regulated environments, reproducibility includes preserving data versions, code, random seeds where relevant, model configurations, computational diagnostics, and decision thresholds. Probabilistic uncertainty should be communicated honestly. A narrow posterior produced by a biased or dependent sample is not evidence of genuine certainty.

Current Developments and Best Practice

Modern Bayesian systems increasingly combine probabilistic programming with automatic differentiation, GPU computation, scalable approximate inference, and simulation-based inference. Bayesian neural networks, Gaussian processes, deep generative models, and probabilistic programming languages extend uncertainty modelling to high-dimensional applications. Simulation-based inference is useful when the likelihood is difficult to evaluate but a simulator can generate data, although its validity depends on the simulator and the quality of diagnostic checks.

Best practice includes:

• define the estimand and decision before fitting the model;
• use priors that are scientifically defensible and conduct prior predictive checks;
• model dependence, hierarchy, missingness, and measurement processes when relevant;
• inspect convergence and effective sample size for MCMC;
• validate variational approximations rather than assuming their uncertainty is exact;
• use posterior predictive checks and out-of-sample evaluation;
• conduct sensitivity analyses for priors, likelihoods, and influential observations;
• report uncertainty, assumptions, limitations, and decision consequences;
• maintain reproducible computational and governance records.

Chapter Summary

Bayesian statistics represents uncertainty about unknown quantities through probability distributions. A prior expresses information before current data, the likelihood describes the data-generating process, and Bayes’ theorem produces the posterior. Conjugate models provide exact updating in simple settings, while posterior predictive distributions support forecasting and model checking. Credible intervals describe posterior probability and differ conceptually from confidence intervals. Hierarchical models enable partial pooling across related groups, improving estimates for sparse groups while representing between-group variation. Bayesian decision theory links posterior uncertainty to actions through loss functions. Complex models require computational methods such as MCMC or variational inference, both of which demand diagnostic evaluation. Responsible Bayesian practice combines mathematical modelling with prior sensitivity, predictive validation, ethical governance, reproducibility, and explicit decision consequences.

Key Terms

Prior distribution: Probability distribution representing uncertainty about a parameter before observing current data.

Likelihood: Probability model for observed data conditional on parameter values.

Posterior distribution: Updated probability distribution for parameters after combining prior and likelihood.

Marginal likelihood: Probability of the observed data after averaging over the prior distribution.

Conjugacy: Property whereby prior and posterior distributions belong to the same family.

Credible interval: Interval containing a specified posterior probability for a parameter.

Posterior predictive distribution: Distribution of future or replicated observations conditional on observed data.

Precision: Reciprocal of variance; a measure of information concentration.

Prior predictive distribution: Distribution of possible data implied by the prior and likelihood before observation.

Hierarchical model: Model containing parameters that are themselves governed by higher-level probability distributions.

Partial pooling: Sharing information across groups while retaining group-specific variation.

Exchangeability: Assumption that group labels do not affect the joint probability structure, conditional on relevant information.

Loss function: Quantification of the cost associated with an action and the true state.

Bayes action: Action that minimises posterior expected loss.

MCMC: Simulation method that uses a Markov chain to approximate a target posterior distribution.

Hamiltonian Monte Carlo: Gradient-based MCMC method for exploring continuous posterior distributions.

Effective sample size: Approximate number of independent draws represented by correlated MCMC samples.

Variational inference: Approximation of a posterior through optimisation over a tractable distributional family.

Posterior predictive check: Comparison between observed data and data simulated from the fitted model.

Prior sensitivity analysis: Examination of how conclusions change under reasonable alternative priors.

Review Questions

Short Answer Questions

1. What are the roles of the prior, likelihood, and posterior in Bayesian inference?

2. Why is the posterior predictive distribution generally more uncertain than a distribution for a parameter alone?

3. State the posterior distribution in the beta–binomial model.

4. What is partial pooling, and why is it useful for small groups?

5. Name two diagnostics used to assess MCMC output.

Long Answer Questions

1. Explain Bayes’ theorem as a framework for statistical modelling, including the interpretation of the marginal likelihood.

2. Derive the beta–binomial posterior and discuss how prior concentration affects learning from data.

3. Compare credible intervals and confidence intervals, including the meanings of their probability statements.

4. Explain how hierarchical models differ from complete-pooling and no-pooling models.

5. Compare MCMC and variational inference in terms of computation, uncertainty representation, and diagnostic requirements.

Analytical / Application Questions

1. A team observes 12 successes in 20 trials and uses a Beta(2, 3) prior. Derive the posterior distribution and explain how you would report uncertainty about the success probability.

2. A hospital has unusually high performance based on only five cases, while another has average performance based on 2,000 cases. Explain how a hierarchical model could affect their estimates and why this may be preferable to ranking raw rates.

3. A fraud classifier has very different costs for false positives and false negatives. Describe how to use a posterior distribution and loss function to select an operational decision threshold.

22. Multiple Choice Questions with Answers

Question 1

Which expression correctly represents the Bayesian posterior, up to proportionality?

A. p(θ|y) ∝ p(θ)p(y|θ)

B. p(θ|y) ∝ p(θ)/p(y|θ)

C. p(θ|y) ∝ p(y)/p(θ)

D. p(θ|y) ∝ p(y|θ)/p(θ)

Correct Answer: A. p(θ|y) ∝ p(θ)p(y|θ)

Question 2

If p has a Beta(α, β) prior and y successes are observed in n binomial trials, what is the posterior?

A. Beta(α+n, β+y)

B. Beta(α+y, β+n−y)

C. Beta(α−y, β−n+y)

D. Beta(y, n−y)

Correct Answer: B. Beta(α+y, β+n−y)

Question 3

What does a 95% Bayesian credible interval mean, conditional on the model and data?

A. The parameter lies in the interval in 95% of repeated samples

B. The null hypothesis is false with probability 95%

C. The interval contains 95% posterior probability

D. The future observation lies in the interval with probability 95%

Correct Answer: C. The interval contains 95% posterior probability

Question 4

Why are posterior predictive intervals usually wider than intervals for a conditional mean?

A. They ignore parameter uncertainty

B. They include both parameter uncertainty and future observation noise

C. They always use a less informative prior

D. They are calculated from confidence rather than probability distributions

Correct Answer: B. They include both parameter uncertainty and future observation noise

Question 5

What is the principal effect of partial pooling in a hierarchical model?

A. It forces all groups to have identical estimates

B. It estimates groups independently with no shared information

C. It shrinks weakly informed group estimates toward a population distribution

D. It removes between-group variation from the model

Correct Answer: C. It shrinks weakly informed group estimates toward a population distribution

Question 6

Under squared-error loss, which quantity is the Bayes action?

A. Posterior mean

B. Posterior mode only

C. Prior median

D. Maximum likelihood estimate in every case

Correct Answer: A. Posterior mean

Question 7

Which finding most strongly indicates a possible MCMC problem?

A. Posterior draws have a high effective sample size

B. Multiple chains mix well and have R-hat near 1

C. Trace plots show chains exploring the same region

D. Chains remain in separate regions and show divergent transitions

Correct Answer: D. Chains remain in separate regions and show divergent transitions

Question 8

What is a major risk of variational inference compared with well-converged MCMC?

A. It cannot produce any posterior approximation

B. It may underestimate posterior uncertainty because of a restrictive approximation family

C. It always requires more computation than MCMC

D. It cannot be used with hierarchical models

Correct Answer: B. It may underestimate posterior uncertainty because of a restrictive approximation family

Question 9

In the reliability case, why should the team simulate future failures rather than report only the posterior mean failure rate?

A. Simulation removes all prior assumptions

B. Future decisions depend on predictive uncertainty and possible outcomes

C. The posterior mean is always mathematically invalid

D. Simulation converts a Bayesian model into a frequentist model

Correct Answer: B. Future decisions depend on predictive uncertainty and possible outcomes

Question 10

What is the purpose of a prior predictive check?

A. To assess whether data simulated before observing outcomes are plausible

B. To guarantee that the posterior has converged

C. To select the action with minimum posterior loss

D. To replace evaluation of the likelihood

Correct Answer: A. To assess whether data simulated before observing outcomes are plausible