Bayesian Statistics and Decision Theory

Introduction and Context

Bayesian statistics provides a formal framework for learning from evidence while incorporating existing knowledge. It is especially valuable when data are limited, observations arrive sequentially, parameters are uncertain, or decisions must account explicitly for the consequences of error. In artificial intelligence and machine learning, Bayesian reasoning underlies probabilistic classifiers, diagnostic systems, sequential decision-making, Bayesian optimization, probabilistic programming, and uncertainty-aware prediction.

Classical or frequentist statistics generally treats parameters as fixed but unknown quantities and evaluates procedures through their repeated-sampling properties. Bayesian statistics instead represents uncertainty about an unknown parameter with a probability distribution. This does not mean that the parameter itself is necessarily random in a physical sense; rather, probability expresses the analyst's uncertainty about its possible values.

Decision theory extends statistical inference. Estimating a parameter or predicting an outcome is not always the final goal. A practitioner may need to approve a loan, deploy a medical intervention, reject a defective component, or trigger a cybersecurity response. Such actions have consequences, and rational decisions depend on both uncertainty and the relative costs of possible errors.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain prior, likelihood, posterior, and posterior predictive distributions.

2. Apply Bayes' theorem to parameter estimation and prediction.

3. Use conjugate models, including Beta–Binomial and Normal models.

4. Distinguish credible intervals from confidence intervals.

5. Explain hierarchical models and partial pooling.

6. Compare models using posterior predictive reasoning and marginal likelihood.

7. Formulate decisions using utility, loss, and risk.

8. Recognize limitations, computational issues, and ethical risks in Bayesian analysis.

1. The Bayesian Framework

Let $\theta$ denote an unknown parameter and let $y$ denote observed data. Bayes' theorem gives

$$p(\theta\mid y)=\frac{p(y\mid\theta)p(\theta)}{p(y)}.$$

Here:

• $p(\theta)$ is the prior distribution, representing uncertainty before observing the current data;
• $p(y\mid\theta)$ is the likelihood, describing how probable the observed data are under a particular parameter value;
• $p(\theta\mid y)$ is the posterior distribution, representing updated uncertainty after observing the data;
• $p(y)=\int p(y\mid\theta)p(\theta)d\theta$ is the marginal likelihood or evidence.

The posterior is proportional to prior times likelihood:

$$p(\theta\mid y)\propto p(y\mid\theta)p(\theta).$$

This proportional form is often sufficient for deriving a posterior because the normalizing constant can be obtained afterward or avoided during simulation-based computation.

The likelihood is not a probability distribution over $\theta$ unless normalized as such. It is a function of $\theta$ for fixed observed data. Confusing likelihood with a posterior is a common conceptual error: the likelihood does not by itself express prior information or normalized uncertainty about the parameter.

Sequential Updating

If data arrive in stages and observations are conditionally independent given $\theta$, then

$$p(\theta\mid y_1,\ldots,y_n)\propto p(\theta)\prod_{i=1}^{n}p(y_i\mid\theta).$$

The posterior after one stage can become the prior for the next stage. Thus Bayesian learning is naturally sequential and does not require re-deriving the entire analysis conceptually.

2. Priors: Sources, Forms, and Sensitivity

A prior may be based on previous experiments, domain expertise, physical constraints, historical data, or a deliberately weak-information specification. Priors should be justified rather than selected merely because they produce convenient mathematics.

A prior can be:

• Informative, when substantial reliable prior knowledge exists;
• Weakly informative, when implausible values are discouraged without strongly determining the result;
• Diffuse, when uncertainty is broad;
• Empirical, when estimated partly from related data;
• Hierarchical, when prior parameters are themselves modeled probabilistically.

A prior is not automatically objective or subjective. Even a supposedly noninformative prior encodes assumptions, such as a scale, parameterization, or boundary constraint. Analysts should examine whether conclusions change under scientifically plausible alternative priors. This is called prior sensitivity analysis.

For a probability $\theta$ between zero and one, the Beta distribution is a useful prior:

$$p(\theta)=\frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)},\quad 0<\theta<1,$$

where $\alpha,\beta>0$ and $B(\alpha,\beta)$ is the beta function. Its mean is $\alpha/(\alpha+\beta)$, and its concentration increases as $\alpha+\beta$ increases. The interpretation of $\alpha$ and $\beta$ as prior successes and failures is useful in some contexts, but it is not universally literal.

3. Conjugate Models and Bayesian Estimation

A prior is conjugate to a likelihood when the posterior belongs to the same distributional family. Conjugacy is computationally convenient, although it is not a requirement of Bayesian analysis.

3.1 Beta–Binomial Model

Suppose $Y\mid\theta\sim\operatorname{Binomial}(n,\theta)$ and $\theta\sim\operatorname{Beta}(\alpha,\beta)$. If $y$ successes are observed, then

$$\theta\mid y\sim\operatorname{Beta}(\alpha+y,\beta+n-y).$$

The posterior mean is

$$E[\theta\mid y]=\frac{\alpha+y}{\alpha+\beta+n}.$$

The Bayesian posterior mean combines prior information with data. Its interpretation depends on the prior and should not be described automatically as the true success probability.

For squared-error loss, the Bayes estimator is the posterior mean. For absolute-error loss, a posterior median is optimal; for zero–one loss, a posterior mode is often relevant, subject to technical details.

Worked Example 1: Defect Probability

A manufacturing process produces 8 defective items in a sample of 100. Before sampling, engineers use a $\operatorname{Beta}(2,98)$ prior, whose mean is $0.02$. The posterior is

$$\theta\mid y\sim\operatorname{Beta}(10,190).$$

The posterior mean is

$$\frac{10}{10+190}=0.05.$$

The observed sample proportion is $0.08$, while the posterior mean is $0.05$ because the prior pulls the estimate toward 0.02. The posterior distribution, rather than only its mean, describes remaining uncertainty. A credible interval can be calculated from Beta quantiles; it is not generally obtained by simply applying a normal approximation without checking its adequacy.

3.2 Normal Model with Known Variance

Suppose observations satisfy

$$Y_i\mid\mu\sim N(\mu,\sigma^2),\qquad i=1,\ldots,n,$$

where $\sigma^2$ is known, and use the prior $\mu\sim N(\mu_0,\tau_0^2)$. Then

$$\mu\mid y\sim N(\mu_n,\tau_n^2),$$

where

$$\tau_n^2=\left(\frac{1}{\tau_0^2}+\frac{n}{\sigma^2}\right)^{-1},$$

and

$$\mu_n=\tau_n^2\left(\frac{\mu_0}{\tau_0^2}+\frac{n\bar y}{\sigma^2}\right).$$

The posterior mean is a precision-weighted average of the prior mean and sample mean. Precision means inverse variance. These equations assume independent observations, a normal likelihood, known common variance, and a normal prior.

Worked Example 2: Sensor Calibration

A sensor has measurement error with known standard deviation $\sigma=2$ units. A prior calibration gives $\mu_0=10$ and $\tau_0=3$. Ten new readings have mean $\bar y=11$. Then

$$\tau_n^2=\left(\frac{1}{9}+\frac{10}{4}\right)^{-1}=\frac{1}{2.6111}\approx0.383,$$

so the posterior standard deviation is approximately $0.619$. The posterior mean is

$$\mu_n=0.383\left(\frac{10}{9}+\frac{10(11)}{4}\right)\approx10.85.$$

The data receive greater weight because their precision, $10/4=2.5$, exceeds the prior precision, $1/9\approx0.111$. A central 95% credible interval under this normal posterior is approximately

$$10.85\pm1.96(0.619),$$

or $(9.64,12.06)$ units. This interval has a direct posterior interpretation: conditional on the model and prior, the posterior probability that $\mu$ lies in this interval is approximately 0.95.

4. Predictive Distributions

Inference about a parameter is different from prediction of a future observation. The posterior predictive distribution is

$$p(\tilde y\mid y)=\int p(\tilde y\mid\theta)p(\theta\mid y)d\theta.$$

It averages predictions over posterior uncertainty in $\theta$. Ignoring parameter uncertainty produces overly confident predictions.

For a Beta–Binomial model, the probability that the next Bernoulli trial is a success is

$$P(\tilde Y=1\mid y)=E[\theta\mid y]=\frac{\alpha+y}{\alpha+\beta+n}.$$

For multiple future trials, the posterior predictive distribution is Beta–Binomial rather than an ordinary Binomial distribution, because uncertainty about $\theta$ induces dependence among future outcomes after integrating out $\theta$.

Posterior predictive checks compare replicated data generated from the fitted model with observed data. Important discrepancies may reveal nonlinearity, outliers, dependence, heterogeneity, or an unsuitable likelihood. A model that predicts the average well may still fail to represent tails or subgroup behavior.

5. Credible Intervals and Uncertainty Communication

A 95% Bayesian credible interval is an interval containing 95% posterior probability, conditional on the model, prior, and data. A frequentist 95% confidence interval is a procedure that covers the fixed parameter in 95% of repeated samples under specified assumptions. The two intervals may be numerically similar, but their interpretations differ.

Common Bayesian intervals include:

• Equal-tailed intervals, excluding equal posterior probability from both tails;
• Highest posterior density intervals, containing the most plausible values for a specified posterior mass.

For skewed or multimodal posteriors, interval choice and graphical presentation matter. Reporting only a posterior mean can conceal meaningful asymmetry or multiple plausible explanations.

6. Hierarchical Bayesian Models

A hierarchical model represents parameters at multiple levels. For example, outcomes from departments may be modeled as

$$Y_{ij}\mid\theta_j\sim p(y\mid\theta_j),$$

with department-specific parameters

$$\theta_j\mid\mu,\tau\sim p(\theta\mid\mu,\tau),$$

and hyperparameters $\mu,\tau$ given their own priors.

Hierarchical models enable partial pooling. Departments with little data are shrunk toward the population distribution, while departments with substantial evidence retain more individual variation. Complete pooling assumes all departments have one common parameter; no pooling estimates each department independently. Partial pooling often provides a useful compromise.

Shrinkage is not an arbitrary correction. It reflects the model's judgment that extreme estimates based on small samples are less reliable than estimates supported by substantial data. However, a badly chosen population structure can overshrink genuine subgroup differences. Group definitions, missing data, and dependence require careful substantive justification.

7. Model Comparison and Model Averaging

Bayesian model comparison can use:

1. Posterior predictive performance, assessing how well models reproduce relevant data features or predict held-out observations;

2. Marginal likelihood, $p(y\mid M)=\int p(y\mid\theta,M)p(\theta\mid M)d\theta$;

3. Bayes factors, ratios of marginal likelihoods for two models;

4. Approximate predictive criteria such as leave-one-out cross-validation or widely applicable information criteria.

Marginal likelihood automatically averages fit over the prior parameter space. Consequently, diffuse priors can strongly affect Bayes factors: a model that spreads prior probability over many implausible values may be penalized. Bayes factors therefore require carefully justified priors and should not be treated as universally superior to predictive validation.

Bayesian model averaging combines predictions across models:

$$p(\tilde y\mid y)=\sum_m p(\tilde y\mid y,M_m)P(M_m\mid y).$$

This can represent model uncertainty rather than pretending that one selected model is certainly correct.

8. Decision Theory: Utility, Loss, and Risk

Statistical inference becomes decision analysis when an action must be selected. Let $a$ be an action and $\theta$ the unknown state. A loss function $L(a,\theta)$ measures the cost of taking action $a$ when the state is $\theta$. Utility is often defined as the negative of loss.

The posterior expected loss of action $a$ is

$$\rho(a\mid y)=E[L(a,\theta)\mid y].$$

A Bayes action minimizes posterior expected loss:

$$a^*(y)=\arg\min_a E[L(a,\theta)\mid y].$$

The risk function evaluates a decision rule over repeated data generation:

$$R(\theta,\delta)=E_{Y\mid\theta}[L(\delta(Y),\theta)].$$

Risk is a property indexed by the true state, whereas posterior expected loss is conditional on observed data and the posterior distribution. A decision rule can be excellent for one loss function and poor for another.

Threshold Decisions

Suppose an intervention is taken if a disease probability exceeds a threshold. Let action $T$ mean treat and $N$ mean do not treat. If treatment costs $C_T$ when disease is absent and failure to treat costs $C_F$ when disease is present, then treatment is preferred when

$$C_TP(D=0\mid y)<C_FP(D=1\mid y).$$

Equivalently, treat when

$$P(D=1\mid y)>\frac{C_T}{C_T+C_F},$$

assuming no additional treatment benefit term and zero loss for correct decisions. This threshold is not necessarily 0.5. If failure to treat is much more costly than unnecessary treatment, the threshold is lower.

9. Illustrative Case: Bayesian Predictive Maintenance

Background. An engineering firm monitors vibration from industrial pumps. A high vibration level may indicate an impending failure, but an unnecessary shutdown is expensive.

Problem. The firm must decide whether to shut down a pump for inspection. Historical evidence suggests that 10% of pumps with a particular operating profile fail within 30 days. A new sensor reading has a likelihood ratio of 6 for failure versus no failure. Suppose the prior odds of failure are $0.10/0.90=0.1111$.

Analysis. Bayes' theorem in odds form gives posterior odds:

$$\text{posterior odds}=\text{prior odds}\times\text{likelihood ratio}=0.1111\times6=0.6667.$$

Therefore, the posterior failure probability is

$$P(F\mid y)=\frac{0.6667}{1+0.6667}=0.40.$$

Assume an unnecessary shutdown costs 1 unit, while failing to shut down when failure occurs costs 4 units. Shutdown is preferred when

$$1\times P(F^c\mid y)<4\times P(F\mid y).$$

At posterior probability 0.40, the expected loss of shutdown is $0.60$, while the expected loss of no shutdown is $1.60$. The rational action under this loss specification is shutdown.

Approach and outcome. The decision is driven not only by the sensor reading but also by the base rate and consequences of errors. If the shutdown cost rose to 3 units, its expected loss would be $1.80$, exceeding the no-shutdown loss of $1.60$; the recommended action would change.

Learning points. First, posterior probabilities depend on both prior prevalence and diagnostic evidence. Second, classification thresholds should reflect costs rather than default to 0.5. Third, sensor calibration, likelihood-ratio stability, and changing operating conditions must be monitored. Fourth, a decision system should expose uncertainty and allow engineers to audit assumptions.

10. Computation and Implementation

Simple conjugate models can be solved analytically. More complex models commonly use:

• Monte Carlo sampling, drawing from the posterior to approximate expectations;
• Markov chain Monte Carlo, including Gibbs sampling and Hamiltonian Monte Carlo;
• Variational inference, which optimizes an approximate posterior and is often faster but may underestimate uncertainty;
• Sequential Monte Carlo, useful for evolving systems and online inference.

Convergence diagnostics, effective sample size, trace plots, divergent-transition checks, and sensitivity analyses are essential. Numerical output is not evidence of correctness by itself. Software should be accompanied by transparent model definitions, data preprocessing documentation, reproducible random seeds where appropriate, and independent validation.

11. Limitations, Ethics, and Governance

Bayesian conclusions are conditional on the likelihood, prior, data quality, and computational approximation. A posterior distribution cannot repair selection bias, measurement error, label leakage, confounding, or a systematically misspecified model. “The data updated the prior” does not imply that the result is objective or unbiased.

In high-stakes AI systems, prior choices may encode historical discrimination. A hierarchical model may improve prediction for small groups while also imposing assumptions that obscure minority-specific patterns. Decision thresholds can distribute false positives and false negatives unequally. Governance should therefore include subgroup evaluation, documentation of priors and losses, human oversight, monitoring for dataset shift, privacy protection, and mechanisms for contesting automated decisions.

When data are used from people, informed consent, confidentiality, data minimization, and lawful processing are relevant. Posterior probabilities should not be presented as certainties, especially when users may misunderstand them. Organizations should distinguish predictive accuracy from fairness, explainability, and decision legitimacy.

12. Best Practices and Contemporary Developments

Effective Bayesian practice includes:

1. Define the estimand and decision before selecting a model.

2. Match the likelihood to the data-generating process.

3. Justify priors in domain-relevant units.

4. Conduct prior and posterior predictive checks.

5. Report full uncertainty, not only point estimates.

6. Test sensitivity to plausible priors and loss functions.

7. Validate predictive performance on held-out or future-like data.

8. Inspect subgroup performance and calibration.

9. Check computational convergence and approximation quality.

10. Document assumptions for technical and nontechnical stakeholders.

Contemporary Bayesian work increasingly combines probabilistic programming, scalable approximate inference, causal modeling, Gaussian processes, Bayesian deep learning, and active learning. In machine learning, epistemic uncertainty represents uncertainty about model parameters or limited knowledge, while aleatoric uncertainty represents irreducible variability in observations. These categories are useful but can be difficult to separate in practice. Bayesian methods can support uncertainty-aware deployment, but they do not automatically guarantee calibrated probabilities or safe decisions.

Chapter Summary

Bayesian statistics updates prior uncertainty using observed evidence through Bayes' theorem. The posterior combines the prior and likelihood and supports estimation, interval construction, prediction, and sequential learning. Conjugate models such as Beta–Binomial and Normal–Normal models provide analytically tractable examples, while hierarchical models enable partial pooling across related groups. Posterior predictive distributions incorporate both observation noise and parameter uncertainty. Credible intervals have a posterior probability interpretation that differs from the repeated-sampling interpretation of confidence intervals.

Model comparison should consider predictive performance, marginal likelihood, prior sensitivity, and model uncertainty. Decision theory connects inference to action through loss, utility, posterior expected loss, and risk. The optimal action depends on consequences, so a probability threshold need not equal 0.5. Bayesian systems are powerful but conditional on assumptions about data, priors, likelihoods, computation, and governance. In AI and professional practice, transparent uncertainty communication, calibration, fairness assessment, and monitoring are essential.

Key Terms

Prior: Distribution representing uncertainty before current data are observed.

Likelihood: Probability model for observed data viewed as a function of parameters.

Posterior: Updated distribution of unknown quantities after observing data.

Marginal likelihood: Probability of the observed data after averaging over the prior.

Evidence: Another name for marginal likelihood in model comparison.

Conjugacy: Property that prior and posterior belong to the same distributional family.

Posterior predictive distribution: Distribution of future observations conditional on current data.

Credible interval: Interval containing a specified posterior probability.

Hierarchical model: Model with parameters organized across multiple levels.

Partial pooling: Sharing information across groups while retaining group-specific variation.

Shrinkage: Movement of uncertain group estimates toward a population-level distribution.

Hyperparameter: Parameter governing a prior or higher-level distribution.

Loss function: Numerical cost assigned to an action and state of nature.

Utility: Quantified desirability, often the negative of loss.

Bayes action: Action minimizing posterior expected loss.

Risk function: Expected loss of a decision rule at a specified true state.

Bayes factor: Ratio of marginal likelihoods for two competing models.

Prior predictive check: Assessment of data plausibility before observing the data.

Posterior predictive check: Assessment of whether fitted-model replications resemble observed data.

Epistemic uncertainty: Uncertainty associated with limited knowledge or model parameters.

Review Questions

Short Answer Questions

1. What are the roles of the prior, likelihood, and posterior in Bayesian inference?

2. Why is the posterior predictive distribution generally preferable to plugging in a single parameter estimate?

3. State one important difference between a credible interval and a confidence interval.

4. What is partial pooling in a hierarchical model?

5. Why might a rational decision threshold differ from 0.5?

Long Answer Questions

1. Derive the Beta–Binomial posterior and explain the interpretation of its parameters.

2. Explain how prior precision and data precision determine the posterior mean in the Normal–Normal model.

3. Compare complete pooling, no pooling, and partial pooling for grouped data.

4. Discuss the strengths and limitations of Bayes factors for model comparison.

5. Explain how prior sensitivity, posterior predictive checks, and computational diagnostics support responsible Bayesian practice.

Analytical / Application Questions

1. A classifier assigns a prior disease probability of 0.04. Its positive test has likelihood ratio 8. Compute the posterior probability after a positive result and explain whether treatment should follow without considering treatment costs.

2. A company estimates the probability of a component defect using a Beta(1,19) prior. In 50 tests, 4 defects are observed. Derive the posterior distribution and posterior mean, then explain the effect of the prior.

3. A hospital must choose between two interventions with different false-positive and false-negative costs. Design a Bayesian decision rule and explain how you would evaluate calibration, subgroup fairness, and sensitivity to the cost assumptions.

Multiple Choice Questions with Answers

Question 1

Which expression correctly represents the Bayesian posterior distribution?

A. p(theta | y) is proportional to p(theta) divided by p(y | theta)

B. p(theta | y) is proportional to p(y | theta) multiplied by p(theta)

C. p(theta | y) equals p(theta) plus p(y | theta)

D. p(theta | y) depends only on the likelihood

Correct Answer: B. p(theta | y) is proportional to p(y | theta) multiplied by p(theta)

Question 2

If theta has a Beta(alpha, beta) prior and y successes are observed in n Binomial trials, what is the posterior?

A. Beta(alpha + n, beta + y)

B. Beta(alpha - y, beta - n + y)

C. Beta(alpha + y, beta + n - y)

D. Beta(alpha beta, n)

Correct Answer: C. Beta(alpha + y, beta + n - y)

Question 3

For squared-error loss, which Bayes estimator is generally optimal?

A. Posterior mean

B. Posterior variance

C. Prior mode

D. Likelihood maximum only

Correct Answer: A. Posterior mean

Question 4

What does a 95% Bayesian credible interval mean under a specified model and prior?

A. 95% of future samples will lie inside the interval

B. The fixed parameter changes randomly in 95% of repeated samples

C. The interval has 95% posterior probability of containing the parameter

D. The likelihood is 95% at every point in the interval

Correct Answer: C. The interval has 95% posterior probability of containing the parameter

Question 5

What is the main purpose of a hierarchical model?

A. To force all groups to have identical parameters

B. To share information across groups while allowing group variation

C. To eliminate the need for priors

D. To guarantee unbiased predictions

Correct Answer: B. To share information across groups while allowing group variation

Question 6

A prior disease probability is 0.10 and a test has likelihood ratio 6. What is the approximate posterior disease probability after a positive result?

A. 0.06

B. 0.40

C. 0.60

D. 0.90

Correct Answer: B. 0.40

Question 7

Which quantity is minimized by a Bayes action?

A. Prior variance

B. Posterior expected loss

C. Marginal likelihood

D. Sample size

Correct Answer: B. Posterior expected loss

Question 8

Why can diffuse priors be problematic for Bayes factors?

A. They always make the posterior improper

B. They can spread prior mass over many values and strongly affect marginal likelihood

C. They remove all model uncertainty

D. They make likelihoods impossible to calculate

Correct Answer: B. They can spread prior mass over many values and strongly affect marginal likelihood

Question 9

What does a posterior predictive check compare?

A. Prior means with posterior means only

B. Observed data with data replicated from the fitted posterior model

C. Two loss functions without data

D. A confidence interval with a credible interval

Correct Answer: B. Observed data with data replicated from the fitted posterior model

Question 10

If failing to treat is much more costly than unnecessary treatment, the optimal treatment threshold will generally be:

A. Lower than 0.5

B. Exactly 0.5

C. Higher than 1

D. Independent of costs

Correct Answer: A. Lower than 0.5