Probabilistic and Bayesian Machine Learning

Introduction and Context

Machine-learning systems rarely operate with complete information. Measurements may be noisy, labels may be missing, observations may be biased, and future events are intrinsically uncertain. A probabilistic approach represents this uncertainty explicitly rather than returning only a single prediction. Bayesian machine learning goes further by combining prior knowledge with observed evidence and producing a posterior distribution over unknown quantities.

This perspective is especially important in medical diagnosis, financial risk, autonomous systems, cybersecurity, scientific modelling, and resource planning. In such applications, an estimate without uncertainty can be unsafe: a classifier that is 60% confident should not be treated like one that is 99% confident, even if both select the same class.

This chapter develops graphical models, Bayesian inference, latent-variable methods, expectation-maximization (EM), variational inference, and Monte Carlo methods. It also considers calibration, computational trade-offs, governance, and risk-aware decision-making.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain probability distributions, conditional independence, Bayesian inference, and posterior predictive reasoning.

2. Construct and interpret directed and undirected graphical models.

3. Formulate latent-variable models and derive the EM algorithm.

4. Distinguish exact inference, variational inference, and Monte Carlo inference.

5. Apply Bayesian reasoning to decisions involving asymmetric costs and uncertainty.

6. Evaluate practical, computational, ethical, and governance limitations of probabilistic models.

1. Probabilistic Foundations

A random variable represents an uncertain quantity. A discrete variable has a probability mass function (PMF), while a continuous variable has a probability density function (PDF). For a variable X, probabilities satisfy P(X=x) >= 0 and, in the discrete case, sum_x P(X=x)=1. A continuous density need not be less than one at every point, but its integral over the entire domain equals one.

The joint distribution describes several variables together. Marginalization obtains a distribution over a subset:

P(X) = sum_z P(X,Z) for discrete Z,

or the corresponding integral for continuous Z. Conditional probability is defined, when P(Y)>0, by

P(X|Y) = P(X,Y)/P(Y).

The product rule gives P(X,Y)=P(X|Y)P(Y), and Bayes' rule follows as

P(X|Y) = P(Y|X)P(X)/P(Y).

Here P(X) is the prior, P(Y|X) is the likelihood, P(X|Y) is the posterior, and P(Y) is the evidence or marginal likelihood. In continuous models these expressions refer to densities, with probabilities obtained by integration.

A key distinction is between parameter uncertainty and predictive uncertainty. Parameter uncertainty concerns an unknown parameter theta, while predictive uncertainty concerns a future observation Y*. The posterior predictive distribution is

p(y|D) = integral p(y|theta,D)p(theta|D)dtheta,

where D is observed data. This averages predictions over plausible parameter values rather than using only a point estimate.

Independence and Conditional Independence

Variables X and Y are independent if p(x,y)=p(x)p(y). They are conditionally independent given Z if

X independent of Y given Z, written X ⟂ Y | Z,

which means p(x,y|z)=p(x|z)p(y|z). Conditional independence is central because it allows high-dimensional distributions to factorize compactly. It is also a modelling assumption: an incorrect independence structure can produce overconfident or biased predictions.

2. Bayesian Modelling and Decision Theory

A Bayesian model consists of a likelihood p(D|theta), a prior p(theta), and the posterior

p(theta|D) = p(D|theta)p(theta) / p(D),

where p(D)=integral p(D|theta)p(theta)dtheta. The denominator ensures normalization and is often computationally difficult. With independent observations conditional on theta,

p(D|theta)=product_i p(y_i|x_i,theta).

A prior is not merely an arbitrary opinion. It can encode physical constraints, historical evidence, sparsity, exchangeability, or regularization. However, prior assumptions must be documented, justified, and subjected to sensitivity analysis.

Conjugacy

A prior is conjugate to a likelihood when the posterior belongs to the same distributional family. For Bernoulli data, a Beta prior is conjugate. If theta is a success probability and theta ~ Beta(alpha,beta), with s successes and f failures, then

theta|D ~ Beta(alpha+s,beta+f).

The posterior mean is (alpha+s)/(alpha+beta+s+f). This is a smoothed estimate; the prior contributes alpha-1 and beta-1 pseudo-counts under the usual Beta parameterization. Conjugacy is valuable for teaching and some production systems, but lack of conjugacy does not invalidate Bayesian modelling.

Decisions Under Uncertainty

Prediction and decision are different. Let a be an action and L(a,y) a loss incurred when the true state is y. The Bayesian decision minimizes posterior expected loss:

a* = argmin_a E[L(a,Y)|D] = argmin_a integral L(a,y)p(y|D)dy.

For binary disease screening with action “treat” and “do not treat,” treatment should depend on treatment cost, missed-disease cost, and false-positive cost—not solely on the most probable class. This framework supports risk-aware thresholds, abstention, selective prediction, and value-of-information analysis.

3. Probabilistic Graphical Models

A graphical model represents variables as nodes and probabilistic relationships as edges. It separates structural assumptions from numerical parameters and provides a language for inference and causal reasoning.

Directed Acyclic Graphs

A Bayesian network is a directed acyclic graph (DAG) with factorization

p(x_1,...,x_n)=product_i p(x_i | pa_i),

where pa_i denotes the parents of node X_i. For example, if Weather W influences Traffic T and Accident A, while Traffic also influences Accident, the factorization is p(W)p(T|W)p(A|T,W).

The graph implies conditional independences through d-separation. A chain X -> Z -> Y implies X ⟂ Y | Z. A fork X <- Z -> Y also implies conditional independence given Z. A collider X -> Z <- Y behaves differently: X and Y may be marginally independent but become dependent when conditioning on Z or one of its descendants. Conditioning on a collider can create selection bias.

A DAG can encode causal hypotheses when its arrows have causal interpretation, but a statistical DAG alone does not prove causality. Causal conclusions require assumptions such as consistency, appropriate measurement, and control of confounding.

Markov Random Fields

An undirected graphical model, or Markov random field, represents symmetric relationships. Its distribution can be written as

p(x)=1/Z exp[-E(x)],

where E(x) is an energy function and Z is the partition function that normalizes the distribution. The partition function may be computationally expensive, especially in dense or high-dimensional networks. Markov random fields are common in image denoising, spatial models, and structured prediction.

Inference Tasks

Inference may be marginal, such as computing p(X_i|evidence), conditional, such as p(X|Y), or maximum a posteriori (MAP), such as argmax_x p(x|evidence). Exact inference is feasible in trees and graphs with low treewidth, using belief propagation or variable elimination. In loopy graphs, exact methods may be exponential in treewidth, motivating approximate methods.

4. Latent-Variable Models

A latent variable is not directly observed but explains patterns in observed data. Examples include disease status, customer segment, topic, speaker identity, and hidden state in a time series. A latent-variable model specifies a joint distribution p(X,Z|theta), where X is observed and Z is hidden.

Mixture Models

A finite mixture model assumes that each observation arises from one of K components. For a Gaussian mixture model (GMM),

p(x)=sum_{k=1}^K pi_k N(x|mu_k,Sigma_k),

where pi_k >= 0, sum_k pi_k=1, and Sigma_k is positive definite. The latent indicator Z_i identifies the component responsible for observation x_i. The posterior responsibility is

r_ik = P(Z_i=k|x_i) = pi_k N(x_i|mu_k,Sigma_k) / sum_j pi_j N(x_i|mu_j,Sigma_j).

Responsibilities express soft membership; they are not automatically calibrated probabilities of a real-world class unless the model and data-generating interpretation support that claim.

A GMM can perform density estimation and clustering, but it has assumptions: a finite number of components, Gaussian component shapes, and often local independence conditional on the component. It may also suffer from singular covariance estimates, label switching, and sensitivity to initialization.

5. Expectation-Maximization

The EM algorithm estimates parameters in models with latent variables or incomplete data. Let the complete-data likelihood be p(X,Z|theta), but only X be observed. The observed-data log-likelihood is

log p(X|theta)=log sum_Z p(X,Z|theta),

or an integral for continuous Z. The logarithm of a sum or integral makes direct optimization difficult.

EM alternates two steps:

E-step: Using current parameters theta_old, compute the posterior q(Z)=p(Z|X,theta_old).

M-step: Maximize the expected complete-data log-likelihood:

Q(theta|theta_old)=E_{p(Z|X,theta_old)}[log p(X,Z|theta)].

Set theta_new to maximize Q. Under standard regularity conditions, each exact EM iteration does not decrease the observed-data likelihood, although it may converge to a local maximum or saddle point rather than the global maximum.

GMM EM Updates

For N observations, define r_ik as above and N_k=sum_i r_ik. The updates are

pi_k = N_k/N,

mu_k = (1/N_k)sum_i r_ik x_i,

Sigma_k = (1/N_k)sum_i r_ik(x_i-mu_k)(x_i-mu_k)^T.

These formulas assume the usual maximum-likelihood GMM with full covariance matrices and no additional regularization. In practice, covariance floors, diagonal covariance restrictions, priors, or Bayesian regularization may be required.

Illustrative Numerical Example

Suppose two one-dimensional Gaussian components have equal weights, means 0 and 4, and common standard deviation 1. For observation x=1, the two normal densities are proportional to exp(-0.5(1-0)^2)=exp(-0.5) and exp(-0.5(1-4)^2)=exp(-4.5). Therefore

r_1 = exp(-0.5)/(exp(-0.5)+exp(-4.5)) ≈ 0.9820,

and r_2 ≈ 0.0180. The observation is assigned mostly, but not completely, to the first component. Hard assignment would discard this uncertainty.

6. Variational Inference

Variational inference (VI) converts inference into optimization. Let p(Z|X) be an intractable posterior and q(Z) a tractable approximation. The evidence lower bound (ELBO) is

log p(X) = L(q) + KL(q(Z)||p(Z|X)),

where

L(q)=E_q[log p(X,Z)]-E_q[log q(Z)]

and KL denotes non-negative Kullback–Leibler divergence. Consequently, L(q) <= log p(X), and maximizing the ELBO minimizes KL divergence from q to the posterior in the chosen direction.

For mean-field VI, q(Z)=product_j q_j(Z_j). Coordinate ascent updates have the form

log q_j^*(Z_j) = E_{q_{-j}}[log p(X,Z)] + constant,

where q_{-j} denotes all factors except q_j. In modern models, stochastic-gradient VI and amortized inference use neural networks to output variational parameters for each observation.

VI is generally faster and more scalable than high-fidelity Monte Carlo, but it can underestimate posterior variance, especially when the posterior is multimodal or strongly correlated. The selected variational family therefore matters as much as the optimizer.

7. Monte Carlo Methods

Monte Carlo methods approximate expectations using samples. If theta^(s) are samples from p(theta|D), then

E[f(theta)|D] ≈ (1/S)sum_{s=1}^S f(theta^(s)).

For independent samples, the standard error decreases approximately as 1/sqrt(S). In Markov chain Monte Carlo (MCMC), samples are dependent. The effective sample size is therefore smaller than the raw number of draws and depends on autocorrelation.

Important Methods

Rejection sampling draws from an easy proposal and accepts samples according to a ratio. It is conceptually simple but inefficient in high dimensions or when the proposal poorly overlaps the target.

Importance sampling estimates an expectation under p using samples from proposal q and weights w(theta)=p(theta)/q(theta). It requires q to cover the support of p; highly variable weights cause weight degeneracy.

Metropolis-Hastings proposes theta' from a proposal distribution q(theta'|theta) and accepts it with probability

alpha = min(1, [p(theta'|D)q(theta|theta')] / [p(theta|D)q(theta'|theta)]).

The unknown normalizing constant cancels. The chain must be diagnosed for burn-in, mixing, stationarity, autocorrelation, and convergence across multiple chains.

Hamiltonian Monte Carlo (HMC) uses gradients and auxiliary momentum to move efficiently through continuous posterior distributions. It can outperform random-walk methods but requires differentiable models and careful tuning.

Particle filters use weighted sequential samples for state-space models and online inference. They are useful when latent states evolve over time, although particle impoverishment and computational cost can be significant.

8. A Practical Bayesian Modelling Workflow

A robust workflow begins with the decision, not the algorithm. Define the target quantity, action space, loss function, acceptable uncertainty, and operational constraints. Then:

1. Specify variables, measurement processes, and causal or predictive assumptions.

2. Choose likelihoods compatible with data type: binary, count, continuous, categorical, censored, or survival data.

3. Select priors using domain knowledge and weakly informative defaults where appropriate.

4. Identify computational requirements and whether exact inference is possible.

5. Fit the model using EM, VI, MCMC, or a combination.

6. Check numerical diagnostics and posterior predictive behavior.

7. Evaluate calibration, discrimination, sharpness, and decision utility.

8. Conduct sensitivity analysis for priors, missingness, outliers, and alternative structures.

9. Monitor the model after deployment for drift and changing base rates.

Posterior predictive checks simulate replicated data from the fitted model and compare them with observed data. A model can have a high likelihood while failing to reproduce important structure, such as tail behavior or subgroup differences.

9. Applied Case: Bayesian Triage for Hospital Readmission

This is an illustrative case, not a report of a specific hospital.

Background and Problem

A hospital wants to identify patients at high risk of readmission within 30 days. The action is to offer an intensive discharge-support program. Capacity is limited, and false positives consume scarce staff time, while false negatives may lead to avoidable harm.

Model and Analysis

Let Y_i indicate readmission, X_i contain clinical and social variables, and theta be model parameters. A Bayesian logistic model may specify

P(Y_i=1|X_i,theta)=sigmoid(X_i^T theta),

where sigmoid(t)=1/(1+exp(-t)). Priors can be placed on coefficients, with hierarchical terms for hospitals or clinical departments to partially pool estimates. Partial pooling is useful when some departments have limited data: estimates are regularized toward a population distribution without forcing all departments to be identical.

Missing variables must not be silently treated as ordinary zeros. The model should represent missingness where plausible, document data provenance, and assess whether missingness differs by demographic group. Posterior samples generate a distribution of risk for each patient rather than a single score.

Decision Rule

Suppose the expected cost of a missed high-risk patient is C_FN and the cost of offering support to a low-risk patient is C_FP. If support prevents readmission with an estimated benefit B and the posterior probability of readmission is p, an action is justified when its expected benefit exceeds its expected cost. The exact threshold depends on all costs, program effectiveness, and capacity constraints; it is not universally 0.5.

Outcome and Governance

The hospital should evaluate calibration separately by age, sex, ethnicity, language, and socioeconomic indicators, while recognizing that fairness metrics can conflict. It should report uncertainty, permit clinician review, and avoid using the score as an automatic denial of care. Monitoring should detect changes in admission policy, coding, population, and intervention effects. If the model influences access to services, an audit trail, explanation process, appeal mechanism, and periodic independent review are appropriate.

Learning Points

The case illustrates that Bayesian modelling supports more than probability estimation. It links uncertainty to action, supports hierarchical learning, makes assumptions explicit, and encourages sensitivity analysis. Nevertheless, a mathematically sophisticated posterior cannot correct biased labels, unmeasured confounding, inequitable access, or an inappropriate decision policy.

10. Limitations, Ethics, and Contemporary Developments

Probabilistic models are conditional on their data-generating assumptions. Distribution shift can invalidate posterior predictions. A narrow prior can dominate small datasets; an overly diffuse prior can create unstable estimates. Identifiability problems arise when different parameter values produce the same observable distribution. Mixture models additionally suffer from label non-identifiability: permuting component labels leaves the likelihood unchanged.

Uncertainty itself has multiple sources: aleatoric uncertainty arises from irreducible variation, while epistemic uncertainty reflects limited knowledge or data. These should not be conflated. Approximate inference may add computational uncertainty, and data collection may introduce selection and measurement bias.

Ethically, probabilistic outputs can create an illusion of objectivity. Risk scores may encode historical discrimination, expose sensitive attributes, or be repurposed beyond their validated context. Responsible use requires data minimization, privacy protection, documentation, subgroup evaluation, human oversight, contestability, and monitoring. Regulatory obligations vary by jurisdiction and application, but high-impact systems commonly require traceability, validation, transparency, and risk management.

Contemporary research includes probabilistic programming, normalizing flows, diffusion-based generative models, deep ensembles, Bayesian neural networks, conformal prediction, causal Bayesian networks, and simulation-based inference. These methods expand modelling capacity but do not remove the need for well-defined estimands, reliable data, diagnostics, and decision analysis. In deep learning, exact Bayesian inference is usually impractical; ensembles, Laplace approximations, variational methods, and scalable sampling provide different compromises between calibration, cost, and flexibility.

Best Practices

• Begin with the decision and loss function, then choose a probabilistic model.
• Make independence, causal, missing-data, and measurement assumptions explicit.
• Use priors that are defensible, interpretable, and tested through sensitivity analysis.
• Prefer posterior predictive checks over reliance on a single fit statistic.
• Diagnose MCMC chains and monitor variational approximation quality.
• Regularize covariance matrices and guard against numerical underflow in mixture models.
• Report intervals or distributions alongside point predictions.
• Evaluate calibration and decision utility, not accuracy alone.
• Test performance across relevant subgroups and under temporal or geographic shift.
• Preserve human accountability when predictions affect rights, health, employment, or access to resources.

Chapter Summary

Probabilistic machine learning represents uncertainty through distributions, conditional relationships, and posterior predictions. Bayesian inference combines priors with likelihoods to produce posterior beliefs, while decision theory converts those beliefs into actions under explicit losses. Graphical models encode conditional independence and enable structured reasoning; DAGs support factorization and, under additional assumptions, causal analysis, whereas Markov random fields represent symmetric dependencies through energy functions.

Latent-variable models explain unobserved structure. EM alternates posterior estimation and expected complete-data optimization, with Gaussian-mixture responsibilities providing a central example. Variational inference replaces difficult posterior computation with ELBO optimization and is scalable but may underestimate uncertainty. Monte Carlo methods approximate expectations through sampling and provide flexible inference, although convergence and computational diagnostics are essential.

Responsible Bayesian machine learning requires attention to identifiability, prior sensitivity, calibration, distribution shift, fairness, privacy, and decision consequences. The most useful model is not necessarily the most complex one; it is the model whose assumptions, uncertainty, and operational risks are understood well enough to support an appropriate decision.

Key Terms

Bayesian inference: Updating prior beliefs using observed data through Bayes’ rule.

Prior: Distribution expressing beliefs or structural information before observing current data.

Likelihood: Probability model for observed data conditional on parameters.

Posterior: Distribution of unknown quantities after incorporating data.

Posterior predictive: Distribution of future observations averaged over posterior uncertainty.

Conditional independence: Independence between variables given another set of variables.

Bayesian network: A DAG-based probabilistic model factorized by parent relationships.

Markov random field: An undirected graphical model defined through compatible local dependencies.

D-separation: Graphical criterion for determining conditional independence in a DAG.

Latent variable: An unobserved variable introduced to explain observed patterns.

Mixture model: Distribution formed as a weighted combination of component distributions.

Responsibility: Posterior probability that a mixture component generated an observation.

Expectation-maximization: Iterative method alternating latent-variable inference and parameter optimization.

ELBO: Evidence lower bound optimized in variational inference.

Variational inference: Approximation of a posterior by optimizing a tractable distribution.

Monte Carlo method: Numerical method using random samples to approximate expectations.

MCMC: Sampling framework that constructs a Markov chain with a target stationary distribution.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Epistemic uncertainty: Uncertainty caused by limited knowledge or data.

Aleatoric uncertainty: Irreducible variability inherent in observations or processes.

Review Questions

Short Answer Questions

1. Distinguish a prior, likelihood, posterior, and posterior predictive distribution.

2. What does conditional independence mean, and why is it useful in graphical models?

3. State the E-step and M-step of the EM algorithm.

4. What is the ELBO, and why is it called a lower bound?

5. Explain the difference between epistemic and aleatoric uncertainty.

Long Answer Questions

1. Explain how a Bayesian network factorizes a joint distribution and discuss d-separation using chain, fork, and collider structures.

2. Derive the responsibility expression for a Gaussian mixture model and explain its interpretation.

3. Compare exact inference, variational inference, and MCMC with respect to accuracy, scalability, and diagnostics.

4. Discuss how Bayesian decision theory differs from selecting the most probable class.

5. Evaluate the principal ethical and governance risks of deploying a probabilistic risk model in healthcare.

Analytical / Application Questions

1. A fraud detector estimates a posterior fraud probability for each transaction, but investigation capacity is limited. Design a decision policy using asymmetric losses, capacity constraints, calibration, and abstention.

2. A two-component mixture model produces nearly singular covariance matrices during EM. Diagnose possible causes and propose mathematically and operationally justified remedies.

3. Design a Bayesian modelling and validation plan for a prediction problem with missing data, subgroup performance differences, temporal drift, and high costs for false negatives.

Multiple Choice Questions with Answers

Question 1

Which quantity is proportional to the posterior distribution in Bayesian inference?

A. Prior divided by likelihood

B. Likelihood multiplied by prior

C. Evidence divided by prior

D. Likelihood minus prior

Correct Answer: B. Likelihood multiplied by prior

Question 2

In a Bayesian network, the joint distribution is generally factorized as:

A. A product of each variable conditioned on its parents

B. A sum of all marginal distributions

C. A product of pairwise distances

D. A ratio of priors to likelihoods

Correct Answer: A. A product of each variable conditioned on its parents

Question 3

In a Gaussian mixture model, a responsibility represents:

A. The covariance determinant of a component

B. The prior probability of an observation

C. The posterior probability that a component generated an observation

D. The number of mixture components

Correct Answer: C. The posterior probability that a component generated an observation

Question 4

What is the principal purpose of the E-step in EM?

A. To compute the posterior distribution of latent variables using current parameters

B. To delete observations with low likelihood

C. To select the number of features

D. To calculate a model’s test accuracy

Correct Answer: A. To compute the posterior distribution of latent variables using current parameters

Question 5

Which statement about the ELBO is correct?

A. It is always greater than the log evidence

B. It equals the likelihood for every variational distribution

C. It is a lower bound on log evidence and is optimized in variational inference

D. It is defined only for Gaussian models

Correct Answer: C. It is a lower bound on log evidence and is optimized in variational inference

Question 6

Why can a collider create dependence between two otherwise independent variables?

A. Conditioning on the collider or its descendant can open a path

B. Colliders always remove all paths

C. A collider forces equal marginal means

D. Conditioning on a collider changes a DAG into an undirected graph

Correct Answer: A. Conditioning on the collider or its descendant can open a path

Question 7

For independent Monte Carlo samples, increasing the sample size by a factor of four approximately changes standard error by what factor?

A. It doubles

B. It is halved

C. It is quadrupled

D. It does not change

Correct Answer: B. It is halved

Question 8

Which issue is especially associated with mean-field variational inference?

A. It always requires exact integration

B. It can underestimate posterior variance and ignore correlations

C. It cannot optimize an objective

D. It is restricted to discrete observations

Correct Answer: B. It can underestimate posterior variance and ignore correlations

Question 9

A Bayesian decision rule selects the action that:

A. Has the highest prior probability

B. Minimizes posterior expected loss

C. Maximizes training accuracy regardless of costs

D. Has the largest parameter estimate

Correct Answer: B. Minimizes posterior expected loss

Question 10

Which diagnostic most directly checks whether a fitted probabilistic model can reproduce important features of observed data?

A. Posterior predictive checking

B. Random label permutation only

C. Parameter sorting

D. Removing all uncertainty intervals

Correct Answer: A. Posterior predictive checking