Probabilistic and Bayesian Machine Learning

Introduction

Machine-learning systems rarely operate with complete information. Sensor readings may be noisy, labels may be incomplete, and future events are inherently uncertain. A prediction such as “the patient has disease” or “the loan will default” is therefore incomplete unless it also communicates confidence and the consequences of being wrong. Probabilistic machine learning represents uncertainty explicitly and supports decisions based on distributions rather than single point predictions.

Bayesian machine learning combines probability theory with learning from data. It treats unknown quantities—such as model parameters, class labels, or missing values—as random variables and updates beliefs when evidence becomes available. This chapter develops Bayesian inference, probabilistic graphical models, latent-variable methods, approximate inference, and risk-sensitive prediction.

Learning Objectives

By the end of this chapter, a learner should be able to:

• distinguish aleatoric uncertainty from epistemic uncertainty;
• formulate likelihoods, priors, posteriors and posterior predictive distributions;
• apply Bayes’ theorem and interpret conjugate models;
• explain Bayesian networks, Markov random fields and factor graphs;
• analyse latent-variable models such as Gaussian mixtures and hidden Markov models;
• compare exact, sampling-based and variational approximate inference;
• evaluate probabilistic predictions using proper scoring rules and calibration;
• make decisions using expected loss rather than accuracy alone;
• identify ethical, computational and governance issues in probabilistic AI.

1. Probability as a Language for Uncertainty

A probability model assigns values between 0 and 1 to uncertain events. For an event A, P(A) is its probability. If A and B are events, the conditional probability of A given B is

The product rule follows directly:

For mutually exclusive and exhaustive hypotheses H₁,…,Hₖ, the law of total probability is

Bayes’ theorem is obtained by equating two forms of the joint probability:

Here H is a hypothesis and D is observed data. P(H) is the prior, P(D | H) is the likelihood, P(D) is the evidence or marginal likelihood, and P(H | D) is the posterior. The evidence normalises the posterior so that its probabilities sum or integrate to one.

A probability distribution may be discrete, such as a categorical distribution, or continuous, such as a Gaussian distribution. For a continuous variable X with density p(x), probabilities are obtained by integration; p(x) itself is not a probability and may exceed 1 when measured in a sufficiently small unit.

Sources of uncertainty

• Aleatoric uncertainty is variability inherent in the process, such as measurement noise or random demand. More training data may not remove it.
• Epistemic uncertainty results from limited knowledge, such as uncertain parameters or insufficient coverage of a population. Additional representative data can reduce it.
• Distribution shift occurs when the deployment distribution differs from the training distribution. A confident posterior under the wrong model may still be unreliable.

2. Bayesian Inference for Parametric Models

Let D = {x₁,…,xₙ} be observations and θ the unknown parameter vector. A Bayesian model specifies a prior p(θ), a likelihood p(D | θ), and the posterior

where

The integral applies to continuous parameters; a sum is used for discrete parameters. The posterior combines prior knowledge with evidence. It is not simply a confidence interval and should not be interpreted as the long-run frequency of θ being random in a frequentist parameterisation.

Posterior prediction

For a future observation x*, Bayesian prediction averages over parameter uncertainty:

For a class label y and features x, the corresponding expression is

This is often safer than substituting a single estimate θ̂ because it includes uncertainty about the parameters.

Point estimates and Bayesian estimates

The maximum a posteriori estimate is

Using Bayes’ theorem, this is equivalent to maximising log likelihood plus log prior:

A maximum-likelihood estimate omits the prior. A posterior mean, median or full posterior may be preferable when uncertainty is important. MAP estimation is therefore a point approximation, not a replacement for posterior inference.

Worked example: Beta–Binomial inference

Suppose a quality-control engineer observes 8 successful products in 10 trials. Let θ be the unknown probability of success. Use a Beta(2,2) prior:

The Binomial likelihood is proportional to θ⁸(1−θ)². Multiplying prior and likelihood gives

For a Beta(a,b) distribution, the mean is a/(a+b), so the posterior mean is 10/14 ≈ 0.714. The prior mean was 2/4 = 0.5, and the data move the estimate toward the observed success rate 0.8 while retaining prior uncertainty.

The posterior predictive probability that the next product succeeds is

The Binomial coefficient is included when calculating the probability of a complete future sample, but it does not depend on θ and therefore does not affect the posterior kernel.

Conjugacy

A prior is conjugate to a likelihood when the posterior belongs to the same distributional family. Examples include Beta–Binomial, Gamma–Poisson and Gaussian–Gaussian models. Conjugacy provides analytical convenience but is not a requirement of Bayesian modelling. A convenient prior should not be selected if it is substantively inappropriate.

3. Probabilistic Graphical Models

A probabilistic graphical model represents conditional relationships using a graph. It can make assumptions explicit, reduce the number of parameters, and support inference in complex systems.

Bayesian networks

A Bayesian network is a directed acyclic graph (DAG). Each node is a random variable and each node has a conditional distribution given its parents. The joint distribution factorises as

For example, if Weather influences Traffic and Traffic influences ArrivalTime, then the graph expresses a structured dependence rather than asserting that all variables are independent. The graph encodes conditional independences through d-separation. These independences are modelling assumptions and must be assessed against domain knowledge and data.

A Bayesian network can support diagnostic reasoning, prediction and intervention analysis, but ordinary observational conditioning is not automatically equivalent to intervention. Causal claims require additional assumptions, such as an appropriate causal graph and absence or control of relevant confounding.

Markov random fields

A Markov random field uses an undirected graph. Its distribution is commonly written as

where C is a set of cliques, ψ_c is a non-negative potential over clique variables, and Z = ∑_x ∏_c ψ_c(x_c) (or the corresponding integral) is the partition function. Potentials need not be individually normalised. Computing Z can be difficult, particularly in densely connected graphs.

Factor graphs

A factor graph is bipartite: variable nodes connect to factor nodes. It represents

and is useful for message passing, including belief propagation. Exact belief propagation is guaranteed on trees; on graphs with loops, loopy belief propagation is approximate and may fail to converge or may produce inaccurate marginals.

4. Latent-Variable Models

A latent variable is not directly observed but helps explain observed data. Latent models are useful for clustering, sequence analysis, missing-data problems and representation learning. They are also vulnerable to identifiability problems: different parameter values may imply the same observed distribution.

Gaussian mixture models

A Gaussian mixture with K components introduces a latent assignment zᵢ for each observation xᵢ:

where πₖ ≥ 0, ∑ₖπₖ = 1, μₖ is the component mean, and Σₖ is a positive-definite covariance matrix. The posterior responsibility of component k is

The expectation–maximisation (EM) algorithm alternates:

1. E-step: calculate responsibilities rᵢₖ using current parameters.
2. M-step: update parameters using weighted observations, for example πₖ = (1/n)∑ᵢrᵢₖ and μₖ = (∑ᵢrᵢₖxᵢ)/(∑ᵢrᵢₖ).

EM increases the likelihood at each iteration under standard conditions, but it may converge to a local maximum or a degenerate solution in which a covariance collapses. Multiple initialisations, covariance regularisation and model-selection criteria can help.

Hidden Markov models

A hidden Markov model (HMM) contains hidden states zₜ and observations xₜ. The first-order assumptions are

The forward algorithm computes filtering probabilities efficiently; the Viterbi algorithm finds the most probable state sequence; and Baum–Welch, an EM procedure, estimates transition and emission parameters. HMMs are applied to speech, activity recognition and equipment monitoring, but the Markov assumption may be too restrictive for long-range dependencies.

5. Approximate Bayesian Inference

Exact inference is tractable only for certain structures and distributions. In realistic models, the posterior may involve high-dimensional integrals or an intractable normalising constant.

Monte Carlo methods

If samples θ⁽¹⁾,…,θ⁽ᴺ⁾ are drawn from p(θ | D), posterior expectations can be approximated by

Markov chain Monte Carlo (MCMC) constructs a Markov chain whose stationary distribution is the posterior. Metropolis–Hastings proposes θ′ from q(θ′|θ) and accepts it with probability

The ratio is valid when proposal and target densities are evaluated consistently. Burn-in, mixing, autocorrelation and convergence diagnostics must be considered. Hamiltonian Monte Carlo can explore continuous, correlated posteriors more efficiently, but requires differentiable models and careful tuning.

Variational inference

Variational inference selects a tractable distribution q(θ) to approximate the posterior. It minimises the Kullback–Leibler divergence

Because the evidence is often intractable, optimisation uses the evidence lower bound (ELBO):

Since log p(D) = ELBO(q) + KL(q || p), maximising the ELBO reduces the divergence from q to the posterior. Mean-field variational inference assumes q(θ)=∏ⱼqⱼ(θⱼ), which is computationally convenient but can underestimate posterior variance and miss dependence.

Laplace approximation

Near a posterior mode θ̂, the log posterior can be approximated quadratically. If H is the negative Hessian of the log posterior at θ̂, then

This requires a well-defined local mode and a positive-definite Hessian. It performs poorly for skewed, multimodal or boundary-constrained posteriors.

6. Evaluating Probabilistic Predictions

Accuracy alone ignores confidence and the cost of errors. A classifier that assigns probabilities 0.51 and 0.99 to two correct predictions has the same accuracy, but the second may be much more useful if it is calibrated.

A model is calibrated when among cases assigned probability p, the event occurs approximately p of the time. Calibration can be assessed with reliability diagrams, expected calibration error, and calibration intercept and slope. Calibration methods include Platt scaling, isotonic regression and Bayesian calibration, but calibration must be evaluated on held-out data.

Common proper scoring rules include:

• Log loss: for a true class y and predicted probability p(y|x), −log p(y|x). It strongly penalises assigning near-zero probability to the outcome that occurs.
• Brier score: for binary y ∈ {0,1} and prediction p, (p−y)². Lower is better.
• Continuous ranked probability score (CRPS): evaluates a full predictive distribution for a continuous outcome and rewards both calibration and concentration.

A scoring rule is proper if the expected score is optimised by reporting the true predictive distribution. Proper scores discourage strategic overconfidence, although no metric replaces analysis of the deployment context.

7. Decision-Making Under Uncertainty

Prediction and decision are distinct. Let actions be a ∈ A, outcomes y, and loss L(a,y). The Bayesian decision rule minimises posterior expected loss:

For continuous outcomes, replace the sum by an integral. Under 0–1 loss, choosing the most probable class is optimal. Under asymmetric loss, it may be rational to choose a less probable class. For example, if missing a fire has much greater cost than a false alarm, an alarm threshold should be lower than 0.5.

A useful professional workflow is: define actions and losses; estimate the predictive distribution; calculate expected losses; conduct sensitivity analysis; and monitor realised outcomes. Probabilities should not be presented without clarifying the reference population, time horizon and decision threshold.

8. Illustrative Case: Bayesian Predictive Maintenance

Background

An industrial manufacturer monitors vibration from pumps. Failures are expensive, but unnecessary maintenance also interrupts production. A deterministic classifier previously produced many false alarms and gave no indication of uncertainty.

Problem

The engineering team needs a probability that a pump will fail within 14 days, together with a maintenance decision that reflects unequal costs. Sensors are noisy, and only a small number of failures have been recorded.

Analysis and approach

An illustrative Bayesian model uses vibration summaries x, operating temperature and pump age. A logistic likelihood models failure probability:

Weakly informative priors are placed on β, and posterior samples are obtained using HMC. The team reports posterior predictive probabilities rather than only posterior means. It evaluates log loss and calibration on a time-ordered test set to avoid leakage from future maintenance records.

Suppose a pump has posterior predictive failure probability 0.18. Let the loss of an unnecessary maintenance action be 1 unit and the loss of an undetected failure be 8 units. If maintenance is selected, expected loss is 1. If no maintenance is selected, expected loss is 8 × 0.18 = 1.44. Maintenance is therefore preferred. The threshold for maintenance is 1/(1+8) ≈ 0.111, not 0.5.

Outcome

The decision policy reduces expected operational loss in simulation, while uncertainty intervals identify pumps for which more sensor data are valuable. Engineers retain authority to override recommendations when safety conditions or model limitations apply.

Learning points

The case demonstrates that calibrated probability, posterior uncertainty, asymmetric loss and operational constraints must be considered together. A model with high classification accuracy can still produce poor maintenance decisions if it is miscalibrated or if the threshold ignores costs.

9. Limitations, Ethics and Governance

Bayesian models do not eliminate bias; they formalise assumptions and can encode biased priors or historical data. In high-impact domains, practitioners should document data provenance, prior selection, missingness mechanisms, protected-group performance, calibration and uncertainty. Sensitive decisions require human oversight, contestability and appropriate legal compliance.

A posterior probability may appear precise even when the model omits important variables or deployment shift. Robustness checks should include alternative priors, posterior predictive checks, subgroup evaluation and stress tests under plausible distribution changes. Missing-not-at-random data require explicit modelling rather than automatic imputation.

Privacy is also relevant. Bayesian updating can expose sensitive information when repeated observations are combined. Access controls, aggregation, secure computation and, where suitable, differential privacy should be evaluated. Reproducibility requires versioned data and code, recorded random seeds where relevant, model cards, decision thresholds and monitoring plans.

10. Contemporary Developments and Best Practice

Modern probabilistic deep-learning systems use neural networks to parameterise distributions, latent representations or likelihoods. Bayesian neural networks, deep ensembles, stochastic variational methods and conformal prediction address different aspects of uncertainty; none should be assumed universally superior. Diffusion and generative models also define complex predictive distributions, although likelihood evaluation and calibration remain challenging.

Best practice is to:

1. define the prediction target, population and time horizon;
2. separate aleatoric and epistemic uncertainty where possible;
3. choose a likelihood appropriate to the data-generating process;
4. make prior assumptions transparent and conduct sensitivity analysis;
5. validate calibration, sharpness and decision utility;
6. use temporal or group-aware validation when ordinary random splits leak information;
7. diagnose approximate-inference convergence and approximation error;
8. monitor drift and recalibrate after deployment;
9. communicate uncertainty in language suitable for the decision-maker;
10. preserve human accountability for consequential decisions.

Chapter Summary

Probabilistic machine learning represents uncertain outcomes and unknown parameters through probability distributions. Bayes’ theorem updates prior beliefs using data, producing a posterior and a posterior predictive distribution. Graphical models encode conditional structure through Bayesian networks, Markov random fields and factor graphs. Latent-variable models such as Gaussian mixtures and HMMs explain unobserved structure, while EM provides a practical optimisation method.

Exact inference is often difficult, motivating MCMC, variational inference and Laplace approximations. These methods involve different trade-offs in accuracy, computation and uncertainty representation. Probabilistic predictions must be assessed for calibration and proper scoring performance, not merely classification accuracy. Finally, Bayesian decision theory converts predictive distributions into actions by minimising expected loss, making it particularly valuable for risk-sensitive applications.

Key Terms

Bayesian inference: Updating a prior distribution using observed evidence.

Prior: Distribution representing beliefs about an unknown quantity before current data.

Likelihood: Probability model for observed data conditional on parameters.

Posterior: Distribution of unknown quantities after observing data.

Evidence: Marginal probability of observed data used to normalise the posterior.

Posterior predictive: Distribution of future observations after averaging over posterior uncertainty.

Aleatoric uncertainty: Irreducible randomness in the data-generating process.

Epistemic uncertainty: Uncertainty caused by limited knowledge or data.

Bayesian network: Directed acyclic graphical model with local conditional distributions.

Markov random field: Undirected graphical model defined through clique potentials.

Factor graph: Bipartite representation of variables and factors.

Latent variable: Unobserved variable introduced to explain observed data.

Gaussian mixture model: Weighted mixture of Gaussian component distributions.

Hidden Markov model: Sequential model with hidden states and observed emissions.

MCMC: Sampling method that constructs a chain with the posterior as stationary distribution.

Variational inference: Approximation of a posterior by optimising a tractable distribution.

Calibration: Agreement between predicted probabilities and observed frequencies.

Proper scoring rule: Scoring rule whose expected optimum is truthful probability reporting.

Expected loss: Average decision cost under a predictive distribution.

Distribution shift: Difference between training and deployment data distributions.

Review Questions

Short Answer Questions

1. Distinguish aleatoric and epistemic uncertainty.
2. Define prior, likelihood, posterior and evidence.
3. Why is posterior prediction preferable to prediction using only a point estimate in some applications?
4. State one difference between a Bayesian network and a Markov random field.
5. What does calibration mean for a probabilistic classifier?

Long Answer Questions

1. Derive Bayes’ theorem from the product rule and explain every term in the result.
2. Explain Gaussian mixture models and describe the E-step and M-step of EM.
3. Compare MCMC, variational inference and Laplace approximation in terms of accuracy and computational cost.
4. Explain how conditional independence supports factorisation in probabilistic graphical models.
5. Discuss why expected loss may produce a decision different from the most probable class.

Analytical / Application Questions

1. A medical test has sensitivity 0.90, specificity 0.95 and disease prevalence 0.02. Use Bayes’ theorem to calculate the probability of disease given a positive test, and explain why prevalence matters.
2. A fraud detector predicts a 0.12 probability of fraud. A false investigation costs 2 units, while missing fraud costs 20 units. Derive the decision threshold and determine whether investigation is preferred.
3. Design a validation and monitoring plan for a Bayesian predictive-maintenance model operating under seasonal distribution shift. Include calibration, scoring rules, uncertainty checks and governance controls.

Multiple Choice Questions with Answers

Question 1

Which quantity is updated by observed data in Bayesian inference?

A. The prior distribution

B. The posterior distribution

C. The sample space

D. The loss function

Correct Answer: B. The posterior distribution

Question 2

What does the posterior predictive distribution represent?

A. Predictions using only the maximum-likelihood parameter

B. Predictions averaged over posterior parameter uncertainty

C. The prior probability of the training data

D. The normalising constant alone

Correct Answer: B. Predictions averaged over posterior parameter uncertainty

Question 3

Which graph is required to be directed and acyclic in a Bayesian network?

A. The factor graph

B. The Markov random field

C. The dependency graph

D. The covariance matrix

Correct Answer: C. The dependency graph

Question 4

In a Gaussian mixture model, what does a responsibility represent?

A. The variance of the complete dataset

B. The posterior probability that a component generated an observation

C. The learning rate used by EM

D. The number of mixture components

Correct Answer: B. The posterior probability that a component generated an observation

Question 5

Which algorithm alternates between estimating latent assignments and updating parameters?

A. Expectation–maximisation

B. Gradient clipping

C. Principal-component analysis

D. Bootstrap aggregation

Correct Answer: A. Expectation–maximisation

Question 6

What is a principal limitation of mean-field variational inference?

A. It always requires labelled data

B. It may underestimate posterior variance and ignore dependencies

C. It cannot optimise an objective function

D. It is restricted to binary outcomes

Correct Answer: B. It may underestimate posterior variance and ignore dependencies

Question 7

Under 0–1 loss, the Bayes-optimal classification decision is generally the class with the:

A. Smallest prior probability

B. Largest posterior predictive probability

C. Largest training-set variance

D. Smallest sample size

Correct Answer: B. Largest posterior predictive probability

Question 8

For a binary event, a Brier score is calculated as:

A. The absolute value of the log likelihood

B. The squared difference between predicted probability and outcome

C. The ratio of prior to evidence

D. The posterior mean divided by its variance

Correct Answer: B. The squared difference between predicted probability and outcome

Question 9

If investigating fraud costs 2 units and missing fraud costs 20 units, the investigation threshold under the case’s loss convention is:

A. 0.02

B. 0.10

C. 0.50

D. 0.90

Correct Answer: B. 0.10

Question 10

Which uncertainty is most directly associated with limited training data?

A. Aleatoric uncertainty

B. Epistemic uncertainty

C. Measurement units

D. Deterministic uncertainty

Correct Answer: B. Epistemic uncertainty