13. Probabilistic Graphical, Latent and Sequential Models

Introduction and context

Machine-learning data are rarely collections of independent observations. A medical diagnosis depends on symptoms, tests, age, and disease history; a word depends on neighbouring words and grammatical structure; a financial event may be influenced by an unobserved economic regime. Models that represent such relationships require more than a probability distribution over isolated variables. They require a language for expressing dependence, uncertainty, hidden causes, and temporal evolution.

Probabilistic graphical models provide that language. A graph describes which variables interact directly, while probability theory specifies how uncertainty is distributed across them. Bayesian networks represent directed, often causal, relationships. Markov random fields represent undirected compatibility relationships. Latent-variable models introduce unobserved factors that explain patterns in observed data. Mixture models describe heterogeneous populations, and the expectation-maximization algorithm estimates their parameters when hidden assignments are involved. Markov chains and related stochastic processes extend these ideas to sequential data, where the present depends on the past.

These models are important in postgraduate machine learning because they connect mathematical structure with inference and decision-making. They also clarify a distinction that is frequently overlooked: a model may predict well without representing causality, and a compact graph may simplify computation without proving that the omitted relationships are unimportant.

Learning objectives

After studying this chapter, the learner should be able to:

1. Explain conditional independence and factorization in probabilistic graphical models.

2. Construct and interpret Bayesian networks and Markov random fields.

3. Distinguish observed, latent, and missing variables.

4. Derive and apply mixture-model and expectation-maximization procedures.

5. Analyse Markov chains, transition matrices, stationary distributions, and stochastic processes.

6. Select appropriate models for structured and sequential data.

7. Evaluate identifiability, computational complexity, uncertainty, and modelling assumptions.

8. Discuss professional, ethical, and governance issues in probabilistic AI systems.

The probabilistic foundation: dependence, independence, and factorization

Let a random vector be denoted by $X=(X_1,\ldots,X_n)$. Its joint distribution $p(x_1,\ldots,x_n)$ contains all marginal and conditional distributions. Directly estimating this joint distribution is difficult because the number of possible configurations grows rapidly with the number of variables. Graphical models reduce this burden by encoding conditional independences.

Two variables $X$ and $Y$ are independent when $p(x,y)=p(x)p(y)$, written $X\perp Y$. Conditional independence given $Z$ is written $X\perp Y\mid Z$ and means

$p(x,y\mid z)=p(x\mid z)p(y\mid z)$.

The chain rule always permits the factorization

$p(x_1,\ldots,x_n)=\prod_{i=1}^{n}p(x_i\mid x_1,\ldots,x_{i-1})$.

A graphical model becomes valuable when its structure allows many of these conditioning sets to be reduced. This provides three benefits: fewer parameters, interpretable assumptions, and often more efficient inference. The price is model misspecification if the graph fails to reflect important relationships.

A useful distinction is between statistical dependence and causal influence. An arrow in a Bayesian network can encode a causal hypothesis, but a graph learned only from observational correlations does not automatically establish causality. Causal interpretation requires assumptions such as appropriate interventions, temporal ordering, and control of confounding.

Bayesian networks: directed representations of structured uncertainty

A Bayesian network is a directed acyclic graph (DAG) whose nodes represent random variables and whose directed edges represent direct dependence assumptions. If $\operatorname{pa}(X_i)$ denotes the parents of node $X_i$, the joint distribution factorizes as

$p(x_1,\ldots,x_n)=\prod_{i=1}^{n}p(x_i\mid \operatorname{pa}(X_i))$.

Consider the network $D\rightarrow S\leftarrow R$, where $D$ is disease, $S$ is symptom, and $R$ is a risk factor. It expresses that disease and risk factor are independent before observing the symptom if no other connection exists, but they can become statistically associated after conditioning on their common effect. This phenomenon is called explaining away. Graphical reasoning must therefore distinguish chains, forks, and colliders.

In a chain $X\rightarrow Y\rightarrow Z$, conditioning on $Y$ blocks the association between $X$ and $Z$. In a fork $X\leftarrow Y\rightarrow Z$, conditioning on the common cause $Y$ also blocks the path. In a collider $X\rightarrow Y\leftarrow Z$, the path is normally blocked, but conditioning on $Y$ or on a descendant of $Y$ can open it. These rules are formalized by d-separation. If all paths between two sets of variables are blocked according to d-separation, the corresponding conditional independence follows from the DAG under the Markov condition.

Inference and learning in Bayesian networks

Inference may be predictive, diagnostic, or intercausal. Predictive inference computes effects from causes, such as $p(S\mid D)$. Diagnostic inference reverses the direction of reasoning, such as $p(D\mid S)$, usually using Bayes' theorem. Exact inference methods include variable elimination and belief propagation on trees. In densely connected or loopy networks, exact inference can be computationally expensive; approximate methods include Monte Carlo sampling, variational inference, and loopy belief propagation.

Parameter learning is relatively straightforward when all variables are observed: maximum likelihood estimates can often be obtained from empirical counts for discrete conditional probability tables. With hidden variables, the likelihood contains a sum over unobserved configurations, which motivates expectation-maximization. Structure learning is more difficult because it searches over graphs and must balance fit, complexity, and prior knowledge. Score-based methods use criteria such as Bayesian information criterion, whereas constraint-based methods test conditional independences.

Markov random fields and energy-based modelling

A Markov random field (MRF), or undirected graphical model, represents symmetric relationships. Let $G=(V,E)$ be an undirected graph. A positive distribution can be written through clique potentials as

$p(x)=\frac{1}{Z}\prod_{C\in\mathcal C}\psi_C(x_C),

where $\psi_C$ is a non-negative potential over clique $C$, and $Z$ is the partition function that normalizes the distribution. Equivalently, with energy functions,

$p(x)=\frac{1}{Z}\exp[-E(x)].

Unlike a Bayesian network, an MRF does not assign a generative direction to an edge. It is often appropriate when the goal is to express compatibility among neighbouring objects rather than a causal process. In image denoising, for example, neighbouring pixels are encouraged to take similar values while observations are constrained by noisy measurements. In a conditional random field, the model directly represents $p(y\mid x)$, making it useful for structured prediction such as sequence labelling and image segmentation.

The local Markov property states that a variable is conditionally independent of non-neighbours given its neighbours. This makes local computation possible, but global normalization through $Z$ can be difficult. Approximate inference and learning are therefore central in large MRFs. The choice between directed and undirected models depends on whether a generative factorization, causal interpretation, symmetric compatibility, or conditional prediction is most important.

Latent variables and mixture models

A latent variable is not directly observed but is introduced to explain regularities in observed data. Latent variables may represent a physical state, customer segment, topic, disease subtype, or behavioural regime. They are not necessarily fictitious; they are model constructs whose credibility depends on predictive performance, interpretability, and external validation.

Suppose observations $x_1,\ldots,x_N$ arise from one of $K$ unobserved components. Let $z_i\in\{1,\ldots,K\}$ be the component assignment, with mixing proportions $\pi_k$, where $\pi_k\geq0$ and $\sum_k\pi_k=1$. A finite mixture model is

$p(x_i)=\sum_{k=1}^{K}\pi_k f(x_i\mid\theta_k).

For a Gaussian mixture model (GMM), $f$ is a multivariate normal density:

$p(x_i)=\sum_{k=1}^{K}\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)$.

The latent assignment gives the complete-data model

$p(x_i,z_i=k)=\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k).

Mixture models can represent multimodal distributions more flexibly than a single Gaussian. However, a component is not automatically a true natural category. Components may overlap, depend on feature scaling, or change when $K$ changes. Labels are also interchangeable: permuting component indices leaves the likelihood unchanged, a property called label switching.

Expectation-maximization

Maximum likelihood for latent-variable models is difficult because the observed-data likelihood is

$\ell(\Theta)=\sum_{i=1}^{N}\log\sum_{k=1}^{K}p(x_i,z_i=k\mid\Theta)$,

and the logarithm of a sum prevents direct separation of component parameters. The expectation-maximization (EM) algorithm alternates between estimating hidden assignments and updating parameters.

In the E-step, calculate responsibilities

$\gamma_{ik}=p(z_i=k\mid x_i,\Theta^{old})

=\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{\sum_{j=1}^{K}\pi_j\mathcal N(x_i\mid\mu_j,\Sigma_j)}.$

These are soft assignments: an observation may belong partly to several components. Define $N_k=\sum_i\gamma_{ik}$. In the M-step, update

$\pi_k=\frac{N_k}{N},\qquad

\mu_k=\frac{1}{N_k}\sum_i\gamma_{ik}x_i,$

and

$\Sigma_k=\frac{1}{N_k}\sum_i\gamma_{ik}(x_i-\mu_k)(x_i-\mu_k)^T.$

The two steps repeat until the likelihood or parameters change by less than a chosen tolerance. EM does not generally find the global maximum; it monotonically improves the likelihood at each iteration but may converge to a local optimum. Multiple initializations, regularization, covariance constraints, and sensible stopping criteria are therefore important. Singular covariance matrices can cause unbounded likelihood, particularly when a component collapses around one observation. A minimum variance or positive-definite regularizer helps prevent this failure.

The deeper principle behind EM is the evidence lower bound (ELBO). For any distribution $q(z)$,

$\log p(x)\geq \mathbb E_q[\log p(x,z)]-\mathbb E_q[\log q(z)].$

The E-step chooses the posterior distribution over latent variables, while the M-step maximizes the expected complete-data log-likelihood. This interpretation connects EM with variational inference and modern latent-variable learning.

Markov chains and sequential dependence

A discrete-time stochastic process is a sequence of random variables $X_0,X_1,\ldots$. It satisfies the first-order Markov property when

$p(X_{t+1}\mid X_t,X_{t-1},\ldots,X_0)=p(X_{t+1}\mid X_t).

The present state contains all information from the past needed to predict the next state. This is a modelling assumption, not a universal fact. Higher-order dependence can be converted to first-order form by defining a state as a tuple of recent observations, although the state space then becomes larger.

For a finite Markov chain, the transition matrix $P$ has entries $P_{ij}=p(X_{t+1}=j\mid X_t=i)$. Each row sums to one. If the initial distribution is a row vector $\pi_0$, then the distribution after $t$ steps is $\pi_t=\pi_0P^t$. A stationary distribution $\pi$ satisfies

$\pi=\pi P,\qquad \sum_i\pi_i=1.$

A chain is irreducible if every state can eventually be reached from every other state. It is aperiodic if returns do not occur only at fixed multiples of a period. A finite irreducible, aperiodic chain has a unique stationary distribution and converges to it from any initial distribution. This distinction matters: a stationary distribution describes long-run behaviour, whereas a transition matrix describes one-step dynamics.

Markov chains support filtering, prediction, simulation, and expected hitting-time analysis. In a random walk, for instance, transition probabilities determine how probability mass spreads over a graph. In reinforcement learning, the transition model describes how actions alter states, although the resulting process is usually a Markov decision process because actions and rewards are included.

Hidden Markov models

A hidden Markov model (HMM) contains a latent state sequence $Z_1,\ldots,Z_T$ that follows a Markov chain, and observations $X_t$ conditionally independent of other observations given $Z_t$. Its joint distribution is

$p(z_{1:T},x_{1:T})=p(z_1)\prod_{t=2}^{T}p(z_t\mid z_{t-1})\prod_{t=1}^{T}p(x_t\mid z_t).

The forward algorithm computes filtered state probabilities efficiently using dynamic programming. The backward algorithm computes information flowing from future observations. Together, forward-backward inference gives smoothed probabilities $p(Z_t\mid x_{1:T})$. The Viterbi algorithm finds the most probable state sequence, which is not necessarily obtained by choosing the most probable state independently at each time. HMM parameters can be learned with Baum-Welch, an EM procedure.

HMMs are useful for speech recognition, activity recognition, biological sequence analysis, predictive maintenance, and regime detection. Their assumptions—discrete or finite state representation, first-order dependence, and conditional independence of observations—can be restrictive. State-space models with continuous latent states, Kalman filters, particle filters, and recurrent or transformer-based architectures provide alternatives when richer dynamics are required.

A unifying view of inference

Graphical models turn a global probability distribution into local factors. Inference then consists of combining factors and marginalizing or maximizing over variables. Sum-product computation produces marginal probabilities; max-product computation produces most probable assignments. Dynamic programming is efficient when the graph has a tree-like structure. The relevant structural measure for exact inference is often treewidth: graphs with small treewidth permit manageable elimination, whereas densely connected graphs may make exact inference intractable.

Three broad approximation strategies are common. Monte Carlo methods draw samples and estimate expectations, but may mix slowly in complex distributions. Variational methods optimize a tractable approximation, often producing efficient but biased estimates. Sequential Monte Carlo represents a changing distribution with weighted particles and is especially suitable for nonlinear temporal systems. Model selection must therefore consider not only statistical fit but also inference cost, uncertainty calibration, and the consequences of approximation error.

Applications and professional use

In healthcare, Bayesian networks can combine symptoms, laboratory tests, and prior risk to support diagnosis. A responsible system should expose uncertainty, distinguish correlation from intervention, and avoid treating a probability as a definitive diagnosis. In finance, Markov-switching models can represent latent volatility or economic regimes, but structural breaks and feedback effects make historical transition probabilities unstable. In manufacturing, HMMs and state-space models can detect hidden equipment conditions from sensor streams and support preventive maintenance.

In natural-language processing, latent topic models represent documents as mixtures of topics, while sequence models capture syntactic or semantic context. In computer vision, MRFs and conditional random fields impose spatial coherence in segmentation. In customer analytics, mixture models can identify probabilistic segments, but segment labels should not be used as immutable identities or as proxies for protected characteristics.

For organizations, the model-selection question is not simply “Which model has the highest likelihood?” Decision-makers should ask whether the states or components are interpretable, whether the data-generating process is stable, whether errors are costly for particular groups, and whether the system can be audited and updated. A simpler calibrated model may be preferable to a more expressive model whose assumptions and uncertainty cannot be communicated.

Illustrative case study: predictive maintenance with a hidden Markov model

Background

A manufacturer operates industrial pumps equipped with vibration, temperature, and pressure sensors. Failures are costly, but labelled examples of each stage of degradation are scarce. Engineers believe that pumps move through unobserved operating conditions such as healthy, mildly degraded, and severely degraded.

Situation and problem

A threshold rule on vibration produces many false alarms because vibration varies with load. The organization needs an early-warning system that uses several signals and recognizes persistence over time.

Analysis

An HMM is proposed. The latent state $Z_t$ represents equipment condition, while $X_t$ contains sensor features. The transition matrix expresses that a healthy pump is likely to remain healthy, degradation usually progresses gradually, and recovery from severe degradation is unlikely without maintenance. Emission distributions describe the sensor patterns associated with each state. The first-order Markov assumption is tested by examining whether recent histories improve prediction after the current inferred state is known.

Approach and solution

Engineers initialize the states using maintenance records and clustering, then estimate transition and emission parameters using Baum-Welch on long sensor sequences. The forward algorithm produces the probability that the pump is currently in each state. A maintenance alert is issued when the probability of the severe-degradation state exceeds a threshold selected according to the relative costs of missed failures and unnecessary interventions. The Viterbi path is used for retrospective interpretation rather than as the only operational signal.

Data are split chronologically to avoid leakage from future operating conditions. Performance is assessed through lead time, false-alarm rate, calibration, and cost-weighted maintenance outcomes. Engineers review alerts to detect sensor drift and changes after equipment redesign.

Outcome

The model provides a probabilistic warning several observation periods before many failures in the evaluation set and reduces reliance on a single noisy sensor. Its value is not merely the inferred label; it is the combination of uncertainty, temporal persistence, and decision thresholds. When operating conditions change, transition and emission parameters require recalibration.

Learning points

The case illustrates that latent states are useful hypotheses rather than directly observed truths. It also shows why temporal validation, domain-informed structure, calibration, and cost-sensitive decisions are essential. An alerting model should support engineers, not conceal uncertainty or automatically authorize expensive maintenance.

Challenges, limitations, and common mistakes

A central difficulty is identifiability. Different parameter values or latent-state labels may yield the same observed distribution. Latent components can be statistically convenient but substantively ambiguous. Overfitting occurs when too many components, states, or edges are introduced relative to the data. Underfitting occurs when important heterogeneity or temporal dependence is suppressed.

Common mistakes include interpreting every edge as causal, conditioning on a collider without recognizing the induced dependence, using a stationary distribution when a chain is not ergodic, and selecting the number of mixture components solely by visual appeal. EM may be stopped at a poor local optimum, while unconstrained Gaussian covariances may become singular. In sequential applications, random train-test splitting can leak information across adjacent time points and produce unrealistic performance.

Numerical issues are also important. Probabilities multiplied across long sequences underflow, so implementations use log probabilities or scaling. Covariance matrices must remain positive definite. Approximate inference may be systematically biased, and uncertainty intervals should reflect both parameter uncertainty and model uncertainty where feasible.

Ethics, governance, and responsible use

Probabilistic outputs can appear objective while encoding biased data, inappropriate priors, or unequal measurement quality. A posterior probability is conditional on the model and evidence; it is not a guarantee. High-stakes systems should document variables, training populations, missing-data procedures, priors, validation conditions, and known failure modes.

Latent representations create particular governance risks. A hidden “risk state” may be used to ration services without a transparent justification. Proxy variables may reproduce discrimination even when protected attributes are excluded. Organizations should conduct subgroup calibration and error analysis, establish human review and appeal procedures, restrict secondary use, and monitor drift. Where personal or sensor data are involved, privacy, retention, access control, and applicable data-protection requirements must be addressed.

Current developments and best practices

Modern probabilistic machine learning combines graphical structure with deep neural networks. Neural parameterizations can make emissions, transition probabilities, or conditional distributions more expressive, producing deep state-space models and neural HMMs. Probabilistic programming systems allow analysts to specify generative models and perform inference using sampling or variational techniques. Differentiable programming makes it possible to train structured probabilistic components jointly with representation learners.

Best practice begins with a clearly stated data-generating and decision problem. Choose variables and graph structure using domain knowledge and empirical checks; distinguish causal claims from predictive ones; standardize and inspect data; use regularization and multiple initializations; validate chronologically for sequential data; compare against simple baselines; assess calibration and subgroup performance; and document assumptions. Finally, conduct sensitivity analysis: determine whether conclusions change under plausible alternative structures, priors, missing-data mechanisms, or transition dynamics.

Chapter Summary

Probabilistic graphical models represent complex joint distributions through graphs and local factors. Bayesian networks use directed acyclic graphs and conditional factorization, while Markov random fields express undirected compatibility through clique potentials. Conditional independence and d-separation support interpretation and inference, although graphical association should not automatically be treated as causal evidence.

Latent-variable models explain observed patterns through hidden causes or assignments. Mixture models represent heterogeneous populations, and EM alternates between posterior responsibility calculations and parameter maximization. Its likelihood improvement does not guarantee a global optimum, so initialization, regularization, and model selection matter. Markov chains describe stochastic evolution through transition matrices; stationary behaviour depends on properties such as irreducibility and aperiodicity. HMMs combine latent Markov states with conditionally independent observations and support filtering, smoothing, decoding, and learning.

These models are valuable in diagnosis, segmentation, language, finance, and predictive maintenance. Their responsible use requires attention to identifiability, computational complexity, temporal leakage, calibration, fairness, privacy, and the distinction between latent statistical structure and verified real-world categories.

Key Terms

Conditional independence: Independence between variables given the value of one or more additional variables.

Bayesian network: A directed acyclic graph with a probability distribution factorized according to parent relationships.

Directed acyclic graph (DAG): A directed graph containing no directed cycle.

d-Separation: A graphical criterion for determining conditional independence in a DAG.

Collider: A node receiving arrows from two paths, such as $X\rightarrow Y\leftarrow Z$.

Markov random field: An undirected graphical model representing symmetric conditional dependencies.

Clique potential: A non-negative function assigned to a clique in an undirected graphical model.

Partition function: The normalizing constant that makes an energy-based distribution sum or integrate to one.

Latent variable: A variable that is not directly observed but is inferred from observed data.

Mixture model: A probability distribution formed as a weighted combination of component distributions.

Responsibility: The posterior probability that an observation belongs to a particular mixture component.

Expectation-maximization: An iterative method alternating between latent-variable inference and parameter maximization.

Identifiability: The property that model parameters or structures can be uniquely determined from the observed distribution.

Markov chain: A stochastic process in which the next state depends on the past only through the current state.

Transition matrix: A matrix whose entries specify one-step state-transition probabilities.

Stationary distribution: A distribution unchanged by application of the transition matrix.

Hidden Markov model: A model with a latent Markov state sequence generating conditionally independent observations.

Filtering: Estimating the current latent state using observations up to the current time.

Smoothing: Estimating a past latent state using observations from the entire sequence.

Treewidth: A graph-structural measure related to the complexity of exact inference.

Review Questions

Short Answer Questions

1. What conditional independence does a Bayesian-network factorization encode?

2. Why can conditioning on a collider create dependence between its parent variables?

3. Define a latent variable and give one machine-learning example.

4. What is the difference between a stationary distribution and a transition matrix?

5. Distinguish filtering from smoothing in a hidden Markov model.

Long Answer Questions

1. Explain Bayesian networks, d-separation, and the difference between statistical dependence and causal interpretation.

2. Derive the EM updates for the mixing proportions and means of a Gaussian mixture model.

3. Compare Bayesian networks and Markov random fields in terms of graph structure, factorization, and applications.

4. Explain the conditions under which a finite Markov chain converges to a unique stationary distribution.

5. Describe the forward, backward, and Viterbi procedures in an HMM and explain how their objectives differ.

Analytical / Application Questions

1. A fraud-detection model contains the structure $F\rightarrow A\leftarrow I$, where $F$ is fraud, $A$ is an alert, and $I$ is an innocent high-risk behaviour. Analyse why conditioning on the alert can make fraud and innocent behaviour statistically associated.

2. A two-component GMM repeatedly produces a nearly singular covariance for one component. Diagnose the likely cause and propose at least three remedies.

3. Design an evaluation plan for an HMM used to predict machine failure. Specify how you would prevent temporal leakage, choose an alert threshold, assess calibration, and monitor drift.

22. Multiple Choice Questions with Answers

Question 1

In a Bayesian network, the joint distribution is factorized primarily in terms of each variable's:

A. Non-neighbours

B. Parents

C. Descendants only

D. Marginal variance

Correct Answer: B. Parents

Question 2

For the structure X → Y ← Z, what generally happens when Y is conditioned on?

A. The path remains blocked and X and Z become more independent

B. X and Z become dependent through the collider

C. X and Z become identical variables

D. The graph becomes acyclic

Correct Answer: B. X and Z become dependent through the collider

Question 3

Which expression is the Gaussian-mixture responsibility of component k for observation xi?

A. The prior probability πk alone

B. The likelihood of xi summed over all observations

C. The posterior p(zi=k|xi) obtained by normalizing component-specific weighted likelihoods

D. The covariance determinant of component k

Correct Answer: C. The posterior p(zi=k|xi) obtained by normalizing component-specific weighted likelihoods

Question 4

What is the principal purpose of the E-step in EM for a mixture model?

A. To calculate posterior probabilities of latent assignments

B. To remove all latent variables from the model

C. To maximize the transition matrix

D. To select the number of observed features

Correct Answer: A. To calculate posterior probabilities of latent assignments

Question 5

Why can EM for a Gaussian mixture model produce different solutions on different runs?

A. The likelihood is always linear in all parameters

B. The algorithm may converge to different local optima depending on initialization

C. Responsibilities cannot be normalized

D. Gaussian densities have no parameters

Correct Answer: B. The algorithm may converge to different local optima depending on initialization

Question 6

A finite Markov chain is most likely to converge to one unique stationary distribution when it is:

A. Reducible and periodic

B. Irreducible and aperiodic

C. Deterministic and reducible

D. Unnormalized and finite

Correct Answer: B. Irreducible and aperiodic

Question 7

In an HMM, smoothing differs from filtering because smoothing uses:

A. Only the current observation

B. Only observations before the estimated time

C. Observations from the entire sequence, including future observations

D. No transition probabilities

Correct Answer: C. Observations from the entire sequence, including future observations

Question 8

Which model is most directly designed to represent symmetric compatibility among neighbouring image pixels?

A. Markov random field

B. Independent Gaussian model

C. Naive Bayes classifier

D. One-state Markov chain

Correct Answer: A. Markov random field

Question 9

Why is random train-test splitting problematic for many sequential datasets?

A. It always reduces the number of variables

B. It can leak information from future time periods into training

C. It prevents calculation of a likelihood

D. It guarantees underfitting

Correct Answer: B. It can leak information from future time periods into training

Question 10

A nearly singular covariance in a GMM can often be addressed by:

A. Allowing variances to become exactly zero

B. Removing the normalization constraint on mixing proportions

C. Adding covariance regularization or imposing minimum-variance constraints

D. Replacing responsibilities with arbitrary labels

Correct Answer: C. Adding covariance regularization or imposing minimum-variance constraints