Joint Distributions and Multivariate Probability

Introduction and Context

Many real problems involve several random quantities observed together. A machine-learning system may use age, income, and transaction frequency to predict fraud; a medical model may combine blood pressure, cholesterol, and age; a robot may estimate position from multiple noisy sensors. Probability models for such situations must describe not only each variable separately, but also how variables behave together.

A joint distribution describes the simultaneous behavior of two or more random variables. From it, we can obtain marginal distributions, conditional distributions, measures of dependence, and predictions based on observed information. These ideas form a mathematical foundation for regression, Bayesian inference, hidden-variable models, clustering, graphical models, and multivariate statistical learning.

This chapter develops the principal concepts for two random variables and then extends them to vectors and multivariate normal models.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define joint, marginal, and conditional distributions for discrete and continuous variables.

2. Determine whether a proposed joint probability model is valid.

3. Explain independence and distinguish it from zero correlation.

4. Calculate covariance, correlation, conditional expectation, and conditional variance.

5. Apply transformations of random variables using the Jacobian method.

6. Describe the multivariate normal distribution and its covariance structure.

7. Explain how joint distributions support probabilistic graphical models and AI applications.

8. Interpret model assumptions, limitations, and ethical consequences in applied settings.

1. Joint Distributions

Let X and Y be random variables defined on the same probability space. Their joint distribution specifies probabilities for combinations of values of X and Y.

1.1 Discrete joint distributions

For discrete variables, the joint probability mass function (pmf) is

\[

p_{X,Y}(x,y)=P(X=x,Y=y).

\]

A valid joint pmf must satisfy

\[

p_{X,Y}(x,y)\geq 0

\]

for every pair (x,y), and

\[

\sum_x\sum_y p_{X,Y}(x,y)=1.

\]

The probability of an event A involving possible pairs is

\[

P((X,Y)\in A)=\sum_{(x,y)\in A}p_{X,Y}(x,y).

\]

The order of summation is immaterial when the terms are nonnegative.

1.2 Continuous joint distributions

For continuous variables, a joint probability density function (pdf) f_{X,Y}(x,y) satisfies

\[

f_{X,Y}(x,y)\geq 0,

\qquad

\int_{-\infty}^{\infty}\int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dx\,dy=1.

\]

Probabilities are obtained by integration over regions. For a region R,

\[

P((X,Y)\in R)=\iint_R f_{X,Y}(x,y)\,dx\,dy.

\]

For continuous variables, the value f_{X,Y}(x,y) is a density, not the probability that X=x and Y=y. In fact, under an ordinary continuous model, P(X=x,Y=y)=0 for every single point.

1.3 Joint cumulative distribution function

For either discrete or continuous variables, the joint cumulative distribution function (CDF) is

\[

F_{X,Y}(x,y)=P(X\leq x,Y\leq y).

\]

The joint CDF is nondecreasing in each argument and approaches 1 as both arguments approach infinity. For a continuous model with sufficient smoothness,

\[

f_{X,Y}(x,y)=\frac{\partial^2}{\partial x\,\partial y}F_{X,Y}(x,y).

\]

The CDF is especially useful when the event boundaries are irregular or when variables are mixed discrete and continuous.

2. Marginal Distributions

A marginal distribution describes one variable without specifying the value of the other. It is obtained by summing or integrating the joint distribution over the unwanted variable.

For discrete variables,

\[

p_X(x)=\sum_y p_{X,Y}(x,y),

\qquad

p_Y(y)=\sum_x p_{X,Y}(x,y).

\]

For continuous variables,

\[

f_X(x)=\int_{-\infty}^{\infty}f_{X,Y}(x,y)\,dy,

\qquad

f_Y(y)=\int_{-\infty}^{\infty}f_{X,Y}(x,y)\,dx.

\]

The marginal distributions are important because they describe individual behavior, while the joint distribution contains both individual behavior and association.

Worked Example 1: Marginalization

Suppose X and Y are discrete with joint pmf

\[

p_{X,Y}(x,y)=\frac{x+y}{18},\qquad x,y\in\{1,2,3\}.

\]

The total is

\[

\frac{1}{18}\sum_{x=1}^3\sum_{y=1}^3(x+y)=\frac{1}{18}(18)=1,

\]

so the pmf is valid. For x=1,

\[

p_X(1)=\frac{(1+1)+(1+2)+(1+3)}{18}=\frac{9}{18}=\frac12.

\]

For general x,

\[

p_X(x)=\sum_{y=1}^3\frac{x+y}{18}=\frac{3x+6}{18}=\frac{x+2}{6}.

\]

Thus p_X(1)=1/2, p_X(2)=2/3, and p_X(3)=5/6 would incorrectly sum to 2. This reveals that the claimed normalizing total was wrong. Rechecking,

\[

\sum_{x=1}^3\sum_{y=1}^3(x+y)=3(1+2+3)+3(1+2+3)=36.

\]

Therefore the valid pmf is actually p_{X,Y}(x,y)=(x+y)/36. Consequently,

\[

p_X(x)=\frac{3x+6}{36}=\frac{x+2}{12},

\]

which gives probabilities 1/4, 1/3, and 5/12, summing to 1. This example illustrates why normalization should always be checked before marginalization.

3. Conditional Distributions and Bayes’ Rule

A conditional distribution describes one variable when information about another variable is available.

For discrete variables, provided p_Y(y)>0,

\[

p_{X|Y}(x|y)=P(X=x\mid Y=y)=\frac{p_{X,Y}(x,y)}{p_Y(y)}.

\]

For continuous variables, provided f_Y(y)>0,

\[

f_{X|Y}(x|y)=\frac{f_{X,Y}(x,y)}{f_Y(y)}.

\]

The denominator ensures that the conditional distribution integrates or sums to one for the fixed value of y.

The multiplication rule is

\[

p_{X,Y}(x,y)=p_{X|Y}(x|y)p_Y(y)

\]

for discrete variables, with the analogous density relationship for continuous variables. Bayes’ rule follows directly:

\[

p_{Y|X}(y|x)=\frac{p_{X|Y}(x|y)p_Y(y)}{p_X(x)}.

\]

For continuous variables, replace pmfs by densities wherever the relevant density is positive. Bayes’ rule is central to classification and diagnosis: the posterior distribution combines a prior belief with evidence through a likelihood.

Worked Example 2: Conditional probability

Suppose a joint pmf is given by

\[

p_{X,Y}(x,y)=\frac{x+y}{36},\qquad x,y\in\{1,2,3\}.

\]

From the previous calculation, p_Y(2)=(2+2)/12=1/3. Therefore

\[

P(X=3\mid Y=2)=\frac{p_{X,Y}(3,2)}{p_Y(2)}

=\frac{5/36}{1/3}=\frac{5}{12}.

\]

The conditional probabilities for X=1,2,3 given Y=2 are respectively 3/12, 4/12, and 5/12, which sum to one.

4. Independence and Dependence

X and Y are independent if knowing one provides no probabilistic information about the other. For discrete variables,

\[

p_{X,Y}(x,y)=p_X(x)p_Y(y)

\]

for every x and y. For continuous variables, the same factorization applies to densities wherever defined:

\[

f_{X,Y}(x,y)=f_X(x)f_Y(y).

\]

Equivalent statements include

\[

P(X\in A,Y\in B)=P(X\in A)P(Y\in B)

\]

for all suitable events A and B, and

\[

f_{X|Y}(x|y)=f_X(x)

\]

whenever the conditional density exists.

Independence is a strong condition. It implies zero covariance when the relevant moments exist, but zero covariance does not generally imply independence. For example, let X be uniformly distributed on (-1,1) and let Y=X^2. Then Y is completely determined by X, so the variables are dependent. However,

\[

E[X]=0,

\qquad E[XY]=E[X^3]=0,

\]

so Cov(X,Y)=0. The relationship is nonlinear and is not detected by covariance.

5. Covariance and Correlation

For random variables with finite second moments, the covariance is

\[

\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])]

=E[XY]-E[X]E[Y].

\]

Covariance measures the direction of linear association. A positive value indicates that large values of one variable tend to occur with large values of the other; a negative value indicates an opposing linear tendency. Its numerical magnitude depends on measurement units, so it is not directly comparable across differently scaled variables.

The variance is the special case

\[

\operatorname{Var}(X)=\operatorname{Cov}(X,X).

\]

The correlation coefficient is

\[

\rho_{X,Y}=\frac{\operatorname{Cov}(X,Y)}{\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}},

\]

provided both variances are positive and finite. It is dimensionless and satisfies -1≤ρ≤1. Values near ±1 indicate strong linear association, whereas a value near zero indicates weak linear association, not necessarily independence.

A useful identity for linear combinations is

\[

\operatorname{Var}(aX+bY)=a^2\operatorname{Var}(X)+b^2\operatorname{Var}(Y)+2ab\operatorname{Cov}(X,Y),

\]

where a and b are constants. If X and Y are independent, the covariance term is zero.

6. Conditional Expectation and Variance

The conditional expectation of X given Y=y is

\[

E[X\mid Y=y]=\sum_x x p_{X|Y}(x|y)

\]

for discrete X, or

\[

E[X\mid Y=y]=\int_{-\infty}^{\infty}x f_{X|Y}(x|y)\,dx

\]

for continuous X. It is a function of y and represents the mean prediction of X after observing Y.

The law of total expectation states

\[

E[X]=E[E[X\mid Y]]=\int E[X\mid Y=y]f_Y(y)\,dy,

\]

with summation replacing integration in the discrete case.

Conditional variance measures residual uncertainty:

\[

\operatorname{Var}(X\mid Y=y)=E[(X-E[X\mid Y=y])^2\mid Y=y].

\]

The law of total variance decomposes overall uncertainty into predictable and residual components:

\[

\operatorname{Var}(X)=E[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(E[X\mid Y]).

\]

This identity is important in prediction. The first term is average uncertainty remaining after observing Y; the second is variation in the conditional mean across values of Y.

7. Transformations of Random Variables

A transformation creates new variables from existing ones. Let

\[

U=g_1(X,Y),\qquad V=g_2(X,Y).

\]

If the transformation is one-to-one and differentiable, and the inverse is x=x(u,v), y=y(u,v), then

\[

f_{U,V}(u,v)=f_{X,Y}(x(u,v),y(u,v))\left|J\right|,

\]

where

\[

J=\frac{\partial(x,y)}{\partial(u,v)}

\]

is the determinant of the inverse Jacobian matrix. The support must also be transformed correctly.

Worked Example 3: Sum and difference

Let X and Y be independent Uniform(0,1) variables. Define U=X+Y and V=X-Y. The original joint density is 1 on the unit square and zero elsewhere. The inverse transformation is

\[

X=\frac{U+V}{2},\qquad Y=\frac{U-V}{2}.

\]

The inverse Jacobian determinant has absolute value 1/2. Hence

\[

f_{U,V}(u,v)=\frac12

\]

for points satisfying 0≤(u+v)/2≤1 and 0≤(u-v)/2≤1, and zero otherwise. The support is a diamond-shaped region in the (u,v)-plane. This example shows why both the Jacobian and transformed support are necessary.

If a transformation is not one-to-one, the densities from all valid inverse branches must be added. If differentiability fails or the transformation maps a continuous pair onto a lower-dimensional set, a more specialized treatment may be required.

8. Multivariate Random Vectors

A random vector is written

\[

\mathbf X=(X_1,X_2,\ldots,X_p)^T.

\]

Its mean vector is

\[

\boldsymbol\mu=E[\mathbf X],

\]

and its covariance matrix is

\[

\boldsymbol\Sigma=E[(\mathbf X-\boldsymbol\mu)(\mathbf X-\boldsymbol\mu)^T].

\]

The entries are

\[

\Sigma_{ij}=\operatorname{Cov}(X_i,X_j).

\]

A covariance matrix is symmetric and positive semidefinite: for every vector a,

\[

a^T\boldsymbol\Sigma a\geq 0.

\]

The variance of a linear combination is

\[

\operatorname{Var}(a^T\mathbf X)=a^T\boldsymbol\Sigma a.

\]

A covariance matrix may be singular when variables are exact linear combinations of one another. Such a matrix cannot be inverted, which affects some estimation and density formulas.

9. The Multivariate Normal Distribution

A p-dimensional random vector has a nondegenerate multivariate normal distribution if its density is

\[

f(\mathbf x)=\frac{1}{(2\pi)^{p/2}|\boldsymbol\Sigma|^{1/2}}

\exp\left[-\frac12(\mathbf x-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right],

\]

where μ is a p×1 mean vector and Σ is a positive-definite p×p covariance matrix. The determinant and inverse require Σ to be nonsingular.

The quadratic expression in the exponent measures standardized multivariate distance. Contours of constant density are ellipses in two dimensions and ellipsoids in higher dimensions.

Important properties include:

• Every linear combination aᵀX is normally distributed.
• Every marginal distribution is normal.
• Conditional distributions are normal.
• For jointly normal variables, zero covariance implies independence. This implication is special to the joint normal family and is not true in general.

For a jointly normal partition

\[

\begin{pmatrix}X_1\\X_2\end{pmatrix}

\sim N\left(

\begin{pmatrix}\mu_1\\\mu_2\end{pmatrix},

\begin{pmatrix}\Sigma_{11}&\Sigma_{12}\\\Sigma_{21}&\Sigma_{22}\end{pmatrix}

\right),

\]

provided Σ₂₂ is invertible, the conditional distribution is

\[

X_1\mid X_2=x_2\sim N\left(

\mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2),

\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}

\right).

\]

The conditional covariance is the Schur complement. It does not depend on the observed value x₂, while the conditional mean changes linearly with x₂.

10. Joint Distributions and Probabilistic Graphical Models

A probabilistic graphical model represents a high-dimensional joint distribution through a graph. The representation can make conditional independence explicit and reduce the number of parameters.

In a directed acyclic graph (DAG), a Bayesian network factorizes the joint distribution as

\[

p(x_1,\ldots,x_p)=\prod_{i=1}^p p(x_i\mid x_{\operatorname{pa}(i)}),

\]

where pa(i) denotes the parents of node i. For example, if A influences B and C, and B and C are conditionally independent given A,

\[

p(A,B,C)=p(A)p(B\mid A)p(C\mid A).

\]

An undirected graphical model, or Markov random field, often has the form

\[

p(\mathbf x)=\frac{1}{Z}\prod_{C\in\mathcal C}\psi_C(\mathbf x_C),

\]

where ψ_C are nonnegative potential functions, C indexes cliques, and Z is a normalizing constant. Unlike a Bayesian-network factorization, potentials need not individually be probabilities.

Graphical models support inference, diagnosis, prediction, and causal reasoning, but a graph alone does not automatically establish causation. Causal interpretation requires assumptions about the data-generating process, interventions, confounding, and measurement.

11. Applied Case: Predictive Maintenance

Background

Consider an illustrative manufacturing system with two sensors: vibration amplitude X and motor temperature Y. Engineers want to estimate the probability of failure within the next week.

Problem

Vibration and temperature are correlated because mechanical friction can increase both. Treating them as independent may underestimate the probability of simultaneous abnormal readings and distort failure-risk estimates.

Analysis

A historical dataset is used to estimate the joint distribution of (X,Y), separately for machines that fail and machines that do not. A bivariate normal model may be reasonable after suitable transformation, but this assumption must be checked using scatterplots, marginal diagnostics, outlier analysis, and residual assessment. The covariance matrix captures the association between sensors. A conditional probability such as

\[

P(\text{failure}\mid X=x,Y=y)

\]

can then be estimated using Bayes’ rule or a discriminative model.

Approach and Solution

The engineering team first standardizes measurement units, handles missing readings, and verifies sensor calibration. It estimates class-specific means and covariance matrices. The system then computes a posterior failure probability and sends an alert when the probability exceeds a threshold selected using the costs of false alarms and missed failures.

Outcome

A joint model can identify combinations of moderately high vibration and temperature that are more concerning than either measurement considered alone. However, performance must be evaluated on future-like data, not only on the training sample.

Learning Points

1. Dependence may contain operational information.

2. Covariance is useful but captures only linear association.

3. Model validation and threshold selection are decision problems, not merely mathematical exercises.

4. Alerts should be auditable and should not conceal uncertainty from engineers.

5. Sensor data may raise privacy, worker-monitoring, and safety-governance concerns.

12. Practical and Ethical Considerations

Joint models are vulnerable to poor data quality, sampling bias, missing-not-at-random observations, outliers, and distribution shift. A covariance estimated from a small sample can be unstable, particularly when the number of variables approaches or exceeds the sample size. Regularization, dimensionality reduction, robust estimation, or domain-specific models may be necessary.

Correlation should not be interpreted as causation. In AI systems, dependence can arise from protected characteristics, historical discrimination, or data-collection practices. A model may use apparently harmless variables that act as proxies for sensitive attributes. Practitioners should document variables, assumptions, missingness mechanisms, validation populations, and limitations. Privacy protections are particularly important when joint distributions combine records across people, locations, or time.

Best practice includes:

• checking normalization and support conditions;
• plotting pairwise relationships before choosing a model;
• distinguishing marginal, conditional, and causal statements;
• testing independence assumptions rather than accepting them automatically;
• validating covariance estimates and positive-semidefiniteness;
• reporting uncertainty and sensitivity to model choices;
• monitoring models after deployment for drift and unequal error rates.

Chapter Summary

Joint distributions describe several random variables simultaneously and provide the basis for marginal and conditional analysis. Marginal distributions are obtained by summing or integrating over unwanted variables; conditional distributions are obtained by normalizing a joint distribution. Independence is expressed through factorization, but zero covariance alone does not establish independence. Covariance measures linear co-movement, while correlation provides a unit-free standardized measure.

Conditional expectation gives the best mean prediction under squared-error loss, and the laws of total expectation and total variance decompose prediction and uncertainty. Transformations require careful use of inverse mappings, Jacobians, and transformed supports. Random vectors summarize multivariate means and covariances, while the multivariate normal distribution provides tractable marginal, conditional, and linear-transformation results. Finally, probabilistic graphical models exploit conditional independence to represent complex joint distributions efficiently. In practical AI and engineering systems, these tools must be combined with validation, uncertainty communication, ethical oversight, and awareness of causal limitations.

Key Terms

Joint distribution: Probability model describing two or more random variables together.

Joint pmf: Probability mass function for discrete variable combinations.

Joint pdf: Density function for a continuous random vector.

Joint CDF: Probability P(X≤x,Y≤y).

Marginal distribution: Distribution of one variable obtained from a joint distribution.

Conditional distribution: Distribution of one variable given information about another.

Independence: Condition under which variables provide no probabilistic information about one another.

Covariance: Measure of linear co-movement between two variables.

Correlation: Standardized covariance ranging from -1 to 1.

Conditional expectation: Mean of a variable given observed information.

Conditional variance: Remaining variance after conditioning on information.

Random vector: Ordered collection of random variables.

Covariance matrix: Matrix containing pairwise covariances.

Positive semidefinite: Matrix property aᵀΣa≥0 for every vector a.

Multivariate normal distribution: Normal probability model for a random vector.

Jacobian: Determinant measuring local change of coordinates in a transformation.

Bayesian network: Directed graphical representation of a factorized joint distribution.

Conditional independence: Independence of variables after conditioning on other variables.

Schur complement: Matrix expression giving conditional covariance in a multivariate normal model.

Probabilistic graphical model: Graph-based representation of a joint probability distribution.

Review Questions

Short Answer Questions

1. What is the difference between a joint distribution and a marginal distribution?

2. State the factorization condition for independence.

3. Why does zero covariance not generally imply independence?

4. What does the covariance matrix represent for a random vector?

5. What role does the Jacobian play in a change-of-variables calculation?

Long Answer Questions

1. Explain how marginal and conditional distributions are derived from a joint density, including the conditions required for valid formulas.

2. Compare covariance and correlation, including their interpretations, units, limitations, and relationship to independence.

3. Derive and interpret the law of total variance in the context of prediction.

4. Describe the main properties of the multivariate normal distribution and explain why its conditional distributions are useful.

5. Explain how a Bayesian network factorizes a joint distribution and discuss why graphical structure should not automatically be interpreted as causal evidence.

Analytical / Application Questions

1. A fraud-detection model uses transaction amount and transaction frequency. Explain how dependence between these variables could affect risk estimates and propose an appropriate validation strategy.

2. Let X and Y be jointly distributed continuous variables. Design a change-of-variables procedure for U=X+Y and V=X−Y, specifying the inverse transformation, Jacobian, and support-checking steps.

3. A predictive-maintenance system reports high accuracy but performs poorly after deployment. Analyze how marginal changes, altered dependence, covariance instability, and missing-data mechanisms could explain the failure, and recommend corrective actions.

Multiple Choice Questions with Answers

Question 1

Which condition must a discrete joint pmf satisfy?

A. It must be negative for some outcomes

B. Its total sum over all pairs must equal 1

C. Each probability must exceed 1

D. Its marginal distributions must be uniform

Correct Answer: B. Its total sum over all pairs must equal 1

Question 2

How is the marginal pmf p_X(x) obtained from a discrete joint pmf?

A. By multiplying p_{X,Y}(x,y) by y

B. By differentiating with respect to y

C. By summing p_{X,Y}(x,y) over all y

D. By setting y equal to zero

Correct Answer: C. By summing p_{X,Y}(x,y) over all y

Question 3

If p_Y(y)>0, which expression gives p_{X|Y}(x|y)?

A. p_X(x)/p_Y(y)

B. p_{X,Y}(x,y)/p_Y(y)

C. p_Y(y)/p_{X,Y}(x,y)

D. p_X(x)p_Y(y)

Correct Answer: B. p_{X,Y}(x,y)/p_Y(y)

Question 4

Which statement is generally true?

A. Zero correlation always implies independence

B. Independence always implies zero covariance when moments exist

C. Correlation can exceed 1

D. Covariance is always dimensionless

Correct Answer: B. Independence always implies zero covariance when moments exist

Question 5

If Var(X)=4, Var(Y)=9, and Cov(X,Y)=1, what is Var(X+Y)?

A. 12

B. 13

C. 15

D. 17

Correct Answer: C. 15

Question 6

What does E[X|Y=y] represent?

A. The unconditional variance of X

B. The mean of X conditional on Y=y

C. The probability that X equals y

D. The covariance of X and Y

Correct Answer: B. The mean of X conditional on Y=y

Question 7

In a valid one-to-one change of variables, the absolute Jacobian determinant accounts for:

A. The change in local area or volume

B. The sample mean

C. The number of observations

D. The correlation coefficient

Correct Answer: A. The change in local area or volume

Question 8

For a jointly multivariate normal vector, zero covariance between two components implies:

A. They are uniformly distributed

B. They are independent

C. Their variances are zero

D. Their means are equal

Correct Answer: B. They are independent

Question 9

Which matrix contains pairwise covariances of a random vector?

A. The covariance matrix

B. The identity matrix only

C. The Jacobian matrix only

D. The design matrix only

Correct Answer: A. The covariance matrix

Question 10

In a Bayesian network, the joint distribution is commonly factorized as:

A. A sum of all marginal means

B. A product of conditional distributions given parent variables

C. A product of correlation coefficients only

D. A ratio of variances

Correct Answer: B. A product of conditional distributions given parent variables