6. Probability and Random Variables

Introduction and Context

Machine-learning systems operate under uncertainty. A medical model may estimate whether a patient has a disease, a bank may assess the probability that a borrower will default, and a recommender system may predict whether a user will click an item. In each case, the available data are incomplete, noisy, or variable. Probability provides the mathematical language for representing and reasoning about this uncertainty.

Probability is not merely a method for assigning percentages. It supports a disciplined framework for describing possible outcomes, updating beliefs when evidence is observed, quantifying variation, and evaluating predictive models. Random variables then translate uncertain outcomes into numerical quantities that can be summarized through distributions, expectations, variances, and covariances.

For machine learning, these ideas are foundational. Classification probabilities, likelihood functions, loss expectations, sampling assumptions, Bayesian inference, confidence estimates, and generalization theory all depend on probability. This chapter develops the principal concepts needed to model uncertainty and introduces the laws of large numbers that connect theoretical probability with observed data.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define sample spaces, events, probability measures, and probability axioms.

2. Compute conditional probabilities and apply Bayes’ theorem.

3. Distinguish independent events from mutually exclusive events.

4. Define discrete and continuous random variables and interpret probability distributions.

5. Use probability mass functions, probability density functions, and cumulative distribution functions.

6. Describe important distributions used in machine learning.

7. Calculate and interpret expectation, variance, covariance, and correlation.

8. Apply the laws of large numbers to sampling and model evaluation.

9. Recognize common probabilistic modelling errors and ethical risks.

The Probabilistic Model: Outcomes, Events, and Axioms

A probabilistic model begins with a sample space, denoted by \(\Omega\), which is the set of all possible outcomes of an experiment. An experiment need not be a laboratory activity. It may be a model prediction, a customer transaction, or the result of selecting one observation from a dataset.

An event is a subset of the sample space. If a diagnostic test produces either a positive or negative result, the event “positive result” contains the corresponding outcome. Events may contain one outcome or many outcomes. The complement of an event \(A\), written \(A^c\), consists of all outcomes in \(\Omega\) that are not in \(A\).

A probability measure \(P\) assigns a number to events. It must satisfy three Kolmogorov axioms:

• \(P(A)\geq 0\) for every event \(A\).
• \(P(\Omega)=1\).
• If \(A_1,A_2,\ldots\) are mutually exclusive events, then \(P(\bigcup_i A_i)=\sum_iP(A_i)\).

Several useful results follow from these axioms. The probability of an impossible event is zero, \(P(\varnothing)=0\). The complement rule is

\[P(A^c)=1-P(A).\]

For two events, the addition rule is

\[P(A\cup B)=P(A)+P(B)-P(A\cap B).\]

The subtraction term prevents outcomes in both events from being counted twice. If events are mutually exclusive, then \(P(A\cap B)=0\), and the rule reduces to simple addition.

In a finite sample space in which all outcomes are equally likely, probability can be calculated as the ratio of favourable outcomes to total outcomes. This approach is useful for simple games of chance, but machine-learning problems usually involve unequal probabilities, continuous measurements, or complex dependence. In such settings, probability must be specified through a distribution or a model rather than by counting outcomes.

Conditional Probability and the Logic of Evidence

The probability of an event may change when information about another event becomes available. The conditional probability of \(A\) given \(B\) is

\[P(A\mid B)=\frac{P(A\cap B)}{P(B)},\qquad P(B)>0.\]

The condition restricts attention to cases in which \(B\) has occurred. For example, the probability that a transaction is fraudulent may differ substantially when the transaction originates from an unusual location. Conditional probability therefore formalizes evidence and context.

Rearranging the definition gives the multiplication rule:

\[P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A).\]

For a collection of events, the chain rule expresses a joint probability as a product of conditional probabilities:

\[P(A_1,\ldots,A_n)=P(A_1)P(A_2\mid A_1)\cdots P(A_n\mid A_1,\ldots,A_{n-1}).\]

This rule is central to probabilistic graphical models and sequence models because it decomposes a complex joint distribution into interpretable conditional components.

Bayes’ Theorem

Equating the two forms of the multiplication rule produces Bayes’ theorem:

\[P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}.\]

Here, \(P(A)\) is the prior probability, \(P(B\mid A)\) is the likelihood, \(P(B)\) is the evidence or marginal probability, and \(P(A\mid B)\) is the posterior probability. Bayes’ theorem updates a prior belief in light of observed evidence.

If \(A_1,\ldots,A_k\) form a mutually exclusive and exhaustive partition of the sample space, the denominator can be computed using the law of total probability:

\[P(B)=\sum_{i=1}^{k}P(B\mid A_i)P(A_i).\]

Consider a disease-screening example. Suppose disease prevalence is 1%, a test has sensitivity 90%, and its false-positive rate is 5%. For a randomly selected person,

\[P(D)=0.01,\quad P(+\mid D)=0.90,\quad P(+\mid D^c)=0.05.\]

Therefore,

\[P(D\mid +)=\frac{0.90(0.01)}{0.90(0.01)+0.05(0.99)}\approx0.154.\]

Although the test is reasonably accurate, a positive result implies only about a 15.4% probability of disease because the disease is rare. This base-rate effect is essential when interpreting alerts, anomaly detectors, and classification outputs.

Independence: A Modelling Assumption, Not a Default Fact

Events \(A\) and \(B\) are independent if

\[P(A\cap B)=P(A)P(B).\]

When \(P(B)>0\), this is equivalent to \(P(A\mid B)=P(A)\). Learning that \(B\) occurred does not change the probability of \(A\).

Independence differs from mutual exclusivity. Mutually exclusive events cannot occur together, whereas independent events do not influence one another. If two events have positive probability and are mutually exclusive, they cannot be independent because their intersection has probability zero while the product of their probabilities is positive.

Random variables may also be independent. Variables \(X\) and \(Y\) are independent when their joint distribution factorizes:

\[P(X\in A,Y\in B)=P(X\in A)P(Y\in B).\]

The independence assumption simplifies modelling and computation, but it must be justified. The naive Bayes classifier, for example, assumes conditional independence of features given the class. This assumption is often false in a literal sense, yet the method can still perform well when the resulting decision boundaries are useful.

Random Variables and Their Distributions

A random variable is a function that assigns a numerical value to each outcome in a sample space. It is called discrete when it takes a finite or countably infinite set of values, such as the number of fraudulent transactions in a batch. It is continuous when it can take values in an interval, such as temperature, height, or a model’s predicted score.

A discrete random variable is described by a probability mass function (PMF),

\[p_X(x)=P(X=x),\]

where \(p_X(x)\geq0\) and \(\sum_xp_X(x)=1\).

A continuous random variable is described by a probability density function (PDF) \(f_X(x)\). Probabilities are obtained by integration:

\[P(a\leq X\leq b)=\int_a^b f_X(x)\,dx.\]

A density is not itself a probability; it may exceed one if concentrated over a sufficiently small interval. For a continuous variable, the probability of any exact point is zero, so \(P(X=x)=0\).

Both discrete and continuous variables have a cumulative distribution function (CDF):

\[F_X(x)=P(X\leq x).\]

The CDF is non-decreasing, approaches zero as \(x\) approaches negative infinity, and approaches one as \(x\) approaches positive infinity. It is especially useful for computing quantiles and comparing distributions.

A joint distribution describes two or more random variables simultaneously. From it, marginal distributions are obtained by summing or integrating over the other variables. Conditional distributions describe one variable after another has been observed. These operations are fundamental in generative modelling and Bayesian inference.

Common Distributions in Machine Learning

Bernoulli and Binomial Distributions

A Bernoulli random variable represents one trial with outcomes 1 and 0. If the probability of success is \(p\), then

\[P(X=1)=p,\qquad P(X=0)=1-p.\]

The sum of \(n\) independent Bernoulli trials follows a binomial distribution, written \(X\sim\operatorname{Binomial}(n,p)\). It models the number of successes in a fixed number of trials:

\[P(X=k)=\binom nkp^k(1-p)^{n-k}.\]

Binary classification labels are often modelled using Bernoulli distributions, while counts of successes in batches may use binomial distributions.

Categorical and Multinomial Distributions

A categorical variable takes one of \(K\) categories with probabilities \(p_1,\ldots,p_K\), where \(\sum_kp_k=1\). Multiclass classification commonly models the target label using a categorical distribution. The softmax function converts model scores into such probabilities:

\[p_k=\frac{e^{z_k}}{\sum_j e^{z_j}}.\]

A multinomial distribution generalizes the binomial distribution to counts across several categories.

Poisson Distribution

The Poisson distribution models the number of events occurring in a fixed interval when events occur at an average rate \(\lambda\):

\[P(X=k)=e^{-\lambda}\frac{\lambda^k}{k!}.\]

It is often used for event counts, such as service requests or equipment failures, although its assumptions should be checked. In a Poisson distribution, both mean and variance equal \(\lambda\).

Uniform and Gaussian Distributions

A continuous uniform distribution assigns equal density over an interval. It can represent a bounded quantity when no value within the interval is initially preferred, although this interpretation must be substantively justified.

The Gaussian or normal distribution is written \(X\sim\mathcal N(\mu,\sigma^2)\), with density

\[f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right].\]

Its symmetry and mathematical convenience explain its frequent use in noise models, regression, and approximate sampling theory. The parameters \(\mu\) and \(\sigma^2\) are its mean and variance. Standardization produces the standard normal variable \(Z=(X-\mu)/\sigma\).

Exponential and Related Distributions

The exponential distribution models waiting times between events in a Poisson process. With rate \(\lambda\), its density is \(f(x)=\lambda e^{-\lambda x}\) for \(x\geq0\). It has the memoryless property: the additional waiting time does not depend on how long one has already waited. This property is useful in reliability and queueing models but is not appropriate for every real waiting-time process.

Expectation: The Long-Run Centre of a Distribution

The expectation or expected value of a discrete random variable is

\[E[X]=\sum_xxp_X(x),\]

and for a continuous variable,

\[E[X]=\int_{-\infty}^{\infty}xf_X(x)\,dx,\]

provided the quantity is well defined. Expectation is a probability-weighted average, not necessarily a value that the variable can actually take.

A crucial property is linearity:

\[E[aX+bY+c]=aE[X]+bE[Y]+c.\]

This holds whether or not \(X\) and \(Y\) are independent. More generally, the law of the unconscious statistician states that

\[E[g(X)]=\sum_xg(x)p_X(x)\]

for discrete variables, with integration replacing summation for continuous variables. This permits the expectation of a transformation without explicitly deriving the distribution of the transformation.

In machine learning, expected loss is central. If a predictor produces \(\hat Y=f(X)\) and the loss is \(L(Y,f(X))\), its population risk is

\[R(f)=E[L(Y,f(X))].\]

Training data provide an empirical approximation to this theoretical quantity.

Variance, Covariance, and Dependence

Variance measures dispersion around the mean:

\[\operatorname{Var}(X)=E[(X-E[X])^2].\]

The computational form is

\[\operatorname{Var}(X)=E[X^2]-(E[X])^2.\]

The standard deviation is \(\sqrt{\operatorname{Var}(X)}\), expressed in the same units as the variable. Variance is sensitive to extreme observations because deviations are squared.

For constants \(a\) and \(b\),

\[\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X).\]

The covariance of \(X\) and \(Y\) is

\[\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])].\]

Equivalently,

\[\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y].\]

Positive covariance indicates that large values of one variable tend to accompany large values of the other; negative covariance indicates an opposing tendency. Covariance depends on measurement units, so correlation is often easier to compare:

\[\operatorname{Corr}(X,Y)=\frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}.\]

Correlation lies between -1 and 1. Zero covariance does not generally imply independence, although independence does imply zero covariance when the relevant expectations exist. This distinction is important: nonlinear dependence may be invisible to correlation.

For a random vector \(X\), the covariance matrix is

\[\Sigma=E[(X-E[X])(X-E[X])^T].\]

It is symmetric and positive semidefinite. Covariance matrices are used in principal component analysis, multivariate Gaussian models, uncertainty propagation, and optimization. Poorly estimated covariance matrices can be singular or unstable, particularly when the number of features is large relative to the sample size.

The Laws of Large Numbers and Empirical Evidence

The law of large numbers explains why averages from repeated observations can approximate population expectations. Let \(X_1,X_2,\ldots\) be independent and identically distributed random variables with finite mean \(\mu\). The sample mean is

\[\bar X_n=\frac1n\sum_{i=1}^nX_i.\]

The weak law states that \(\bar X_n\) converges in probability to \(\mu\) as \(n\) increases. Informally, for every positive \(\epsilon\),

\[P(|\bar X_n-\mu|>\epsilon)\to0.\]

The strong law states that the sample mean converges almost surely to \(\mu\), a stronger form of convergence. These results justify estimating expected losses, class proportions, and average outcomes using sufficiently large representative samples.

The laws do not say that every finite sample is close to the population mean, nor do they eliminate systematic sampling bias. More data reduce random error under suitable assumptions; they do not repair biased labels, data leakage, distribution shift, or dependent observations. In machine learning, the law of large numbers supports empirical risk minimization, but generalization also requires attention to model complexity and the relationship between training and deployment distributions.

An Illustrative Case Study: Probabilistic Fraud Screening

Background

Consider a fictional online payment company that uses a machine-learning model to flag potentially fraudulent transactions. The model produces a score interpreted as the probability of fraud.

Situation and Problem

Fraud represents a small fraction of all transactions. Management notices that many flagged transactions are legitimate and questions whether the model is reliable. A team must distinguish model quality from the effects of the low base rate.

Analysis

Suppose 0.5% of transactions are fraudulent. The detector identifies 95% of fraudulent transactions but incorrectly flags 2% of legitimate transactions. For 100,000 transactions, approximately 500 are fraudulent. The model flags about 475 fraudulent transactions and 1,990 legitimate transactions. Thus, among 2,465 flagged transactions, the proportion that are truly fraudulent is approximately

\[\frac{475}{2465}\approx19.3\%.\]

This is the positive predictive value, which is a posterior probability. It is not equal to sensitivity. The distinction follows directly from Bayes’ theorem.

Approach and Solution

The company should calibrate predicted probabilities on a validation set that reflects deployment conditions, report a confusion matrix and precision-recall measures, and choose an operating threshold according to the costs of missed fraud, unnecessary reviews, and customer disruption. It should also examine whether false-positive rates differ across customer groups. Human reviewers can be assigned cases near the decision boundary, while high-confidence cases may be handled automatically subject to audit controls.

Outcome

The revised system does not necessarily maximize the number of alerts. Instead, it produces risk estimates, documents the decision threshold, monitors calibration and drift, and periodically retrains using confirmed outcomes. This makes the system more useful operationally and more defensible from a governance perspective.

Learning Points

The case illustrates base-rate effects, conditional probability, the difference between sensitivity and predictive value, threshold-dependent decisions, and the need to evaluate probability quality rather than accuracy alone.

Practical and Professional Applications

Probability is used in Bayesian diagnosis, credit-risk assessment, demand forecasting, speech recognition, reinforcement learning, reliability engineering, A/B testing, and portfolio risk. In supervised learning, a probabilistic model can express uncertainty about labels rather than returning only a hard class. In generative models, the joint distribution supports simulation and missing-data analysis. In decision systems, expected utility combines uncertain outcomes with their consequences.

Probabilistic outputs must be interpreted carefully. A probability of 0.8 should mean that events with comparable conditions occur approximately 80% of the time under the relevant reference population. This property is called calibration. A model may rank cases correctly while being poorly calibrated. Conversely, calibration alone does not guarantee useful discrimination.

Modern software libraries automate sampling, density evaluation, optimization, and Bayesian computation. Monte Carlo methods approximate expectations by averages from simulated samples. Markov chain Monte Carlo, variational inference, and probabilistic programming extend these ideas to models whose exact calculations are difficult. Automation does not remove the need to specify assumptions, inspect data, and validate results.

Limitations, Risks, and Common Mistakes

Several errors repeatedly arise in probabilistic reasoning. The first is confusing \(P(A\mid B)\) with \(P(B\mid A)\). The second is ignoring base rates when interpreting a test or classifier. The third is treating correlation as causation or independence. The fourth is using a familiar distribution without checking whether its support, shape, and dependence assumptions fit the application.

Small samples can produce unstable estimates of rare-event probabilities. Class imbalance can make accuracy misleading. Dependence between observations reduces the effective information in a dataset and may invalidate simple standard-error calculations. Distribution shift can make a well-calibrated model unreliable after deployment.

Probability models also encode choices about what is measured, whose data are included, and which errors matter. A model trained on historically biased decisions may reproduce those biases. Probabilistic risk scores should not be treated as objective facts about individuals. Organizations should document assumptions, retain audit trails, test subgroup performance, protect sensitive information, and provide appropriate human oversight for high-impact decisions.

Emerging Directions and Best Practices

Current machine-learning practice increasingly emphasizes uncertainty quantification. Predictive intervals, conformal prediction, Bayesian neural networks, ensembles, and distribution-free calibration methods seek to communicate when a model may be unreliable. These approaches differ in assumptions and guarantees; none should be presented as universally superior.

Good practice includes:

• Define the random experiment and target population before selecting a model.
• Separate prior information, observed evidence, and assumptions.
• Check whether independence and distributional assumptions are defensible.
• Report uncertainty, calibration, and relevant error rates rather than accuracy alone.
• Use validation data that resemble deployment data.
• Distinguish statistical uncertainty from uncertainty caused by distribution shift or poor measurement.
• Monitor predictions after deployment and update the model when conditions change.

Chapter Summary

Probability provides axioms and rules for representing uncertainty. Events are subsets of a sample space, while conditional probability describes how evidence changes an assessment. Bayes’ theorem combines prior information and likelihood to produce posterior probabilities. Independence is a simplifying assumption and must not be confused with mutual exclusivity.

Random variables map uncertain outcomes to numbers. Their distributions may be represented through PMFs, PDFs, and CDFs. Bernoulli, binomial, categorical, multinomial, Poisson, Gaussian, uniform, and exponential distributions appear frequently in machine-learning applications. Expectation summarizes a probability-weighted centre, variance measures dispersion, and covariance describes joint variation. The laws of large numbers explain why sample averages can estimate population expectations, while also showing why large samples cannot correct systematic bias. Sound probabilistic practice requires attention to calibration, dependence, base rates, data quality, ethical impact, and deployment conditions.

Key Terms

Sample space: Set of all possible outcomes of a random experiment.

Event: A subset of the sample space to which a probability can be assigned.

Probability measure: A function assigning values from zero to one to events while satisfying the probability axioms.

Conditional probability: Probability of one event given that another event has occurred.

Bayes’ theorem: Rule for computing a posterior probability from a prior, likelihood, and evidence.

Prior probability: Probability assigned before incorporating newly observed evidence.

Likelihood: Probability of observed evidence conditional on a proposed event or parameter.

Posterior probability: Updated probability after incorporating evidence.

Independence: Property whereby occurrence of one event does not alter the probability of another.

Random variable: Numerical function defined on outcomes of a random experiment.

PMF: Function assigning probabilities to values of a discrete random variable.

PDF: Density function whose integrals over intervals give probabilities for a continuous variable.

CDF: Function giving the probability that a random variable is less than or equal to a specified value.

Expectation: Probability-weighted average or theoretical mean of a random variable.

Variance: Expected squared deviation from the mean.

Covariance: Measure of joint directional variation between two random variables.

Correlation: Standardized covariance ranging from -1 to 1.

Calibration: Agreement between predicted probabilities and observed frequencies.

Law of large numbers: Result stating that sample averages approach population expectations under suitable conditions.

Base rate: Prevalence or prior frequency of an event in the relevant population.

Review Questions

Short Answer Questions

1. State the three axioms of probability.

2. Define conditional probability and give its multiplication-rule form.

3. Distinguish independence from mutual exclusivity.

4. What is the difference between a PMF and a PDF?

5. Define covariance and explain what its sign indicates.

Long Answer Questions

1. Derive Bayes’ theorem from the multiplication rule and explain each term using a classification example.

2. Explain the Bernoulli, binomial, categorical, Poisson, and Gaussian distributions and identify an appropriate machine-learning application for each.

3. Discuss expectation, variance, and covariance, including their principal algebraic properties.

4. Explain the weak and strong laws of large numbers and discuss their relevance to empirical risk estimation.

5. Describe why calibration and base rates are important when interpreting probabilistic predictions.

Analytical / Application Questions

1. A disease has prevalence 2%. A test has sensitivity 90% and a false-positive rate of 4%. Calculate the probability that a person with a positive result has the disease.

2. A fraud model flags 3% of legitimate transactions and detects 92% of fraud. If fraud prevalence is 1%, calculate the approximate fraction of flagged transactions that are fraudulent.

3. Explain why a large training dataset may still produce unreliable deployment probabilities when the deployment population differs from the training population.

22. Multiple Choice Questions with Answers

Question 1

A screening test has sensitivity 90%, false-positive rate 5%, and disease prevalence 1%. What is the approximate probability that a person with a positive result has the disease?

A. 1.0%

B. 15.4%

C. 50.0%

D. 90.0%

Correct Answer: B. 15.4%

Question 2

Which condition defines independence of events A and B?

A. P(A∩B)=0

B. P(A∪B)=1

C. P(A∩B)=P(A)P(B)

D. P(A|B)=P(B|A)

Correct Answer: C. P(A∩B)=P(A)P(B)

Question 3

For a continuous random variable X with density f, which expression gives P(a≤X≤b)?

A. f(a)+f(b)

B. ∫a^b f(x)dx

C. P(X=a)P(X=b)

D. E[X] over [a,b]

Correct Answer: B. ∫a^b f(x)dx

Question 4

If X and Y are independent and have finite expectations, which statement is necessarily true?

A. Cov(X,Y)=0

B. Corr(X,Y)=1

C. X and Y have equal means

D. X+Y is constant

Correct Answer: A. Cov(X,Y)=0

Question 5

Which distribution is most directly suited to modelling the number of successes in n independent trials with common success probability p?

A. Poisson

B. Binomial

C. Exponential

D. Uniform

Correct Answer: B. Binomial

Question 6

What does calibration of a predicted probability of 0.8 mean?

A. The model is correct for exactly 80% of individual cases

B. The prediction has 80% sensitivity

C. Comparable cases experience the event approximately 80% of the time

D. The model has 80% accuracy overall

Correct Answer: C. Comparable cases experience the event approximately 80% of the time

Question 7

Which statement about variance is correct?

A. Var(aX+b)=aVar(X)+b

B. Var(aX+b)=a²Var(X)

C. Variance has the same units as X

D. Variance cannot be zero

Correct Answer: B. Var(aX+b)=a²Var(X)

Question 8

The law of large numbers implies that, under suitable assumptions, the sample mean:

A. Equals the population mean for every finite sample

B. Approaches the population mean as sample size increases

C. Becomes independent of the observations

D. Eliminates systematic sampling bias

Correct Answer: B. Approaches the population mean as sample size increases

Question 9

A probability density value greater than 1 is:

A. Always impossible

B. Valid if the density integrates correctly to one

C. Evidence that the variable is discrete

D. Equivalent to a probability greater than one

Correct Answer: B. Valid if the density integrates correctly to one

Question 10

Why can accuracy be misleading for a rare-event classifier?

A. Accuracy does not use predictions

B. Rare events cannot be modelled probabilistically

C. A model can achieve high accuracy by mostly predicting the common class

D. Accuracy is always lower than precision

Correct Answer: C. A model can achieve high accuracy by mostly predicting the common class