Probability Foundations and Random Variables

Introduction

Data science is fundamentally concerned with reasoning under uncertainty. Measurements may be noisy, observations may be incomplete, and future outcomes cannot be known with certainty. Probability provides the mathematical language for representing uncertainty, combining evidence, quantifying risk, and constructing statistical and machine-learning models.

Probability is not merely a method for calculating gambling odds. It underlies classification, forecasting, Bayesian inference, experimental design, anomaly detection, reliability analysis, Monte Carlo simulation, and decision-making. This chapter develops the foundations required to model uncertain events and numerical outcomes rigorously.

Learning Objectives

By the end of this chapter, you should be able to:

• apply the probability axioms and standard counting rules;
• distinguish conditional probability, independence, and mutual exclusivity;
• derive and use Bayes’ theorem;
• define discrete and continuous random variables;
• interpret probability mass functions, density functions, and distribution functions;
• calculate expectation, variance, covariance, and correlation;
• use important probability distributions in data-science applications;
• explain moment-generating and probability-generating functions;
• identify modelling assumptions, limitations, and ethical risks in probabilistic systems.

1. Probability Spaces and Events

A probability model consists of three components:

1. Sample space \Omega: the set of all possible outcomes.
2. Event A: a subset of the sample space representing an outcome or collection of outcomes.
3. Probability measure P: a function assigning a number between 0 and 1 to events.

For a finite experiment involving one die, \Omega={1,2,3,4,5,6}. The event of obtaining an even number is A={2,4,6}.

Probability axioms

For every event A:

The entire sample space is certain:

If A₁,A₂,... are mutually exclusive events, then:

These axioms imply several useful results:

where A^c is the complement of A. For two events:

The subtraction avoids counting the intersection twice. If events are mutually exclusive, P(A∩B)=0, and the addition rule simplifies to P(A∪B)=P(A)+P(B).

Counting and equally likely outcomes

When all outcomes are equally likely,

where |A| is the number of favourable outcomes and |Ω| is the number of possible outcomes. The number of ways to select r objects from n objects without regard to order is

This formula applies when sampling is without replacement and each subset is treated as distinct, not when order matters.

2. Conditional Probability and Independence

Conditional probability expresses the probability of A when information about B is available:

Rearranging gives the multiplication rule:

The order can also be reversed:

Therefore,

This is Bayes’ theorem. Here, P(A) is the prior probability, P(B|A) is the likelihood, P(B) is the evidence, and P(A|B) is the posterior probability.

If B₁,...,B_k partition the sample space, then the law of total probability is

Substitution into Bayes’ theorem produces the practical form:

Independence

Events A and B are independent if knowing one does not change the probability of the other:

provided the probabilities are defined. Equivalently, P(A|B)=P(A) when P(B)>0.

Independence is different from mutual exclusivity. Mutually exclusive events cannot occur together, whereas independent events do not influence each other. Two events with positive probability cannot be both mutually exclusive and independent, because mutual exclusivity gives intersection probability zero while independence gives a positive product.

For random variables, pairwise independence means every pair is independent; mutual independence is stronger and requires every finite subset to satisfy the product rule.

Worked example: medical screening

Suppose a disease affects 1% of a population. A test detects disease with probability 0.95 and gives a positive result for a healthy person with probability 0.05. Let D denote disease and + denote a positive test.

Using Bayes’ theorem:

The total probability of a positive result is

Thus,

A positive result implies approximately a 16.1% probability of disease, not 95%. The difference arises from the low base rate and illustrates the importance of prior probabilities.

3. Random Variables

A random variable is a function that assigns a numerical value to each outcome in a sample space. It is commonly written as X or Y. A random variable is not itself random in the computational sense; uncertainty concerns which outcome, and therefore which value, will occur.

A discrete random variable has a finite or countably infinite set of possible values, such as the number of failed servers. A continuous random variable can take values in an interval, such as response time or temperature.

Distribution functions

The cumulative distribution function (CDF) of X is

A CDF is non-decreasing, approaches 0 as x approaches negative infinity, and approaches 1 as x approaches positive infinity.

For a discrete variable, the probability mass function (PMF) is

It satisfies p_X(x)≥0 and ∑_x p_X(x)=1. The CDF is obtained by summing the PMF up to x.

For a continuous variable, a probability density function (PDF) f_X(x) satisfies

with f_X(x)≥0 and

For a continuous variable, P(X=x)=0 for every individual point. Density is not probability; probability is area under the density curve.

Important distributions

For a Bernoulli variable,

If X counts successes in n independent Bernoulli trials with common probability p, then

The binomial assumptions are fixed trial count, two outcomes per trial, common success probability, and independence.

For a Poisson variable,

The parameter λ is both the mean and variance. The Poisson model is appropriate for counts under assumptions such as a constant event rate and independent occurrences; overdispersion or clustering may invalidate it.

The normal density is

The standardized variable is

When X is normal, Z has a standard normal distribution with mean 0 and variance 1. Standardization supports comparison across variables measured on different scales.

4. Expectation, Variance, and Moments

The expectation or mean represents a probability-weighted long-run average. For a discrete random variable,

For a continuous random variable,

For a function g(X),

in the discrete case, or E[g(X)]=∫g(x)f_X(x)dx in the continuous case.

Expectation is linear, even without independence:

Variance measures squared dispersion around the mean:

where μ=E[X]. The standard deviation is σ_X=√Var(X), expressed in the same units as X. For constants a and b,

If X and Y are independent, then

Without independence, covariance must be included.

Covariance and correlation

Covariance measures joint linear variation:

The correlation coefficient is

when both standard deviations are positive. It lies between −1 and 1. Zero covariance indicates no linear association, but generally does not prove independence. Independence implies zero covariance when the relevant moments exist.

For a weighted sum,

This formula is central to portfolio risk, ensemble modelling, sensor fusion, and error propagation.

Worked example: a discrete distribution

Let X take values 0, 1, and 2 with probabilities 0.2, 0.5, and 0.3. Then

Also,

Therefore,

and the standard deviation is √0.49=0.7.

5. Generating Functions

Generating functions encode a distribution in a function, allowing probabilities and moments to be derived algebraically.

For a non-negative integer-valued random variable, the probability-generating function (PGF) is

provided the series converges. Its derivatives at s=1 give factorial moments:

and

Thus Var(X)=G″_X(1)+G′_X(1)−[G′_X(1)]². For independent variables, the PGF of their sum is the product of their PGFs.

The moment-generating function (MGF) is

when finite in an interval around t=0. The nth derivative at zero gives the nth raw moment:

For independent X and Y, M_X+Y(t)=M_X(t)M_Y(t). MGFs can uniquely identify distributions when they exist near zero, but not every distribution has an MGF. The characteristic function, φ_X(t)=E[e^itX], always exists and is more general, though it is complex-valued.

6. A Data-Science Case: Fraud-Alert Calibration

Background. A payment platform uses a classifier to flag transactions for review. Fraud is rare, but false alerts impose substantial costs on customers and investigators.

Problem. In a validation population, 0.2% of transactions are fraudulent. The model flags 90% of fraudulent transactions and incorrectly flags 1% of legitimate transactions. What proportion of flagged transactions are actually fraudulent?

Analysis. Let F represent fraud and A represent an alert. Then P(F)=0.002, P(A|F)=0.90, and P(A|F^c)=0.01.

Bayes’ theorem gives

Only about 15.3% of alerts represent fraud. The model has high sensitivity, but the low base rate means most alerts are false positives.

Approach and outcome. The platform should not evaluate the classifier using sensitivity alone. It should report precision, review workload, expected financial loss, and performance across customer groups. Threshold selection can be framed as an expected-cost problem. If review costs, fraud losses, and customer-harm costs are specified, a threshold can be chosen to minimize expected loss rather than maximize a single technical metric.

Learning points. Base rates matter; conditional probabilities are directional; model performance depends on deployment prevalence; and probabilistic outputs should be calibrated before being interpreted as risks.

7. Limitations, Governance, and Best Practice

Probability models simplify reality. Independence may fail because users, devices, or transactions influence one another. Stationarity may fail when behaviour changes. Sampling bias can make estimated probabilities unrepresentative. A model can be mathematically correct yet operationally misleading if its population, time period, or measurement process is misunderstood.

Best practice includes:

• define the random experiment and target population explicitly;
• distinguish population probabilities from sample estimates;
• check whether independence, identical distribution, and fixed-rate assumptions are defensible;
• report uncertainty intervals or distributions rather than unsupported point certainty;
• validate calibration, subgroup performance, and distribution shift;
• avoid interpreting correlation as causation;
• document priors, likelihoods, data sources, and missing-data treatment;
• protect sensitive attributes and assess disparate error rates.

In high-impact applications such as lending, employment, healthcare, and policing, probabilistic scores can affect rights and opportunities. Governance therefore requires transparency, contestability, human oversight, privacy protection, and monitoring for discriminatory outcomes. AI systems should not present uncertain predictions as facts.

Chapter Summary

Probability axioms provide the formal foundation for reasoning about uncertain events. Conditional probability updates beliefs in light of evidence, while Bayes’ theorem combines prior information with likelihoods. Independence is a product relationship and must not be confused with mutual exclusivity.

Random variables map outcomes to numbers. Their PMFs, PDFs, and CDFs describe distributions. Expectation summarizes central tendency, variance measures dispersion, and covariance describes joint linear behaviour. Standard distributions such as Bernoulli, binomial, Poisson, normal, uniform, and exponential support common data-science models, but each depends on assumptions. Generating functions provide algebraic tools for moments and sums of independent variables. In applied work, base rates, calibration, data quality, distribution shift, fairness, and governance are as important as mathematical computation.

Key Terms

Probability: A numerical measure of uncertainty.

Sample space: The set of all possible outcomes.

Event: A subset of the sample space.

Complement: Outcomes not belonging to an event.

Conditional probability: Probability given that another event has occurred.

Independence: A condition in which one event does not alter the probability of another.

Mutual exclusivity: A condition in which events cannot occur together.

Bayes’ theorem: A rule for reversing conditional probabilities using priors and evidence.

Random variable: A numerical function of an uncertain outcome.

PMF: A function assigning probabilities to values of a discrete random variable.

PDF: A density whose area over an interval gives continuous probability.

CDF: The probability that a random variable is less than or equal to a specified value.

Expectation: The probability-weighted average of a random variable.

Variance: The expected squared deviation from the mean.

Covariance: A measure of joint linear variation between two variables.

Correlation: Standardized covariance ranging from −1 to 1.

Bernoulli distribution: A distribution for one binary trial.

Binomial distribution: A distribution for the number of successes in independent Bernoulli trials.

Poisson distribution: A distribution for event counts at a specified rate.

Moment-generating function: A function whose derivatives encode raw moments.

Review Questions

Short Answer Questions

1. State the three probability axioms.
2. Distinguish conditional probability from independence.
3. What is the difference between a PMF and a PDF?
4. Define covariance and correlation.
5. Why can a positive diagnostic result have a low posterior probability when the disease is rare?

Long Answer Questions

1. Derive Bayes’ theorem from the multiplication rule and explain each term.
2. Compare the Bernoulli, binomial, Poisson, and normal distributions, including their assumptions and applications.
3. Explain expectation, variance, and standard deviation, and derive the identity Var(X)=E[X²]−E[X]².
4. Discuss why independence assumptions are useful in data science and how their violation affects conclusions.
5. Explain probability-generating and moment-generating functions and their usefulness for sums of independent random variables.

Analytical / Application Questions

1. A screening test has sensitivity 0.92, a false-positive rate of 0.04, and disease prevalence of 0.5%. Calculate the posterior probability of disease after a positive result and interpret it.
2. A service receives an average of 3 independent requests per minute. Using a Poisson model, calculate the probability of exactly 5 requests in one minute and state assumptions that could make the model inappropriate.
3. Two risk factors have standard deviations 4 and 6 and correlation 0.25. For a combined score S=2X−Y, derive its variance in terms of the given quantities and explain the effect of covariance.

Multiple Choice Questions with Answers

Question 1

Which expression defines conditional probability when P(B)>0?

A. P(A|B)=P(A)+P(B)

B. P(A|B)=P(A∩B)/P(B)

C. P(A|B)=P(A)P(B)

D. P(A|B)=P(B)/P(A)

Correct Answer: B. P(A|B)=P(A∩B)/P(B)

Question 2

If two events are independent, which relationship must hold?

A. P(A∩B)=0

B. P(A∪B)=1

C. P(A∩B)=P(A)P(B)

D. P(A)=P(B)

Correct Answer: C. P(A∩B)=P(A)P(B)

Question 3

A disease prevalence is 0.01, sensitivity is 0.95, and false-positive rate is 0.05. What is approximately P(D|+)?

A. 0.010

B. 0.050

C. 0.161

D. 0.950

Correct Answer: C. 0.161

Question 4

For a continuous random variable with a density, what is P(X=x) for an individual point under the usual model?

A. 0

B. 1

C. f_X(x)

D. The mean of X

Correct Answer: A. 0

Question 5

Which condition is required for a binomial model?

A. A continuously measured outcome

B. Independent trials with a common success probability

C. A changing success probability at every trial

D. Events occurring at a constant continuous-time rate

Correct Answer: B. Independent trials with a common success probability

Question 6

If E[X]=1.1 and E[X²]=1.7, what is Var(X)?

A. 0.49

B. 0.60

C. 1.21

D. 2.80

Correct Answer: A. 0.49

Question 7

Which statement about correlation is correct?

A. It always proves causation.

B. It can exceed 1 for strongly related variables.

C. It is standardized covariance and lies between −1 and 1.

D. Zero correlation always implies independence.

Correct Answer: C. It is standardized covariance and lies between −1 and 1.

Question 8

For a Poisson distribution with rate λ, which statement is correct?

A. Its mean is λ and its variance is λ.

B. Its mean is zero for every λ.

C. It can take negative integer values.

D. It requires a fixed number of trials.

Correct Answer: A. Its mean is λ and its variance is λ.

Question 9

What does the first derivative of a PGF at s=1 provide?

A. The median

B. E[X]

C. The standard deviation

D. The probability of X=0 only

Correct Answer: B. E[X]

Question 10

Which statement best describes Bayes’ theorem in applied modelling?

A. It removes the need for prior information.

B. It converts a posterior into a sample space.

C. It updates a prior probability using evidence and a likelihood.

D. It applies only to continuous random variables.

Correct Answer: C. It updates a prior probability using evidence and a likelihood.