3. Random Variables and Probability Distributions

Introduction and Context

Probability provides a language for uncertainty; a random variable converts uncertain outcomes into numerical quantities that can be analysed, modelled, and used for decision-making. In data science, this connection is fundamental. A customer’s next purchase, the number of defects in a production batch, the time until a machine fails, and the classification score assigned to an image are all uncertain quantities. Random variables and probability distributions provide formal methods for describing such quantities before, during, and after data collection.

A probability distribution does more than list possible values. It specifies how probability is allocated across those values. This allocation enables the calculation of expected outcomes, variability, tail risk, percentiles, and probabilities of events. It also supports statistical inference, simulation, predictive modelling, quality control, reliability analysis, and machine learning.

This chapter develops the principal ideas needed to work with discrete and continuous random variables. It examines probability mass and density functions, cumulative distribution functions, expectation, variance, moments, quantiles, and important distributions including Bernoulli, binomial, geometric, Poisson, uniform, normal, exponential, and related models. The emphasis is not only on formulae but also on selecting, interpreting, and critically evaluating probability models.

Learning Objectives

After studying this chapter, you should be able to:

1. Define random variables and distinguish discrete from continuous variables.

2. Use probability mass functions, probability density functions, and cumulative distribution functions.

3. Calculate and interpret expectation, variance, standard deviation, covariance, and moments.

4. Explain the assumptions and applications of major discrete and continuous distributions.

5. Use quantiles, percentiles, and standardisation in data analysis.

6. Select an appropriate distribution for a practical problem and identify model limitations.

7. Apply distributions to simulation, risk analysis, operations, and predictive analytics.

From Random Outcomes to Numerical Variables

A random experiment is a process whose outcome cannot be known with certainty in advance, although the possible outcomes and their probabilities may be described. The sample space, denoted by \(\Omega\), is the set of all possible outcomes. A random variable \(X\) is a function that assigns a numerical value to every outcome in the sample space.

The word “variable” can be misleading. A random variable is not merely a column of observed numbers; it is a probabilistic model for the numerical result of an uncertain process. For example, if three components are tested, \(X\) may denote the number that fail. Before testing, \(X\) is random. After testing, its observed value is fixed.

Random variables may be classified as follows:

• A discrete random variable takes a finite or countably infinite set of values, such as 0, 1, 2, and so on.
• A continuous random variable can take values in intervals of the real line, such as operating temperature or delivery time.
• A mixed random variable has both discrete and continuous components, as may occur when a service time can be exactly zero for customers who abandon immediately and positive for others.

The distinction matters because probabilities are represented differently. For a discrete variable, probability is assigned to individual values. For a continuous variable, probability is assigned to intervals; the probability of any exact single value is ordinarily zero.

Probability Functions and Distribution Functions

Probability mass functions

For a discrete random variable \(X\), the probability mass function (PMF) is

\[p_X(x)=P(X=x).\]

A valid PMF satisfies

\[p_X(x)\geq 0, \qquad \sum_x p_X(x)=1.\]

The probability that \(X\) belongs to a set \(A\) is obtained by summing the masses of the values in that set:

\[P(X\in A)=\sum_{x\in A}p_X(x).\]

Suppose a help desk receives 0, 1, 2, or 3 urgent calls in an hour, with probabilities 0.10, 0.25, 0.40, and 0.25 respectively. The probability of at least two calls is \(0.40+0.25=0.65\).

Probability density functions

A continuous random variable is described by a probability density function (PDF), written \(f_X(x)\). A PDF must satisfy

\[f_X(x)\geq 0, \qquad \int_{-\infty}^{\infty}f_X(x)\,dx=1.\]

Probabilities are areas under the density curve:

\[P(a\leq X\leq b)=\int_a^b f_X(x)\,dx.\]

The value \(f_X(x)\) is not itself a probability and may exceed 1 when the measurement scale is sufficiently narrow. For a continuous variable, \(P(X=x)=0\) for every particular value \(x\), although intervals can have positive probability.

Cumulative distribution functions

The cumulative distribution function (CDF) is defined for every random variable by

\[F_X(x)=P(X\leq x).\]

For a discrete variable, the CDF is a step function. For a continuous variable, it is obtained by integrating the PDF:

\[F_X(x)=\int_{-\infty}^{x}f_X(t)\,dt.\]

The CDF is particularly useful for interval probabilities:

\[P(a<X\leq b)=F_X(b)-F_X(a).\]

Quantiles are also defined through the CDF. The \(q\)-quantile is a value \(x_q\) such that \(F_X(x_q)\geq q\), with an appropriate convention when the distribution has jumps. The median is the 0.5 quantile; the first and third quartiles are the 0.25 and 0.75 quantiles.

Expectation, Variability, and Moments

Expectation

The expected value, or mean, of a discrete random variable is

\[E[X]=\sum_x x p_X(x),\]

provided the sum is well defined. For a continuous random variable,

\[E[X]=\int_{-\infty}^{\infty}x f_X(x)\,dx.\]

Expectation is a probability-weighted average, not necessarily a value that the variable can actually take. If a fair die is rolled, its expected value is 3.5 even though 3.5 cannot appear on one roll.

For a function \(g(X)\), the law of the unconscious statistician gives

\[E[g(X)]=\sum_x g(x)p_X(x)\]

for a discrete variable, or

\[E[g(X)]=\int g(x)f_X(x)\,dx\]

for a continuous variable. This result allows quantities such as \(E[X^2]\) to be calculated directly from the distribution.

Expectation is linear:

\[E[aX+bY]=aE[X]+bE[Y],\]

whether or not \(X\) and \(Y\) are independent. This property is central in aggregate forecasting and portfolio analysis.

Variance and standard deviation

Variance measures expected squared deviation from the mean:

\[\operatorname{Var}(X)=E[(X-\mu)^2],\qquad \mu=E[X].\]

The computational form is

\[\operatorname{Var}(X)=E[X^2]-[E[X]]^2.\]

Standard deviation is \(\sigma=\sqrt{\operatorname{Var}(X)}\), and unlike variance it has the same units as the original variable. Variance is never negative and is zero only when the variable is constant with probability one.

For constants \(a\) and \(b\),

\[\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X).\]

Adding a constant changes location but not variability; multiplying by a constant changes the standard deviation by its absolute value.

Higher moments and shape

The raw moment of order \(r\) is \(E[X^r]\). The central moment of order \(r\) is \(E[(X-\mu)^r]\). The second central moment is variance. The third central moment is related to skewness, and the fourth to kurtosis.

A standardized third central moment is

\[\gamma_1=\frac{E[(X-\mu)^3]}{\sigma^3},\]

which describes asymmetry. Positive skewness generally indicates a longer right tail. A common excess-kurtosis measure is

\[\gamma_2=\frac{E[(X-\mu)^4]}{\sigma^4}-3.\]

These measures should be interpreted cautiously, particularly with small samples or extreme observations. Quantiles and visualisations may provide more stable descriptions of skewed data.

Covariance and dependence

For two random variables,

\[\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])].\]

The correlation is

\[\rho_{XY}=\frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y},\]

when both standard deviations are positive. Correlation measures linear association, not general dependence. Zero correlation does not always imply independence. If \(X\) and \(Y\) are independent, then \(E[XY]=E[X]E[Y]\) and their covariance is zero, subject to the existence of the relevant expectations.

Important Discrete Distributions

Bernoulli distribution

A Bernoulli trial has two outcomes, conventionally coded as success (1) and failure (0). If the probability of success is \(p\), then

\[P(X=x)=p^x(1-p)^{1-x},\qquad x\in\{0,1\}.\]

Its mean and variance are

\[E[X]=p,\qquad \operatorname{Var}(X)=p(1-p).\]

Examples include whether a transaction is fraudulent, whether a customer renews, or whether a sensor detects a fault. The two outcomes need not be equally likely, and “success” is simply the outcome being counted.

Binomial distribution

The binomial distribution counts successes in \(n\) independent Bernoulli trials with a common success probability \(p\):

\[P(X=k)=\binom{n}{k}p^k(1-p)^{n-k},\qquad k=0,1,\ldots,n.\]

Its mean and variance are \(np\) and \(np(1-p)\). The model requires a fixed number of trials, two outcomes per trial, constant success probability, and independence. Sampling without replacement from a small finite population may violate independence; a hypergeometric distribution may then be more suitable.

Geometric distribution

If independent Bernoulli trials continue until the first success, the number of trials required, \(X\), has the geometric distribution:

\[P(X=k)=(1-p)^{k-1}p,\qquad k=1,2,\ldots.\]

Its mean is \(1/p\), and its variance is \((1-p)/p^2\). The distribution is memoryless: given that the first success has not yet occurred, the remaining waiting-time distribution is unchanged. A different convention counts failures before the first success; analysts must state which definition they use.

Poisson distribution

The Poisson distribution models a count of events in a fixed interval when events occur independently at a constant average rate \(\lambda\):

\[P(X=k)=e^{-\lambda}\frac{\lambda^k}{k!},\qquad k=0,1,2,\ldots.\]

Both its mean and variance equal \(\lambda\). It is useful for modelling incoming calls, website requests, or defects per unit, provided the rate is reasonably stable and events are not strongly clustered. If events occur at rate \(\lambda\) per hour, a four-hour interval has Poisson mean \(4\lambda\).

The Poisson distribution can approximate a binomial distribution when \(n\) is large and \(p\) is small, with \(\lambda=np\). The approximation is not a substitute for checking assumptions.

Related count models

The negative binomial distribution can model the number of trials or failures required to obtain a specified number of successes. It is also frequently used for overdispersed count data, where the variance exceeds the mean. A binomial or Poisson model may be inadequate when observations are heterogeneous or correlated. In applied analytics, negative binomial regression is often preferred to Poisson regression when overdispersion is present.

Important Continuous Distributions

Uniform distribution

A continuous variable uniformly distributed on \([a,b]\) has constant density

\[f(x)=\frac{1}{b-a},\qquad a\leq x\leq b.\]

Its mean and variance are

\[E[X]=\frac{a+b}{2},\qquad \operatorname{Var}(X)=\frac{(b-a)^2}{12}.\]

The uniform model is appropriate when all values in a bounded interval are considered equally plausible, or as a simple model for random-number generation. It should not be adopted merely because the variable has known minimum and maximum values.

Normal distribution

The normal distribution has density

\[f(x)=\frac{1}{\sigma\sqrt{2\pi}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right],\qquad \sigma>0.\]

It is symmetric about \(\mu\), with mean, median, and mode equal to \(\mu\), and variance \(\sigma^2\). The standard normal distribution is \(N(0,1)\). A normal variable can be standardised as

\[Z=\frac{X-\mu}{\sigma}.\]

Standardisation permits probabilities to be obtained from the standard normal CDF, often denoted by \(\Phi\). For example,

\[P(X\leq x)=\Phi\left(\frac{x-\mu}{\sigma}\right).\]

The empirical 68–95–99.7 rule states that approximately 68%, 95%, and 99.7% of a normal distribution lie within one, two, and three standard deviations of its mean. These are approximations, not universal rules. The normal model is often justified by measurement mechanisms or by the central limit theorem, but individual data need not be normal for sample means to be approximately normal under suitable conditions.

Exponential distribution

The exponential distribution models a waiting time until the next event in a Poisson process. With rate \(\lambda>0\), its density and CDF are

\[f(x)=\lambda e^{-\lambda x},\quad x\geq0,\]

\[F(x)=1-e^{-\lambda x},\quad x\geq0.\]

Its mean is \(1/\lambda\), and its variance is \(1/\lambda^2\). It is memoryless:

\[P(X>s+t\mid X>s)=P(X>t).\]

This property is mathematically convenient but may be unrealistic for ageing equipment, where failure risk changes with operating time. The Weibull distribution is often more flexible because its shape parameter can represent increasing, decreasing, or constant hazard.

Gamma and lognormal distributions

The gamma distribution generalises the exponential distribution and can model the waiting time until several Poisson events occur. It is positive-valued and right-skewed, making it useful for claim sizes, service times, and accumulated waiting times.

A lognormal variable is one whose logarithm is normally distributed. It is also positive and right-skewed. Income, transaction values, and biological measurements may sometimes be better represented by a lognormal model than a normal model. Choice should be supported by domain knowledge and diagnostics rather than by convenience.

Choosing and Using a Distribution

Distribution selection is a modelling decision. A useful workflow is:

1. Define the random quantity and its measurement unit.

2. Determine whether it is a count, binary outcome, bounded measurement, waiting time, or unrestricted measurement.

3. Examine support: possible values may be nonnegative, bounded, integer-valued, or real-valued.

4. Consider the data-generating mechanism and assumptions such as independence, constant rate, or symmetry.

5. Estimate parameters using appropriate methods, commonly maximum likelihood or method of moments.

6. Compare the fitted model with data using graphical diagnostics, probability plots, goodness-of-fit measures, and validation on relevant outcomes.

7. Assess sensitivity: determine whether decisions change under plausible alternative distributions.

A histogram alone is not sufficient. Bin choices can conceal tails, and a large sample may reveal statistically significant deviations that have little practical effect. Conversely, a visually plausible model may underestimate rare but consequential events. Tail probabilities, quantiles, and decision-specific loss should receive explicit attention.

Applications in Data Science and Professional Practice

Probability distributions appear throughout the data-science lifecycle. In exploratory analysis, they summarise uncertainty and identify skewness, outliers, and unusual concentration. In predictive modelling, a model may estimate a full conditional distribution rather than only a point prediction. A demand forecast, for example, can provide the probability that demand exceeds inventory rather than merely its expected value.

In operations, Poisson counts support staffing and capacity planning, while exponential or gamma waiting-time models support queue and reliability analysis. In finance and insurance, quantiles and tail probabilities inform loss limits and reserves. In manufacturing, binomial models describe defect counts and normal models may describe dimensional variation. In digital services, distributions of latency and request counts guide service-level objectives and infrastructure allocation.

Simulation is another major application. A fitted distribution can generate synthetic observations for stress testing, Monte Carlo estimation, and scenario analysis. Simulation results are only as credible as the assumptions and parameter estimates behind them. Random generation does not convert a poor model into a reliable forecast.

Modern software libraries calculate PMFs, CDFs, quantiles, random samples, and fitted parameters. They reduce arithmetic errors but do not decide whether a Poisson process is appropriate or whether a normal approximation is defensible. Reproducible analysis should record the distribution, parameterisation, software version, random seed where relevant, and model-validation procedure.

Illustrative Case Study: Modelling Support-Centre Workload

Background

A software company wants to plan staffing for its technical support centre. The centre receives requests throughout the day, and managers need to estimate the probability that more than 20 requests will arrive in a one-hour period.

Situation and problem

Historical records suggest that the average number of requests in a one-hour interval is 16. However, managers have noticed that requests sometimes arrive in bursts after system releases. A simple Poisson model may be useful, but its assumptions must be examined.

Analysis

Under a Poisson model with \(\lambda=16\), the request count \(X\) has mean and variance both equal to 16. The probability of more than 20 requests is

\[P(X>20)=1-P(X\leq20),\]

which can be computed using a statistical software package. The resulting probability can be translated into a staffing target, but the model should first be checked against historical intervals.

Suppose the observed variance is substantially greater than 16. This overdispersion suggests that the constant-rate, independent-event assumption is inadequate. Possible causes include release-related clustering, different demand levels across time periods, or unobserved heterogeneity. A negative binomial model or a time-varying Poisson model may be more suitable.

Approach and solution

The analyst segments data by hour of day and release status, compares Poisson and negative binomial fits, and evaluates out-of-sample tail predictions. The centre then uses the model that best predicts high-volume intervals, not merely the model with the best average fit. A simulation incorporates predicted hourly rates and tests staffing policies under ordinary and release-day conditions.

Outcome

The analysis demonstrates that a single average rate can understate congestion risk. A model that represents changing rates and overdispersion produces more realistic upper-tail estimates and supports a flexible staffing plan, such as reserving additional agents during releases.

Learning points

The case illustrates four principles: the mean alone does not determine a distribution; variance is a diagnostic for count models; tail probabilities can matter more than average performance; and model selection must reflect the operational decision. It also shows why domain knowledge and data-generating mechanisms are as important as formulae.

Limitations, Risks, and Common Mistakes

A frequent mistake is confusing a PDF value with a probability. Probabilities require an area or an interval calculation for continuous variables. Another is assuming independence because observations are recorded in separate rows; time series, clustered customers, and repeated measurements often remain dependent.

Analysts also use the normal distribution for bounded, strongly skewed, or count-valued data without checking support. A normal model can assign negative probabilities to quantities that cannot be negative. Likewise, a Poisson model is often applied to counts with excess zeros, changing rates, or variance much larger than the mean.

Parameter uncertainty is another risk. Fitting a distribution does not mean its parameters are known exactly. Confidence intervals, predictive intervals, bootstrap procedures, or Bayesian posterior distributions may be needed. Outliers can substantially alter estimated means and variances, especially in heavy-tailed settings.

Ethically, distributional assumptions can affect access to resources, insurance premiums, credit decisions, and employee evaluation. Analysts should document assumptions, quantify uncertainty, test subgroup performance, and avoid presenting model-based probabilities as objective facts. Sensitive applications require governance, auditability, privacy protection, and human review where consequences are significant.

Current Developments and Best Practices

Contemporary analytics increasingly uses probabilistic forecasting, which produces calibrated distributions rather than single predictions. Quantile regression, Bayesian hierarchical models, survival models, probabilistic graphical models, and neural forecasting systems can represent uncertainty at several levels. Generative models and simulation-based inference also rely on the ability to describe and sample from complex distributions.

These methods do not eliminate foundational distributional reasoning. Good practice remains to define the target quantity clearly, inspect support and dependence, distinguish aleatory uncertainty from uncertainty about parameters or models, validate calibration, and evaluate performance in the tails when decisions concern rare events. Where no standard distribution is adequate, empirical, mixture, hierarchical, or semiparametric models may be preferable.

Chapter Summary

A random variable represents the numerical outcome of an uncertain experiment. Discrete variables use probability mass functions, while continuous variables use densities and interval probabilities; the cumulative distribution function applies to both. Expectation describes probability-weighted location, variance describes dispersion, and higher moments and quantiles describe shape and tail behaviour. Covariance and correlation describe aspects of joint variation.

Bernoulli and binomial distributions model binary trials and counts of successes. Geometric distributions model trials until a first success, while Poisson distributions model event counts under a constant-rate process. Uniform, normal, exponential, gamma, lognormal, and Weibull-related models describe important continuous phenomena. Each distribution depends on assumptions about support, independence, rate, shape, and the data-generating mechanism.

In data science, distributions support forecasting, simulation, reliability, capacity planning, risk analysis, and probabilistic machine learning. A responsible analyst chooses models using both statistical evidence and domain knowledge, validates decision-relevant quantities, reports parameter uncertainty, and recognises that a convenient distribution may fail in the tails or under dependence.

Key Terms

Random variable: A function assigning numerical values to outcomes of a random experiment.

Discrete random variable: A random variable with finite or countably infinite possible values.

Continuous random variable: A random variable taking values over intervals of the real line.

Probability mass function: A function assigning probabilities to individual values of a discrete variable.

Probability density function: A nonnegative function whose interval areas give probabilities for a continuous variable.

Cumulative distribution function: The function \(F(x)=P(X\leq x)\) for a random variable.

Expectation: The probability-weighted average or mean of a random variable.

Variance: The expected squared deviation from the mean.

Standard deviation: The positive square root of variance.

Moment: An expected power of a variable or its deviation from the mean.

Quantile: A value below which a specified proportion of a distribution lies.

Covariance: A measure of joint linear variation between two variables.

Correlation: A standardized measure of linear association.

Bernoulli distribution: A distribution for one binary trial.

Binomial distribution: The distribution of successes in a fixed number of independent Bernoulli trials.

Geometric distribution: The distribution of trials required to obtain the first success.

Poisson distribution: A count distribution for events occurring at a constant average rate.

Normal distribution: A symmetric continuous distribution determined by mean and variance.

Exponential distribution: A continuous waiting-time distribution with the memoryless property.

Overdispersion: A condition in which observed variance exceeds that expected under a specified model.

Review Questions

Short Answer Questions

1. What is the difference between a discrete random variable and a continuous random variable?

2. Why is a PDF value not itself a probability?

3. State the relationship between variance and the second raw moment.

4. What assumptions are required for a binomial model?

5. Explain the memoryless property of the exponential distribution.

Long Answer Questions

1. Explain PMFs, PDFs, and CDFs, including how each is used to calculate probabilities.

2. Derive and interpret the mean and variance of a Bernoulli random variable.

3. Compare the binomial, geometric, and Poisson distributions in terms of their mechanisms and applications.

4. Explain standardisation for a normal random variable and discuss the role of the central limit theorem.

5. Discuss a systematic workflow for selecting and validating a probability distribution in data science.

Analytical / Application Questions

1. A call centre receives an average of 12 calls per ten minutes. Explain when a Poisson model would be appropriate and identify two diagnostics that could reveal its inadequacy.

2. A delivery-time variable is positive and strongly right-skewed. Compare the suitability of normal, exponential, gamma, and lognormal models for this problem.

3. A quality team observes 100 independent items, each with a 0.03 probability of defect. Formulate a suitable model for the number of defects, calculate its mean and variance, and discuss whether a Poisson approximation might be reasonable.

22. Multiple Choice Questions with Answers

Question 1

A continuous random variable has density f(x). Which expression gives the probability that it lies between a and b?

A. f(b)-f(a)

B. ∫a^b f(x) dx

C. f(a)f(b)

D. P(X=a)+P(X=b)

Correct Answer: B. ∫a^b f(x) dx

Question 2

If X has mean 5 and variance 9, what are the mean and variance of Y=2X+4?

A. Mean 14, variance 13

B. Mean 10, variance 18

C. Mean 14, variance 36

D. Mean 9, variance 36

Correct Answer: C. Mean 14, variance 36

Question 3

Which situation most directly satisfies the binomial model?

A. Counting arrivals during a variable-rate day

B. Counting successes in 20 independent trials with the same success probability

C. Measuring the time until a machine fails

D. Measuring a temperature that varies continuously

Correct Answer: B. Counting successes in 20 independent trials with the same success probability

Question 4

A Poisson random variable has parameter λ=7. Which statement is correct?

A. Its mean is 7 and variance is 49

B. Its mean is 49 and variance is 7

C. Its mean and variance are both 7

D. Its standard deviation is 7 and mean is 1/7

Correct Answer: C. Its mean and variance are both 7

Question 5

For a Bernoulli random variable with success probability p, which variance is correct?

A. p^2

B. 1-p

C. p(1-p)

D. p/(1-p)

Correct Answer: C. p(1-p)

Question 6

Which distribution is most directly associated with the waiting time until the next event in a constant-rate Poisson process?

A. Binomial

B. Exponential

C. Uniform

D. Geometric

Correct Answer: B. Exponential

Question 7

A fitted count model has observed variance much greater than its mean. Which concern is most relevant if a Poisson model was used?

A. Underdispersion caused by a bounded continuous variable

B. Overdispersion indicating that the Poisson assumptions may fail

C. The count must follow a normal distribution

D. The expected value must be zero

Correct Answer: B. Overdispersion indicating that the Poisson assumptions may fail

Question 8

If Z=(X−μ)/σ, what is the purpose of this transformation when X is normally distributed?

A. It makes X discrete

B. It changes the distribution to uniform

C. It standardises X to a normal variable with mean 0 and variance 1

D. It removes all sampling uncertainty

Correct Answer: C. It standardises X to a normal variable with mean 0 and variance 1

Question 9

Which statement about quantiles is correct?

A. The median is always equal to the mean

B. The 0.75 quantile has approximately 75% of the distribution at or below it

C. A quantile can only be defined for discrete variables

D. Quantiles measure only variance

Correct Answer: B. The 0.75 quantile has approximately 75% of the distribution at or below it

Question 10

A support centre models hourly requests with a single Poisson rate, but release periods create clustered arrivals. What is the most appropriate response?

A. Keep the model because the mean is sufficient

B. Replace all counts with continuous measurements

C. Investigate time-varying rates or an overdispersed count model

D. Assume the requests are uniformly distributed

Correct Answer: C. Investigate time-varying rates or an overdispersed count model