Probability Distributions and Statistical Models

Introduction and Context

Probability distributions are mathematical models for describing uncertainty. In artificial intelligence and machine learning, uncertainty arises because data are incomplete, measurements are noisy, observations vary across individuals, and future events cannot be known with certainty. A probability distribution provides a systematic way to represent possible outcomes, their probabilities, and important summaries such as the mean and variance.

A statistical model connects a probability distribution to a data-generating process. For example, a model may represent the number of customer arrivals per minute, the time until a machine fails, the intensity of an image pixel, or the prediction errors of a regression model. Selecting a suitable distribution requires understanding the type of variable, the measurement process, the independence structure, and the assumptions that can reasonably be defended.

This chapter develops the principal discrete and continuous probability distributions used in statistical analysis and AI: Bernoulli, binomial, geometric, Poisson, uniform, exponential, gamma, beta, normal, lognormal, chi-square, t, and F distributions. It also explains statistical models, parameter interpretation, model checking, and practical limitations.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish discrete and continuous random variables and interpret probability mass and density functions.

2. Select an appropriate distribution for a real-world or AI-related problem.

3. Calculate probabilities, expectations, variances, and quantiles for common distributions.

4. Explain the assumptions underlying each distribution and recognize their limitations.

5. Apply normal, chi-square, t, and F distributions in statistical inference.

6. Use simulation and diagnostic methods to evaluate statistical models.

7. Discuss ethical and professional implications of probabilistic modeling.

1. Random Variables and Distributional Models

A random variable X assigns a numerical value to each outcome of a random experiment. A discrete random variable takes countable values, whereas a continuous random variable can take values in an interval or collection of intervals.

For a discrete variable, the probability mass function (PMF) is

P(X = x) = p(x),

where p(x) >= 0 and the sum of p(x) over all possible x equals 1. For a continuous variable, the probability density function (PDF) f(x) satisfies f(x) >= 0 and

integral from minus infinity to infinity of f(x) dx = 1.

For a continuous variable, probabilities are areas, not point heights:

P(a <= X <= b) = integral from a to b of f(x) dx.

The cumulative distribution function (CDF) is

F(x) = P(X <= x).

The expected value describes the distribution’s center. For a discrete variable, E[X] = sum x p(x); for a continuous variable, E[X] = integral x f(x) dx. The variance is

Var(X) = E[(X - mu)^2] = E[X^2] - mu^2,

where mu = E[X]. The standard deviation is the positive square root of the variance.

A parameter is a fixed but usually unknown quantity describing a population or data-generating process. A statistic is calculated from a sample. Statistical modeling uses observed data to estimate parameters and assess whether the chosen model is credible.

2. Discrete Probability Distributions

2.1 Bernoulli Distribution

A Bernoulli trial has exactly two outcomes, conventionally called success and failure. If X = 1 denotes success and X = 0 denotes failure, then X has a Bernoulli distribution with success probability p, written X ~ Bernoulli(p), where 0 <= p <= 1.

P(X = x) = p^x(1-p)^(1-x), for x in {0,1}.

Its mean and variance are

E[X] = p,

Var(X) = p(1-p).

The model assumes one trial, two mutually exclusive outcomes, and a clearly defined success probability. Examples include whether a transaction is fraudulent, whether an image contains a pedestrian, or whether a server request fails. Logistic regression models a Bernoulli response by allowing p to depend on explanatory variables through the logistic function:

p(x) = 1 / [1 + exp(-(beta_0 + beta_1 x_1 + ... + beta_k x_k))].

2.2 Binomial Distribution

The binomial distribution counts successes in n Bernoulli trials. If trials are independent and each has the same success probability p, then X ~ Binomial(n,p), with

P(X = k) = C(n,k) p^k(1-p)^(n-k),

for k = 0,1,...,n, where C(n,k) = n!/[k!(n-k)!].

The mean and variance are

E[X] = np,

Var(X) = np(1-p).

The fixed number of trials and constant p are essential assumptions. If observations are dependent, or probabilities differ substantially among trials, the binomial model may be inappropriate. For example, testing 100 independent images for a defect can be binomial, but sampling without replacement from a small finite batch is better modeled by a hypergeometric distribution.

Worked example. Suppose a classifier correctly labels an independent image with probability 0.8. For n = 10 images, the probability of exactly 8 correct labels is

P(X=8) = C(10,8)(0.8)^8(0.2)^2

= 45(0.16777216)(0.04)

= 0.301989888.

Thus, the probability is approximately 0.302. This calculation treats the accuracy as constant and predictions as independent; both assumptions should be checked in practice.

2.3 Geometric Distribution

The geometric distribution models the number of trials until the first success. Under independent Bernoulli trials with constant success probability p, if X counts the trial on which the first success occurs,

P(X = k) = (1-p)^(k-1)p, for k = 1,2,...

and

E[X] = 1/p, Var(X) = (1-p)/p^2.

It has the memoryless property: the probability of waiting additional trials does not depend on how many failures have already occurred. This property is often unrealistic when fatigue, learning, maintenance, or changing demand affects future events. A geometric model may describe the number of attempts until an independent API request succeeds, provided the success probability remains stable.

A related convention counts failures before the first success; then the support is 0,1,2,... and the PMF is (1-p)^k p. The convention must always be stated.

2.4 Poisson Distribution

The Poisson distribution models the number of events in a fixed interval of time, space, area, or volume when events occur independently at a constant average rate. If X ~ Poisson(lambda), then

P(X=k) = exp(-lambda) lambda^k / k!, for k = 0,1,2,...

and

E[X] = Var(X) = lambda.

Here lambda is the expected number of events in the specified interval. If the rate is r events per unit and the interval has length t, then lambda = rt. The Poisson model is appropriate for rare-event counts under approximate independence and a stable rate. Examples include incoming support tickets per hour, defects per metre of cable, or requests arriving at a service endpoint.

The Poisson distribution is not appropriate merely because data are counts. Excessive zeros, clustering, changing rates, or dependence may require zero-inflated, negative binomial, Hawkes-process, or time-varying-rate models.

3. Continuous Probability Distributions

3.1 Uniform Distribution

A continuous variable X is uniform on [a,b], written X ~ Uniform(a,b), if all values in the interval are equally plausible in the density sense:

f(x) = 1/(b-a), for a <= x <= b,

and 0 otherwise. Its mean and variance are

E[X] = (a+b)/2,

Var(X) = (b-a)^2/12.

For a <= c <= d <= b,

P(c <= X <= d) = (d-c)/(b-a).

Uniform models are useful for random initialization, simulation, and situations where only lower and upper bounds are defensible. They should not be used automatically for bounded data merely because the variable has a finite range.

3.2 Exponential Distribution

The exponential distribution models a nonnegative waiting time until the next event in a Poisson process. If X ~ Exponential(lambda), where lambda > 0 is a rate per unit time,

f(x) = lambda exp(-lambda x), for x >= 0.

Its mean and variance are

E[X] = 1/lambda,

Var(X) = 1/lambda^2.

Its survival function is P(X > x) = exp(-lambda x). It is memoryless: P(X>s+t | X>s) = P(X>t). This is useful for simple queueing and reliability models, but it excludes aging and wear-out effects. If a component becomes more likely to fail as it ages, a Weibull model may be more suitable.

3.3 Gamma Distribution

The gamma distribution generalizes the exponential distribution and can model positive, skewed quantities. Using shape alpha > 0 and rate beta > 0,

f(x) = [beta^alpha / Gamma(alpha)] x^(alpha-1) exp(-beta x), for x > 0.

Its mean and variance are

E[X] = alpha/beta,

Var(X) = alpha/beta^2.

The parameterization must be stated because some texts use scale theta = 1/beta. When alpha = 1, the gamma distribution becomes exponential with rate beta. A gamma variable can represent the waiting time until alpha events in a Poisson process when alpha is a positive integer, and gamma distributions are widely used as priors for rates in Bayesian models.

3.4 Beta Distribution

The beta distribution is defined on [0,1] and is useful for proportions and probabilities. With shape parameters alpha > 0 and beta > 0,

f(x) = x^(alpha-1)(1-x)^(beta-1) / B(alpha,beta), for 0 < x < 1,

where B(alpha,beta) is the beta function that normalizes the density. Its mean and variance are

E[X] = alpha/(alpha+beta),

Var(X) = alpha beta / [(alpha+beta)^2(alpha+beta+1)].

The shapes alpha and beta control concentration and skewness. A Beta(1,1) distribution is uniform on [0,1]. The beta distribution is especially important in Bayesian modeling of an unknown Bernoulli probability because the beta family is conjugate to the binomial likelihood. It is not automatically appropriate for every percentage: percentages based on different denominators can have different uncertainties.

4. The Normal and Lognormal Distributions

4.1 Normal Distribution

The normal distribution is written X ~ N(mu,sigma^2), where sigma > 0. Its density is

f(x) = [1/(sigma sqrt(2 pi))] exp[-(x-mu)^2/(2 sigma^2)], for minus infinity < x < infinity.

The parameters are the mean mu and variance sigma^2. Standardization gives

Z = (X-mu)/sigma,

and Z ~ N(0,1). Therefore,

P(a <= X <= b) = Phi((b-mu)/sigma) - Phi((a-mu)/sigma),

where Phi is the standard normal CDF.

The normal distribution is symmetric and unbounded. It often arises as an approximation to sums or averages through the central limit theorem, under suitable conditions. It is used for measurement errors, aggregate demand, residuals, and latent variables. However, a normal model can assign positive probability to impossible negative values and can understate extreme events when data are heavy-tailed.

Worked example. If a measurement is modeled as N(100, 15^2), the probability that it lies between 85 and 130 is

P(85 <= X <= 130) = Phi(2.00) - Phi(-1.00).

Using Phi(2.00) approximately 0.9772 and Phi(-1.00) approximately 0.1587 gives 0.8185. Thus, approximately 81.85% of measurements fall in this interval under the model.

4.2 Lognormal Distribution

A positive variable X is lognormal if ln(X) is normally distributed. If ln(X) ~ N(mu,sigma^2), then X is lognormal. Its mean and variance are

E[X] = exp(mu + sigma^2/2),

Var(X) = [exp(sigma^2)-1] exp(2mu+sigma^2).

The distribution is right-skewed and suitable for quantities formed by multiplicative effects, such as incomes, file sizes, response times, and some biological measurements. The arithmetic mean of X is not exp(mu); exp(mu) is the median. Confusing the mean on the log scale with the mean on the original scale is a common modeling error.

5. Chi-Square, t, and F Distributions

These distributions are central to classical statistical inference.

5.1 Chi-Square Distribution

If Z_1,...,Z_n are independent standard normal variables, then

Q = Z_1^2 + ... + Z_n^2

has a chi-square distribution with n degrees of freedom, written chi-square(n). Its mean is n and its variance is 2n. It is nonnegative and right-skewed, though it becomes more symmetric as n increases.

For a sample from a normal population, the standardized sample variance satisfies

(n-1)S^2/sigma^2 ~ chi-square(n-1),

where S^2 is the usual sample variance and n-1 is the degrees of freedom. This result requires independent observations from a normal population. Chi-square distributions also arise in goodness-of-fit tests and tests of independence in contingency tables, where expected cell counts should generally be sufficiently large for the approximation to be reliable.

5.2 Student’s t Distribution

If Z ~ N(0,1), V ~ chi-square(nu), and Z and V are independent, then

T = Z / sqrt(V/nu)

has a t distribution with nu degrees of freedom. It is symmetric around zero, has heavier tails than the standard normal, and approaches the standard normal as nu increases.

For a normal population with unknown variance,

T = (X-bar - mu)/(S/sqrt(n)) ~ t(n-1).

The t distribution is used for confidence intervals and tests concerning means when the population standard deviation is unknown. For strongly nonnormal data or very small samples, the nominal t procedure may be unreliable; robust or resampling methods may be preferable.

5.3 F Distribution

If V_1 ~ chi-square(nu_1) and V_2 ~ chi-square(nu_2) are independent, then

F = (V_1/nu_1)/(V_2/nu_2)

has an F distribution with numerator and denominator degrees of freedom nu_1 and nu_2. It is nonnegative and generally right-skewed. The F distribution is used to compare variances and in analysis of variance (ANOVA) and regression model comparison.

In one-way ANOVA, the statistic is

F = MS_between / MS_within,

where mean squares are variance estimates associated with between-group and within-group variation. The classical F test assumes independent observations, normally distributed errors, and equal population variances across groups. Independence is especially important; clustered observations can produce misleadingly small p-values.

6. Choosing and Evaluating a Distribution

Distribution selection should begin with the data-generating mechanism rather than with a convenient formula.

1. Identify the variable type. Is it binary, a count, a waiting time, a proportion, an unrestricted measurement, or a positive skewed measurement?

2. Identify support. A normal distribution permits all real values; a Poisson distribution permits nonnegative integers; a beta distribution is restricted to [0,1].

3. Consider the process. Fixed trials suggest binomial; arrivals suggest Poisson; waiting times suggest exponential or gamma; multiplicative variation suggests lognormal.

4. Check dependence and heterogeneity. Repeated observations from one person, device, or location may not be independent. Changing rates may require hierarchical or time-varying models.

5. Estimate parameters using suitable methods. Maximum likelihood selects parameters that make the observed data most plausible under the model. Bayesian methods combine a likelihood with a prior distribution to obtain a posterior distribution.

6. Diagnose fit. Histograms, empirical CDFs, Q-Q plots, residual plots, probability plots, and posterior predictive checks can reveal skewness, outliers, heavy tails, zero inflation, and dependence.

A good fit to historical data does not prove that the model is true. It may fail under distribution shift, changing behavior, new technology, or adversarial manipulation.

7. Statistical Models in AI and Machine Learning

A statistical model often takes the form

data = systematic component + random component.

In linear regression, for example,

Y_i = beta_0 + beta_1 x_i1 + ... + beta_k x_ik + epsilon_i,

where errors may be modeled as independent N(0,sigma^2) variables under the classical model. The normal error assumption is mainly relevant to exact inference; least-squares prediction can still be useful when errors are not normal, provided other assumptions and goals are considered.

For classification, a Bernoulli or binomial likelihood supports probabilistic predictions. For event counts, Poisson regression can model

log(lambda_i) = beta_0 + beta_1 x_i1 + ... + beta_k x_ik,

where lambda_i is the expected count for observation i. The logarithm ensures lambda_i is positive. If the variance is substantially greater than the mean, known as overdispersion, a negative binomial model or another alternative may be more appropriate.

In Bayesian AI, beta, gamma, and normal distributions are commonly used as prior or posterior distributions. The choice of prior can affect decisions when data are scarce, so prior assumptions should be documented and subjected to sensitivity analysis.

8. Illustrative Case: Predictive Maintenance for Industrial Pumps

Background. A manufacturer operates 2,000 industrial pumps and wants to predict failures and schedule maintenance. The data include daily failure counts, operating temperatures, time between failures, and binary alerts generated by a machine-learning model.

Problem. Engineers initially model all failure-related variables as normal. This produces negative predicted waiting times, poorly calibrated alert probabilities, and an apparent excess of failures in a few facilities.

Analysis. Failure occurrence for each pump on a day is binary, so a Bernoulli model is appropriate for individual outcomes. The number of failures across a stable group of pumps can be modeled with a binomial distribution, but only if the pumps have similar probabilities and outcomes are sufficiently independent. Daily failures across the whole facility are counts; a Poisson model is a starting point, but the observed variance is four times the mean, indicating overdispersion. A negative binomial model is therefore investigated. Time between failures is positive and right-skewed; exponential and gamma models are compared. The exponential model’s memoryless assumption is rejected because older pumps fail more often. A gamma or Weibull model better captures aging. Alert probabilities are modeled with a beta distribution for uncertainty calibration, while the classifier itself produces Bernoulli probabilities conditional on sensor features.

Approach and solution. Engineers stratify by pump type, include facility-level random effects, and examine residuals and calibration plots. They use a gamma-type survival model for time-to-failure, a count model allowing overdispersion for facility failures, and logistic regression for alert outcomes. Predictions are evaluated using calibration, precision-recall performance, expected maintenance cost, and false-negative rates rather than accuracy alone.

Outcome. The revised system reduces missed high-cost failures while avoiding unnecessary replacement of reliable pumps. Maintenance staff retain authority to override automated recommendations when sensor faults or safety concerns are present.

Learning points. The case demonstrates that distribution choice must follow the measurement process, that dependence and heterogeneity matter, and that predictive performance should be assessed against operational costs and safety objectives. A statistically plausible model is not automatically a responsible decision system.

9. Limitations, Ethics, and Best Practices

Probability models simplify reality. Common limitations include incorrect independence assumptions, parameter instability, outliers, selection bias, measurement error, omitted variables, and distribution shift. Model uncertainty should be distinguished from random variability: even if parameters were known exactly, future outcomes would remain uncertain; when parameters are estimated, there is additional uncertainty.

In AI systems, probabilistic outputs can affect access to credit, healthcare, employment, insurance, and public services. A model that is well calibrated overall may be poorly calibrated for a protected subgroup. Practitioners should evaluate subgroup performance, privacy risks, disparate error rates, explainability, and the consequences of false positives and false negatives. Sensitive data should be collected and used lawfully, securely, and only for justified purposes.

Best practices include:

• State the variable, units, support, parameters, and assumptions explicitly.
• Use a distribution because of a defensible process, not because it is familiar.
• Separate training, validation, and test data when evaluating predictive systems.
• Inspect graphical diagnostics and perform sensitivity analysis.
• Test independence, stationarity, and overdispersion where relevant.
• Report uncertainty intervals rather than only point predictions.
• Consider cost-sensitive and subgroup-specific evaluation.
• Monitor deployed models for drift and recalibrate when necessary.
• Preserve reproducible records of data, transformations, model versions, and decisions.

Contemporary Developments

Modern probabilistic machine learning combines classical distributions with flexible models. Mixture models represent heterogeneous populations by combining several component distributions. Hierarchical models allow related groups to share information while retaining group-specific effects. Bayesian computation uses Markov chain Monte Carlo and variational inference to estimate complex posterior distributions. Deep generative models learn high-dimensional distributions, while conformal prediction provides distribution-free prediction sets under exchangeability assumptions.

These methods expand modeling capability but do not remove the need for careful assumptions. More flexible models can be harder to interpret, more computationally demanding, and vulnerable to data leakage or distribution shift. Classical distributions remain valuable because they provide transparent baselines, interpretable parameters, and foundations for inference.

Chapter Summary

Probability distributions model uncertainty in discrete and continuous variables. Bernoulli distributions represent one binary trial; binomial distributions count successes in a fixed number of independent trials; geometric distributions describe trials until first success; and Poisson distributions describe event counts under a stable rate. Uniform, exponential, gamma, and beta distributions model bounded values, waiting times, positive skewed quantities, and proportions. Normal distributions model symmetric measurements and support many approximations, while lognormal distributions describe positive multiplicative variation. Chi-square, t, and F distributions underpin inference about variances, means, and variance ratios.

Effective model selection requires attention to support, units, data-generating mechanisms, independence, parameter stability, and diagnostics. In AI, probability distributions support classification, regression, count modeling, Bayesian inference, uncertainty quantification, and decision-making. Models must be evaluated not only statistically but also ethically and operationally, with attention to calibration, subgroup effects, privacy, safety, and distribution shift.

Key Terms

Random variable: A numerical function assigning values to outcomes of a random experiment.

Probability mass function: A function giving probabilities for the values of a discrete random variable.

Probability density function: A nonnegative function whose area over an interval gives a continuous probability.

Cumulative distribution function: The probability that a random variable is less than or equal to a specified value.

Expected value: The probability-weighted mean or long-run average of a random variable.

Variance: The expected squared distance from the mean.

Bernoulli distribution: Distribution for one binary trial with success probability p.

Binomial distribution: Distribution of successes in n independent Bernoulli trials.

Geometric distribution: Distribution of trials until the first success.

Poisson distribution: Distribution of event counts in a fixed interval at a constant average rate.

Uniform distribution: Distribution with constant density over a specified interval.

Exponential distribution: Distribution of waiting time to the next event in a Poisson process.

Gamma distribution: Flexible positive distribution characterized by shape and rate or scale.

Beta distribution: Distribution on [0,1] commonly used for probabilities and proportions.

Normal distribution: Symmetric bell-shaped distribution defined by mean and variance.

Lognormal distribution: Distribution whose logarithm is normally distributed.

Degrees of freedom: Number of independent pieces of information available for estimating variation.

Overdispersion: Variability in count data greater than that predicted by a Poisson model.

Calibration: Agreement between predicted probabilities and observed frequencies.

Distribution shift: Change in the data-generating distribution between development and deployment.

Review Questions

Short Answer Questions

1. What is the difference between a probability mass function and a probability density function?

2. State the assumptions of a binomial distribution.

3. Why is the exponential distribution called memoryless?

4. What does the rate parameter lambda represent in a Poisson distribution?

5. Distinguish the mean of a lognormal variable from the mean of its logarithm.

Long Answer Questions

1. Explain the Bernoulli, binomial, and geometric distributions, including their relationships, assumptions, and applications.

2. Compare the Poisson, exponential, and gamma distributions in terms of the events or waiting times they model.

3. Explain the normal, chi-square, t, and F distributions and describe their roles in statistical inference.

4. Develop a principled procedure for selecting and evaluating a distribution for an AI dataset.

5. Discuss the ethical and operational risks of using probabilistic predictions in high-impact decisions.

Analytical / Application Questions

1. A service receives an average of 12 independent requests per hour. Explain when a Poisson model would be appropriate and calculate the probability of exactly 10 requests in one hour.

2. A classifier is tested on 20 independent observations with constant accuracy 0.9. Formulate a binomial model and calculate the probability of exactly 18 correct predictions.

3. A machine’s time between failures is positive and becomes more variable as the machine ages. Compare exponential, gamma, and lognormal modeling choices, and propose diagnostics for selecting among them.

Multiple Choice Questions with Answers

Question 1

Which distribution models the number of successes in a fixed number of independent trials with constant success probability?

A. Binomial

B. Poisson

C. Geometric

D. Exponential

Correct Answer: A. Binomial

Question 2

If X ~ Bernoulli(p), which expression gives Var(X)?

A. p

B. p(1-p)

C. 1/p

D. p^2

Correct Answer: B. p(1-p)

Question 3

For X ~ Poisson(lambda), what is Var(X)?

A. lambda^2

B. 1/lambda

C. lambda

D. sqrt(lambda)

Correct Answer: C. lambda

Question 4

If X ~ Exponential(lambda), what is E[X]?

A. lambda

B. lambda^2

C. 1/lambda

D. 1-lambda

Correct Answer: C. 1/lambda

Question 5

Which distribution is naturally defined on the interval [0,1] and is often used for probabilities?

A. Normal

B. Beta

C. Gamma

D. Chi-square

Correct Answer: B. Beta

Question 6

If X ~ N(mu,sigma^2), which transformation has a standard normal distribution?

A. X/mu

B. (X-mu)/sigma

C. (X-sigma)/mu

D. X-mu/sigma^2

Correct Answer: B. (X-mu)/sigma

Question 7

A binomial random variable has n = 10 and p = 0.8. What is its expected value?

A. 0.8

B. 2

C. 8

D. 10.8

Correct Answer: C. 8

Question 8

Which distribution is used in the statistic (X-bar - mu)/(S/sqrt(n)) when sampling from a normal population with unknown variance?

A. Uniform

B. Poisson

C. Student’s t

D. Beta

Correct Answer: C. Student’s t

Question 9

If a count dataset has variance substantially greater than its mean, which issue is indicated for a basic Poisson model?

A. Underflow

B. Overdispersion

C. Symmetry

D. Truncation

Correct Answer: B. Overdispersion

Question 10

If ln(X) is normally distributed, then X has which distribution?

A. Lognormal

B. Gamma

C. Geometric

D. F

Correct Answer: A. Lognormal