3. Probability and Statistical Foundations

Introduction and Context

Insurance is an organized response to uncertainty. An insurer cannot know in advance which policyholders will die, suffer illness, experience a motor accident, or submit a property claim. It can, however, study patterns in relevant populations and use probability and statistics to estimate the frequency and financial consequences of uncertain events. These estimates support premium rating, reserving, underwriting, reinsurance, solvency assessment, claims management, and strategic decisions.

Probability provides a language for uncertainty, while statistics provides methods for learning about uncertainty from data. Actuarial work requires both. Probability models describe possible future outcomes; statistical methods estimate the parameters of those models and evaluate whether observed evidence is consistent with assumptions. This chapter develops the foundations needed for later study of mortality models, risk theory, life contingencies, credibility, and actuarial finance.

Learning Objectives

After studying this chapter, a student should be able to:

1. Apply conditional probability, independence, and Bayes’ theorem to insurance problems.

2. Define random variables and calculate probabilities, expectations, variances, covariances, and correlations.

3. Use probability-generating, moment-generating, and related functions.

4. Distinguish and apply important discrete and continuous probability distributions.

5. Explain sampling, estimation, confidence intervals, and hypothesis testing.

6. Interpret correlation and regression models in actuarial contexts.

7. Recognize model risk, data limitations, ethical issues, and the role of technology in statistical practice.

Probability as a Model of Insurance Uncertainty

A probability model consists of a sample space, a collection of events, and a probability measure. The sample space, denoted by \(\Omega\), contains all possible outcomes. An event is a subset of \(\Omega\), and its probability lies between zero and one. For events \(A\) and \(B\), the principal rules are

\[

P(A^c)=1-P(A),\qquad P(A\cup B)=P(A)+P(B)-P(A\cap B).

\]

The second expression avoids double-counting outcomes belonging to both events. In a claims portfolio, \(A\) might mean that a policy generates at least one claim and \(B\) that the claim exceeds a specified amount.

Conditional probability measures the likelihood of one event when information about another event is available:

\[

P(A\mid B)=\frac{P(A\cap B)}{P(B)},\quad P(B)>0.

\]

The multiplication rule follows directly:

\[

P(A\cap B)=P(A\mid B)P(B).

\]

If information about \(B\) does not alter the probability of \(A\), the events are independent, so that \(P(A\mid B)=P(A)\) and \(P(A\cap B)=P(A)P(B)\). Independence is a strong assumption. For example, two policyholders may not be independent if they live in the same flood zone or are insured under the same employer scheme.

Bayes’ Theorem and Updating Risk

Suppose \(B_1, B_2,\ldots,B_k\) are mutually exclusive and exhaustive events. Bayes’ theorem states

\[

P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum_{i=1}^{k}P(A\mid B_i)P(B_i)}.

\]

It combines prior information with new evidence. An insurer may begin with a prior probability that a claim is fraudulent and revise it after observing unusual documentation, claimant behaviour, or loss characteristics. The prior probability must not be confused with the likelihood of evidence: a screening test that is accurate among fraud cases may still produce many false positives when fraud is uncommon.

Bayesian reasoning is useful in underwriting, medical insurance, fraud detection, catastrophe assessment, and credibility theory. Its quality depends on the relevance of the prior distribution and the reliability of the evidence.

Random Variables and Their Numerical Summaries

A random variable assigns a numerical value to each outcome. A discrete random variable has a finite or countably infinite set of possible values; the number of claims in a month is an example. A continuous random variable can take values over intervals; claim severity and time to failure are common examples.

For a discrete variable \(X\), the probability mass function is \(p_X(x)=P(X=x)\). For a continuous variable, a density \(f_X(x)\) satisfies

\[

P(a\leq X\leq b)=\int_a^b f_X(x)\,dx,

\]

and a single point has probability zero. In both cases, the cumulative distribution function is \(F_X(x)=P(X\leq x)\).

The expected value is the probability-weighted average:

\[

E[X]=\sum_x xp_X(x)\quad\text{or}\quad E[X]=\int_{-\infty}^{\infty}xf_X(x)\,dx.

\]

For a function \(g\), \(E[g(X)]\) is obtained by replacing \(x\) with \(g(x)\). This result, often called the law of the unconscious statistician, permits calculation without first deriving the distribution of \(g(X)\).

Variance measures dispersion around the mean:

\[

\operatorname{Var}(X)=E[(X-E[X])^2]=E[X^2]-(E[X])^2.

\]

The standard deviation is the square root of variance. In insurance, the mean may represent expected claims cost, while the standard deviation indicates volatility and the need for capital or risk transfer.

For two random variables,

\[

\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])],

\]

and the correlation coefficient is

\[

\rho_{XY}=\frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}.

\]

Correlation lies between -1 and 1 and measures linear association. Zero correlation does not generally imply independence. Variance also obeys

\[

\operatorname{Var}(aX+bY)=a^2\operatorname{Var}(X)+b^2\operatorname{Var}(Y)+2ab\operatorname{Cov}(X,Y).

\]

This formula explains why diversification reduces aggregate uncertainty when risks are not perfectly positively correlated.

Conditional expectation is especially important in actuarial work. The law of total expectation and the law of total variance are

\[

E[X]=E[E(X\mid Y)],

\]

\[

\operatorname{Var}(X)=E[\operatorname{Var}(X\mid Y)]+\operatorname{Var}(E[X\mid Y]).

\]

The first term in the second expression represents average within-group variation; the second represents variation between groups.

Generating Functions and Aggregate Risk

The probability-generating function (PGF) of a non-negative integer-valued variable is

\[

G_X(s)=E[s^X].

\]

Its derivatives at one provide factorial moments: \(G'_X(1)=E[X]\). If \(X\) and \(Y\) are independent, the PGF of their sum is \(G_{X+Y}(s)=G_X(s)G_Y(s)\).

The moment-generating function (MGF), when it exists near zero, is

\[

M_X(t)=E[e^{tX}].

\]

Derivatives at zero give ordinary moments: \(M'_X(0)=E[X]\) and \(M''_X(0)=E[X^2]\). The cumulant-generating function is \(K_X(t)=\log M_X(t)\); cumulants add for independent sums. These functions simplify the analysis of aggregate claims, approximations, and compound distributions. Their limitations matter: an MGF may not exist for every distribution, and mathematical tractability does not establish empirical suitability.

Distributions Used in Actuarial Models

The Bernoulli distribution describes one trial with success probability \(p\). A sum of \(n\) independent Bernoulli trials has a binomial distribution, with

\[

P(X=k)=\binom nk p^k(1-p)^{n-k},\quad E[X]=np,\quad \operatorname{Var}(X)=np(1-p).

\]

It is appropriate for a fixed number of policies, each producing or not producing a claim under approximately identical and independent conditions.

The geometric distribution models the number of trials until the first success, while the negative binomial distribution generalizes this to the number of trials or failures until a specified number of successes. The hypergeometric distribution applies to sampling without replacement and is therefore relevant when a finite portfolio is sampled without replacement.

The Poisson distribution is widely used for claim counts:

\[

P(N=k)=e^{-\lambda}\frac{\lambda^k}{k!},\quad k=0,1,\ldots,

\]

with mean and variance both equal to \(\lambda\). The equality of mean and variance is a diagnostic property, not a universal truth. Overdispersion may indicate heterogeneity, dependence, or an inadequate model. A negative binomial model can accommodate variance exceeding the mean. A Poisson process additionally assumes independent increments and a constant event rate; these assumptions should be tested rather than accepted automatically.

The uniform distribution represents equal density over an interval and is often useful for simple approximation. The exponential distribution has density \(f(x)=\lambda e^{-\lambda x}\), for \(x\geq0\), and mean \(1/\lambda\). It is memoryless, a property sometimes useful in waiting-time models but often unrealistic for ageing equipment or human mortality.

The normal distribution is symmetric and determined by mean \(\mu\) and variance \(\sigma^2\). Standardization gives \(Z=(X-\mu)/\sigma\). The central limit theorem explains why standardized sums of many suitably behaved independent variables tend approximately to normality, even when individual observations are not normal. It does not justify normality automatically for highly skewed claim severities or dependent observations.

The lognormal distribution is useful for positive, right-skewed quantities because \(\log X\) is normal. Gamma and Weibull distributions offer flexible models for positive loss amounts and waiting times. The Pareto distribution, with its heavy right tail, is important for large claims and catastrophe exposures. Tail choice strongly affects reinsurance pricing, extreme quantiles, and solvency capital.

Sampling, Estimation, and Statistical Inference

A population is the complete set of units of interest; a sample is the observed subset. A parameter, such as a population mean or claim probability, describes the population. A statistic, such as the sample mean \(\bar X\), is calculated from data and used to estimate the parameter.

Simple random sampling gives each sample of a specified size an equal chance of selection. Stratified sampling divides the population into meaningful groups and samples within each group; it can improve precision when groups differ materially. Cluster sampling selects groups rather than individual units and may be operationally efficient but statistically less precise. Convenience sampling is vulnerable to selection bias and is generally unsuitable for reliable inference.

An estimator is unbiased if its expected value equals the parameter. The sample mean is unbiased for the population mean under random sampling. Consistency means that the estimator approaches the true parameter as sample size increases. Efficiency concerns comparative variability among estimators. In practice, a biased estimator may sometimes be preferred if its reduction in variance produces lower overall mean squared error.

For approximately normal observations with unknown variance, a confidence interval for the mean uses the Student \(t\) distribution:

\[

\bar X\pm t_{\alpha/2,n-1}\frac{s}{\sqrt n}.

\]

A 95% confidence interval is a procedure that captures the fixed parameter in 95% of repeated samples; it is not strictly correct to say that the parameter has a 95% probability of lying in one calculated frequentist interval. Bootstrap methods can estimate sampling uncertainty when analytical formulas are unreliable, though they cannot repair biased or unrepresentative data.

Hypothesis Testing and Regression

A hypothesis test begins with a null hypothesis \(H_0\), an alternative hypothesis \(H_1\), a test statistic, and a significance level. A p-value is the probability, assuming \(H_0\) is true, of observing a result at least as extreme as the one obtained. It is not the probability that \(H_0\) is true. A Type I error rejects a true null hypothesis; a Type II error fails to reject a false null hypothesis. Statistical significance should be considered alongside effect size, uncertainty, cost, and practical importance.

For comparing an observed claim frequency with an assumed rate, a binomial or Poisson test may be appropriate. A chi-square test can assess categorical association or goodness of fit. Multiple testing increases the chance of false discoveries and requires suitable control procedures.

Regression models quantify relationships between a response and explanatory variables. In simple linear regression,

\[

Y=\beta_0+\beta_1X+\varepsilon.

\]

The slope estimates the expected change in \(Y\) associated with one unit of \(X\), conditional on the model and other assumptions. Important assumptions include appropriate functional form, independent errors, stable variance, and limited influential observations. Residual plots are essential for diagnosis.

Insurance data often require generalized linear models (GLMs). A Poisson or negative-binomial model may model claim counts, while a gamma model with a log link may model positive claim severity. A logistic model estimates the probability of a binary outcome such as claim occurrence or lapse. Exposure, offsets, rating factors, interactions, credibility information, and fairness constraints must be considered. Association is not causation, and predictive accuracy does not by itself establish a legitimate underwriting rationale.

Illustrative Case Study: Revising a Motor Portfolio Frequency Model

Background

An insurer has 100,000 private motor policies. Its existing pricing model assumes an average of 0.10 claims per policy-year and uses a Poisson frequency distribution. Recent data are available by driver age, territory, vehicle type, and annual mileage.

Situation and Analysis

The portfolio recorded 11,800 claims, giving an observed frequency of 0.118. The difference from 0.10 may reflect random variation, genuine risk deterioration, changes in exposure mix, or reporting changes. Analysts first verify exposure definitions, duplicate records, missing mileage, claim notification delays, and whether the period includes unusual weather.

The variance of monthly portfolio counts substantially exceeds the mean. This suggests that a homogeneous Poisson model may be inadequate. Examination by territory shows that high-mileage urban risks have both higher frequency and greater variability. A negative-binomial GLM with an exposure offset is therefore considered. The team separates training and validation periods, examines residuals, and tests whether mileage and territory interact.

Approach and Solution

The insurer estimates a model of the form

\[

\log E[N_i]=\log(e_i)+\beta_0+\beta_1\text{Mileage}_i+\beta_2\text{Territory}_i+\cdots,

\]

where \(e_i\) is policy exposure. It compares Poisson and negative-binomial fits using out-of-sample performance, calibration, interpretability, and stability. The model is supplemented by governance review to ensure that variables are legally permissible, explainable, and not unjustifiably discriminatory.

Outcome

The analysis indicates that the original overall frequency understated expected claims for some segments but overstated it for others. The revised model improves calibration and identifies that dispersion arises partly from unobserved heterogeneity. Management adopts the model with monitoring thresholds, periodic recalibration, and a manual referral process for unusual cases.

Learning Points

The case demonstrates that a sample average is not automatically an adequate risk model. Exposure must be measured consistently, variance must be examined, assumptions must be validated, and statistical improvement must be balanced against fairness, explainability, operational feasibility, and regulatory expectations.

Professional Practice, Governance, and Emerging Analytics

Actuaries must distinguish data quality from model sophistication. Common failures include treating dependent risks as independent, ignoring exposure, confusing correlation with causation, extrapolating beyond observed data, selecting a distribution solely because it fits the centre of the data, and reporting p-values without practical interpretation. Small samples, missing data, censoring, truncation, changing claim definitions, and regime changes can invalidate apparently precise results.

Governance should document data sources, assumptions, transformations, model purpose, validation evidence, limitations, approval responsibilities, and monitoring plans. Sensitive personal information must be collected and used lawfully, securely, and only for legitimate purposes. Automated pricing and claims systems require controls for discrimination, explainability, cybersecurity, drift, and human oversight. Proxy variables may reproduce protected characteristics even when those characteristics are excluded explicitly.

Modern actuarial practice uses statistical programming, reproducible workflows, cloud platforms, dashboards, machine learning, and real-time telematics. Machine-learning methods can detect nonlinearities and interactions, but they may sacrifice interpretability and can amplify historical bias. The appropriate standard is not maximum predictive complexity; it is a model that is sufficiently accurate, stable, explainable, governed, and fit for its decision purpose.

Best-Practice Checklist

A sound actuarial analysis should:

• define the decision, population, exposure, and outcome before modelling;
• inspect data quality, missingness, outliers, dependence, and selection effects;
• use probability assumptions that match the mechanism and data;
• distinguish estimation uncertainty from process variability;
• validate models on data not used for fitting;
• report effect sizes, confidence intervals, sensitivity tests, and limitations;
• consider fairness, privacy, explainability, and regulatory requirements;
• monitor performance after implementation and revise assumptions when evidence changes.

Chapter Summary

Probability formalizes uncertain events through sample spaces, conditional probabilities, independence, and Bayes’ theorem. Random variables translate outcomes into numerical quantities, while expectation, variance, covariance, and correlation summarize their behaviour. Generating functions facilitate the study of sums and aggregate claims. Bernoulli, binomial, Poisson, negative binomial, exponential, normal, lognormal, gamma, Weibull, and Pareto distributions provide useful but assumption-dependent models.

Statistical inference uses samples to estimate parameters and quantify uncertainty. Confidence intervals and hypothesis tests must be interpreted carefully, particularly with large datasets and multiple comparisons. Regression and generalized linear models connect insurance outcomes with rating factors, but require diagnostic testing and professional judgement. Across all applications, data quality, model validation, ethics, governance, and monitoring are as important as mathematical technique.

Key Terms

Probability: A numerical measure of the likelihood of an event.

Conditional probability: The probability of an event given that another event has occurred.

Independence: A relationship in which occurrence of one event does not change the probability of another.

Bayes’ theorem: A rule for updating probabilities using new evidence.

Random variable: A numerical function defined on outcomes of a random experiment.

Probability mass function: Probabilities assigned to values of a discrete random variable.

Probability density function: A function whose integral over an interval gives a continuous probability.

Cumulative distribution function: The probability that a random variable is at most a specified value.

Expectation: The probability-weighted average value of a random variable.

Variance: The expected squared deviation from the mean.

Covariance: A measure of joint directional variation between two variables.

Correlation: Standardized covariance measuring linear association.

Moment-generating function: The expected value of an exponential transformation used to derive moments.

Poisson distribution: A count distribution with equal mean and variance under its basic model.

Central limit theorem: A result describing approximate normality of suitably standardized sums.

Estimator: A statistic used to estimate a population parameter.

Confidence interval: An interval produced by a procedure designed to capture a parameter with stated long-run frequency.

p-value: The probability under the null hypothesis of a result at least as extreme as the observed result.

Regression: A method for modelling a response in relation to explanatory variables.

Generalized linear model: A regression framework allowing non-normal responses and link functions.

Review Questions

Short Answer Questions

1. State the multiplication rule for two events and explain its actuarial interpretation.

2. Distinguish a probability mass function from a probability density function.

3. What does the variance of an aggregate claims amount measure?

4. Define a p-value and explain one common misinterpretation of it.

5. Why may a negative-binomial model be preferred to a Poisson model for claim counts?

Long Answer Questions

1. Explain Bayes’ theorem and illustrate how it can be used in insurance fraud screening.

2. Discuss expectation, variance, covariance, and correlation, including their relevance to diversification.

3. Compare the binomial, Poisson, normal, lognormal, gamma, and Pareto distributions in actuarial applications.

4. Explain the stages of statistical estimation and hypothesis testing for an insurance portfolio.

5. Discuss the assumptions, interpretation, and governance requirements of an insurance GLM.

Analytical / Application Questions

1. A portfolio has 10,000 independent policies, each with claim probability 0.02. Identify a suitable count model, calculate the expected claim count and variance, and explain whether the model’s assumptions should be reviewed.

2. A fraud alert is more common among fraudulent claims but fraud is rare overall. Explain why the positive predictive value may still be low and identify the probabilities needed for a Bayes calculation.

3. An insurer observes that claim-count variance is three times its mean. Analyse possible explanations and recommend a modelling and validation strategy.

22. Multiple Choice Questions with Answers

Question 1

If P(A)=0.20, P(B|A)=0.50, and P(B|A^c)=0.10, what is P(A|B)?

A. 0.20

B. 0.50

C. 0.556

D. 0.714

Correct Answer: C. 0.556

Question 2

For a Poisson random variable with parameter λ=4, which statement is correct?

A. Its mean is 4 and variance is 16.

B. Its mean and variance are both 4.

C. Its mean is 16 and variance is 4.

D. Its variance is always less than its mean.

Correct Answer: B. Its mean and variance are both 4.

Question 3

Which quantity is represented by Var(X)=E[X²]−(E[X])²?

A. The expected value

B. The standard error

C. The variance

D. The covariance

Correct Answer: C. The variance

Question 4

If two risks have zero correlation, which conclusion is necessarily valid?

A. They are independent.

B. They have identical means.

C. They have no linear association.

D. Their aggregate variance is zero.

Correct Answer: C. They have no linear association.

Question 5

A portfolio count has variance substantially greater than its mean. Which model issue is most directly suggested?

A. Underdispersion

B. Overdispersion relative to a basic Poisson model

C. Perfect independence

D. A continuous outcome

Correct Answer: B. Overdispersion relative to a basic Poisson model

Question 6

Which statement about a 95% frequentist confidence interval is most accurate?

A. There is a 95% probability that the fixed parameter lies in this particular interval.

B. The procedure captures the parameter in 95% of repeated samples under its assumptions.

C. The sample mean is correct with probability 0.95.

D. The null hypothesis is true with probability 0.95.

Correct Answer: B. The procedure captures the parameter in 95% of repeated samples under its assumptions.

Question 7

A p-value of 0.01 means that:

A. The null hypothesis has a 1% probability of being true.

B. The alternative hypothesis has a 99% probability of being true.

C. Assuming the null hypothesis, results at least this extreme have probability 0.01.

D. The estimated effect is necessarily economically important.

Correct Answer: C. Assuming the null hypothesis, results at least this extreme have probability 0.01.

Question 8

Which distribution is especially suitable for a positive, right-skewed variable whose logarithm is approximately normal?

A. Lognormal

B. Bernoulli

C. Uniform

D. Hypergeometric

Correct Answer: A. Lognormal

Question 9

In a claims GLM, an exposure offset is primarily used to:

A. Remove all sampling error

B. Account for differing amounts of exposure among observations

C. Guarantee independence of claims

D. Convert a continuous response into a binary response

Correct Answer: B. Account for differing amounts of exposure among observations

Question 10

Which statement best describes machine learning in actuarial modelling?

A. Greater predictive complexity automatically makes a model more suitable.

B. Validation, explainability, fairness, and governance remain necessary.

C. Machine learning eliminates the need for probability assumptions.

D. A model with high training accuracy cannot be biased.

Correct Answer: B. Validation, explainability, fairness, and governance remain necessary.