Statistical Foundations and Generalized Linear Models

Introduction

Insurance pricing and reserving are fundamentally statistical activities. An insurer observes incomplete, variable, and often skewed information about future claims. Statistical modelling provides a disciplined way to describe uncertainty, estimate relationships, test evidence, and produce predictions that can be translated into premiums, reserves, underwriting rules, and capital decisions.

Generalized linear models (GLMs) are central to insurance analytics because they combine statistical interpretability with operational usefulness. They allow the expected claim outcome to depend on rating factors while accommodating non-normal distributions, unequal exposure, nonlinear transformations, and interactions. Common applications include claim frequency, claim severity, probability of claim, and pure premium.

This chapter develops the foundations required to construct and evaluate Poisson, negative binomial, Gamma, Tweedie, and logistic models. It also addresses offsets, interactions, regularization, diagnostics, governance, and practical implementation.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain probability distributions, expectation, variance, estimation, and hypothesis testing in an insurance context.
2. Distinguish ordinary linear regression from generalized linear modelling.
3. Select and interpret Poisson, negative binomial, Gamma, Tweedie, and logistic models.
4. Use exposure offsets and interpret coefficients, interactions, and link functions.
5. Apply regularization and diagnostic techniques to improve model reliability.
6. Evaluate the professional, ethical, regulatory, and managerial implications of predictive insurance models.

1. Probability and Insurance Data

A random variable represents an uncertain quantity. In insurance, examples include the number of claims in a policy period, the cost of a claim, or whether a policyholder reports at least one claim. A probability distribution describes the possible values and their probabilities.

The expected value of a random variable X is its probability-weighted average:

For a continuous variable, probabilities are represented by a density and the corresponding expression is E[X] = ∫ x f(x) dx. The variance measures dispersion around the mean:

where μ = E[X]. The standard deviation is √Var(X), expressed in the same units as X.

Insurance distributions are rarely symmetric. Claim counts are non-negative integers and often contain many zeros. Claim costs are non-negative, right-skewed, and may have extreme observations. Aggregate losses combine a discrete frequency process with a continuous severity process and can be highly skewed.

Frequency, severity, and pure premium

Let N denote the number of claims during an exposure period and let Z₁, Z₂, …, Z_N denote individual claim amounts. Aggregate loss is:

If frequency and severity are independent, with E[N] = λ and E[Z] = μ_Z, then:

The expected loss per exposure unit is commonly called pure premium or expected loss cost. Independence is not essential for defining pure premium, but it is required for the simple product formula above. Coverage limits, deductibles, inflation, claim handling expenses, and policy conditions must be reflected in the definition of severity.

2. Important Probability Distributions

Bernoulli and binomial distributions

A Bernoulli variable Y takes value 1 when an event occurs and 0 otherwise. If P(Y = 1) = p, then E[Y] = p and Var(Y) = p(1 − p). This is appropriate for claim occurrence at policy level, provided the outcome is binary.

A binomial variable counts successes in n independent Bernoulli trials with common probability p. Insurance applications include the number of claims among a fixed number of comparable exposure units, although independence and equal-probability assumptions must be examined.

Poisson distribution

The Poisson distribution models a non-negative integer count. If N ~ Poisson(λ), then:

Here λ > 0 is both the mean and variance. The equal-mean-and-variance property is called equidispersion. Poisson models are useful for claim counts when events are reasonably independent and the rate is stable conditional on rating variables.

Negative binomial distribution

Insurance frequency data commonly exhibit overdispersion, where Var(N) exceeds E[N]. A negative binomial model introduces additional heterogeneity and can be parameterized as:

where μ is the conditional mean and α ≥ 0 is an overdispersion parameter. When α approaches zero, the model approaches Poisson. The negative binomial model is often preferable when policyholders differ in unobserved risk or claims cluster over time.

Gamma distribution

The Gamma distribution is continuous and positive, making it useful for claim severity. Under a shape–scale parameterization with shape k > 0 and scale θ > 0:

Some software uses a shape–rate parameterization; then the rate is 1/θ. Analysts must identify the parameterization before interpreting estimates.

Tweedie distribution

A Tweedie distribution with power parameter p between 1 and 2 has a non-negative mass at zero and a continuous positive component. It is therefore useful for aggregate loss per exposure when many observations have zero loss and positive losses are skewed. Its variance function is:

where μ is the mean, φ > 0 is a dispersion parameter, and 1 < p < 2. The Tweedie model is not a substitute for understanding data structure: zero claims and positive aggregate losses must still be examined for coverage, reporting, and data-quality effects.

3. Estimation and Hypothesis Testing

A parameter is a population quantity, whereas an estimator is calculated from a sample. The maximum likelihood estimator (MLE) chooses parameters that maximize the likelihood of the observed data. For independent observations y₁, …, y_n, the likelihood is:

and analysts usually maximize the log-likelihood:

The log transformation does not change the maximizing parameter because logarithm is monotonic, but it improves numerical stability.

A confidence interval expresses sampling uncertainty under the model and sampling assumptions. A commonly used approximate interval is estimate ± critical value × standard error. It is not a guarantee that the fixed parameter lies within a randomly changing interval in every repeated sample; its interpretation is frequency-based.

A hypothesis test begins with a null hypothesis H₀ and an alternative hypothesis H₁. A p-value is the probability, under H₀, of observing a result at least as extreme as the one obtained. It is not the probability that H₀ is true. Statistical significance does not establish business significance, causality, or fairness.

For nested models, a likelihood-ratio statistic is often used:

Under regularity conditions and a true reduced model, G² is approximately chi-squared with degrees of freedom equal to the difference in parameter counts. Boundary parameters, sparse data, and model misspecification can invalidate this approximation.

4. Generalized Linear Models

A GLM has three components:

1. Random component: the distribution of the response, such as Poisson, Gamma, or binomial.
2. Systematic component: a linear predictor formed from explanatory variables.
3. Link function: a function connecting the conditional mean to the linear predictor.

For observation i, the linear predictor is:

and the link relationship is:

where μᵢ = E[Yᵢ | xᵢ]. The inverse link converts the predictor into a mean: μᵢ = g⁻¹(ηᵢ).

The log link is appropriate for positive means and gives multiplicative effects:

A coefficient β_j has multiplicative interpretation exp(β_j) for a one-unit increase in x_j, holding other variables constant. The percentage change is approximately 100β_j% for small β_j and exactly 100[exp(β_j) − 1]% for a one-unit increase.

Frequency models and offsets

Suppose policy i has exposure e_i, measured in policy-years or another consistent unit. If λ_i is the claim rate per exposure unit, expected claims are e_iλ_i. A Poisson frequency model is:

The term log(e_i) is an offset: its coefficient is fixed at one. Equivalently, the model for the rate is:

Exposure must be positive and measured consistently. An offset should not be used merely because a variable is available; it represents a known proportional scaling relationship.

Severity models

For positive claim amounts, a Gamma GLM with log link is common:

Severity observations should generally be claim-level and should reflect the same payment basis. A deductible or limit creates censoring or transformation and must be handled explicitly.

Logistic regression

For a binary outcome Y_i, logistic regression models the log odds:

where p_i = P(Y_i = 1 | x_i). The odds ratio for a one-unit increase in x_j is exp(β_j). Logistic regression is suitable for claim occurrence, fraud referral, lapse, or litigation indicators, but it models probability rather than claim count or cost.

5. Interactions, Nonlinearity, and Regularization

An interaction allows the effect of one variable to depend on another. For example:

The effect of age on the log mean is β₁ + β₃vehicle, not simply β₁. Main effects should normally be retained when an interaction is included, even if their individual p-values are not significant. Centering continuous variables can make the intercept and main effects more interpretable, although it does not remove the interaction.

Insurance risk often varies nonlinearly with age, territory, or prior claims. Options include grouped variables, polynomials, splines, generalized additive models, and monotonic transformations. Binning improves communication but can create discontinuities and instability near boundaries.

Regularization controls instability in high-dimensional or correlated models. Ridge estimation minimizes:

Lasso minimizes:

where λ ≥ 0 controls penalty strength and the intercept is usually not penalized. Ridge shrinks correlated coefficients together; lasso can set some coefficients exactly to zero. Penalties introduce bias deliberately to reduce variance. λ should be selected using validation or cross-validation, performed within a leakage-controlled modelling process.

6. Diagnostics and Model Evaluation

A model is not validated merely because it converges. Important checks include:

• Distributional fit: compare observed and predicted frequencies, severities, and aggregate losses.
• Calibration: group predictions into bands and compare predicted with observed outcomes.
• Residuals: examine Pearson, deviance, and randomized quantile residuals for structure.
• Overdispersion: compare frequency deviance or Pearson statistics with residual degrees of freedom, while recognizing that such ratios are diagnostic rather than definitive tests.
• Influence: identify observations or groups with high leverage or disproportionate impact.
• Stability: compare coefficients and performance across time, geography, product, and socioeconomic groups.
• Discrimination: for binary models, use measures such as ROC-AUC with care; discrimination does not imply calibration.
• Out-of-sample performance: use a temporal or properly grouped holdout set where future deployment is the objective.

For pricing models, relevant measures include deviance, mean absolute error, root mean squared error, Poisson or Gamma deviance, lift charts, and portfolio-level actual-to-expected ratios. Metrics must match the business decision and the loss function. A model optimized for individual predictive accuracy may not produce an appropriate portfolio premium structure.

A common diagnostic pattern is systematic underprediction for high-risk policies. Possible causes include missing variables, insufficient nonlinear structure, exposure errors, claims development, or a wrong response distribution. Remedies should be based on investigation rather than automatic increases in complexity.

7. Worked Example: Frequency, Severity, and Pure Premium

Consider an illustrative motor portfolio. A fitted frequency model gives:

where λ is expected claims per policy-year and I(rural) equals 1 for a rural policy and 0 otherwise. For an urban policy, λ = exp(−1.50) ≈ 0.2231 claims per year. For a rural policy, λ = exp(−1.30) ≈ 0.2725 claims per year. The rural-to-urban frequency ratio is exp(0.20) ≈ 1.221, meaning approximately 22.1% higher expected frequency, conditional on the model.

Suppose a Gamma severity model estimates an expected covered claim cost of 1,800 currency units. The estimated pure premium for the rural policy is:

This is expected loss cost, not necessarily the final premium. Expenses, commissions, reinsurance, risk margin, taxes, profit provision, deductibles, limits, and capital costs may be added or modelled separately. The calculation also assumes that the frequency and severity definitions are compatible and that the product of their conditional means is an acceptable approximation.

8. Applied Case: Illustrative Home Insurance Portfolio

Background

An insurer writes 600,000 home policies with unequal policy-year exposures. Claims are sparse, claim costs are highly skewed, and recent underwriting changes have altered the mix of business.

Problem

The existing frequency model is Poisson and produces a Pearson dispersion ratio of 1.9. It also underpredicts claims in older properties. A single average claim cost is used for pricing, creating weak differentiation by construction type and protection features.

Analysis

The analytics team first verifies exposure dates, duplicate claims, reopened claims, inflation treatment, and whether claim payments are gross or net of recoveries. It then fits Poisson and negative binomial frequency models using a log exposure offset. A likelihood comparison, out-of-time validation, and calibration plots indicate that the negative binomial model better represents overdispersion. A Gamma log-link model is fitted to positive claim severities, with a separate treatment for large losses and policy limits.

The team adds a construction-age spline and an interaction between construction age and electrical protection. The interaction is retained only after checking adequate observations across the relevant cells and confirming that the relationship is stable in the holdout period.

Approach and solution

The final indicated loss cost is produced by multiplying conditional frequency and severity predictions, with a separate large-loss and catastrophe framework. A governance committee reviews variable definitions, documentation, fairness indicators, and the reasonableness of relativities. Regularization is tested as a sensitivity analysis, but interpretability and stability lead to a moderately sized GLM rather than a substantially more complex black-box model.

Outcome

Out-of-time calibration improves, particularly for older properties. The insurer does not rely solely on a global accuracy score: it monitors actual-to-expected loss ratios by territory, construction age, protection class, and renewal cohort. Rates are implemented with caps and floors where required by regulation and business policy.

Learning points

The case illustrates that model choice is inseparable from data quality, exposure definition, temporal validation, and governance. Overdispersion is evidence for investigation, not an automatic justification for a negative binomial model. Interactions can improve risk differentiation but require sufficient data and careful communication.

9. Limitations, Ethics, and Governance

GLMs are interpretable but not automatically correct. Omitted variables can create bias; correlated predictors complicate attribution; historical claims may reflect past underwriting practices; and reported claims may be influenced by access, incentives, or service processes. A statistically significant coefficient may encode structural disadvantage rather than a legitimate risk mechanism.

Insurance regulation commonly requires actuarial justification, documentation, reproducibility, filing compliance, and controls over unfair discrimination. The precise requirements vary by jurisdiction. Sensitive characteristics and their proxies require particular scrutiny. Fairness cannot be reduced to one universal metric because calibration, equal error rates, and parity criteria can conflict when base rates differ.

Good governance includes a documented purpose, data lineage, variable dictionary, approval thresholds, validation, monitoring, change control, reproducible code, access controls, and an explanation process. Models should be reviewed for drift, performance degradation, emerging peril, and changes in claims handling. Privacy principles require data minimization, lawful use, security, retention control, and transparent communication.

Artificial intelligence can support feature engineering, anomaly detection, and model monitoring, but it does not eliminate actuarial responsibility. A complex model should be adopted only when its incremental value is demonstrated and its risks can be controlled. GLMs often remain valuable as benchmarks, challenger models, and transparent production models.

10. Best-Practice Workflow

A defensible GLM workflow is:

1. Define the decision, response variable, exposure, time period, and unit of analysis.
2. Establish data lineage and reconcile records to financial and claims systems.
3. Explore distributions, missingness, zeros, outliers, truncation, and leakage.
4. Specify a baseline model before adding complexity.
5. Select distribution and link using subject-matter reasoning and diagnostics.
6. Fit with offsets where exposure is a known scaling quantity.
7. Test nonlinearities and interactions with adequate support and business rationale.
8. Validate temporally and across material segments.
9. Assess calibration, stability, fairness, and sensitivity to assumptions.
10. Document limitations, obtain governance approval, implement controls, and monitor after deployment.

Chapter Summary

Probability distributions provide the language for modelling uncertain claims. Frequency is commonly represented by Poisson or negative binomial models, while positive severity is often modelled with Gamma distributions. Tweedie models can represent aggregate losses containing both zeros and positive continuous amounts. Logistic regression models binary outcomes through log odds.

GLMs combine a response distribution, a linear predictor, and a link function. The log link produces multiplicative insurance relativities, and offsets incorporate known exposure. Interactions and nonlinear terms can improve realism, while ridge and lasso regularization manage instability. Estimation, hypothesis testing, residual analysis, calibration, temporal validation, and governance are all necessary for responsible deployment. A model’s value depends not only on predictive performance but also on data quality, interpretability, fairness, regulatory acceptability, and operational monitoring.

Key Terms

Probability distribution: A rule describing the possible values and probabilities of a random variable.

Expected value: The probability-weighted average outcome.

Variance: A measure of dispersion around the mean.

Frequency: Number of claims per defined exposure period.

Severity: Cost of an individual claim.

Pure premium: Expected loss cost per exposure unit.

Exposure: The quantity of risk or time subject to insurance coverage.

Poisson distribution: A count distribution with equal mean and variance under its basic parameterization.

Overdispersion: Variance greater than the mean in count data.

Negative binomial model: A count model allowing overdispersion.

Gamma distribution: A positive continuous distribution often used for claim severity.

Tweedie distribution: A distribution useful for non-negative aggregate loss with zeros and positive continuous values.

Generalized linear model: A model combining a response distribution, linear predictor, and link function.

Link function: A function relating a conditional mean to a linear predictor.

Offset: A predictor with a fixed coefficient, commonly log exposure.

Interaction: A term allowing one variable’s effect to depend on another.

Regularization: Penalization used to reduce estimation variance and instability.

Calibration: Agreement between predicted and observed outcomes.

Deviance: A likelihood-based measure of discrepancy between a fitted model and observations.

Model governance: Processes controlling model development, validation, approval, use, and monitoring.

Review Questions

Short Answer Questions

1. Why is the Poisson distribution often used for claim frequency?
2. What is the purpose of an exposure offset?
3. Interpret exp(β_j) in a log-link GLM.
4. What is overdispersion, and why is it important?
5. Distinguish claim frequency from claim severity.

Long Answer Questions

1. Explain the three components of a generalized linear model and illustrate each using an insurance example.
2. Compare Poisson and negative binomial frequency models, including their assumptions and diagnostic implications.
3. Explain how Gamma, Tweedie, and logistic models serve different insurance modelling purposes.
4. Discuss interactions, nonlinear terms, and regularization in an interpretable pricing model.
5. Describe a complete model governance and validation framework for a production insurance GLM.

Analytical / Application Questions

1. A portfolio has an expected frequency of 0.18 claims per policy-year and an expected severity of 2,400 currency units. Calculate the pure premium and explain which assumptions are required for this multiplication.
2. A log-link model coefficient for a binary rating factor is 0.35. Calculate and interpret the associated multiplicative relativity and percentage change.
3. A fitted Poisson model shows substantial overdispersion and underprediction for high-risk policies. Propose a structured diagnostic and remediation plan, distinguishing data, distributional, specification, and governance issues.

Multiple Choice Questions with Answers

Question 1

Which GLM component specifies the distribution of the response variable?

A. Random component

B. Link function

C. Offset

D. Penalty term

Correct Answer: A. Random component

Question 2

In a Poisson frequency model, what is the usual purpose of log exposure as an offset?

A. To estimate exposure as a response

B. To account for known scaling of expected claims

C. To remove all overdispersion

D. To convert claims into binary outcomes

Correct Answer: B. To account for known scaling of expected claims

Question 3

A log-link coefficient of 0.20 implies a multiplicative relativity of approximately:

A. 0.20

B. 1.02

C. 1.22

D. 20.00

Correct Answer: C. 1.22

Question 4

Which model is specifically designed for binary outcomes through log odds?

A. Gamma regression

B. Logistic regression

C. Negative binomial regression

D. Tweedie regression

Correct Answer: B. Logistic regression

Question 5

If expected frequency is 0.25 and expected severity is 1,600 currency units, the indicated pure premium is:

A. 400 currency units

B. 640 currency units

C. 1,600 currency units

D. 6,400 currency units

Correct Answer: A. 400 currency units

Question 6

What does overdispersion in a count model mean?

A. The mean is zero

B. The variance exceeds the mean

C. All observations are positive

D. The response is normally distributed

Correct Answer: B. The variance exceeds the mean

Question 7

For a Tweedie model with power parameter between 1 and 2, which data structure is especially appropriate?

A. Only binary outcomes

B. Only strictly positive symmetric outcomes

C. Zeros together with positive continuous aggregate losses

D. Fixed numbers of successes

Correct Answer: C. Zeros together with positive continuous aggregate losses

Question 8

Which regularization method can set some coefficients exactly to zero?

A. Ridge

B. Lasso

C. Ordinary least squares

D. Pearson scaling

Correct Answer: B. Lasso

Question 9

In an interaction model, the effect of one predictor:

A. Must be constant across all observations

B. Depends on the value of another predictor

C. Is always statistically insignificant

D. Cannot be interpreted on the link scale

Correct Answer: B. Depends on the value of another predictor

Question 10

A p-value is best interpreted as:

A. The probability that the null hypothesis is true

B. The probability that the alternative hypothesis is true

C. The probability, under the null, of results at least as extreme as observed

D. The percentage improvement in model accuracy

Correct Answer: C. The probability, under the null, of results at least as extreme as observed