Regression Diagnostics and Generalized Linear Models

Introduction and Context

Regression analysis is useful only when its assumptions, data-generating process, and decision context are sufficiently credible. A high R² does not establish that a model is correctly specified, causally interpretable, or suitable for prediction. Regression diagnostics examine whether residuals, predictors, and influential observations reveal violations that could distort estimates, standard errors, forecasts, or managerial conclusions.

Many business outcomes are not continuous and normally distributed. A customer may churn or remain active; a bank may observe the number of defaults; a retailer may record daily transactions. Generalized linear models (GLMs) extend ordinary least squares by allowing non-normal response distributions and by linking the expected response to a linear predictor through a link function.

This chapter integrates diagnostic reasoning with logistic, Poisson, negative binomial, and related GLMs. The emphasis is not on mechanically obtaining coefficients, but on deciding whether a model is statistically defensible and operationally useful.

Learning Objectives

By the end of this chapter, you should be able to:

1. Diagnose nonlinearity, heteroscedasticity, multicollinearity, autocorrelation, outliers, and influential observations.
2. Distinguish a modeling assumption from a diagnostic symptom and select suitable remedial actions.
3. Interpret leverage, Cook’s distance, studentized residuals, VIF, and heteroscedasticity-robust inference.
4. Explain the structure and estimation logic of generalized linear models.
5. Fit and interpret logistic and count-data models, including offsets and overdispersion.
6. Evaluate predictive performance, calibration, discrimination, and model specification.
7. Apply diagnostics responsibly in a business decision and communicate uncertainty and limitations.

1. The Diagnostic Logic of Regression

For a linear model with n observations and p predictors,

where y_i is the response, x_{ij} is predictor j for observation i, β_j is a population coefficient, and ε_i is the unexplained error. Ordinary least squares (OLS) estimates coefficients by minimizing

The classical assumptions are commonly expressed as linearity in parameters, independent errors, zero conditional mean E(ε_i | X) = 0, constant conditional variance Var(ε_i | X) = σ², and—mainly for exact small-sample tests—normally distributed errors. Normality is not required for OLS coefficients to be unbiased, but it matters for exact t and F inference in small samples. Independence is a property of the data-collection process, not something that a residual plot alone can guarantee.

A residual is e_i = y_i − ŷ_i. Residual analysis should be conducted against fitted values, important predictors, time, groups, and relevant business segments. A useful diagnostic workflow is:

1. Clarify the response type, sampling mechanism, and decision objective.
2. Inspect data quality, missingness, coding, units, and duplicate records.
3. Examine residual and predictor patterns.
4. Quantify possible problems with targeted statistics.
5. Refit plausible alternative specifications.
6. Compare out-of-sample performance and calibration.
7. Report sensitivity to assumptions and influential records.

2. Diagnosing Functional Form and Nonlinearity

A residual-versus-fitted plot should show an approximately patternless cloud centered around zero. Curvature, systematic waves, or runs indicate that the conditional mean may not be linear. A residual-versus-predictor plot is especially useful when a particular variable is suspected.

Possible remedies include transforming a predictor, adding a polynomial term, using restricted cubic splines, or fitting a generalized additive model. A quadratic specification is

Here β_1 is not the constant marginal effect of X. The marginal effect is

The units are those of Y per unit of X. Centering X before squaring, X_c = X − mean(X), can reduce correlation between X_c and X_c² and make β_0 interpretable at the sample mean, but it does not change fitted values.

A transformation must have a substantive rationale. For example, log revenue may be appropriate when proportional rather than absolute changes are meaningful. A log-log model,

interprets β_1 approximately as elasticity: a 1% increase in X is associated with approximately a β_1% change in Y, subject to the model assumptions. Predictions on the original scale require care because E(exp(Z)) generally differs from exp(E(Z)); retransformation bias may require a smearing correction or simulation.

3. Heteroscedasticity

Heteroscedasticity exists when the conditional error variance changes with predictors or fitted values:

where σ_i² is not constant. A funnel-shaped residual plot is a common symptom. Breusch–Pagan and White-type tests can provide evidence, but with large samples they may flag trivial departures and with small samples they may lack power.

Under exogeneity, OLS coefficients can remain unbiased despite heteroscedasticity, but conventional standard errors, confidence intervals, and p-values may be incorrect. Remedies include:

• heteroscedasticity-consistent (HC) standard errors;
• transforming the response or predictors when variance structure supports it;
• weighted least squares when a defensible variance model is available;
• modeling the response with an appropriate GLM.

Weighted least squares minimizes ∑w_i(y_i − ŷ_i)². If Var(ε_i|X_i) is proportional to 1/w_i, the weights can improve efficiency. Arbitrary weights, however, can create bias or misleading precision. Robust standard errors address inference; they do not fix a misspecified conditional mean or produce a new fitted relationship.

4. Multicollinearity

Multicollinearity means that predictors contain overlapping information. Perfect collinearity makes coefficients unidentified; near-collinearity produces unstable estimates, large standard errors, and sensitivity to small data changes. It does not necessarily reduce predictive accuracy, and a low individual p-value is not a reliable test for its absence.

For predictor j, the variance inflation factor is

where R_j² is obtained by regressing predictor j on all the other predictors. A VIF of 5 means the estimated variance of β̂_j is five times the variance it would have under orthogonality, under the relevant design comparison. Rules such as 5 or 10 are heuristics, not universal laws.

Examine coefficient stability, condition indices, domain redundancy, and the correlation structure. Remedies include removing or combining redundant variables, using a theoretically justified index, collecting more informative data, or applying ridge regression or another penalized method. Standardizing variables changes coefficient units but does not eliminate the underlying information overlap. Principal components can reduce collinearity but may weaken interpretability.

5. Autocorrelation and Dependence

For ordered observations, errors may be correlated across time or space. Positive first-order autocorrelation can be represented as

where t indexes time, ρ is the lag-one correlation parameter, and u_t is an innovation. Dependence can make standard errors too small and can cause forecasts to appear more precise than they are.

Plot residuals over time, inspect the autocorrelation function, and consider the data-generating process. The Durbin–Watson statistic is approximately 2(1 − r_1), where r_1 is the sample lag-one residual autocorrelation; values below 2 suggest positive first-order correlation, but the statistic has limitations with lagged dependent variables and irregular sampling.

Appropriate responses include time-series regression with autoregressive errors, Newey–West standard errors under suitable conditions, cluster-robust inference, fixed effects for repeated entities, or explicit dynamic models. Randomly splitting time-ordered data into training and test sets causes leakage; use rolling-origin or forward-chaining validation.

6. Outliers, Leverage, and Influence

An outlier has an unusual response conditional on its predictors. A high-leverage observation has an unusual predictor pattern. An influential observation materially changes estimates when included or removed. These are related but distinct concepts.

The hat matrix is H = X(XᵀX)⁻¹Xᵀ, and leverage for observation i is h_ii, the i-th diagonal element. Average leverage is (p + 1)/n when an intercept and p predictors are included. A common screening rule is h_ii > 2(p + 1)/n, but it is not a definitive cutoff.

A studentized residual scales a residual by an estimated residual standard deviation. Cook’s distance summarizes the change in the fitted coefficient vector when observation i is deleted:

in a common OLS formulation, where MSE is the residual mean square and p + 1 is the number of estimated coefficients. Different software conventions may use internally studentized residuals or equivalent matrix forms, so definitions should be checked.

Never delete a record simply because it is inconvenient. Investigate data-entry errors, unusual but valid customers, fraud, structural breaks, and changes in measurement. Report sensitivity analyses, such as estimates with and without a verified influential case, while preserving the original data and the audit trail.

7. Generalized Linear Models

A GLM consists of three components:

1. A random component: Y_i follows an exponential-family distribution, such as normal, binomial, Poisson, or gamma.
2. A systematic component: η_i = β_0 + β_1x_{i1} + ⋯ + β_px_{ip}.
3. A link function: g(μ_i) = η_i, where μ_i = E(Y_i|X_i).

Thus,

The inverse link transforms the linear predictor into the response mean. Coefficients are commonly estimated by maximum likelihood, often using iteratively reweighted least squares. GLM coefficients are generally not changes in Y per unit X; interpretation occurs on the link scale or through marginal effects on the response scale.

The variance is related to the mean through Var(Y_i|X_i) = φV(μ_i), where V(μ_i) is the variance function and φ is a dispersion parameter. For Poisson models φ is conventionally 1; for binomial models it is also conventionally fixed at 1. A quasi-likelihood or alternative count model may estimate dispersion.

Logistic Regression

For a binary response Y ∈ {0,1}, let π_i = P(Y_i = 1|X_i). Logistic regression uses the logit link:

The odds are π/(1 − π). Holding other variables constant, exp(β_j) is the odds ratio for a one-unit increase in x_j. It is not generally a risk ratio or a probability increase. The probability is

Probability differences depend on the baseline covariates. Therefore, report predicted probabilities or average marginal effects in addition to odds ratios. A classification threshold, such as 0.5, is a decision rule and should reflect costs of false positives and false negatives rather than convention alone.

Model assessment should include a confusion matrix at a stated threshold, sensitivity, specificity, precision, recall, ROC-AUC, precision–recall analysis for rare events, and calibration. ROC-AUC measures ranking ability; it does not establish probability calibration or economic value. Complete or quasi-complete separation can cause very large estimates; penalized likelihood, Bayesian regularization, or redesigned data collection can help.

Poisson Regression and Exposure

For a count response Y_i, Poisson regression assumes

The Poisson assumption implies E(Y_i|X_i) = Var(Y_i|X_i) = μ_i. exp(β_j) is an incidence-rate ratio for a one-unit increase in x_j, holding other predictors constant.

When observations have different exposure times or population at risk, use an offset. If t_i is exposure, then

where ln(t_i) has coefficient fixed at 1. Equivalently, the model describes a rate μ_i/t_i. An offset must represent exposure, not an ordinary predictor whose coefficient is estimated.

Overdispersion and Negative Binomial Models

If the observed conditional variance exceeds the conditional mean, the Poisson model is overdispersed. Causes include omitted heterogeneity, dependence, excess zeros, or an incorrect mean structure. A dispersion statistic, residual diagnostics, and substantive knowledge should be considered together.

A negative binomial model can represent extra-Poisson variation, commonly through

where α ≥ 0 is an overdispersion parameter under this parameterization. Parameterizations differ across software, so the variance formula must be verified. Robust standard errors may protect inference under some misspecifications, but they do not reproduce the full predictive distribution of a negative binomial model.

Other Useful GLMs

Gamma regression with a log link is often useful for positive, right-skewed costs or claim amounts. Binomial models can analyze successes out of a known number of trials. Quasi-binomial and quasi-Poisson models permit a dispersion adjustment but do not provide the same likelihood-based information criteria as fully specified likelihood models. Zero-inflated and hurdle models are specialized approaches requiring a substantive reason for distinct structural-zero and sampling processes; they should not be selected merely because zeros are frequent.

8. GLM Diagnostics and Model Comparison

GLM diagnostics include deviance residuals, Pearson residuals, leverage, influence measures, binned residual plots, and checks for nonlinearity in the linear predictor. Residuals should be examined against fitted values and important covariates. For binomial models, calibration plots and calibration intercept/slope are particularly important. For counts, compare predicted and observed frequency distributions, zero proportions, and tail behavior.

Deviance is based on the comparison between the fitted model and a saturated model. For nested models, a likelihood-ratio comparison may be useful when regularity conditions hold. AIC is

where k is the number of estimated parameters and L̂ is the maximized likelihood. Lower AIC is preferred among comparable likelihood-based models fitted to the same response and data; AIC is not a universal test of truth. BIC replaces 2k with k ln(n), imposing a stronger complexity penalty. Cross-validation is generally more directly connected to predictive performance.

Use train/test or nested cross-validation when tuning models. For imbalanced outcomes, stratification may preserve class proportions, but temporal or grouped dependence must take priority over random splitting. Avoid selecting variables using the full dataset before validation, because this leaks information from the test data.

9. Illustrative Case: Customer-Service Escalations

Background and Problem

A fictional telecommunications firm wants to predict whether a support interaction will escalate within 30 days. The dataset contains 18,000 interactions, monthly subscription charge, tenure, prior complaints, service failures, contact channel, and customer region. The business wants to prioritize proactive retention offers while controlling unnecessary contact.

Analysis

An initial logistic model showed that the coefficient for monthly charge changed substantially when a highly correlated annual-value variable was added. VIF values were 8.4 and 7.9 for the two monetary variables, indicating overlapping information. The team retained monthly charge because it was more current and interpretable. A residual-style binned plot suggested nonlinearity in tenure, so restricted splines were compared with a linear tenure term. Calibration improved with the spline, although the ROC-AUC changed little.

Several cases had high leverage. Investigation found that two were valid enterprise accounts using a different service contract; they were not deleted, but contract type was added and a sensitivity analysis was reported. Because customer interactions were clustered within customers, standard errors were clustered by customer and validation was performed by customer rather than by interaction.

Approach and Outcome

The final model reported odds ratios, predicted probabilities, calibration by decile, and a decision curve based on the estimated cost of an unnecessary offer versus a missed escalation. A threshold below 0.5 was selected because missed escalations were substantially more costly. The firm did not treat the model as an automated eligibility rule: agents reviewed high-impact cases, and protected attributes were examined for disparate error rates.

Learning Points

The case illustrates that predictive quality depends on functional form, dependence, collinearity, influence, validation design, and decision costs. AUC alone would not answer the business question. Diagnostic findings changed both the specification and the governance plan.

10. Professional, Ethical, and Governance Considerations

Diagnostics can reveal data-quality problems and unfair measurement practices, but statistical correction is not a substitute for institutional accountability. For logistic decisions involving credit, employment, insurance, or access to essential services, assess legal requirements, explainability, disparate impact, documentation, human oversight, and appeal mechanisms. Monitor performance after deployment because population drift, policy changes, and feedback loops can invalidate historical relationships.

Maintain a model card or equivalent record containing purpose, population, variables, exclusions, training period, validation design, assumptions, known limitations, thresholds, and monitoring metrics. Protect personal data through minimization, access controls, retention limits, and appropriate anonymization. Do not infer that a predictive association is causal without a valid causal design.

Best-Practice Checklist

• Match the model family to the response and exposure structure.
• Diagnose mean structure before interpreting p-values.
• Plot residuals by fitted value, predictor, time, and important groups.
• Use robust or clustered inference only when its assumptions and target are appropriate.
• Investigate, rather than automatically remove, unusual observations.
• Interpret transformed and link-scale coefficients correctly.
• Validate using splits that respect time, groups, and deployment conditions.
• Report calibration, uncertainty, sensitivity, and decision consequences.
• Document data provenance, model versions, and post-deployment monitoring.

Chapter Summary

Regression diagnostics test whether a model’s functional form, variance structure, independence, predictor information, and influential observations support reliable inference and prediction. Nonlinearity may require transformations, polynomials, splines, or additive terms. Heteroscedasticity affects conventional uncertainty estimates; multicollinearity destabilizes individual coefficients; autocorrelation requires dependence-aware modeling or inference; and outliers, leverage, and influence must be investigated in context.

Generalized linear models combine a response distribution, a linear predictor, and a link function. Logistic regression models binary probabilities through log odds; Poisson regression models counts through a log mean and can use exposure offsets; negative binomial models address extra-Poisson variation. Interpretation should distinguish link-scale effects, odds or rate ratios, and response-scale predictions. Diagnostics, calibration, appropriate validation, ethical review, and governance are essential components of professional analytics.

Key Terms

Diagnostic: A procedure used to assess model assumptions, fit, or sensitivity.

Residual: The observed response minus its fitted value.

Heteroscedasticity: Non-constant conditional error variance.

Multicollinearity: Overlap among predictors that destabilizes coefficient estimates.

Variance inflation factor (VIF): A measure of variance inflation caused by a predictor’s linear relation with other predictors.

Autocorrelation: Correlation among errors ordered in time or another sequence.

Leverage: The extent to which an observation has an unusual predictor configuration.

Influential observation: An observation whose removal materially changes fitted results.

Cook’s distance: A measure summarizing an observation’s influence on fitted coefficients.

Link function: A function connecting the conditional mean to the linear predictor.

Exponential family: A broad distribution class underlying standard GLMs.

Logit: The logarithm of odds, ln[π/(1−π)].

Odds ratio: The multiplicative change in odds associated with a predictor change.

Incidence-rate ratio: The multiplicative change in an event rate for a count-model predictor change.

Offset: A predictor with a fixed coefficient, usually representing exposure on a log scale.

Overdispersion: Conditional variance greater than that implied by a Poisson or binomial model.

Deviance: A likelihood-based measure comparing a fitted model with a saturated model.

Calibration: Agreement between predicted probabilities and observed frequencies.

Separation: A logistic-regression condition in which predictors nearly or completely distinguish outcome classes.

Influence analysis: Systematic assessment of how unusual records affect conclusions.

Review Questions

Short Answer Questions

1. Why can heteroscedasticity make conventional OLS p-values unreliable even when OLS coefficients remain unbiased?
2. Distinguish an outlier, a high-leverage observation, and an influential observation.
3. What does a VIF measure, and why is a cutoff not a universal decision rule?
4. Interpret exp(β_j) in a logistic regression.
5. Why is an exposure offset used in Poisson regression?

Long Answer Questions

1. Develop a diagnostic workflow for a multiple regression model used to forecast customer spending. Explain the plots, statistics, and remedial actions you would use.
2. Compare OLS, logistic, Poisson, and negative binomial models in terms of response type, mean structure, variance assumptions, and interpretation.
3. Explain how autocorrelation affects inference and describe validation and modeling strategies for time-ordered business data.
4. Discuss why calibration and discrimination provide different information in binary classification.
5. Explain how influential observations should be investigated and reported in a regulated analytics project.

Analytical / Application Questions

1. A Poisson model for claim counts has a mean of 2.0 but an observed conditional variance near 7.5. Propose a diagnostic sequence and justify whether a negative binomial, quasi-Poisson, or alternative model may be appropriate.
2. A logistic model gives an odds ratio of 1.40 for a one-unit increase in complaints. Explain why this does not imply a 40-percentage-point increase in probability, and specify what additional quantities should be reported.
3. A model’s AUC is high, but its predicted probabilities are systematically too large. Diagnose the problem and recommend evaluation, recalibration, and governance actions.

Multiple Choice Questions with Answers

Question 1

Which plot most directly reveals whether the conditional mean relationship is misspecified in an OLS model?

A. Residuals versus fitted values

B. Histogram of the response alone

C. Bar chart of a categorical predictor

D. Pie chart of missing values

Correct Answer: A. Residuals versus fitted values

Question 2

If R_j² = 0.80 when predictor j is regressed on the other predictors, what is VIF_j?

A. 0.20

B. 1.25

C. 5.00

D. 80.00

Correct Answer: C. 5.00

Question 3

In logistic regression, exp(β_j) represents which quantity, holding other predictors constant?

A. A percentage-point change in probability

B. An odds ratio for a one-unit predictor increase

C. A guaranteed causal risk ratio

D. The residual variance

Correct Answer: B. An odds ratio for a one-unit predictor increase

Question 4

Which link is standard for Poisson regression?

A. Identity link

B. Probit link

C. Log link

D. Complementary log-log link only

Correct Answer: C. Log link

Question 5

What is the principal purpose of an offset ln(t_i) in a Poisson model?

A. To estimate a coefficient for exposure

B. To account for differing exposure while fixing its coefficient at one

C. To remove all overdispersion

D. To convert a count into a binary response

Correct Answer: B. To account for differing exposure while fixing its coefficient at one

Question 6

Under the conventional Poisson model, which equality is assumed conditionally on predictors?

A. Mean equals variance

B. Mean equals zero

C. Variance equals one for every observation

D. Mean equals median exactly in every sample

Correct Answer: A. Mean equals variance

Question 7

Which statement about heteroscedasticity-consistent standard errors is correct?

A. They always correct a nonlinear conditional mean

B. They can improve inference when variance is non-constant under suitable conditions

C. They delete influential observations

D. They force residuals to be normally distributed

Correct Answer: B. They can improve inference when variance is non-constant under suitable conditions

Question 8

An observation with unusual predictor values but an ordinary response conditional on those predictors is primarily characterized by high:

A. Dispersion

B. Leverage

C. Calibration

D. Autocorrelation

Correct Answer: B. Leverage

Question 9

Which validation strategy is most appropriate for forecasting future monthly demand?

A. Randomly shuffle all months before splitting

B. Use future observations to impute past training data

C. Use forward-chaining or rolling-origin validation

D. Evaluate only in-sample R²

Correct Answer: C. Use forward-chaining or rolling-origin validation

Question 10

For a negative binomial model parameterized with Var(Y|X) = μ + αμ², what does α represent?

A. The intercept

B. An extra-dispersion parameter

C. The exposure time

D. The logistic odds ratio

Correct Answer: B. An extra-dispersion parameter