8. Regression and Generalized Linear Models

Introduction and Context

Regression is one of the central methods of statistical learning. It provides a formal way to describe how an outcome changes in relation to one or more explanatory variables, to quantify uncertainty, to make predictions, and to support decisions. In data science, regression is used in demand forecasting, credit scoring, clinical research, marketing attribution, policy analysis, reliability engineering, and many other domains.

The word regression refers not merely to drawing a line through data. A regression model expresses assumptions about a data-generating process. It distinguishes systematic variation, which may be explained by predictors, from random variation, measurement error, omitted variables, and other sources of uncertainty. Sound regression analysis therefore requires both mathematical modelling and critical assessment of whether the model is appropriate for the problem.

This chapter develops ordinary linear regression, polynomial and multiple regression, interactions and confounding, diagnostics, multicollinearity, heteroscedasticity, and regularization. It then extends the regression framework to generalized linear models (GLMs), including logistic and count-data models.

Learning Objectives

By the end of this chapter, students should be able to:

• formulate and interpret simple and multiple linear regression models;
• distinguish association from causation and identify confounding and interaction;
• fit and interpret polynomial terms and categorical predictors;
• assess assumptions using residual analysis and diagnostic measures;
• explain multicollinearity, heteroscedasticity, influential observations, and appropriate remedies;
• compare ordinary least squares with ridge and lasso regularization;
• explain the structure of generalized linear models;
• fit and interpret logistic, Poisson, and related models;
• evaluate predictive performance without confusing it with causal validity; and
• apply regression responsibly in professional and organizational settings.

From a Line to a Statistical Model

In simple linear regression, an outcome \(Y\) is related to a predictor \(X\) through

\[

Y_i=\beta_0+\beta_1X_i+\varepsilon_i,

\]

where \(\beta_0\) is the intercept, \(\beta_1\) is the expected change in the outcome for a one-unit increase in the predictor, and \(\varepsilon_i\) represents unexplained variation. The conditional mean is

\[

E(Y_i\mid X_i)=\beta_0+\beta_1X_i.

\]

The usual least-squares estimates choose coefficients that minimize the sum of squared residuals:

\[

\sum_{i=1}^{n}(y_i-\hat y_i)^2.

\]

A residual is \(e_i=y_i-\hat y_i\). The fitted line describes the mean relationship, not necessarily every individual observation. A large residual is not automatically evidence of a bad model; it may reflect natural variability. However, systematic residual patterns indicate that the model has failed to represent some important feature of the data.

Under the classical linear model, useful inferential assumptions include linearity in the coefficients, independent errors, constant error variance, and (for small-sample tests and intervals) approximately normal errors. Normality of the predictors is not required. Nor is normality of the outcome itself required when sample sizes are adequate and the other assumptions are reasonable.

The coefficient of determination, \(R^2\), is

\[

R^2=1-\frac{\text{SSE}}{\text{SST}},

\]

where SSE is the residual sum of squares and SST is the total sum of squares. It measures the proportion of variation in the observed outcome explained by the fitted model relative to an intercept-only model. A high \(R^2\) does not prove causation, guarantee accurate extrapolation, or establish that the model is useful for a particular decision.

Multiple Regression

With several predictors, the model becomes

\[

Y_i=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}+\varepsilon_i.

\]

Here, \(\beta_j\) is the expected change in \(Y\) associated with a one-unit increase in \(X_j\), holding the other included predictors constant. This conditional interpretation is valuable but can be misleading if the predictors are strongly related or if the variables held constant are consequences of the exposure being studied.

Categorical predictors are represented using indicator variables. If a variable has \(k\) categories, a model generally includes \(k-1\) indicators and selects one reference category. The coefficient for an indicator compares its category with the reference category, conditional on the remaining variables.

A coefficient estimate is commonly accompanied by a standard error, confidence interval, and p-value. A confidence interval communicates a range of parameter values compatible with the data and model assumptions. Statistical significance does not measure practical importance; a small effect may be precisely estimated in a large dataset, while an important effect may be uncertain in a small one.

Shape, Interactions, and Confounding

Polynomial Regression

A curved mean relationship can be modelled by including powers of a predictor:

\[

E(Y\mid X)=\beta_0+\beta_1X+\beta_2X^2.

\]

Although the model is nonlinear in \(X\), it remains linear in its coefficients and can be estimated by least squares. The marginal effect is

\[

\frac{dE(Y\mid X)}{dX}=\beta_1+2\beta_2X.

\]

Thus, \(\beta_1\) alone is not the overall effect of \(X\). Centering \(X\) before creating powers can improve interpretability and reduce numerical correlation between terms. High-degree polynomials often behave erratically near the boundaries and should not be used simply because they improve in-sample fit. Splines or other flexible methods may be preferable.

Interactions

An interaction means that the effect of one predictor depends on the level of another. For two variables,

\[

E(Y\mid X,Z)=\beta_0+\beta_1X+\beta_2Z+\beta_3XZ.

\]

The effect of \(X\) is \(\beta_1+\beta_3Z\), not simply \(\beta_1\). If \(Z=0\) is not meaningful, centering or otherwise choosing a meaningful reference value is important. Main effects should normally remain in a model containing their interaction, even when an individual main-effect test is not statistically significant.

Confounding and Causal Caution

A confounder is a variable associated with both a predictor and an outcome that creates or distorts their observed association. For example, an analysis may find that training hours are associated with productivity, but employee experience may influence both training participation and productivity. Adjusting for experience may change the training coefficient substantially.

Confounding is not solved mechanically by adding every available variable. Adjustment can introduce bias when conditioning on a collider, a variable influenced by two other variables. Causal modelling should therefore begin with subject-matter knowledge, a clear estimand, and preferably a causal diagram or design strategy. Regression alone generally estimates conditional associations, not causal effects.

Diagnosing a Fitted Model

Model diagnosis is an iterative process rather than a final ceremonial step. Important tools include:

• Residual-versus-fitted plots: reveal curvature, unequal spread, and unusual observations.
• Residuals versus predictors: reveal omitted nonlinear structure or interactions.
• Normal Q–Q plots: assess whether residuals have strongly non-normal tails.
• Scale–location plots: help identify changing residual variance.
• Leverage: identifies observations with unusual predictor values.
• Cook’s distance and related measures: assess whether a case has substantial influence on fitted coefficients.

An outlier is unusual in the outcome; a high-leverage point is unusual in predictor space; an influential observation materially changes the fitted model. These are different concepts. Deleting a point merely because it is inconvenient is bad practice. The analyst should verify data quality, investigate the substantive context, conduct sensitivity analyses, and report how conclusions change.

Heteroscedasticity

Heteroscedasticity occurs when the conditional variance of errors changes with predictors or fitted values. It does not necessarily bias ordinary least-squares coefficient estimates when the conditional mean is correctly specified, but conventional standard errors, confidence intervals, and tests may be unreliable. Remedies include transforming the outcome, using heteroscedasticity-robust standard errors, modelling the variance explicitly, or applying weighted least squares when a defensible variance model is available.

Multicollinearity

Multicollinearity occurs when predictors contain overlapping information. It often produces unstable coefficient estimates, large standard errors, unexpected coefficient signs, and sensitivity to small changes in the data. It does not necessarily reduce predictive accuracy. Variance inflation factors can be used as diagnostic indicators, but no universal cutoff replaces substantive judgement.

Possible responses include removing redundant variables, combining measurements, collecting more informative data, centering terms in polynomial or interaction models, or using ridge regression. Dropping a variable solely because of a high diagnostic value may damage interpretation when the variable is a theoretically essential confounder.

Validation and Prediction

A model should be evaluated on data not used for fitting. A train-test split or cross-validation estimates out-of-sample performance. For continuous outcomes, common measures include mean squared error, root mean squared error, and mean absolute error. Cross-validation must be designed around the data-generating process: time series should preserve temporal order, and observations from the same person or organization should not be split across training and test sets when that would cause leakage.

Prediction intervals are wider than confidence intervals for the mean because they include both uncertainty in the estimated mean and individual outcome variability. Extrapolation beyond the observed predictor range is especially risky, even when a model fits well within the sample.

Regularization and the Bias–Variance Trade-off

When there are many predictors, correlated features, or a relatively small sample, unconstrained least squares can overfit. Regularization shrinks coefficients toward zero by adding a penalty to the fitting objective.

Ridge regression minimizes

\[

\sum_i(y_i-\hat y_i)^2+\lambda\sum_j\beta_j^2,

\]

while lasso minimizes

\[

\sum_i(y_i-\hat y_i)^2+\lambda\sum_j|\beta_j|.

\]

The intercept is usually not penalized, and predictors should generally be standardized before penalization. Ridge tends to retain correlated predictors with smaller coefficients. Lasso can set some coefficients exactly to zero and therefore perform variable selection, although its selection can be unstable among highly correlated features. Elastic net combines both penalties.

The tuning parameter \(\lambda\) is selected using validation or cross-validation. Regularization introduces bias in exchange for reduced variance and often improved prediction. Penalized coefficients should not be interpreted like ordinary least-squares estimates without considering the penalty and the modelling objective.

Generalized Linear Models

Ordinary linear regression is unsuitable when the outcome is binary, a count, or otherwise restricted. A generalized linear model has three components:

1. a random component specifying the outcome distribution;

2. a systematic component, \(\eta=X\beta\), formed from predictors; and

3. a link function connecting the conditional mean \(\mu=E(Y\mid X)\) to \(\eta\):

\[

g(\mu)=X\beta.

\]

GLMs are commonly fitted by maximum likelihood, often using iteratively reweighted least squares. Their coefficients are interpreted on the link scale unless transformed into a more intuitive scale.

Logistic Regression

For a binary outcome, logistic regression models the log-odds:

\[

\log\left(\frac{p}{1-p}\right)=\beta_0+\beta_1X_1+\cdots+\beta_pX_p.

\]

Exponentiating a coefficient gives an odds ratio. Holding other variables constant, a one-unit increase in \(X_j\) multiplies the odds by \(e^{\beta_j}\). An odds ratio is not generally a risk ratio, especially when the outcome is common. Predicted probabilities are often more accessible to decision-makers than log-odds or odds ratios.

Performance should be assessed using discrimination and calibration. The ROC curve and area under the curve summarize ranking ability, while calibration examines agreement between predicted probabilities and observed frequencies. A classification threshold should reflect the relative costs of false positives and false negatives rather than being selected automatically at 0.5.

Count Models

Poisson regression models a non-negative count using

\[

\log(\mu_i)=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}.

\]

Exponentiating \(\beta_j\) gives an incidence-rate ratio for a one-unit increase in a predictor. If observations have different exposure times, an offset such as \(\log(t_i)\) can be included with coefficient fixed at one. The Poisson model assumes that conditional variance equals the conditional mean. Overdispersion, in which variance exceeds the mean, may require quasi-Poisson or negative binomial modelling. Excess zeros may motivate hurdle or zero-inflated approaches, but these should be justified by the data-generating process rather than selected automatically.

Other GLMs include gamma regression for positive, right-skewed outcomes and inverse Gaussian models for certain duration or cost data. The choice should reflect the outcome’s support, variance structure, scientific meaning, and predictive purpose.

An Illustrative Case: Hospital Readmission Risk

Background and problem

Suppose a hospital wants to estimate the probability that a patient will be readmitted within 30 days of discharge. The data include age, previous admissions, length of stay, comorbidity score, discharge destination, and whether a follow-up appointment was scheduled. The objective is operational: identify patients who may benefit from additional support, not to label patients as personally responsible for readmission.

Analysis and approach

A logistic regression is appropriate because the outcome is binary. The analyst first defines the prediction time and ensures that only information available before discharge is used. Variables recorded after discharge would create leakage. A model is fitted on training data and assessed on temporally later data. Calibration is examined because a hospital needs trustworthy probabilities for allocating scarce follow-up resources.

Suppose previous admissions has a positive coefficient. Its exponentiated coefficient represents the multiplicative change in odds associated with one additional previous admission, conditional on the other included predictors. The hospital should not interpret this as proof that previous admissions causes readmission. It may partly reflect unmeasured illness severity or social conditions.

The analyst checks for nonlinearity in age, interactions between discharge destination and follow-up care, missing-data patterns, and subgroup calibration. If the model performs less well for a demographic group, the organization investigates data quality, access differences, and clinical processes rather than simply lowering or raising a threshold for that group.

Outcome and learning points

The model becomes one component of a triage process. Staff review high-risk cases, and an evaluation compares outcomes under the new process with outcomes under usual care. The case demonstrates that a statistically adequate model is not automatically a beneficial intervention. Valid deployment requires temporal validation, governance, fairness assessment, human oversight, and monitoring for changes in patient populations and clinical practice.

Professional and Ethical Practice

Regression analysis can influence lending, employment, insurance, healthcare, public services, and resource allocation. Analysts should document the target population, data provenance, missing-data treatment, feature construction, exclusions, model version, validation design, and intended use. Sensitive attributes may be relevant for auditing fairness even when they are excluded from prediction.

Privacy protection includes data minimization, access controls, secure storage, and careful reporting of small groups. A model should not be deployed merely because it has a high accuracy or \(R^2\). Stakeholders need to understand uncertainty, limitations, operating thresholds, and consequences of errors. Model cards, reproducible pipelines, version control, drift monitoring, and periodic review support responsible operation.

Common mistakes include interpreting correlation as causation, reporting only p-values, ignoring nonlinear effects, fitting many transformations after inspecting the test set, treating missingness as harmless, deleting influential cases without investigation, and using accuracy for highly imbalanced classification. Automated machine-learning systems do not remove the need for design, diagnostics, domain expertise, or accountability.

Emerging Directions

Contemporary regression practice increasingly combines interpretable statistical models with flexible machine-learning methods. Generalized additive models, tree ensembles, boosting, Bayesian regression, and causal machine learning can capture complex structure, but their use should be matched to the objective. Explainability methods describe model behaviour but do not automatically establish causal explanations. Probabilistic forecasting, conformal prediction, distribution shift detection, and privacy-preserving analytics are also developing areas. The enduring principle is that predictive performance, interpretability, fairness, and operational value must be evaluated together.

Best-Practice Workflow

A defensible regression project generally follows this sequence:

1. Define the outcome, prediction or causal estimand, population, and decision context.

2. Inspect data quality, missingness, measurement units, sampling, and leakage risks.

3. Use subject knowledge to specify plausible predictors, transformations, and interactions.

4. Fit a transparent baseline before considering more complex models.

5. Diagnose residuals, dependence, variance, collinearity, influence, and calibration.

6. Use cross-validation or appropriate holdout data for performance assessment.

7. Compare alternatives using both statistical evidence and practical consequences.

8. Quantify uncertainty and conduct sensitivity and subgroup analyses.

9. Document the model and deploy with monitoring, governance, and a plan for revision.

Chapter Summary

Regression models quantify relationships between outcomes and predictors and support both explanation and prediction. Linear regression estimates conditional means through least squares, while multiple regression permits adjustment for several variables. Polynomial terms and interactions represent curvature and effect modification, but coefficients must be interpreted in context. Confounding requires causal reasoning rather than indiscriminate adjustment.

Residual diagnostics reveal nonlinearity, heteroscedasticity, non-normality, leverage, and influential observations. Multicollinearity can destabilize individual coefficients, and regularization can improve prediction by trading a little bias for lower variance. Generalized linear models extend regression to binary, count, and other non-normal outcomes through a distribution and link function. Logistic coefficients describe changes in log-odds, while Poisson coefficients describe changes in log rates. Valid application requires appropriate validation, ethical governance, transparent communication, and continuous monitoring.

Key Terms

Regression: A statistical method for modelling an outcome as a function of explanatory variables.

Conditional mean: The expected value of an outcome given specified predictor values.

Residual: The difference between an observed outcome and its fitted value.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Multiple regression: Regression involving two or more predictors.

Polynomial regression: A linear-in-parameters model containing powers of a predictor.

Interaction: A relationship in which the effect of one predictor depends on another.

Confounding: Distortion of an association by a variable related to both exposure and outcome.

Multicollinearity: Strong dependence or overlap among predictors.

Heteroscedasticity: Non-constant conditional variance of model errors.

Leverage: The degree to which an observation has an unusual predictor pattern.

Influential observation: A case whose removal substantially changes model results.

Regularization: Penalization used to control coefficient size and reduce overfitting.

Ridge regression: Regression using an L2, or squared-coefficient, penalty.

Lasso regression: Regression using an L1, or absolute-coefficient, penalty.

Generalized linear model: A model combining an outcome distribution, linear predictor, and link function.

Logistic regression: A GLM for binary outcomes using the logit link.

Odds ratio: The ratio of odds associated with a specified change in a predictor.

Poisson regression: A log-link GLM commonly used for count outcomes.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Review Questions

Short Answer Questions

1. What does a multiple-regression coefficient mean when other predictors are held constant?

2. Distinguish an outlier, a high-leverage observation, and an influential observation.

3. Why can heteroscedasticity make ordinary standard errors unreliable?

4. What is the purpose of a link function in a generalized linear model?

5. Why should predictors usually be standardized before ridge or lasso regression?

Long Answer Questions

1. Explain the assumptions of ordinary linear regression and describe diagnostic methods for assessing them.

2. Discuss interactions, polynomial terms, and the interpretation difficulties they create.

3. Compare ridge regression and lasso regression in terms of objective, coefficient behaviour, and use cases.

4. Explain logistic regression, including odds, odds ratios, probability prediction, discrimination, and calibration.

5. Describe how confounding differs from multicollinearity and why each requires a different response.

Analytical / Application Questions

1. A sales model has a strong residual funnel shape. What does this suggest, and what analyses or remedies would you consider?

2. A count model for emergency visits shows variance far greater than its mean and many zero observations. Which modelling issues arise, and what alternatives could be investigated?

3. A hospital prediction model performs well overall but is poorly calibrated for one demographic group. Design a responsible investigation and improvement plan.

22. Multiple Choice Questions with Answers

Question 1

In a multiple linear regression, the coefficient for X1 is interpreted as:

A. The unconditional change in Y for a one-unit change in X1

B. The expected change in Y for a one-unit change in X1, holding included predictors constant

C. The percentage of variation explained by X1 alone

D. The probability that X1 causes Y

Correct Answer: B. The expected change in Y for a one-unit change in X1, holding included predictors constant

Question 2

In a model containing X, Z, and XZ, the marginal effect of X is:

A. β1

B. β1 + β2

C. β1 + β3Z

D. β0 + β1X

Correct Answer: C. β1 + β3Z

Question 3

Which pattern most directly suggests heteroscedasticity?

A. Residuals forming a widening funnel as fitted values increase

B. Predictors having different measurement units

C. A categorical variable having three levels

D. A high value of the intercept

Correct Answer: A. Residuals forming a widening funnel as fitted values increase

Question 4

What is a primary effect of multicollinearity?

A. It always biases predictions upward

B. It makes the outcome binary

C. It can produce unstable coefficients and inflated standard errors

D. It guarantees heteroscedasticity

Correct Answer: C. It can produce unstable coefficients and inflated standard errors

Question 5

Which statement about lasso regression is correct?

A. It uses only a squared-coefficient penalty

B. It can shrink some coefficients exactly to zero

C. It cannot be used with standardized predictors

D. It always outperforms ridge regression

Correct Answer: B. It can shrink some coefficients exactly to zero

Question 6

Logistic regression models which quantity linearly in the predictors?

A. The raw probability without transformation

B. The odds ratio itself

C. The log-odds of the outcome

D. The residual variance

Correct Answer: C. The log-odds of the outcome

Question 7

If a logistic-regression coefficient is β, exp(β) represents:

A. A change in probability of exactly β

B. An odds multiplier for a one-unit predictor increase, conditional on other variables

C. The model’s accuracy

D. The residual standard deviation

Correct Answer: B. An odds multiplier for a one-unit predictor increase, conditional on other variables

Question 8

A Poisson model with an exposure-time offset is designed primarily to model:

A. A binary outcome

B. A continuous outcome with unrestricted negative values

C. A count or event rate accounting for differing exposure times

D. The correlation between two predictors

Correct Answer: C. A count or event rate accounting for differing exposure times

Question 9

Why is a random train-test split inappropriate in some time-series regression problems?

A. It prevents coefficient estimation

B. It can allow future information to influence training and produce leakage

C. It always increases heteroscedasticity

D. It removes the intercept

Correct Answer: B. It can allow future information to influence training and produce leakage

Question 10

Which statement best reflects responsible interpretation of a regression association?

A. A statistically significant coefficient proves causation

B. A high R² proves the model will extrapolate accurately

C. Causal interpretation requires design and subject-matter reasoning beyond regression alone

D. A small p-value establishes practical importance

Correct Answer: C. Causal interpretation requires design and subject-matter reasoning beyond regression alone