5. Regression and Generalized Linear Models

Introduction and Context

Regression analysis is one of the principal ways in which researchers explain variation, evaluate relationships, predict outcomes, and support decisions. In business research, regression may be used to estimate how advertising expenditure relates to sales, whether employee engagement predicts turnover, how credit characteristics relate to default risk, or whether a managerial intervention changes organizational performance. Its value, however, depends less on producing a coefficient table than on formulating an appropriate model, identifying credible assumptions, diagnosing weaknesses, and interpreting results in relation to the research design.

The classical linear model is often introduced first, but many important outcomes are not continuous and unbounded. A customer either churns or remains; a firm records a number of patent applications; a survey respondent selects an ordered satisfaction category. Generalized linear models (GLMs) extend regression to such outcomes by combining a probability model for the response with a systematic component and a link function. This chapter develops the logic of linear, logistic, count, ordinal, and related models, while addressing interactions, nonlinearities, confounding, endogeneity, heteroskedasticity, multicollinearity, specification, and regularization.

Regression is not automatically causal. A sophisticated model fitted to observational data may still reflect selection bias, omitted variables, reverse causality, or measurement error. Doctoral-level statistical practice therefore requires simultaneous attention to estimation, identification, interpretation, and governance.

Learning Objectives

By the end of this chapter, readers should be able to:

1. Explain the common structure of linear and generalized linear models.

2. Select regression models appropriate to continuous, binary, count, and ordinal outcomes.

3. Interpret coefficients, marginal effects, odds ratios, incidence-rate ratios, and predicted probabilities.

4. Model interactions and nonlinear relationships without producing misleading conclusions.

5. Diagnose heteroskedasticity, multicollinearity, influential observations, misspecification, and dependence.

6. Distinguish confounding from mediation and recognize threats from endogeneity.

7. Compare maximum likelihood, robust inference, generalized estimating equations, mixed models, and regularization.

8. Evaluate regression results ethically and communicate uncertainty responsibly.

A Common Architecture for Regression Models

Let $Y_i$ denote an outcome and $X_i$ a vector of predictors. A regression model specifies a conditional distribution for $Y_i$ given $X_i$ and relates its expected value to predictors. In ordinary least squares (OLS),

$$Y_i=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}+\varepsilon_i,$$

where $\beta_j$ represents the expected change in the outcome associated with a one-unit increase in $X_j$, holding other included variables constant. The phrase “holding constant” is a conditional statement, not evidence that a causal effect has been established.

A GLM has three linked elements. First is the random component, which identifies a distribution such as Gaussian, binomial, Poisson, or negative binomial. Second is the systematic component, or linear predictor, $\eta_i=X_i\beta$. Third is the link function $g$, which connects the conditional mean $\mu_i=E(Y_i|X_i)$ to the predictors:

$$g(\mu_i)=\eta_i.$$

The link ensures that fitted values respect the outcome’s scale. For example, probabilities remain between zero and one, while Poisson means remain positive. Parameters are commonly estimated by maximum likelihood, although quasi-likelihood and Bayesian methods are also widely used.

Linear Regression and Its Assumptions

OLS is appropriate when the outcome is approximately continuous and the conditional mean is reasonably linear in the predictors. The key assumptions are more precise than the informal claim that “the data must be normal.” The conditional mean should be correctly specified; observations should be independent or dependence should be explicitly modeled; regressors should not be perfectly collinear; and, for conventional standard errors, errors should have constant conditional variance. Normality of errors is mainly relevant to exact small-sample inference, not to unbiasedness itself.

Important diagnostics include residual-versus-fitted plots, partial-residual plots, leverage, Cook’s distance, and tests or graphical assessments of heteroskedasticity. A statistically significant Breusch–Pagan test may indicate nonconstant variance, but substantive inspection is essential. Heteroskedasticity-consistent (robust) standard errors can improve inference about coefficients, whereas they do not repair omitted variables, nonlinearities, or biased point estimates. If observations are clustered—for example, employees within firms—cluster-robust standard errors or multilevel models may be appropriate.

Binary Outcomes: Logistic Regression

For a binary outcome $Y\in\{0,1\}$, logistic regression models the log odds of success:

$$\log\left(\frac{p_i}{1-p_i}\right)=X_i\beta,$$

where $p_i=P(Y_i=1|X_i)$. Exponentiating $\beta_j$ gives an odds ratio. An odds ratio of 1.40 means that a one-unit increase in $X_j$ multiplies the odds by 1.40, conditional on the other variables. It does not mean that probability increases by 40 percentage points. Probability changes depend on the baseline probability and values of other covariates; therefore, predicted probabilities and average marginal effects are often more interpretable.

Probit regression uses the cumulative normal distribution rather than the logistic distribution. In many applications the substantive conclusions are similar, but coefficients are on different scales. Model assessment can include calibration plots, Brier scores, discrimination measures such as the area under the ROC curve, and out-of-sample performance. Classification accuracy alone is often misleading when outcomes are imbalanced or when the costs of errors differ.

Counts, Rates, and Exposure

Count outcomes are nonnegative integers. Poisson regression uses

$$\log(\mu_i)=X_i\beta,$$

and implies that the conditional variance equals the conditional mean. Exponentiated coefficients are incidence-rate ratios (IRRs). An IRR of 1.10 indicates a 10 percent multiplicative change in the expected count for a one-unit predictor increase, holding other variables constant.

When observations have different exposure times or population at risk, an offset is required. For example, workplace incidents should be modeled relative to hours worked, not merely as raw counts. The offset enters the model with a coefficient fixed at one, commonly as $\log(\text{exposure})$.

Overdispersion occurs when the observed conditional variance exceeds the Poisson variance. It may arise from unobserved heterogeneity, clustering, or excess zeros. Negative binomial regression introduces an additional dispersion parameter. Zero-inflated and hurdle models can be useful when the data-generating process plausibly contains distinct structural-zero and count-generating mechanisms; they should not be selected solely because they improve a fit statistic.

Ordinal and Other GLMs

Ordinal outcomes possess an order but not necessarily equal distances between categories. The proportional-odds model estimates cumulative logits:

$$\log\left[\frac{P(Y\leq k)}{P(Y>k)}\right]=\alpha_k-X\beta,$$

where each threshold has its own intercept but the predictor effects are constrained to be equal across thresholds. This proportional-odds assumption should be assessed. If it is implausible, partial proportional-odds, multinomial, adjacent-category, or continuation-ratio models may be preferable. Treating a five-point rating as continuous can sometimes be a useful approximation, but it should be justified rather than automatic.

Multinomial logistic regression applies when categories are nominal and have no defensible order. Gamma or inverse-Gaussian GLMs can model positive, right-skewed continuous outcomes such as claim sizes. Survival models address time-to-event outcomes and censoring; although not always classified as GLMs, they extend the same principle of linking predictors to an outcome distribution or hazard.

Interactions, Nonlinearity, and Interpretation

An interaction exists when the association between one predictor and the outcome depends on another predictor. In a linear model,

$$Y=\beta_0+\beta_1X+\beta_2Z+\beta_3XZ+\varepsilon,$$

$\beta_3$ is the change in the slope of $X$ for a one-unit change in $Z$. When an interaction is included, main effects are conditional: $\beta_1$ is the effect of $X$ when $Z=0$. Centering variables can make this reference point more meaningful, although centering does not remove collinearity or alter fitted values.

In logistic and other nonlinear models, the coefficient on an interaction term is not generally the interaction effect on the probability scale. Researchers should calculate predicted probabilities or marginal effects at substantively relevant combinations of predictors and display them graphically. Confidence intervals should accompany such displays.

Nonlinearity can be represented with polynomial terms, splines, transformations, or generalized additive models. A quadratic term should not be interpreted independently of its linear term. Splines provide flexible curves while controlling smoothness, but excessive flexibility can overfit. The choice between a linear and nonlinear specification should be guided by theory, graphical evidence, cross-validation, and transparent sensitivity analysis rather than by repeated significance testing.

Confounding, Mediation, and Endogeneity

A confounder is a variable associated with both the predictor and outcome that, if omitted, distorts the association of interest. Adjustment should be based on a causal diagram or substantive theory, not simply on whether a variable changes a coefficient by an arbitrary percentage. A mediator lies on the pathway from exposure to outcome. Controlling for it changes the estimand from a total effect toward a direct effect and may introduce bias if the mediator is affected by unmeasured variables.

Endogeneity occurs when a predictor is correlated with the error term. Major sources include omitted variables, simultaneity, reverse causality, and measurement error. Consequences may include biased and inconsistent estimates. Remedies depend on the research design: randomized assignment, fixed effects, instrumental variables, regression discontinuity, difference-in-differences, lagged designs, or control-function approaches. An instrumental variable must be relevant to the endogenous predictor and satisfy an exclusion restriction, a condition that cannot be established by a statistical test alone. Weak instruments can produce severe finite-sample problems.

Regression adjustment should therefore be presented as conditional association unless the identification assumptions are explicit and defensible. Longitudinal data, natural experiments, and sensitivity analyses can strengthen causal claims but do not eliminate the need for design-based reasoning.

Multicollinearity, Specification, and Model Comparison

Multicollinearity occurs when predictors contain overlapping information. It does not necessarily bias OLS coefficients, but it increases standard errors, makes estimates unstable, and complicates interpretation. Variance inflation factors are diagnostic aids, not universal decision rules. Remedies include combining conceptually related measures, collecting more informative data, using dimension reduction, or applying regularization. Dropping a theoretically important variable merely to reduce a VIF can create omitted-variable bias.

Specification errors include omitted nonlinear terms, inappropriate link functions, incorrect interactions, omitted exposure, and unsuitable outcome distributions. Residual patterns, calibration, influence diagnostics, theory, and out-of-sample validation should be considered together. Information criteria such as AIC compare likelihood and complexity; BIC imposes a stronger complexity penalty. Neither criterion proves causal validity.

Influential cases deserve investigation rather than automatic deletion. A high-leverage observation may be valid and substantively important. Analysts should report sensitivity analyses with and without influential observations, document data corrections, and distinguish data error from genuine but unusual behavior.

Regularization and Predictive Modeling

When predictors are numerous or highly correlated, regularization can improve prediction. Ridge regression minimizes squared error plus a penalty proportional to the squared coefficients. Lasso uses an absolute-value penalty and can set some coefficients exactly to zero. Elastic net combines both penalties and is often useful when predictors occur in correlated groups. Penalized logistic and count models apply analogous principles.

The penalty parameter must be selected using cross-validation or a pre-specified validation strategy. Preprocessing, feature selection, and imputation must occur within training folds to prevent information leakage. Regularization improves predictive stability but does not solve confounding or make selected variables causal determinants. Inference after selection requires care, and predictive importance should not be confused with theoretical importance.

A Research Workflow for Regression Analysis

A defensible workflow begins with a research question and an estimand: prediction, conditional association, total effect, direct effect, or another clearly defined quantity. The analyst then examines measurement, missingness, outcome distribution, sampling, dependence, and the plausible data-generating process. A primary model should be specified before extensive outcome-driven experimentation where feasible.

The analysis proceeds through estimation, diagnostics, sensitivity analysis, validation, and communication. Missing data require attention to the missingness mechanism; complete-case analysis may be biased and inefficient. Multiple imputation can be appropriate under defensible assumptions, with the imputation model including variables related to missingness and the outcome. Results should include effect estimates, uncertainty intervals, sample size, model specification, coding decisions, and relevant diagnostics. Reproducible scripts, versioned data, and a transparent analysis record are essential doctoral research practices.

Illustrative Case: Predicting Customer Churn

Background and problem

A fictional telecommunications company wishes to understand and predict whether monthly customers will cancel service within the next quarter. The database contains churn status, contract type, tenure, monthly charges, service complaints, usage, and customer-support contacts. Management initially proposes linear regression because it is familiar.

Analysis

Churn is binary, so logistic regression is more appropriate than OLS for estimating conditional probabilities. The research team defines the prediction window, excludes information recorded after that window, and checks whether customers have unequal observation periods. Contract type and tenure are theoretically important, while complaints may partly reflect dissatisfaction preceding cancellation. If the purpose is prediction, complaints may be useful even though they would be problematic as a control in a total-effect causal analysis.

The team models tenure flexibly using a spline and examines an interaction between contract type and monthly charges. Results are reported as predicted probabilities for representative customers rather than only log-odds coefficients. Calibration is assessed on held-out data, and the classification threshold is chosen by considering the relative costs of contacting a likely churner and contacting a loyal customer.

Diagnostics reveal that customer-support contacts are strongly correlated with complaints. Rather than automatically deleting one variable, the analysts compare models, explain the overlap, and assess stability. They also test whether performance differs across customer segments. A model with excellent overall discrimination but poor calibration for low-income customers would not be acceptable for automated retention decisions.

Outcome and learning points

The company uses the model as a decision-support tool, not an automatic cancellation or pricing mechanism. Retention offers are evaluated through a randomized experiment, because a predictive association does not establish that an offer will prevent churn. The case demonstrates the importance of matching the outcome to the model, separating prediction from causal explanation, interpreting probabilities, validating out of sample, and governing algorithmic use.

Challenges, Ethics, and Governance

Regression can reproduce historical discrimination when protected characteristics or their proxies encode unequal treatment. Removing a sensitive variable does not necessarily remove bias, because geographic, behavioral, or socioeconomic variables may function as proxies. Fairness assessment should reflect the decision context and include error rates, calibration, access, and potential harms. Privacy, purpose limitation, data minimization, security, and retention should guide data use.

Researchers should avoid p-hacking through undisclosed model searching, selective reporting, arbitrary transformations, and repeated subgroup analyses. A statistically significant coefficient may be practically trivial, while a wide interval may indicate insufficient information rather than evidence of no effect. Automated tools and AI-assisted coding can accelerate analysis but require human review, reproducible execution, secure handling of confidential data, and verification of generated code. Model cards, data documentation, audit logs, and predefined monitoring rules support responsible deployment.

Current Developments and Best Practice

Contemporary regression practice increasingly combines interpretable statistical models with machine-learning methods, causal machine learning, Bayesian hierarchical models, and explainable prediction. Heterogeneous treatment-effect methods seek to identify which units benefit from an intervention, but they require careful validation and protection against multiple-testing artifacts. Distribution shift and concept drift mean that deployed models should be monitored rather than treated as permanent.

Best practice is to:

• define the outcome, estimand, time horizon, and population clearly;
• select a model consistent with the outcome and data-generating process;
• use theory and causal diagrams to guide adjustment;
• report effect scales that decision-makers can understand;
• inspect residuals, calibration, dependence, influence, missingness, and sensitivity;
• validate predictions using data not used for fitting;
• distinguish statistical significance, practical importance, and causal evidence;
• document code, assumptions, exclusions, transformations, and model changes; and
• assess fairness, privacy, security, and consequences before deployment.

Chapter Summary

Regression models connect outcomes to explanatory variables, but their adequacy depends on the outcome scale, dependence structure, assumptions, research design, and purpose of analysis. OLS models continuous outcomes; logistic models binary outcomes; Poisson and negative binomial models address counts; ordinal models preserve ordered categories; and other GLMs extend the framework through suitable distributions and link functions. Coefficients must be interpreted on the correct scale, often through marginal effects, predicted probabilities, or rate ratios.

Interactions and nonlinearities require conditional interpretation and visual presentation. Confounding, mediation, and endogeneity are distinct causal issues that cannot be solved by mechanical variable selection. Heteroskedasticity, multicollinearity, influence, misspecification, missing data, and dependence require diagnostics and sensitivity analysis. Regularization supports prediction but does not establish causality. Ethical deployment further requires fairness assessment, privacy protection, reproducibility, transparent reporting, and ongoing monitoring.

Key Terms

Generalized linear model: A model combining a response distribution, linear predictor, and link function.

Link function: A function connecting the conditional mean of an outcome to the linear predictor.

Ordinary least squares: Estimation that minimizes the sum of squared residuals.

Logistic regression: A model for binary outcomes using the logit link.

Odds ratio: The multiplicative change in odds associated with a predictor change.

Marginal effect: The change in an expected outcome or probability associated with a predictor change.

Poisson regression: A count model whose canonical mean-variance assumption equates variance and mean.

Incidence-rate ratio: The exponentiated coefficient in a log-link count model.

Offset: A known exposure term included with coefficient fixed at one.

Overdispersion: Variability exceeding that assumed by a specified count model.

Ordinal regression: Regression for ordered categorical outcomes.

Interaction: A relationship in which the effect of one predictor depends on another.

Confounding: Distortion of an association caused by a common cause of predictor and outcome.

Mediation: Transmission of an effect through an intermediate variable.

Endogeneity: Correlation between a predictor and the model error term.

Heteroskedasticity: Nonconstant conditional variance of errors.

Multicollinearity: Strong linear association among predictors.

Regularization: Penalization used to constrain estimates and improve generalization.

Calibration: Agreement between predicted and observed outcome frequencies.

Influential observation: An observation that substantially affects fitted estimates or conclusions.

Review Questions

Short Answer Questions

1. What are the three components of a generalized linear model?

2. Why is an odds ratio not equivalent to a percentage-point change in probability?

3. What is the purpose of an offset in count regression?

4. Distinguish confounding from mediation.

5. Why can robust standard errors not correct omitted-variable bias?

Long Answer Questions

1. Compare OLS, logistic, Poisson, negative binomial, and ordinal regression in terms of outcome type, link, interpretation, and key assumptions.

2. Explain how interactions and nonlinearities should be specified and interpreted in linear and nonlinear models.

3. Discuss endogeneity, its principal sources, and design-based strategies for addressing it.

4. Develop a complete diagnostic and reporting plan for a regression study involving clustered organizational data.

5. Explain how regularization improves prediction and why it should not be interpreted as a causal solution.

Analytical / Application Questions

1. A firm models customer complaints as Poisson counts but observes variance far greater than the mean and many zeros. Propose a diagnostic and modeling strategy, including exposure and alternative models.

2. A logistic model has a strong ROC-AUC but systematically overpredicts risk for one customer segment. What does this reveal, and what should the organization do before deployment?

3. A study estimates the effect of training on productivity while controlling for employee motivation measured after training. Analyze whether this is appropriate for a total-effect question and propose a better strategy.

22. Multiple Choice Questions with Answers

Question 1

Which feature distinguishes a generalized linear model from ordinary linear regression?

A. It always uses maximum likelihood and never least squares

B. It combines a response distribution, linear predictor, and link function

C. It requires normally distributed predictors

D. It can only model binary outcomes

Correct Answer: B. It combines a response distribution, linear predictor, and link function

Question 2

In logistic regression, an odds ratio of 1.40 for a predictor means that, holding other variables constant, a one-unit increase

A. raises probability by 40 percentage points

B. raises the odds by 40 percent

C. raises the log odds by 40 units

D. reduces the odds by 60 percent

Correct Answer: B. raises the odds by 40 percent

Question 3

What is the principal purpose of an offset in a Poisson model?

A. To remove multicollinearity

B. To model unequal exposure or observation time

C. To convert counts into binary outcomes

D. To test proportional odds

Correct Answer: B. To model unequal exposure or observation time

Question 4

A count outcome has conditional variance substantially larger than its mean. Which issue is most directly indicated?

A. Underfitting caused by centering

B. Overdispersion relative to the Poisson model

C. Perfect separation

D. Homoskedasticity

Correct Answer: B. Overdispersion relative to the Poisson model

Question 5

In a linear model containing X, Z, and XZ, the coefficient of X represents

A. The unconditional effect of X for every value of Z

B. The effect of X when Z equals zero

C. The effect of Z when X equals zero

D. The correlation between X and Z

Correct Answer: B. The effect of X when Z equals zero

Question 6

Which statement about robust standard errors is correct?

A. They eliminate omitted-variable bias

B. They correct incorrect outcome distributions

C. They can improve inference under heteroskedasticity

D. They guarantee causal interpretation

Correct Answer: C. They can improve inference under heteroskedasticity

Question 7

Why is an instrumental variable strategy vulnerable when the instrument is weak?

A. It necessarily creates overdispersion

B. It can produce unstable and biased finite-sample estimates

C. It prevents estimation of standard errors

D. It converts an ordinal outcome to continuous form

Correct Answer: B. It can produce unstable and biased finite-sample estimates

Question 8

What does regularization primarily do in predictive modeling?

A. Guarantees that selected variables are causal

B. Constrains coefficients to improve stability and generalization

C. Removes all measurement error

D. Establishes the exclusion restriction

Correct Answer: B. Constrains coefficients to improve stability and generalization

Question 9

For an ordinal outcome, the proportional-odds assumption means that

A. All outcome categories have equal frequencies

B. Predictor effects are constrained to be equal across cumulative thresholds

C. The outcome can be treated as nominal only

D. The residual variance must equal the mean

Correct Answer: B. Predictor effects are constrained to be equal across cumulative thresholds

Question 10

A model has excellent discrimination but poor calibration for a protected group. The most appropriate conclusion is that

A. The model is automatically fair because its AUC is high

B. The model should be deployed without modification

C. Predicted risks do not reliably correspond to observed risks for that group

D. Calibration is irrelevant in decision support

Correct Answer: C. Predicted risks do not reliably correspond to observed risks for that group