Regression and Regularized Predictive Models

Introduction and context

Regression is one of the central methods of business analytics. It provides a formal way to relate an outcome of interest to one or more explanatory variables, estimate the magnitude and direction of those relationships, generate predictions, and support decisions under uncertainty. In supervised learning, the outcome variable is observed during model development. The model learns a mapping from predictors to that outcome and is then evaluated on data not used for fitting.

Regression is especially valuable in business because it supports both prediction and explanation. A retailer may forecast weekly demand, a bank may estimate credit losses, a hospital may predict length of stay, and a human-resources department may estimate employee turnover risk. These uses are related but not identical. A model that predicts accurately need not identify causal effects, and a coefficient that is statistically significant need not be operationally important.

This chapter develops linear regression, polynomial regression, generalized linear models, and regularized methods—ridge, lasso, and elastic net. It emphasizes model assumptions, diagnostics, interpretation, validation, managerial use, and the risks of treating an estimated association as a causal relationship.

Learning objectives

By the end of this chapter, you should be able to:

1. Formulate and interpret linear and polynomial regression models.
2. Explain ordinary least squares and its assumptions.
3. Diagnose nonlinearity, heteroscedasticity, dependence, outliers, and influential observations.
4. Interpret coefficients, uncertainty intervals, and predictive performance.
5. Explain why generalized linear models are needed for non-Gaussian outcomes.
6. Apply and compare ridge, lasso, and elastic-net regularization.
7. Design a leakage-resistant modeling workflow and communicate results responsibly.

1. The regression framework

Let yᵢ denote the observed outcome for case i and let xᵢ₁, …, xᵢₚ denote p predictors. A linear regression model is

Here β₀ is the intercept, βⱼ is the population slope associated with predictor j, and εᵢ is the unexplained error. The model is called linear because it is linear in its unknown coefficients β, not necessarily because every predictor must enter as a simple straight-line relationship.

The conditional mean is

where E(Yᵢ | Xᵢ) is the expected outcome conditional on the predictors. The distinction between the conditional mean and an individual outcome is important: even a correctly specified model generally does not predict every case exactly.

Ordinary least squares

Given n observations, ordinary least squares (OLS) chooses coefficients that minimize the residual sum of squares:

where ŷᵢ is the fitted value and eᵢ = yᵢ − ŷᵢ is the residual. In matrix notation, with design matrix X, outcome vector y, and coefficient vector β, the objective is

If X has full column rank, the OLS estimator is

The inverse exists only when the columns of X are linearly independent. Perfect multicollinearity—for example, including both a variable and a duplicate copy—prevents this calculation. Near multicollinearity may leave the calculation possible but make estimates unstable.

OLS estimates the conditional mean correctly only under suitable specification and sampling conditions. The familiar claim that OLS is “best” requires care. Under the Gauss–Markov conditions, OLS is the best linear unbiased estimator, meaning it has the smallest variance among linear unbiased estimators. Normality is not required for unbiasedness, although it supports exact small-sample t and F inference when combined with other assumptions.

Decomposing variation

With an intercept, total variation in y can be decomposed as

where SST = ∑(yᵢ − ȳ)² is total sum of squares, SSR = ∑(ŷᵢ − ȳ)² is regression sum of squares, and SSE = ∑(yᵢ − ŷᵢ)² is residual sum of squares. The coefficient of determination is

R² is the proportion of in-sample variation explained relative to an intercept-only model. It is not the probability that the model is correct, not a causal measure, and not necessarily a good measure of future predictive performance. Adding predictors cannot decrease training R², even if they add no useful information. Adjusted R² partially accounts for model size:

This expression assumes an intercept and p predictors. For predictive work, cross-validated error is usually more informative than either R² measure.

2. Interpreting coefficients and categorical variables

In a multiple regression, βⱼ is the expected change in y associated with a one-unit increase in xⱼ, holding the other included predictors constant. “Holding constant” describes a conditional association; it does not automatically establish a causal effect.

Units matter. If monthly sales are measured in dollars and advertising expenditure in thousands of dollars, a coefficient of 2.4 means that an additional thousand dollars of advertising is associated with 2.4 additional dollars of monthly sales, conditional on other predictors. This may signal a unit mistake if the business context suggests a different scale.

For a binary variable, coded 0 and 1, its coefficient compares the expected outcomes between the group coded 1 and the reference group coded 0, conditional on other variables. For a categorical variable with k categories, use k − 1 indicator variables when an intercept is included. The omitted category is the reference category.

Interactions allow the effect of one variable to depend on another:

The conditional slope of x is β₁ + β₃z. Therefore β₁ is the effect of x when z = 0, not necessarily its average effect. Centering z can make the reference value more meaningful, but it does not change fitted values when the model is reparameterized consistently.

Standardized coefficients are sometimes used to compare predictors measured on different scales. A standardized slope describes the expected standard-deviation change in y for a one-standard-deviation change in x, conditional on other predictors. Standardization aids comparison but does not make a relationship causal or automatically improve an unregularized OLS fit.

3. Polynomial regression and transformations

A straight-line effect may be too restrictive. Polynomial regression adds powers of a predictor:

Although x² and higher powers make the curve nonlinear in x, the model remains linear in β and can be estimated by OLS. The marginal slope for a quadratic model is

Thus β₁ alone is not the overall effect of x. A negative β₂ can represent diminishing returns or an inverted-U relationship, but the turning point x* = −β₁/(2β₂) should be interpreted only if it lies within the observed and substantively plausible range of x.

High-degree raw polynomials can be unstable, especially near the boundaries of the data. Centering and scaling x, or using orthogonal polynomials or splines, can improve numerical behavior. A flexible curve should not be extrapolated confidently beyond the range used for fitting.

Logarithmic transformations can improve linearity or reduce skewness. In a model log(y) = β₀ + β₁x + ε, β₁ is approximately the proportional change in y for a one-unit increase in x when the effect is small; the exact multiplicative change in the conditional median under standard log-error assumptions is related to e^β₁ − 1. Back-transformation requires care because, generally, E(e^ε) ≠ 1. A naive exponentiation of predicted log outcomes may therefore underestimate the mean on the original scale.

4. Assumptions and diagnosis

Assumptions should be treated as diagnostic questions, not ceremonial requirements.

Linearity in parameters and adequate functional form. The conditional mean should be represented adequately by the chosen predictors, transformations, interactions, or nonlinear terms. Residual-versus-fitted plots and partial-residual plots can reveal systematic curvature.

Independent observations or correctly modeled dependence. Repeated observations from the same customer, store, employee, or time period are not automatically independent. Ignoring clustering can understate uncertainty. Time series may require lag variables, time-series models, or blocked validation.

Exogeneity or zero conditional mean. For unbiased estimation of conditional associations, one commonly requires E(ε|X)=0. Omitted confounders, reverse causality, and measurement error can violate this condition. Regression adjustment alone does not solve an identification problem.

Constant error variance. Homoscedasticity means Var(ε|X)=σ², with the same variance across predictor values. Heteroscedasticity does not necessarily bias OLS coefficients, but conventional standard errors may be wrong. Remedies include heteroscedasticity-robust standard errors, transformation, weighted least squares, or a model with an appropriate variance structure.

Distribution of errors. Normal residuals are mainly relevant to exact finite-sample inference, not to the basic computation of OLS predictions. Q–Q plots and residual histograms are useful, but small departures from normality may matter less than dependence, nonlinearity, or severe outliers.

No perfect multicollinearity. Correlated predictors inflate coefficient uncertainty and make individual coefficients sensitive to small data changes. Variance inflation factors can help diagnose the issue, but there is no universal cutoff that substitutes for subject-matter judgment.

No undue influence. Leverage measures unusual predictor combinations; a high-leverage observation is not automatically harmful. Influence combines leverage with residual size. Cook’s distance is one screening measure, but observations should be investigated rather than deleted mechanically.

A sound diagnostic sequence includes: inspect data quality; plot outcomes and predictors; fit a baseline model; examine residual, Q–Q, leverage, and influence plots; test plausible nonlinearities and interactions; assess dependence and heteroscedasticity; and validate predictive performance on held-out data.

5. Generalized linear models

Linear regression is unsuitable when the outcome distribution and mean–variance relationship are fundamentally different from a continuous Gaussian response. Generalized linear models (GLMs) consist of three components:

1. A random component specifying a distribution from the exponential family, such as Bernoulli, binomial, Poisson, or Gamma.
2. A systematic component, the linear predictor η = Xβ.
3. A link function g connecting the conditional mean μ = E(Y|X) to η:

Logistic regression

For a binary outcome Y ∈ {0,1}, logistic regression uses the logit link:

where p = P(Y = 1|X). Exponentiating βⱼ gives an odds ratio for a one-unit increase in xⱼ, conditional on other predictors:

An odds ratio of 1.30 means 30% higher odds, not necessarily 30% higher probability. The probability change depends on the baseline probability and all other predictor values.

Poisson regression

For count outcomes, a common model is

where μ = E(Y|X). The multiplicative effect for a one-unit increase in xⱼ is e^{βⱼ}. Poisson regression assumes that conditional variance equals the conditional mean. Overdispersion, zero inflation, and excess dependence may require negative-binomial, hurdle, zero-inflated, or other models.

GLMs are estimated by maximum likelihood rather than ordinary least squares. Coefficients are interpreted on the link scale, while managerial decisions often require predicted probabilities, expected counts, or marginal effects on the original scale.

6. Regularization: controlling complexity

When predictors are numerous, correlated, or measured with noise, an unregularized model may fit training data too closely. Regularization adds a penalty to the loss function. Predictors should generally be standardized before penalization so that the penalty treats their coefficients comparably; the intercept is usually not penalized.

Ridge regression

Ridge minimizes squared error plus an L2 penalty:

λ ≥ 0 controls the penalty. At λ = 0, ridge equals OLS. As λ increases, coefficients shrink toward zero but generally do not become exactly zero. Ridge is useful when many predictors have small or shared effects and when multicollinearity makes OLS unstable.

For centered data, ridge has a useful linear-algebra interpretation. If X = UDVᵀ is a singular-value decomposition, the ridge estimator applies shrinkage factors d_k²/(d_k² + λ) to directions in predictor space. Directions with weak information are shrunk more strongly. This explains why ridge can improve out-of-sample prediction at the cost of bias.

Lasso regression

Lasso uses an L1 penalty:

The geometry of the absolute-value constraint permits exact zeros, producing a sparse model. Lasso can support variable selection, but with strongly correlated predictors it may select one variable arbitrarily or distribute selection unpredictably. A zero coefficient means the fitted penalized model excludes that variable; it is not proof that the variable has no causal or scientific relevance.

Elastic net

Elastic net combines both penalties:

where 0 ≤ α ≤ 1. α = 1 gives lasso; α = 0 gives a ridge-like penalty under this scaling. Elastic net is often effective when predictors are correlated and the analyst wants both shrinkage and some sparsity.

The penalty parameter—and α for elastic net—should be selected using cross-validation within the training data. A one-standard-error rule may choose a simpler model whose validation error is close to the minimum. Crucially, preprocessing, feature selection, and hyperparameter tuning must be performed inside resampling folds. Otherwise information from validation data leaks into training.

7. Evaluation and model selection

Separate three goals: explanation, prediction, and decision optimization. For continuous outcomes, common metrics include mean squared error:

and root mean squared error, RMSE = √MSE. Mean absolute error is

RMSE penalizes large errors more heavily; MAE is easier to interpret in outcome units and is more resistant to extreme errors. For binary outcomes, discrimination metrics such as ROC-AUC should be accompanied by calibration measures, confusion-matrix metrics at a specified threshold, and business costs. Accuracy can be misleading under class imbalance.

Random k-fold cross-validation partitions training data into k folds, fits on k−1 folds, and evaluates on the remaining fold, repeating until each fold has served as validation. The final test set must remain untouched until the modeling process is complete. For time-ordered data, use rolling or forward-chaining validation rather than random shuffling.

Prediction intervals quantify uncertainty for a future individual and are wider than confidence intervals for the mean response under the same conditions. In managerial reporting, uncertainty should be translated into ranges, downside exposure, and decision thresholds rather than hidden behind a single point prediction.

8. Worked example: regularized demand prediction

Consider a retailer predicting weekly sales from price, digital advertising, store traffic, promotion indicators, competitor price, and seasonal variables. The outcome is sales in units. The data are split chronologically: the first 80% of weeks form training data and the final 20% form a test set. Numeric predictors are standardized using means and standard deviations calculated from the training period only.

An OLS model produces training RMSE of 410 units and test RMSE of 620 units. Ridge cross-validation selects λ = 8 and produces test RMSE of 535 units. Lasso selects λ = 5 and produces test RMSE of 548 units, setting two weak seasonal coefficients to zero. Elastic net produces test RMSE of 529 units and retains groups of correlated advertising variables together.

The appropriate conclusion is not that elastic net has discovered the true drivers of demand. It has performed best on this held-out period under the stated metric and data-generating conditions. The improvement may reflect reduced variance and better handling of correlated predictors. Managers should examine stability across time windows, promotion policy changes, stockouts, and competitor reactions before deployment.

Suppose the fitted logistic model for a separate promotion-response outcome has β_promotion = 0.40. The conditional odds ratio is e^0.40 ≈ 1.49, so the promotion is associated with approximately 49% higher odds, holding other predictors constant. If baseline probability is 0.20, baseline odds are 0.20/0.80 = 0.25; new odds are approximately 0.25 × 1.49 = 0.3725, giving probability 0.3725/(1 + 0.3725) ≈ 0.271. Thus the probability rises by about 7.1 percentage points, not 49 percentage points.

9. Illustrative case: credit-line management

Background. A fictional regional bank wants to predict whether an existing customer will miss a payment in the next 90 days. Available variables include utilization, payment history, account age, income band, recent inquiries, and recorded customer-service contacts.

Problem. The bank seeks accurate risk ranking while complying with governance requirements. Some variables are highly correlated, the event is rare, and customer records are repeated over time. A model trained on randomly shuffled observations could leak future information or allow the same customer to appear in both training and validation sets.

Analysis. The analytics team defines the prediction date and removes variables unavailable at that date. It uses customer-level, time-aware splitting. Missing-value treatment, scaling, and feature selection are placed inside each training fold. Logistic regression supplies an interpretable baseline. Ridge logistic regression stabilizes correlated payment-history variables; lasso offers a sparse alternative; elastic net is evaluated because correlated behavioral measures are expected to act as groups. Performance is assessed using PR-AUC, calibration, recall at a review capacity, and expected cost—not ROC-AUC alone.

Approach and solution. The selected model is calibrated on a validation period and compared with a simpler scorecard. The bank documents variable definitions, coefficient signs, data lineage, drift indicators, approval thresholds, and an override process. Sensitive attributes are not automatically removed from governance analysis: they may be needed to test disparate impact, even if excluded from prediction.

Outcome. The regularized model improves ranking and reduces unstable coefficients, but deployment is conditional on monitoring. A sudden change in underwriting policy, economic conditions, or data capture could alter calibration. Human review remains necessary for borderline cases and adverse-action explanations.

Learning points. Regularization is a predictive-control method, not a substitute for causal reasoning, fair lending analysis, data governance, or operational validation. The unit of splitting must match the unit of decision. A model is a sociotechnical system whose consequences extend beyond its validation score.

10. Limitations, ethics, and contemporary practice

Regression models can encode historical discrimination, reproduce measurement inequalities, and create feedback loops. A variable such as prior service contact may act as a proxy for unequal access rather than underlying need. Analysts should assess subgroup performance, calibration, error rates, missingness, and the consequences of false positives and false negatives. Fairness criteria can conflict; selecting one requires institutional and legal judgment.

Privacy, data minimization, access control, retention limits, and documentation are essential. In regulated contexts, organizations may need explainability, adverse-decision reasons, audit trails, and human oversight. Explainability should distinguish descriptive summaries from causal explanations. Post hoc explanations of a complex model do not automatically establish that its predictions are fair or reliable.

Modern implementations use pipelines, reproducible environments, versioned datasets, and automated monitoring. Distributed computation and specialized solvers enable high-dimensional regularized regression. However, technical sophistication does not compensate for poor target definition, leakage, unstable labels, or inappropriate validation. Explainable boosting, generalized additive models, Bayesian regression, quantile regression, and hierarchical models may be preferable when nonlinear effects, uncertainty, or group structure are central.

Best practice is to begin with a transparent baseline, establish a leakage-resistant evaluation design, inspect data-generating processes, compare regularized alternatives, quantify uncertainty, and communicate limitations. The strongest model is not necessarily the most complex one; it is the model whose performance, assumptions, governance, and operational purpose are aligned.

Chapter Summary

Regression models estimate conditional relationships between outcomes and predictors. OLS minimizes squared residuals and is interpretable under appropriate specification and data conditions, but its assumptions must be diagnosed rather than assumed. Coefficients require attention to units, reference categories, interactions, transformations, and the distinction between association and causation. Polynomial terms and transformations can represent nonlinear patterns, while GLMs extend regression to binary, count, and other non-Gaussian outcomes through a distribution, linear predictor, and link function.

Ridge regression shrinks correlated coefficients toward zero, lasso can produce sparse solutions, and elastic net combines both behaviors. Regularization trades a little bias for potentially substantial variance reduction and improved prediction. Cross-validation must be nested within a workflow that prevents leakage, and time-dependent or grouped data require matching validation designs. Model evaluation should reflect the decision context, include calibration and subgroup analysis where relevant, and account for the costs of errors. Responsible deployment requires documentation, privacy protection, fairness assessment, monitoring, and human governance.

Key Terms

Regression: A method for modeling an outcome as a function of predictors.

Conditional mean: The expected outcome given specified predictor values.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Residual: The observed outcome minus its fitted value.

Multicollinearity: Strong dependence among predictors that can destabilize coefficient estimates.

Heteroscedasticity: Nonconstant conditional variance of the errors.

Leverage: A measure of how unusual an observation’s predictor values are.

Influence: The extent to which an observation affects fitted results.

Polynomial regression: Regression using powers of predictors to represent curvature.

Interaction: A term allowing one predictor’s effect to depend on another predictor.

Generalized linear model: A model combining a response distribution, linear predictor, and link function.

Logit link: The logarithm of odds, used commonly for binary outcomes.

Odds ratio: The multiplicative change in odds associated with a predictor change.

Regularization: Penalization used to control model complexity.

Ridge regression: Regression using an L2 coefficient penalty.

Lasso: Regression using an L1 coefficient penalty that can create zero coefficients.

Elastic net: A model combining L1 and L2 penalties.

Cross-validation: Resampling used to estimate out-of-sample performance or tune hyperparameters.

Data leakage: Use of information during training that would not be available at prediction time.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Review Questions

Short Answer Questions

1. What does a slope coefficient represent in a multiple regression model?
2. Why can training R² be misleading for model selection?
3. Distinguish ridge regression from lasso regression.
4. What are the three components of a generalized linear model?
5. Why should scaling be performed within, rather than before, cross-validation folds?

Long Answer Questions

1. Explain the principal OLS assumptions and describe a practical diagnostic for each.
2. Compare linear, polynomial, logistic, and Poisson regression, including the types of outcomes each is designed to model.
3. Explain how regularization can improve prediction while making coefficient interpretation more difficult.
4. Discuss why a statistically strong regression relationship should not automatically be interpreted as causal.
5. Design a complete model-development workflow for a business problem involving repeated observations and potential data leakage.

Analytical / Application Questions

1. A model has excellent training performance but substantially worse test performance. Diagnose possible causes and propose a revised validation and regularization strategy.
2. A logistic coefficient for a campaign indicator is 0.60, and the baseline event probability is 0.10. Calculate the approximate odds ratio and the predicted probability for an otherwise comparable treated case, then explain why the probability increase is not 60 percentage points.
3. A lasso model selects one of five highly correlated advertising variables. Explain why this may occur and recommend how a manager should interpret and validate the result.

Multiple Choice Questions with Answers

Question 1

Which objective does ordinary least squares minimize?

A. The sum of absolute residuals

B. The sum of squared residuals

C. The number of predictors

D. The variance of the predictors

Correct Answer: B. The sum of squared residuals

Question 2

In a multiple regression, a coefficient is ordinarily interpreted as the expected change in the outcome associated with a one-unit predictor increase while:

A. All observations are identical

B. The outcome is held constant

C. Other included predictors are held constant

D. The residual is forced to zero

Correct Answer: C. Other included predictors are held constant

Question 3

Which regularization method can set some fitted coefficients exactly equal to zero?

A. Ridge

B. Lasso

C. Ordinary least squares

D. Unpenalized polynomial regression

Correct Answer: B. Lasso

Question 4

For logistic regression, exponentiating a coefficient gives a:

A. Difference in probabilities independent of baseline risk

B. Ratio of variances

C. Conditional odds ratio

D. Residual standard deviation

Correct Answer: C. Conditional odds ratio

Question 5

If β = 0.40 in a logistic model, the associated odds ratio is approximately:

A. 0.40

B. 1.00

C. 1.49

D. 2.40

Correct Answer: C. 1.49

Question 6

Which issue is most directly indicated by a systematic curve in a residual-versus-fitted plot?

A. Possible functional-form misspecification

B. Perfect calibration

C. Absence of leverage

D. Correct independence

Correct Answer: A. Possible functional-form misspecification

Question 7

In a generalized linear model, the link function connects the conditional mean to the:

A. Sample size

B. Linear predictor

C. Residual rank

D. Penalty coefficient only

Correct Answer: B. Linear predictor

Question 8

Why are predictors commonly standardized before ridge or lasso fitting?

A. To make the outcome binary

B. To ensure the intercept is penalized

C. To put predictors on comparable scales for penalization

D. To eliminate all correlation among predictors

Correct Answer: C. To put predictors on comparable scales for penalization

Question 9

Which validation design is generally most appropriate for forecasting future time periods?

A. Randomly shuffling all observations

B. Forward-chaining or rolling validation

C. Using the test set repeatedly for tuning

D. Using only training error

Correct Answer: B. Forward-chaining or rolling validation

Question 10

A major danger of data leakage is that it:

A. Always decreases training performance

B. Makes validation performance look unrealistically strong

C. Guarantees causal interpretation

D. Eliminates the need for monitoring

Correct Answer: B. Makes validation performance look unrealistically strong