Regression and Regularized Learning

Introduction and context

Regression is a family of supervised learning methods used to estimate a numerical outcome from one or more explanatory variables. In business analytics, regression supports demand forecasting, pricing, credit-risk estimation, customer-value modelling, staffing, capacity planning, and resource allocation. Its value is not limited to prediction: a well-specified regression model can quantify associations, communicate marginal effects, identify uncertainty, and support decisions under constraints.

Regression should nevertheless be treated as a model of a data-generating process, not as proof that one variable causes another. A coefficient may represent a causal effect only when the design, assumptions, controls, and identification strategy justify that interpretation.

Learning objectives

By the end of this chapter, a learner should be able to:

• formulate and interpret linear and polynomial regression models;
• explain the assumptions behind ordinary least squares;
• distinguish prediction, explanation, and causal interpretation;
• construct and interpret generalized linear models;
• explain ridge, lasso, and elastic-net regularization;
• diagnose nonlinearity, heteroscedasticity, dependence, multicollinearity, and influential observations;
• quantify uncertainty using confidence and prediction intervals;
• select and validate regression models for business applications; and
• communicate model limitations, governance risks, and managerial implications.

1. The regression framework

Let yᵢ denote the outcome for observation i, and let xᵢ₁, …, xᵢₚ denote p predictors. A general regression model can be written as

where f(·) is the systematic component and εᵢ is unexplained variation. The conditional mean is

The distinction between the conditional mean and an individual outcome is important: even a highly accurate model generally cannot predict every customer, transaction, or day perfectly.

Terminology

The response, dependent variable, or target is y. Predictors, features, covariates, or explanatory variables are x variables. A parameter is a population quantity, such as βⱼ; an estimate, such as β̂ⱼ, is calculated from a sample. Residuals are observed errors, defined as eᵢ = yᵢ − ŷᵢ. Fitted values are predictions for observations used to estimate the model.

A model may be used for:

1. Prediction: estimating future or unobserved outcomes.
2. Description: summarizing relationships in observed data.
3. Explanation: assessing whether a proposed mechanism is consistent with evidence.
4. Causal analysis: estimating the effect of an intervention, which normally requires stronger assumptions or an experimental/quasi-experimental design.

2. Ordinary least squares and linear regression

The multiple linear regression model is

Here β₀ is the intercept, βⱼ is the expected change in y associated with a one-unit increase in xⱼ while the other included predictors remain fixed, and εᵢ is the error term. “Holding other variables fixed” is a conditional interpretation, not automatically a causal one.

In matrix notation,

where y is an n × 1 response vector, X is an n × (p + 1) design matrix containing a column of ones, β is a (p + 1) × 1 parameter vector, and ε is an n × 1 error vector.

Ordinary least squares (OLS) chooses coefficients that minimize the residual sum of squares:

In matrix form, RSS(β) = (y − Xβ)ᵀ(y − Xβ). If XᵀX is invertible, the OLS estimator is

The inverse exists when the columns of X are linearly independent. Exact or near-linear dependence creates singularity or unstable estimates.

Why OLS is useful

Under the assumptions that the conditional mean is correctly specified, observations are appropriately independent or dependence is handled, and E(ε | X) = 0, OLS is unbiased for the conditional mean parameters. Under homoscedastic, uncorrelated errors with finite variance, it is the best linear unbiased estimator (the Gauss–Markov result). Normality of errors is not required for unbiasedness; it is useful for exact small-sample t and F inference.

Important assumptions are:

• Linearity in parameters: the model is linear in β, even if transformed predictors are used.
• Exogeneity: E(εᵢ | X) = 0. Omitted variables, reverse causality, and measurement error can violate this condition.
• Appropriate dependence structure: time-series or clustered observations require suitable methods or standard errors.
• Constant conditional variance, when conventional standard errors are used: Var(εᵢ | X) = σ².
• No perfect multicollinearity: no predictor is an exact linear combination of others.
• Correct outcome scale and specification: systematic nonlinearities should not be ignored.

Goodness of fit

The total sum of squares is TSS = ∑(yᵢ − ȳ)², and the coefficient of determination is

For models with an intercept, R² is the proportion of in-sample variation around the mean accounted for by fitted values. It is not a probability that the model is correct and can increase when irrelevant predictors are added. Adjusted R² penalizes model size:

where n is sample size and p is the number of predictors excluding the intercept. For predictive work, cross-validated error is generally more informative than either R² measure.

Common error metrics include

and

MAE is in the units of the response and treats errors linearly. RMSE is also in response units but penalizes large errors more strongly. MAPE is problematic when actual values are zero or close to zero; it should not be used uncritically.

3. Interpretation, transformations, and polynomial regression

The scale of variables determines the meaning of coefficients. If y is revenue in dollars and x is advertising expenditure in dollars, β₁ has units of dollars of revenue per dollar of advertising. A rescaling from dollars to thousands of dollars changes the numerical coefficient but not the underlying relationship.

For a log-transformed response,

β₁ is approximately the percentage change in y associated with a one-unit increase in x when the change is small. More exactly, for a one-unit increase, the multiplicative change is exp(β₁), so the percentage change is 100[exp(β₁) − 1]%. This interpretation assumes y > 0.

For a log-transformed predictor,

β₁/100 is approximately the change in y for a 1% increase in x, provided x > 0 and the approximation is appropriate. With both variables logged, β₁ is an elasticity: a 1% increase in x is associated with approximately a β₁% change in y.

Categorical predictors are represented by indicator variables. If a three-category variable has reference category A, two indicators for B and C are sufficient. Their coefficients compare each category with A, conditional on other model variables. Including all category indicators together with an intercept creates perfect multicollinearity, known as the dummy-variable trap.

Polynomial regression

A curved relationship can be represented by polynomial terms:

This remains linear in the parameters β₀, …, β_d. The marginal effect is

A quadratic term allows the slope to vary with x. High-degree polynomials can oscillate at the boundaries and overfit. Centering x before forming powers often improves numerical stability and makes the intercept more interpretable, though it does not remove correlation entirely. Splines or generalized additive models are often preferable when smooth, local flexibility is needed.

4. Statistical uncertainty and inference

Suppose the classical linear model assumptions hold and errors have variance σ². The estimated residual variance is

The estimated covariance matrix of β̂ is

The standard error of β̂ⱼ is the square root of its diagonal covariance element. A two-sided approximate 100(1 − α)% confidence interval is

where t is a Student t critical value. This interval describes uncertainty in the parameter under the model and sampling assumptions; it is not the range of future individual outcomes.

For a new observation at predictor vector x₀, a mean-response interval reflects uncertainty about E(Y | x₀). A prediction interval additionally includes irreducible outcome error and is therefore wider. Under the classical model, the prediction standard-error factor is

where x₀ includes the intercept. The “1” represents new-observation noise. Intervals become wider for predictor values far from the centre of the training data; extrapolation is especially hazardous.

For heteroscedastic or dependent data, conventional standard errors may be wrong even when coefficient estimates are useful. Heteroscedasticity-consistent, clustered, or time-series-robust standard errors may be appropriate, but they do not repair omitted-variable bias or incorrect functional form.

5. Generalized linear models

Linear regression is unsuitable for outcomes such as binary purchase, count of claims, or positive skewed expenditure. Generalized linear models (GLMs) separate:

• a random component describing the response distribution;
• a systematic component ηᵢ = xᵢᵀβ; and
• a link function g connecting the conditional mean μᵢ = E(Yᵢ | Xᵢ) to ηᵢ.

Logistic regression

For a binary outcome Y ∈ {0,1}, logistic regression uses the logit link:

where pᵢ = P(Yᵢ = 1 | Xᵢ). Equivalently,

exp(βⱼ) is the odds ratio associated with a one-unit increase in xⱼ, conditional on other variables. It is not a constant change in probability; the probability effect depends on the starting p and other predictors. Maximum likelihood, rather than OLS, estimates the coefficients.

Count models

Poisson regression commonly uses

with μᵢ = E(Yᵢ | Xᵢ). exp(βⱼ) is an incidence-rate ratio. The Poisson assumption equates conditional mean and variance; overdispersion requires alternatives such as negative binomial regression or robust inference. Zero-inflated models may be considered only when a defensible structural explanation for excess zeros exists.

6. Regularized learning

OLS can have high variance when predictors are numerous, highly correlated, or when p is close to n. Regularization modifies the optimization objective by penalizing coefficient magnitude. Predictors should normally be standardized before penalization so that the penalty treats them comparably. The intercept is generally not penalized.

Ridge regression

Ridge estimates minimize

where λ ≥ 0 controls shrinkage. At λ = 0, ridge equals OLS. As λ increases, coefficients move toward zero but generally do not become exactly zero. Ridge is effective when many correlated predictors contribute modestly to prediction. It reduces variance at the cost of some bias.

Lasso regression

Lasso minimizes

The absolute-value penalty can set coefficients exactly to zero, providing a form of variable selection. With strongly correlated predictors, lasso may select one arbitrarily or distribute selection unstably. A zero coefficient means the penalized model has excluded that feature under the selected tuning parameter; it is not proof that the feature has no relationship with the outcome.

Elastic net

Elastic net combines both penalties:

where 0 ≤ α ≤ 1. α = 1 gives lasso; α = 0 gives ridge. Elastic net is useful when predictors are numerous and correlated because the L₂ component encourages grouped behaviour while the L₁ component can produce sparsity.

The tuning parameters λ and, for elastic net, α must be selected using validation data or cross-validation. The test set must remain untouched until final evaluation. Feature engineering, imputation, scaling, and selection must be performed within each training fold; otherwise information leakage produces optimistic estimates.

Regularized coefficients are conditional on the penalty and should not be interpreted exactly like unbiased OLS estimates. Classical p-values after data-driven selection are not automatically valid. If inference is the primary objective, pre-specified models, selective-inference methods, bootstrap procedures, or post-selection strategies should be considered.

7. Diagnosing regression models

Diagnostics should combine statistical evidence, visual analysis, domain knowledge, and out-of-sample performance.

Residual structure. Plot residuals against fitted values and important predictors. Curvature suggests missing nonlinear terms; a funnel shape suggests nonconstant variance.

Normality. A Q–Q plot can reveal heavy tails or skewness. Non-normal residuals mainly threaten small-sample inference and prediction intervals, not necessarily the usefulness of conditional-mean predictions.

Multicollinearity. Correlated predictors inflate standard errors and make individual coefficients unstable. Variance inflation factors are often summarized as

where Rⱼ² comes from regressing predictor j on the other predictors. No universal cutoff is valid across contexts. Remedies include combining variables, removing redundant predictors, collecting more informative data, or using ridge/elastic net.

Influence and leverage. Leverage concerns unusual predictor values; influence concerns the effect of an observation on fitted results. Cook’s distance is one diagnostic, but an influential observation is not automatically erroneous. Investigate data quality and business context before deletion.

Dependence and leakage. Random splitting is invalid for many time-series and grouped problems. Use rolling-origin validation for temporal forecasting and group-aware splitting where entities recur. Variables recorded after the decision point can leak future information into training.

Calibration and business loss. A model with good RMSE may be poor for a decision with asymmetric costs. Define the loss function before selecting the model. For logistic models, assess calibration as well as discrimination; a useful ranking model may still produce systematically inaccurate probabilities.

8. Worked example: regularized demand prediction

A retailer has n = 100 stores and p = 20 standardized predictors, including price, promotions, local income, competitor presence, seasonality indicators, and digital traffic. The target is weekly unit demand. The analyst uses a training set, a validation set, and a final test set.

OLS produces training RMSE 82 units and test RMSE 118 units. Ridge regression, with λ chosen by five-fold cross-validation inside the training data, produces training RMSE 91 and test RMSE 101. Lasso produces training RMSE 96 and test RMSE 108, retaining 8 predictors. Elastic net produces training RMSE 92 and test RMSE 99.

The lower training error of OLS but higher test error indicates greater variance or overfitting. Elastic net is preferred for prediction in this comparison, but the difference should be assessed using repeated or paired resampling rather than treated as meaningful solely because one point estimate is lower. If stockout costs are high, the business may optimize an asymmetric decision loss rather than RMSE. Predictions should also be checked for calibration across store types and for systematic errors during promotions.

A coefficient of −0.12 for standardized price in the elastic-net model means that, holding the penalized model’s other standardized predictors fixed, higher standardized price is associated with lower predicted demand. It does not directly mean that a 1% price increase reduces demand by 0.12%; elasticity requires a log–log specification or a carefully derived marginal interpretation.

9. Applied case: customer-retention intervention

Background

This illustrative case concerns a subscription business with monthly customer records. Management wants to identify customers likely to cancel and allocate a limited retention budget.

Problem

The outcome is cancellation within the next 60 days. Available predictors include tenure, recent service contacts, payment history, product usage, and plan type. The business must avoid using information that becomes available only after the intervention decision.

Analysis

The team defines the prediction date and constructs all features using information available at that date. It uses a time-based split because customer behaviour changes across months. Logistic regression provides interpretable odds ratios, while elastic-net logistic regression handles correlated usage measures and produces a more compact model. Calibration, recall among high-risk customers, expected intervention cost, and subgroup performance are evaluated.

Approach and solution

The team estimates cancellation probabilities, ranks customers, and applies a budget constraint. If the retention offer costs C and produces benefit B only for a customer who would otherwise cancel, an intervention is economically attractive when the estimated incremental expected benefit exceeds C. The correct quantity is uplift or incremental treatment effect, not merely cancellation risk. Therefore, where feasible, the firm runs a randomized offer experiment and models treatment response directly.

Outcome

The predictive model improves targeting, but the experiment shows that some high-risk customers are not persuadable, while some moderate-risk customers respond strongly to the offer. The firm changes from “target the highest-risk customers” to “target customers with the highest expected incremental value.”

Learning points

Prediction of an event is different from prediction of response to an action. Leakage, changing customer behaviour, class imbalance, fairness, and intervention costs must be addressed. Model performance should be evaluated at the decision threshold and under the actual resource constraint, not only with a global accuracy score.

10. Governance, ethics, and contemporary practice

Regression systems can reproduce historical discrimination when protected characteristics are encoded directly or indirectly through geography, language, spending capacity, or digital access. Removing a protected variable does not guarantee fairness. Organizations should document purpose, data provenance, feature definitions, validation design, limitations, and monitoring responsibilities.

High-stakes uses require attention to privacy, consent, data minimization, explainability, security, and applicable law. Predictions should not be presented as certainties. Human review must be meaningful rather than a nominal approval step. Monitor drift in feature distributions, relationships, calibration, error rates, and subgroup outcomes. Retraining should be governed by evidence, not by an automatic schedule alone.

Modern regression practice increasingly uses automated pipelines, cross-validation, regularization, gradient boosting, and probabilistic prediction intervals. These tools do not eliminate the need for sound measurement, causal reasoning, leakage prevention, and domain validation. Simple regression can remain preferable when transparency, stability, and auditability matter more than a small predictive improvement.

Best-practice workflow

1. Define the decision, outcome horizon, unit of analysis, and cost of errors.
2. Establish a temporal and causal data boundary.
3. Explore distributions, missingness, outliers, dependence, and measurement quality.
4. Create a defensible baseline, usually a mean, seasonal, or simple linear model.
5. Fit candidate models using preprocessing within resampling folds.
6. Compare out-of-sample metrics aligned with the business objective.
7. Diagnose residuals, calibration, stability, influence, and subgroup performance.
8. Quantify uncertainty and avoid unsupported causal claims.
9. Select thresholds or allocations using the actual resource constraint.
10. Deploy with documentation, monitoring, review, and a plan for model retirement.

Chapter Summary

Regression models express a conditional relationship between an outcome and predictors. OLS estimates linear parameters by minimizing squared residuals and is especially interpretable when its assumptions are credible. Transformations and polynomial terms represent elasticities and nonlinear effects, while GLMs extend regression to binary and count outcomes through link functions and appropriate response distributions. Confidence intervals describe parameter uncertainty; prediction intervals additionally include individual outcome noise.

Regularization trades a small amount of bias for reduced variance. Ridge shrinks correlated predictors without generally removing them, lasso can produce sparse models, and elastic net combines both behaviours. Tuning must be performed using leakage-free cross-validation, and standardized predictors are normally required. Diagnostics should examine functional form, variance, dependence, multicollinearity, influence, calibration, and temporal validity. In business settings, the best model is not necessarily the one with the lowest generic error: decision costs, interventions, fairness, governance, and uncertainty determine whether a model creates value.

Key Terms

Regression: A method for modelling a numerical or distributional outcome using predictors.

Conditional mean: The expected outcome given specified predictor values.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Residual: The observed outcome minus its fitted value.

Exogeneity: The condition that the error has conditional mean zero given predictors.

Multicollinearity: Strong linear association among predictors.

Heteroscedasticity: Predictor-dependent error variance.

Polynomial regression: Regression using powers of a predictor to represent curvature.

Generalized linear model: A model combining a response distribution, linear predictor, and link function.

Logit link: The logarithm of odds, ln[p/(1−p)].

Odds ratio: The multiplicative change in odds associated with a predictor change.

Ridge regression: Regression with an L₂, or squared-coefficient, penalty.

Lasso regression: Regression with an L₁, or absolute-coefficient, penalty.

Elastic net: A regression penalty combining L₁ and L₂ terms.

Regularization: Penalization used to control model complexity and variance.

Cross-validation: Resampling-based estimation of out-of-sample performance and tuning selection.

Data leakage: Use of information unavailable at prediction time.

Confidence interval: An interval procedure for uncertainty about a population parameter.

Prediction interval: An interval for a future individual outcome.

Calibration: Agreement between predicted probabilities and observed frequencies.

Review Questions

5 Short Answer Questions

1. What is the distinction between a fitted value and a residual?
2. State two assumptions required for conventional OLS inference.
3. Why are predictors usually standardized before ridge or lasso regression?
4. Interpret exp(βⱼ) in a logistic regression model.
5. Why can a prediction interval be wider than a confidence interval for the mean response?

5 Long Answer Questions

1. Explain the OLS objective, derive the matrix estimator, and discuss the condition required for its existence.
2. Compare ridge, lasso, and elastic net in terms of penalties, coefficient behaviour, and suitable business contexts.
3. Explain how log transformations and polynomial terms change coefficient interpretation.
4. Develop a diagnostic plan for a regression model showing heteroscedasticity, multicollinearity, and influential observations.
5. Discuss why predictive accuracy does not establish causality, using a business example.

3 Analytical / Application Questions

1. A bank predicts default using customer income, debt, and account activity. Design a leakage-free validation and monitoring strategy for a time-changing portfolio.
2. A retailer has a fixed budget for discounts. Explain why a cancellation-risk model may be inferior to an uplift model for allocating the budget.
3. A dataset contains 500 correlated marketing variables and 200 observations. Recommend a modelling strategy, including preprocessing, tuning, evaluation metrics, and governance safeguards.

Multiple Choice Questions with Answers

Question 1

Which condition is required for the usual OLS estimator to be unbiased for the conditional-mean parameters?

A. The predictors must all be normally distributed.

B. E(εᵢ | X) = 0.

C. The response must be binary.

D. All coefficients must be positive.

Correct Answer: B. E(εᵢ | X) = 0.

Question 2

In a logistic regression, what does exp(βⱼ) represent, conditional on the other predictors?

A. A constant change in probability.

B. The residual variance.

C. An odds ratio for a one-unit increase in xⱼ.

D. The model’s R².

Correct Answer: C. An odds ratio for a one-unit increase in xⱼ.

Question 3

Which regularization method can set some coefficients exactly equal to zero?

A. Ridge regression.

B. Lasso regression.

C. Unpenalized OLS only.

D. Polynomial regression.

Correct Answer: B. Lasso regression.

Question 4

If λ = 0 in ridge regression, the ridge objective reduces to which method, assuming XᵀX is invertible?

A. OLS.

B. Logistic regression.

C. Lasso with maximum sparsity.

D. Poisson regression.

Correct Answer: A. OLS.

Question 5

Why should preprocessing be performed inside each cross-validation training fold?

A. To guarantee that every coefficient is positive.

B. To prevent information leakage from validation observations.

C. To increase the number of observations.

D. To convert a regression problem into classification.

Correct Answer: B. To prevent information leakage from validation observations.

Question 6

A model has RSS = 400 and TSS = 1,000. What is its R²?

A. 0.40

B. 0.60

C. 1.40

D. 2.50

Correct Answer: B. 0.60

Question 7

Which interval includes both uncertainty in the estimated mean response and new-observation noise?

A. A parameter confidence interval only.

B. A prediction interval.

C. A leverage statistic.

D. An odds-ratio interval only.

Correct Answer: B. A prediction interval.

Question 8

What is a principal reason for using ridge regression with highly correlated predictors?

A. It always performs variable selection.

B. It reduces coefficient variance through shrinkage.

C. It requires a binary response.

D. It removes the need for validation.

Correct Answer: B. It reduces coefficient variance through shrinkage.

Question 9

For a count outcome, which issue challenges the basic Poisson model?

A. The response has no possible zeros.

B. Conditional variance substantially exceeds the conditional mean.

C. Predictors are standardized.

D. The intercept is included.

Correct Answer: B. Conditional variance substantially exceeds the conditional mean.

Question 10

Why is targeting only customers with the highest predicted cancellation risk potentially suboptimal for a retention offer?

A. Risk prediction always has zero accuracy.

B. High-risk customers may not be responsive to the intervention.

C. Logistic regression cannot use customer data.

D. Regularization prevents all interventions.

Correct Answer: B. High-risk customers may not be responsive to the intervention.