Regression and Predictive Estimation

Introduction and Context

Regression is a family of statistical and machine-learning methods used to estimate a continuous outcome from one or more explanatory variables. Examples include forecasting electricity demand, estimating house prices, predicting customer lifetime value, modelling manufacturing yield, and quantifying the relationship between advertising expenditure and sales.

Regression has two related purposes. Prediction focuses on accurately estimating unseen outcomes. Explanation or inference focuses on understanding how predictors are associated with the response and whether those associations are statistically credible. A model can be useful for prediction even when individual coefficients are difficult to interpret; conversely, a statistically interpretable model may have limited predictive accuracy outside the observed data.

In supervised learning, a dataset consists of observations (x_i, y_i), where x_i is a vector of predictors and y_i is a continuous response. The task is to learn a function f(x) such that predictions ŷ_i = f(x_i) are accurate for future observations drawn from a comparable data-generating process.

Learning Objectives

By the end of this chapter, you should be able to:

• formulate simple and multiple linear regression models;
• derive and interpret ordinary least-squares estimates;
• extend linear regression with transformations and polynomial terms;
• explain ridge, lasso, and elastic-net regularization;
• distinguish training, validation, and test performance;
• diagnose nonlinearity, heteroscedasticity, dependence, influential observations, and multicollinearity;
• select and interpret appropriate statistical and business metrics;
• communicate regression results while recognizing uncertainty, limitations, and governance risks.

1. Regression as a Predictive Model

A regression model separates systematic structure from unexplained variation. A general representation is

where y_i is the observed response, f(x_i) is the systematic component, and ε_i is the error or irreducible variation. The conditional mean is E(Y|X=x) = f(x) when the model is correctly specified for the mean.

A fitted model produces ŷ_i. The residual is

Residuals are observed training-data discrepancies; future prediction errors are not known at training time. This distinction is essential when evaluating generalization.

Terminology

• Response, target, or dependent variable: the continuous quantity being estimated.
• Predictor, feature, or explanatory variable: an input used to estimate the response.
• Coefficient: a parameter describing the model’s response to a predictor, conditional on other predictors.
• Intercept: the expected response when all predictors equal zero, subject to the meaningfulness of that reference point.
• Fitted value: the prediction for an observation used during fitting.
• Residual: observed response minus fitted response.
• Generalization: performance on new data from the target population.
• Bias–variance trade-off: the balance between systematic approximation error and sensitivity to sampling variation.

2. Simple and Multiple Linear Regression

2.1 Model specification

Simple linear regression uses one predictor:

Here, β₀ is the intercept, β₁ is the slope, and ε_i is the random error. The conditional mean is linear in x. The slope is measured in response units per predictor unit.

Multiple linear regression is

The coefficient β_j represents the expected change in Y associated with a one-unit increase in X_j, holding the other included predictors constant. This conditional interpretation does not automatically establish causation.

In matrix notation, with n observations and p predictors,

where y is an n × 1 response vector, X is an n × (p+1) design matrix including a column of ones, β is the coefficient vector, and ε is the error vector.

2.2 Ordinary least squares

Ordinary least squares (OLS) chooses coefficients that minimize the residual sum of squares:

The notation assumes that x_i includes the intercept component. In matrix form:

If X^TX is invertible, the OLS estimator is

This closed-form expression applies to ordinary least squares with full column rank. In software, QR decomposition or singular-value decomposition is generally preferred to explicitly computing a matrix inverse because it is more numerically stable.

For simple regression, the slope and intercept can be written as

where x̄ and ȳ are sample means. The denominator must be nonzero; otherwise the predictor has no variation and its slope cannot be estimated.

Worked example: advertising and sales

Suppose a fitted model estimates weekly sales, in thousands of units, from digital advertising expenditure, in thousands of currency units:

For advertising expenditure of 5 units, the estimate is

Thus, predicted sales are 30 thousand units. The slope means that an additional thousand currency units of advertising is associated with an estimated increase of 2.4 thousand units of sales, within the observed range and conditional on the model. It does not prove that increasing advertising will cause exactly that increase.

3. Assumptions, Inference, and Interpretation

OLS coefficient estimates can remain computationally defined when assumptions fail, but the assumptions determine interpretation, efficiency, and the validity of standard errors and tests.

1. Linearity in parameters: the conditional mean is correctly represented by the chosen linear combination of terms.
2. Independent observations: errors are not systematically dependent, unless dependence is explicitly modelled.
3. Exogeneity or zero conditional mean: E(ε|X)=0. Predictors must not contain systematic information about the unexplained error. Omitted variables, reverse causality, and measurement error can violate this condition.
4. Constant error variance: Var(ε|X)=σ². This is homoscedasticity. It is not required for unbiased coefficient estimates under stronger conditions, but it affects conventional standard errors and efficiency.
5. Normality of errors: not essential for OLS point estimates. It is mainly useful for exact small-sample t and F procedures when errors are plausibly normal.
6. No perfect multicollinearity: no predictor is an exact linear combination of other predictors.

Under standard Gauss–Markov conditions, OLS is the best linear unbiased estimator: among linear unbiased estimators, it has the smallest covariance matrix. This result does not state that OLS is best among all possible nonlinear or biased estimators, nor does it guarantee good predictions under misspecification.

For a coefficient estimate, a conventional confidence interval is

where SE(β̂_j) is the estimated standard error, t_{1−α/2, n−p−1} is a t critical value, α is the significance level, n is sample size, and p is the number of predictors excluding the intercept. This interval concerns a population parameter under the model and sampling assumptions; it is not automatically a prediction interval for an individual future outcome.

A prediction interval is wider because it includes both uncertainty in the estimated mean and new-observation noise. In the classical simple-regression setting, for a new predictor value x₀, the interval has the form

where S_xx=∑(x_i−x̄)², s is the residual standard deviation, and the term 1 represents new individual error. The formula assumes a correctly specified linear model, independent errors, constant variance, and appropriate use within the data range.

4. Model Fit and Evaluation

4.1 Sum-of-squares decomposition

When an intercept is included, total variation can be decomposed as

where SST=∑(y_i−ȳ)² is total sum of squares, SSR=∑(ŷ_i−ȳ)² is regression sum of squares, and SSE=∑(y_i−ŷ_i)² is error sum of squares.

The coefficient of determination is

It is the proportion of in-sample variation around the sample mean accounted for by fitted values under the stated decomposition. A high R² does not prove causality, guarantee low future error, or demonstrate that all important predictors have been included. Out-of-sample R² can be negative when predictions are worse than using the training or validation mean, depending on the evaluation definition.

Adjusted R² penalizes additional predictors:

It is useful for descriptive comparison of nested linear models under the same response and dataset, but it is not a substitute for out-of-sample validation.

4.2 Prediction metrics

Let e_i=y_i−ŷ_i be evaluation errors for m held-out observations.

Metric choice should follow the decision problem. If a large under-forecast creates disproportionate operational harm, squared-error metrics may be appropriate, but they should be complemented by directional and business-cost analysis. Percentage metrics are unsuitable when actual values can be zero or close to zero. For asymmetric costs, a weighted loss or a quantile model may be more appropriate than ordinary least squares.

4.3 Validation design

A single training score measures fit, not generalization. A typical workflow divides data into training, validation, and test sets. The training set estimates parameters; the validation set supports model and hyperparameter selection; the untouched test set provides a final estimate of expected performance.

For time-ordered data, random splitting can leak future information. Use rolling-origin or expanding-window evaluation. For grouped observations, such as multiple transactions from the same customer, split by customer when the intended deployment involves unseen customers. All preprocessing, feature selection, and imputation learned from data must be fitted within each training fold, not on the full dataset before cross-validation.

5. Nonlinearity and Polynomial Regression

A linear model can represent nonlinear relationships by transforming predictors. A polynomial model is

Although the curve is nonlinear in x, it remains linear in its coefficients and can be estimated using linear regression after constructing the transformed features. The marginal effect is

Therefore, β₁ alone is not the overall effect except at a particular reference point. Centering x before creating powers can improve interpretability and numerical stability, although it does not change the fitted curve when the model space is unchanged.

High-degree polynomials can oscillate strongly near boundaries and extrapolate implausibly. Splines provide a more locally controlled alternative by joining polynomial pieces at knots, subject to continuity constraints. Generalized additive models extend this idea by estimating smooth functions for predictors while retaining additive structure.

6. Regularization and Model Complexity

When predictors are numerous, correlated, or high-dimensional, least squares can have high variance. Regularization adds a penalty to the fitting objective.

Ridge regression

The penalty parameter λ≥0 controls shrinkage. The intercept is usually not penalized, and predictors should generally be standardized so that penalty strength is comparable across units. Ridge usually retains all predictors and is particularly useful under multicollinearity.

Lasso regression

The absolute-value penalty can set some coefficients exactly to zero, producing a sparse model. With strongly correlated predictors, lasso may select one variable arbitrarily or distribute selection unstably; selected variables should not automatically be treated as proven causal drivers.

Elastic net

where 0≤α≤1. α=1 gives lasso and α=0 gives ridge, subject to scaling conventions. λ and α should be selected using training-only cross-validation. Regularization introduces bias intentionally to reduce variance; its success must be judged by validation performance and deployment objectives.

7. Diagnosing Regression Models

Diagnostics should combine numerical summaries, visual inspection, domain knowledge, and validation results.

Residual patterns

A residual-versus-fitted plot should show a structureless cloud around zero. Curvature suggests missing nonlinear terms. A funnel shape suggests nonconstant variance. Clusters or sequences suggest dependence or omitted group effects. A Q–Q plot assesses whether residual quantiles approximately follow a reference normal distribution; deviations matter more for small-sample inference than for large-sample prediction.

Leverage and influence

A high-leverage observation has unusual predictor values. It need not be problematic, but it can strongly affect the fitted line. Influence combines unusual predictors with a large residual. Cook’s distance is a common diagnostic, but there is no universal cutoff that mechanically identifies bad data. Investigate influential points for recording errors, legitimate rare cases, regime changes, or important subpopulations. Deleting observations solely to improve a metric is poor practice.

Heteroscedasticity and dependence

When variance changes with predictors, consider transformations, weighted least squares, robust standard errors, or models designed for the conditional variance. When observations are correlated across time, location, or entity, use appropriate time-series, hierarchical, clustered, or longitudinal methods. Treating repeated measurements as independent can produce overconfident conclusions.

Multicollinearity

Correlated predictors make individual coefficients unstable and inflate standard errors, although predictions may remain accurate. Variance inflation factors can assist diagnosis, but thresholds are context-dependent. Remedies include combining variables, removing redundant features, collecting more informative data, using ridge regression, or reframing the inferential question around groups of predictors.

8. Generalized Regression Models

Ordinary linear regression is designed for continuous responses with an approximately constant conditional variance. Generalized linear models (GLMs) extend regression to non-normal response distributions through a systematic component and a link function:

For a binary response, logistic regression uses the logit link:

where p=P(Y=1|X). Exponentiating β_j gives an odds ratio for a one-unit increase in x_j, holding other predictors constant. Logistic regression estimates probabilities, not continuous quantities, but illustrates the broader principle of generalized regression.

For counts, Poisson regression uses a log link, log(E[Y|X])=β₀+β^TX, while negative binomial models can address overdispersion. For positive, right-skewed continuous outcomes, a Gamma GLM with a suitable link may be more appropriate than transforming the response and then back-transforming without correction. Model selection should reflect the response scale, sampling mechanism, and variance structure.

9. Detailed Applied Case: Predicting Hospital Length of Stay

The following is an illustrative case, not a report of a real hospital.

Background and problem

A hospital wants to predict inpatient length of stay in days at admission to support bed planning and discharge coordination. Available variables include age, admission type, prior admissions, comorbidity score, emergency-department waiting time, and initial laboratory measurements. The hospital must avoid using information unavailable at admission, such as discharge diagnosis or later treatment decisions.

Analysis

Length of stay is positive, right-skewed, and may contain a small number of very long admissions. A baseline model using the historical median provides a reference. Analysts then compare a log-transformed linear model, a Gamma GLM, and a regularized model with nonlinear terms. The split is temporal: earlier admissions are used for training and later admissions for validation and testing. This reflects deployment more realistically than a random split.

Missing laboratory values are imputed using rules fitted within the training data. Predictors are standardized for regularized models. Model selection uses validation MAE and RMSE. MAE reflects typical scheduling error, while RMSE reveals whether a model fails badly on unusually long stays. Calibration is assessed by comparing average predicted and observed stay within risk bands.

Approach and solution

Suppose the selected model has a test MAE of 1.8 days and RMSE of 4.6 days, while the median baseline has MAE of 2.4 days and RMSE of 5.1 days. The model improves average accuracy, but the gap between MAE and RMSE indicates substantial large-error cases. These cases should be investigated rather than hidden by reporting only one metric. Prediction intervals or quantile estimates may support capacity planning better than a single point estimate.

The model card documents the training period, variables, exclusions, evaluation design, subgroup performance, intended use, and limitations. Performance is monitored after deployment because admission practices, coding, patient mix, and clinical pathways may change.

Outcome and learning points

The model is used as decision support, not as an automatic discharge rule. A clinician can override a prediction and record a reason. Subgroup analysis examines whether errors are systematically larger for particular age groups, hospitals, or language communities. The case demonstrates that a technically strong regression model requires leakage control, deployment-aligned validation, uncertainty communication, fairness monitoring, and operational integration.

10. Professional, Ethical, and Governance Considerations

Regression models can reproduce historical inequities. A predictor may be statistically useful yet ethically inappropriate if it encodes unequal access to care, discriminatory practices, or proxies for protected attributes. Fairness cannot be assessed by a single universal metric; organizations must define the relevant harm, population, and decision context.

Data governance should address consent, purpose limitation, access control, retention, provenance, and auditability. Sensitive attributes may be needed for fairness evaluation even when excluded from prediction, subject to lawful and ethical safeguards. Models used in high-impact settings require human oversight, documented uncertainty, monitoring for drift, and a process for contesting decisions.

Interpretability should be matched to use. Coefficients, partial dependence, residual analysis, and local explanations can support understanding, but an explanation is not proof of causation. If an organization needs to estimate the effect of an intervention, causal design, experiments, or quasi-experimental methods may be required rather than ordinary predictive regression.

11. Best Practices and Contemporary Developments

Effective regression practice includes:

• define the decision, prediction horizon, and cost of errors before selecting a metric;
• establish a simple baseline;
• inspect data quality, units, missingness, outliers, and leakage;
• use domain-informed transformations and interaction terms;
• standardize predictors when using penalized models;
• use cross-validation or time-aware validation to select complexity;
• report uncertainty, calibration, subgroup performance, and confidence in the evaluation design;
• preserve a final test set until model selection is complete;
• monitor drift, residual patterns, data availability, and business outcomes after deployment.

Contemporary systems increasingly combine interpretable regression with tree ensembles, neural networks, and probabilistic forecasting. Regression remains valuable as a benchmark, a calibrated component, a transparent model for governance, and a source of structured features. Modern predictive estimation also emphasizes conformal prediction, which can construct distribution-free predictive intervals under exchangeability assumptions, and automated monitoring of data and concept drift. These methods do not remove the need for careful sampling, valid targets, and responsible use.

Chapter Summary

Regression estimates continuous outcomes by relating a response to predictors through a systematic component and an error term. Linear regression uses coefficients that are linear in the parameters, and OLS minimizes squared residuals. Its useful interpretations depend on assumptions concerning linearity, independence, exogeneity, variance, and multicollinearity. Polynomial terms, splines, and generalized models allow more flexible relationships or response distributions. Ridge, lasso, and elastic net control complexity by shrinking coefficients, with hyperparameters selected through validation.

Model quality must be assessed on data representative of deployment, using metrics aligned with business consequences. MAE is interpretable and comparatively robust, RMSE penalizes large errors, and percentage metrics require care near zero. Residual, leverage, influence, heteroscedasticity, dependence, and calibration diagnostics reveal weaknesses that a single score cannot. Finally, regression deployment requires attention to leakage, uncertainty, fairness, privacy, monitoring, and the distinction between prediction and causal explanation.

Key Terms

Regression: Estimation of a response from one or more predictors.

Response variable: The outcome being predicted or explained.

Predictor: An input variable used by a model.

Intercept: Expected response when predictors equal their reference values.

Coefficient: Parameter quantifying a predictor’s conditional association with the response.

Residual: Observed response minus fitted response.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Generalization: Performance on new data from the target population.

Homoscedasticity: Constant conditional error variance.

Heteroscedasticity: Nonconstant conditional error variance.

Multicollinearity: Strong correlation or linear dependence among predictors.

Leverage: Unusualness of an observation’s predictor values.

Influence: The extent to which an observation affects fitted results.

Ridge regression: L2-penalized regression that shrinks coefficients.

Lasso regression: L1-penalized regression that can produce zero coefficients.

Elastic net: A mixture of L1 and L2 regularization.

MAE: Mean absolute error, measured in response units.

RMSE: Square root of mean squared error, measured in response units.

Prediction interval: An interval intended to contain a future individual outcome.

Data leakage: Use of information unavailable at prediction time or improperly shared across evaluation partitions.

Review Questions

Short Answer Questions

1. What is the difference between a residual and a future prediction error?
2. Interpret a multiple-regression coefficient while holding other predictors constant.
3. State two assumptions required for conventional OLS inference.
4. Why should predictors usually be standardized before ridge or lasso regression?
5. Why can MAPE be problematic when actual values are zero or close to zero?

Long Answer Questions

1. Derive the OLS objective and explain the matrix estimator β̂=(X^TX)⁻¹X^Ty, including its applicability conditions.
2. Compare ridge, lasso, and elastic net in terms of penalties, sparsity, multicollinearity, and model selection.
3. Explain how residual, leverage, influence, and heteroscedasticity diagnostics contribute to model assessment.
4. Compare training, validation, test, random, grouped, and time-aware evaluation designs.
5. Discuss why predictive association should not automatically be interpreted as causation, using a practical example.

Analytical / Application Questions

1. A demand model has low training error but much higher validation error. Diagnose likely causes and propose a revised modelling workflow.
2. A hospital model has good overall RMSE but substantially higher errors for one demographic group. Design an investigation and governance response.
3. A sales response is positive, highly skewed, and contains zeros. Recommend candidate models and metrics, explaining the assumptions behind your choices.

Multiple Choice Questions with Answers

Question 1

In ordinary least squares, the fitted coefficients are chosen to minimize which quantity?

A. The sum of absolute residuals

B. The sum of squared residuals

C. The maximum residual only

D. The number of predictors

Correct Answer: B. The sum of squared residuals

Question 2

In a multiple linear regression model, the coefficient of a predictor is interpreted as the expected change in the response for a one-unit increase in that predictor, provided that:

A. The response is binary

B. All other included predictors are held constant

C. The residuals are exactly zero

D. The predictor has zero variance

Correct Answer: B. All other included predictors are held constant

Question 3

Which metric is expressed in the same units as the response and strongly penalizes large errors?

A. RMSE

B. MAPE

C. R²

D. Adjusted R²

Correct Answer: A. RMSE

Question 4

What is a principal characteristic of lasso regression?

A. It always leaves coefficients unchanged

B. It can set some coefficients exactly to zero

C. It requires a binary response

D. It removes the intercept by definition

Correct Answer: B. It can set some coefficients exactly to zero

Question 5

A funnel-shaped pattern in a residual-versus-fitted plot most directly suggests:

A. Perfect multicollinearity

B. Constant error variance

C. Heteroscedasticity

D. An exact causal relationship

Correct Answer: C. Heteroscedasticity

Question 6

If actual values are zero, which commonly used metric is undefined for those observations?

A. MAE

B. MSE

C. RMSE

D. MAPE

Correct Answer: D. MAPE

Question 7

For a time-ordered forecasting problem, which evaluation strategy best avoids using future information to predict the past?

A. Randomly shuffle all observations before splitting

B. Use a temporal or rolling-origin split

C. Fit and test on the same observations

D. Select the model using the test set repeatedly

Correct Answer: B. Use a temporal or rolling-origin split

Question 8

If a fitted model is Ŷ = 10 + 3X, what is the prediction when X = 4?

A. 12

B. 14

C. 22

D. 40

Correct Answer: C. 22

Question 9

Which statement about ridge regression is correct?

A. It uses an L2 penalty and generally shrinks correlated coefficients

B. It uses an L1 penalty and always produces a sparse model

C. It cannot be used with continuous responses

D. It selects hyperparameters using the test set by definition

Correct Answer: A. It uses an L2 penalty and generally shrinks correlated coefficients

Question 10

A prediction interval for an individual future observation is generally wider than a confidence interval for the mean response because it includes:

A. Only coefficient estimation uncertainty

B. New-observation error as well as mean-estimation uncertainty

C. No uncertainty at all

D. Only uncertainty from the intercept

Correct Answer: B. New-observation error as well as mean-estimation uncertainty