Regression and Predictive Estimation

Introduction and context

Regression is a family of statistical and machine-learning methods for estimating a continuous response from one or more explanatory variables. It is used to forecast demand, estimate property prices, predict energy consumption, quantify credit risk, model biological measurements, and support operational decisions. Unlike classification, whose outputs are categories, regression produces numerical predictions and—when properly specified—an estimate of predictive uncertainty.

A regression model is not simply a line fitted through data. It is a set of assumptions about how a response is generated, a procedure for estimating unknown parameters, and an evaluation framework for judging whether predictions are useful. A professional regression analysis therefore addresses four questions:

1. What relationship between inputs and response is plausible?
2. How should the model parameters be estimated?
3. Are the assumptions sufficiently reasonable for the intended use?
4. How accurate, stable, interpretable, and fair are the predictions?

Learning objectives

By the end of this chapter, you should be able to:

• formulate simple and multiple linear regression models;
• estimate coefficients using ordinary least squares and interpret them correctly;
• construct polynomial features without confusing nonlinearity in variables with nonlinearity in parameters;
• explain ridge, lasso, and elastic-net regularization;
• describe generalized linear models for non-Gaussian responses;
• diagnose linearity, independence, constant variance, influential observations, and multicollinearity;
• quantify parameter and predictive uncertainty;
• calculate and interpret common regression metrics;
• select and validate a regression model for a practical application.

1. The regression framework

Let yᵢ denote the response for observation i and let xᵢ₁, …, xᵢₚ denote p predictors. In multiple linear regression,

Here β₀ is the intercept, βⱼ is the partial effect of predictor j, and εᵢ is the unexplained error. The fitted or predicted response is

The distinction between yᵢ and ŷᵢ is essential. The first is observed; the second is estimated. The residual is

The regression function describes the conditional mean:

This does not assert that every observation lies on the fitted surface. It asserts that the average response at a given predictor configuration follows that surface.

Terminology and interpretation

A predictor may be called an explanatory variable, feature, regressor, or independent variable. The response may be called the dependent variable, target, or outcome. In observational data, the word “independent” does not imply that changing a predictor causes the response to change. Causal interpretation requires a research design and assumptions beyond regression fitting.

A coefficient is interpreted conditionally. If β̂₂ = 3.4, then a one-unit increase in x₂ is associated with an estimated 3.4-unit increase in the mean response, holding the other included predictors constant. The units are response units per predictor unit. For a binary indicator, the coefficient compares the predicted means of the indicated group and the reference group, conditional on the other predictors.

2. Ordinary least squares

Ordinary least squares (OLS) estimates coefficients by minimizing the sum of squared residuals:

The square penalizes large errors more heavily than small errors and makes the objective differentiable. In matrix notation, let X be the n × (p + 1) design matrix, including a column of ones, let y be the n × 1 response vector, and let β be the coefficient vector. Then

If X has full column rank, the unique OLS solution is

The full-rank condition means that no column of X is an exact linear combination of the others. In practice, numerical algorithms such as QR decomposition or singular-value decomposition are preferred to explicitly computing an inverse, because they are more stable.

Short derivation

Differentiating the matrix objective gives

Setting this gradient to zero produces the normal equations:

When XᵀX is invertible, solving these equations yields the OLS estimator above.

Assumptions

The main assumptions should be distinguished carefully:

• Linearity in parameters: the conditional mean is correctly represented by a linear combination of the chosen features. Polynomial and interaction terms can still satisfy this condition.
• Exogeneity: E(εᵢ | X) = 0. Predictors must not be systematically related to omitted error components if coefficients are to have the usual unbiased interpretation.
• Independence: errors from different observations are independent, or their dependence is explicitly modelled. Time series and clustered data often violate this assumption.
• Constant conditional variance: Var(εᵢ | X) = σ². This is called homoscedasticity.
• Normality: εᵢ is normally distributed conditional on X. Normality is not required for OLS coefficients to be computed, and large-sample prediction can remain useful without it; it is mainly important for exact small-sample tests and intervals.
• No perfect multicollinearity: predictors must contain enough independent information to estimate separate coefficients.

Under linearity, exogeneity, independence, constant variance, and full rank, OLS is the best linear unbiased estimator (BLUE) in the Gauss–Markov sense. Normality is an additional assumption, not part of the basic Gauss–Markov result.

3. Measures of fit and predictive accuracy

The total variation in the observed response can be decomposed as

where ȳ is the sample mean. The residual sum of squares is

With an intercept, OLS gives SST = SSR + SSE, where SSR is the regression sum of squares. The coefficient of determination is

R² is the proportion of in-sample variation explained relative to a mean-only model. It is not the proportion of future errors eliminated, nor does a high value prove causality. Adding predictors cannot decrease ordinary R², even when they are unhelpful. Adjusted R² addresses this by penalizing unnecessary predictors:

where n is the sample size and p is the number of predictors excluding the intercept.

Common error metrics are:

MAE has the same units as y and treats errors linearly. MSE and RMSE penalize large errors more strongly; RMSE returns to response units. Mean absolute percentage error is

MAPE is inappropriate or unstable when yᵢ is zero or close to zero, and it can distort comparisons across low-valued observations. For strictly positive data, alternatives include symmetric percentage measures or evaluating errors on a logarithmic scale. Metrics must be selected according to the cost of errors and the measurement scale.

Evaluation should use a test set or cross-validation that reflects deployment. Random splitting is unsuitable when future observations must be predicted from past observations; a time-ordered split is then required. Data preprocessing, feature selection, and hyperparameter tuning must be performed within training folds to prevent leakage.

4. Polynomial and interaction regression

A relationship can be curved while remaining linear in its coefficients. A quadratic model is

and a degree-d polynomial is

The model is linear in β₀, …, β_d, so OLS remains applicable after creating transformed features. An interaction model includes a product:

Here the effect of x₁ depends on x₂: ∂E(Y|X)/∂x₁ = β₁ + β₃x₂. Main effects should normally be retained when an interaction is included, so that interpretation follows the hierarchy principle.

High-degree polynomials can oscillate and extrapolate dangerously. Centering or standardizing predictors often improves numerical conditioning and makes the lower-order coefficients more interpretable. Splines or generalized additive models may provide smoother local flexibility with less global instability.

5. Regularized regression

When predictors are numerous, correlated, or measured with noise, an unregularized model may have unstable coefficients and poor generalization. Regularization adds a penalty to the training objective.

Ridge regression

The intercept is generally not penalized. λ ≥ 0 controls shrinkage. With standardized predictors and an unpenalized intercept, the solution has the form

for the penalized coefficient block. Ridge usually retains all predictors and is effective under multicollinearity.

Lasso regression

The absolute-value penalty can set some coefficients exactly to zero, producing a sparse model. Selection can be unstable when highly correlated predictors are interchangeable.

Elastic net

where 0 ≤ α ≤ 1. α = 1 gives lasso and α = 0 gives ridge, subject to scaling conventions. Predictors should generally be standardized before penalization; otherwise the penalty depends on their measurement units. λ and, where applicable, α should be selected by cross-validation. Regularization introduces bias deliberately to reduce variance and improve out-of-sample performance.

6. Generalized linear models

Ordinary linear regression assumes a continuous response with an approximately constant-variance error structure. Generalized linear models (GLMs) extend regression to other response distributions through three components:

1. a random component from an exponential-family distribution;
2. a systematic component, η = Xβ;
3. a link function g relating the conditional mean μ = E(Y|X) to η:

For a binary outcome, logistic regression uses the Bernoulli distribution and the logit link:

where p = P(Y = 1|X). Exponentiating βⱼ gives an odds ratio for a one-unit increase in xⱼ, holding other variables constant. Logistic regression is a GLM for classification, but it belongs in a regression framework because it models a conditional probability.

For counts, Poisson regression uses

and μ is the expected count. An exposure offset, such as log(hours at risk), can be included when modelling rates. Poisson models require the conditional mean and variance to be equal; overdispersion may require quasi-Poisson or negative-binomial methods. For positive, right-skewed continuous responses, a Gamma GLM with a suitable link can be more appropriate than ordinary least squares.

7. Diagnosing assumptions and influential observations

Diagnostics should be performed on training data and interpreted with subject-matter knowledge. A residual-versus-fitted plot can reveal curvature, unequal spread, or groups. A normal Q–Q plot assesses whether residuals are approximately normal; it does not test linearity. Residuals plotted against time or ordering can reveal dependence.

Heteroscedasticity means Var(εᵢ|X) changes with predictors. OLS coefficients may still be unbiased under exogeneity, but conventional standard errors can be wrong. Remedies include transforming the response, using heteroscedasticity-robust standard errors, weighted least squares, or modelling the variance directly. Weighted least squares minimizes

where weights are chosen in relation to the inverse error variance when that variance structure is known or credibly estimated.

Multicollinearity inflates coefficient uncertainty and makes individual effects difficult to separate. It does not necessarily reduce predictive accuracy. Correlation matrices, condition numbers, and variance inflation factors are useful indicators. For predictor j, a common variance inflation factor is

where Rⱼ² is obtained by regressing predictor j on the remaining predictors. Thresholds such as 5 or 10 are heuristics, not universal laws.

An observation may be an outlier in response space, have high leverage because its predictors are unusual, or be influential because its removal changes the fitted model substantially. Leverage, studentized residuals, and Cook’s distance help identify such cases. An influential observation should be investigated for data error, legitimate rare conditions, or a missing subpopulation—not automatically deleted.

8. Uncertainty and prediction intervals

Assume the classical normal linear model with independent errors of variance σ² and a full-rank design. The residual variance estimator is

For a new predictor vector x₀, including a leading 1 for the intercept, the estimated mean response is x₀ᵀβ̂. A 100(1 − α)% confidence interval for the mean response is

A prediction interval for one future observation is wider because it includes both uncertainty in the estimated mean and new individual error:

The term 1 is the key difference. These formulas assume the model is correctly specified and that x₀ is within a credible region of the training data. Extrapolation can make intervals misleading even when their algebra is correct. For regularized and highly nonlinear models, bootstrap, conformal prediction, or other resampling and distribution-free methods may be more appropriate, but their assumptions and coverage guarantees must be checked.

9. Worked example: energy-demand estimation

Suppose a small office records daily electricity demand y in kWh and mean outdoor temperature x in °C. A fitted quadratic model is

For a day with x = 25 °C,

The derivative is

At 25 °C, the slope is −2 kWh per °C. Thus, locally, a one-degree increase is associated with approximately 2 kWh lower predicted demand. At x = 30 °C, the slope is 0 kWh per °C, indicating the minimum of this quadratic occurs at x = 30 °C, because solving −12 + 0.40x = 0 gives x = 30. The interpretation should not be extended beyond the observed temperature range without evidence; a quadratic can rise rapidly outside the data.

Suppose actual demand on the 25 °C day is 260 kWh. The residual is 260 − 245 = 15 kWh. This single residual contributes 15² = 225 kWh² to SSE. Its absolute error is 15 kWh, while its squared error is 225 kWh². These are different quantities and should not be confused.

A sound workflow would split historical days chronologically, fit the model on earlier days, tune polynomial degree or regularization using validation data, evaluate MAE and RMSE on later days, inspect residuals against temperature and calendar variables, and report prediction intervals. If demand depends strongly on occupancy, humidity, weekday, and equipment schedule, temperature alone may be useful but incomplete.

10. Applied case: illustrative hospital equipment forecasting

Background

A hospital wants to forecast weekly maintenance hours for imaging equipment so that engineering staff can be scheduled. Historical records contain equipment age, weekly operating hours, number of emergency procedures, manufacturer, and prior fault counts.

Problem

Management initially requests a model with the lowest RMSE. However, maintenance hours are non-negative, fault counts are skewed, and emergency activity varies by season. A model that performs well on average but systematically underpredicts high-demand weeks could create operational risk.

Analysis and approach

The team first defines the prediction horizon and excludes information unavailable at forecast time. It creates a chronological training, validation, and test split. Missing values are imputed within training folds. Manufacturer is encoded using training data only, and numerical predictors are standardized for regularized models.

A baseline predicts the historical training mean. OLS provides an interpretable benchmark. Ridge regression is fitted because operating hours and emergency procedures are correlated. A lasso model is considered for sparse feature selection. Because the response is a positive, skewed number of hours, the team also evaluates a Gamma GLM with a log link. Residual and calibration analyses are performed by equipment type and workload level.

Outcome

The selected model is not chosen solely by one metric. It has lower test MAE than the baseline, acceptable RMSE, stable performance across equipment groups, and no serious calibration bias. Prediction intervals are supplied to scheduling staff. The model is monitored after deployment because equipment policies, staffing, and procedure volumes can change.

Learning points

• A meaningful baseline is necessary to establish whether modelling adds value.
• Temporal validation is more credible than random splitting for future forecasting.
• A response distribution and operational loss function should influence model choice.
• Correlation among predictors affects coefficient interpretation and stability.
• Subgroup performance and uncertainty matter as much as average accuracy.
• Forecasting maintenance hours does not establish that a predictor causes faults.

11. Professional, ethical, and governance considerations

Regression models can encode historical inequities, proxy sensitive attributes, or produce systematically poorer predictions for smaller groups. Before deployment, practitioners should document data provenance, intended use, exclusions, transformations, metric choices, subgroup performance, and known limitations. Personal and health data require appropriate privacy, access control, retention, and legal safeguards.

Model outputs should be treated as decision support unless validated for autonomous use. Staff need clear explanations of what a coefficient or interval means, and an escalation process for unusual cases. Monitoring should include drift in predictors, changes in residual error, calibration, and subgroup disparities. Re-training should be governed rather than triggered automatically by arbitrary changes in one metric.

12. Best-practice workflow

1. Define the response, forecast horizon, decision cost, and unit of analysis.
2. Inspect data quality, missingness, measurement units, duplicates, and leakage.
3. Establish a simple baseline.
4. Visualize response–predictor relationships and plausible transformations.
5. Fit an interpretable benchmark before adding complexity.
6. Use pipelines so preprocessing occurs inside resampling folds.
7. Compare OLS, polynomial, regularized, or GLM alternatives according to data characteristics.
8. Diagnose residuals, dependence, heteroscedasticity, leverage, and subgroup behaviour.
9. Tune hyperparameters using validation or cross-validation appropriate to the data-generating process.
10. Report test performance with uncertainty, units, and the baseline comparison.
11. Check whether the model is being extrapolated.
12. Document assumptions and monitor the deployed system.

Contemporary developments

Modern regression practice increasingly combines interpretable statistical models with flexible machine-learning estimators such as gradient boosting and neural networks. Even when a flexible model is selected, regression principles remain relevant: leakage control, appropriate loss functions, calibration, uncertainty, drift monitoring, and residual analysis. Explainability tools can describe associations but do not automatically establish causation. Conformal prediction is receiving attention because it can provide finite-sample predictive coverage under exchangeability assumptions; however, dependence, distribution shift, and subgroup coverage require careful examination.

Chapter Summary

Regression estimates a conditional response from predictors and can support both explanation and prediction. OLS minimizes squared residuals and has useful properties under linearity, exogeneity, independence, constant variance, and full rank. Polynomial and interaction terms represent curvature and conditional effects while remaining linear in parameters. Ridge, lasso, and elastic net trade a small amount of bias for improved stability or sparsity and require appropriate scaling and validation. GLMs connect a response distribution and link function to a linear predictor, allowing models for binary, count, and positive continuous outcomes. Reliable practice requires diagnostic assessment, leakage-safe validation, suitable metrics, uncertainty intervals, and awareness of extrapolation. Professional deployment additionally requires fairness analysis, privacy protection, documentation, monitoring, and human oversight.

Key Terms

Regression: Estimation of a response from one or more predictors.

Response: The outcome or target variable being predicted.

Predictor: An input variable used to estimate the response.

Residual: The observed response minus its fitted value.

Ordinary least squares: Estimation by minimizing squared residuals.

Design matrix: Matrix containing the predictor values and usually an intercept column.

Exogeneity: Condition that the conditional mean of the error given predictors is zero.

Homoscedasticity: Constant conditional error variance.

Multicollinearity: Strong linear dependence among predictors.

Polynomial regression: Regression using powers of one or more predictors.

Interaction: A term whose effect depends on the value of another predictor.

Ridge regression: L2-penalized regression that shrinks coefficients toward zero.

Lasso regression: L1-penalized regression that can set coefficients exactly to zero.

Elastic net: Combination of L1 and L2 penalties.

Generalized linear model: Model combining a response distribution, linear predictor, and link function.

MAE: Mean absolute error, in response units.

RMSE: Square root of mean squared error, in response units.

Confidence interval: Interval expressing uncertainty about a population or mean-response quantity.

Prediction interval: Interval for a future individual observation.

Data leakage: Use of information during training that would not be available at prediction time.

Review Questions

Short Answer Questions

1. Distinguish an observed response, a fitted response, and a residual.
2. State the main assumptions of the classical linear regression model.
3. Why should predictors generally be standardized before ridge or lasso regression?
4. Explain the difference between MAE and RMSE.
5. What is the difference between a confidence interval for a mean response and a prediction interval for a future observation?

Long Answer Questions

1. Derive the OLS normal equations and explain the full-rank condition.
2. Compare ridge, lasso, and elastic-net regression, including their penalties, strengths, and limitations.
3. Explain how residual plots, Q–Q plots, leverage, and influence diagnostics assess a regression model.
4. Describe the three components of a generalized linear model and illustrate them with logistic and Poisson regression.
5. Design a leakage-safe workflow for forecasting a continuous business outcome using historical data.

Analytical / Application Questions

1. A model has excellent training RMSE but poor chronological test RMSE. Diagnose at least three possible causes and propose corrective actions.
2. A maintenance model performs well overall but substantially underpredicts high-demand weeks for one equipment group. Which metrics, diagnostics, data checks, and governance actions should be considered?
3. A quadratic demand model predicts a minimum inside the observed temperature range but rises sharply outside it. Explain the mathematical reason, the extrapolation risk, and two alternative modelling strategies.

Multiple Choice Questions with Answers

Question 1

Which quantity is minimized by ordinary least squares?

A. The sum of absolute residuals

B. The sum of squared residuals

C. The maximum residual

D. The percentage error only

Correct Answer: B. The sum of squared residuals

Question 2

In a multiple regression, a coefficient is normally interpreted as the change in the conditional mean response for a one-unit predictor increase while what is held constant?

A. The sample size

B. The error variance

C. The other included predictors

D. The test-set error

Correct Answer: C. The other included predictors

Question 3

Which regularization method can set some coefficients exactly to zero?

A. Ridge regression

B. Lasso regression

C. Unregularized OLS

D. Weighted least squares

Correct Answer: B. Lasso regression

Question 4

Why is MAPE problematic when observed responses are zero or close to zero?

A. It always becomes negative

B. It requires normally distributed errors

C. Its denominator makes the percentage unstable or undefined

D. It has squared response units

Correct Answer: C. Its denominator makes the percentage unstable or undefined

Question 5

Which assumption is specifically associated with constant conditional error variance?

A. Homoscedasticity

B. Exogeneity

C. Full rank

D. Normality of predictors

Correct Answer: A. Homoscedasticity

Question 6

For a quadratic model ŷ = 420 − 12x + 0.20x², what is the prediction when x = 25?

A. 120 kWh

B. 245 kWh

C. 300 kWh

D. 545 kWh

Correct Answer: B. 245 kWh

Question 7

In a generalized linear model, what connects the conditional mean to the linear predictor?

A. A link function

B. A leverage score

C. A residual plot

D. A test split

Correct Answer: A. A link function

Question 8

Why is a prediction interval wider than a confidence interval for the mean response at the same predictor value?

A. It includes uncertainty from a new individual error

B. It always uses a larger sample

C. It excludes parameter uncertainty

D. It assumes zero residual variance

Correct Answer: A. It includes uncertainty from a new individual error

Question 9

Which validation strategy is generally most appropriate when predicting future values from past observations?

A. Randomly shuffling all observations before splitting

B. Using the training error only

C. A chronological or rolling-origin split

D. Selecting the model with the highest training R²

Correct Answer: C. A chronological or rolling-origin split

Question 10

What is the primary effect of multicollinearity on ordinary least-squares coefficient estimates?

A. It guarantees biased predictions

B. It can inflate coefficient uncertainty and destabilize interpretation

C. It makes residuals exactly zero

D. It removes the need for an intercept

Correct Answer: B. It can inflate coefficient uncertainty and destabilize interpretation