5. Regression and Continuous-Outcome Prediction

Introduction and Context

Many decisions in business, public policy, engineering, health care and science depend on predicting a quantity rather than assigning a class. Examples include estimating house prices, forecasting electricity demand, predicting customer lifetime value, modelling blood pressure, and estimating the remaining useful life of a machine. These are continuous-outcome problems. Regression provides the principal statistical and machine-learning framework for relating an outcome to one or more explanatory variables and for generating predictions for new observations.

Regression is more than fitting a line through a collection of points. A useful regression analysis requires the analyst to define the prediction target, understand the data-generating process, specify an appropriate functional form, estimate uncertainty, diagnose violations of assumptions, compare models and communicate results responsibly. In predictive analysis, a model may be valuable even when its coefficients are not interpreted causally. Conversely, a statistically significant coefficient does not guarantee accurate predictions or a meaningful causal conclusion.

This chapter develops linear, polynomial, regularized and generalized regression models. It also examines multicollinearity, residual analysis, uncertainty, validation and professional applications. The emphasis is on the connection between mathematical formulation and practical judgement.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain the purpose, terminology and assumptions of regression analysis.

2. Formulate and interpret simple and multiple linear regression models.

3. Distinguish prediction, association and causal interpretation.

4. Use polynomial terms and transformations to represent non-linear relationships.

5. Explain ridge, lasso and elastic-net regularization and select among them appropriately.

6. Diagnose heteroscedasticity, non-linearity, influential observations and non-normal residuals.

7. Interpret confidence intervals, prediction intervals and cross-validation results.

8. Identify and manage multicollinearity, leakage, overfitting and extrapolation risk.

9. Apply regression methods to professional and organizational problems.

10. Evaluate ethical and governance issues in continuous-outcome prediction.

The Regression Problem

Let Y denote a continuous response and let X1, X2, ..., Xp denote predictors. A regression model describes the conditional behaviour of Y given the predictors. In a simple linear model,

Y_i = beta_0 + beta_1 X_i + epsilon_i,

where beta_0 is the intercept, beta_1 is the slope and epsilon_i represents unexplained variation. In multiple regression,

Y_i = beta_0 + beta_1X_i1 + beta_2X_i2 + ... + beta_pX_ip + epsilon_i.

The fitted value for observation i is written as Y-hat_i. The residual is e_i = Y_i - Y-hat_i. Regression coefficients are estimated from a sample, so they are estimates rather than known population quantities.

The ordinary least squares (OLS) method chooses coefficients that minimize the residual sum of squares:

RSS = sum(Y_i - Y-hat_i)^2.

Squaring prevents positive and negative errors from cancelling and gives greater weight to large errors. OLS is computationally convenient and, under appropriate assumptions, has strong statistical properties. However, minimizing training RSS alone can produce an over-complex model that performs poorly on new data.

A central distinction is between explanation and prediction. In an explanatory analysis, the analyst may be interested in the direction and size of an association, subject to a defensible research design. In a predictive analysis, the primary concern is performance on unseen cases. A model can predict well using variables that are not causes of the outcome, while a causal interpretation generally requires stronger assumptions, careful study design and control of confounding.

Interpreting Linear Models

The intercept represents the expected outcome when all predictors equal zero, although that value may have no practical meaning if zero is outside the observed range. In a multiple regression, beta_j is the expected change in Y associated with a one-unit increase in X_j while the other included predictors are held constant. The phrase “holding other variables constant” describes the model’s conditional association; it does not automatically establish causation.

For a binary predictor, the coefficient represents the difference between the reference group and the coded group, conditional on other predictors. Categorical variables with more than two levels are normally represented with indicator variables, with one category serving as the reference. Interaction terms allow the effect of one predictor to depend on another. For example,

Y = beta_0 + beta_1 experience + beta_2 training + beta_3(experience × training) + epsilon.

Here, the training effect changes with experience. Omitting an important interaction can produce misleading average effects.

Model fit is often summarized using R-squared, the proportion of sample variation explained by the fitted model:

R-squared = 1 - RSS/TSS,

where TSS is the total sum of squares around the sample mean. Adjusted R-squared penalizes the addition of predictors and is useful for descriptive comparison, but neither measure is a substitute for test-set performance. A high R-squared may result from leakage, a narrow sampling range or an overfit model.

Assumptions and Their Practical Meaning

Classical inference for linear regression commonly relies on several assumptions. Linearity means that the conditional mean of the outcome is adequately represented by the specified predictors and functional form. Independence means that errors are not systematically related across observations. This assumption is especially important in time series, panel data, repeated measurements and spatial data. Constant variance, or homoscedasticity, means that the spread of errors is approximately stable across fitted values or predictor levels. Normality of errors is mainly important for small-sample tests and intervals, not necessarily for obtaining useful predictions in large samples.

These assumptions concern the errors, not simply the raw variables. A histogram of the response cannot by itself establish whether a regression model is appropriate. Residual plots are usually more informative. A curved residual pattern suggests non-linearity; a funnel shape suggests changing variance; clusters may indicate dependence or omitted groups; and isolated extreme residuals may identify outliers.

When assumptions fail, possible responses include transforming variables, adding interaction or polynomial terms, using robust standard errors, fitting a generalized linear model, modelling dependence explicitly or choosing a different algorithm. The appropriate response depends on whether the goal is inference, prediction or both.

Representing Non-linearity

A linear model can represent non-linear relationships by transforming predictors. Polynomial regression adds powers of a variable:

Y = beta_0 + beta_1X + beta_2X^2 + ... + beta_kX^k + epsilon.

Although the equation is non-linear in X, it is linear in its coefficients and can be estimated by least squares. A quadratic term may capture a U-shaped or inverted-U relationship. Higher-degree polynomials can fit training data closely but tend to be unstable near the boundaries and may oscillate in implausible ways.

Other transformations include log(X), square root, reciprocal and log(Y). If log(Y) is the response, a coefficient can often be interpreted approximately as a percentage change for a one-unit predictor change, depending on the model specification. Transformations should be motivated by subject knowledge, scale, variance behaviour or exploratory analysis, not selected solely because they improve a training score.

Splines and generalized additive models provide more flexible alternatives. A spline fits low-degree polynomials over intervals joined at knots, usually with smoothness constraints. A generalized additive model expresses the expected outcome as a sum of smooth functions, such as E(Y) = beta_0 + f_1(X_1) + f_2(X_2). These methods can reveal non-linear structure while retaining more interpretability than many black-box models.

Regularization and the Bias–Variance Trade-off

When predictors are numerous or strongly correlated, an unregularized model may have unstable coefficients and poor generalization. Regularization adds a penalty to the objective function. Ridge regression minimizes:

RSS + lambda sum(beta_j^2),

where lambda is non-negative and the intercept is usually not penalized. Ridge shrinks correlated coefficients toward one another and is particularly useful when many predictors contribute small amounts of information.

Lasso regression minimizes:

RSS + lambda sum|beta_j|.

The absolute-value penalty can set some coefficients exactly to zero, producing a sparse model that may assist feature selection. However, when predictors are highly correlated, lasso may select one variable arbitrarily and discard others. Elastic net combines the ridge and lasso penalties and can be useful when sparsity and stability are both desired.

Predictors should generally be standardized before regularization because the penalty depends on coefficient magnitude. The value of lambda should be selected using cross-validation within the training data. A larger penalty usually increases bias but reduces variance. This is an example of the bias–variance trade-off: a model that is too simple underfits, while one that is too flexible overfits.

Regularized coefficients are not interpreted in the same way as OLS coefficients without qualification. Shrinkage introduces bias deliberately to improve out-of-sample performance. A lasso-selected variable is not automatically a causal determinant, and exclusion from a penalized model does not prove that a variable is irrelevant.

Generalized Regression Models

Ordinary linear regression assumes a continuous response with an approximately constant error variance. Generalized linear models extend regression to other outcome distributions through three components: a random component describing the response distribution, a systematic component formed from predictors, and a link function connecting the expected outcome to the linear predictor.

For a binary outcome, logistic regression uses the logit link. For counts, Poisson regression uses a log link, although overdispersed counts may require a negative binomial model. For positive, right-skewed continuous outcomes, a Gamma model with a suitable link may be more appropriate than OLS. These models are called generalized regression models even though not all have normally distributed errors.

A link function changes interpretation. In logistic regression, a coefficient is a change in log-odds, and exponentiating it gives an odds ratio. In a Poisson model with a log link, exponentiated coefficients represent multiplicative changes in the expected count. Model choice should follow the outcome’s scale, support and variance structure rather than convenience alone.

Multicollinearity and Variable Structure

Multicollinearity occurs when predictors contain overlapping information. It does not necessarily reduce predictive accuracy, but it can make individual coefficients unstable, inflate standard errors and complicate interpretation. For example, income, occupational status and expenditure may be strongly related. Small changes in the data can then produce substantial changes in their separate coefficients while leaving predictions relatively stable.

A correlation matrix can reveal pairwise relationships, but multicollinearity may also arise from combinations of variables. The variance inflation factor (VIF) is one diagnostic, calculated for each predictor from the R-squared obtained by regressing that predictor on the others. Large VIF values signal potential instability, although no single threshold should be treated as a universal rule.

Responses include removing redundant variables, combining them into an index, using principal components, collecting more informative data or applying ridge regression. Variables should not be removed solely because they are correlated if they are required by theory, policy or operational use. Interpretation must acknowledge the information structure of the predictors.

Evaluating Predictive Performance

A sound predictive workflow separates data into training and test sets. The training data are used to estimate the model; the test data provide an approximately unbiased assessment of performance on unseen observations. When data are limited, k-fold cross-validation repeatedly partitions the training set and averages an evaluation metric. Time-dependent data require chronological splits rather than random shuffling when future information must not influence the past.

Common regression metrics include mean absolute error (MAE),

MAE = mean(|Y_i - Y-hat_i|),

and root mean squared error (RMSE),

RMSE = square root of mean((Y_i - Y-hat_i)^2).

MAE is easier to interpret in the original units and treats errors linearly. RMSE gives greater emphasis to large errors. Mean absolute percentage error can be misleading when actual values are zero or near zero. R-squared on a test set can be negative if predictions are worse than the baseline mean.

Prediction intervals differ from confidence intervals. A confidence interval describes uncertainty about an average or a population parameter. A prediction interval describes the likely range for an individual future outcome and is therefore wider because it includes both parameter uncertainty and individual random variation. Communicating intervals is essential when decisions involve risk rather than merely a point estimate.

A Practical Modelling Workflow

A professional regression project begins with a precise target definition and a clear prediction horizon. The analyst then audits data types, missingness, units, duplicates, outliers and the timing of variables. Predictors that would only become available after the prediction moment must be excluded; otherwise, target leakage produces unrealistic performance.

Exploratory analysis should examine distributions, relationships, group differences and temporal patterns. A baseline model, such as the training-set mean or a simple domain rule, provides a meaningful comparison. The analyst next creates a preprocessing pipeline so that imputation, scaling, transformations and feature selection are learned only from the training folds. Candidate models are fitted and tuned through cross-validation, followed by evaluation on an untouched test set.

Diagnostics then examine residuals, influential observations, calibration, subgroup errors and stability over time. The final model should be selected not only for accuracy but also for interpretability, maintenance cost, fairness, latency and organizational consequences. A model card or equivalent documentation can record data sources, intended use, limitations, metrics and monitoring requirements.

Applied Case: Predicting Delivery Time for an E-commerce Network

Background

The following is an illustrative case. An online retailer wants to predict delivery time in hours at the moment an order is confirmed. The operations team will use the prediction to allocate capacity and communicate a realistic delivery window.

Problem and Data

Historical records contain destination region, warehouse, item category, order time, package weight, number of items, carrier, weather indicator and actual delivery time. The model must not use dispatch delay or later tracking events because these are unavailable at the prediction moment.

Initial exploration shows that delivery time is right-skewed and increases with distance and package weight. Some predictors are correlated: distance, region and carrier choice overlap substantially. Weekend orders also show a different pattern from weekday orders.

Analysis and Approach

The team creates a chronological training set and a later test set to imitate deployment. Missing weather values are imputed inside the modelling pipeline. Numeric variables are standardized for regularized models, and categorical variables are encoded using training data only. A baseline mean prediction is compared with OLS, a polynomial model for distance, ridge regression, lasso and a Gamma generalized linear model.

Residual plots for OLS show increasing variance for longer delivery times. A log transformation of the response and the Gamma model reduce this problem. Ridge regression performs similarly to the more flexible model but has more stable coefficients when distance and region are included together. Five-fold cross-validation is used within the training period, and the final test set is used once for performance reporting.

The operations team chooses the ridge-based model because its MAE is competitive, its behaviour is stable across regions and its coefficients can be explained to planners. Prediction intervals are reported rather than only point estimates. For unusually remote destinations outside the training range, the system flags the prediction as low confidence instead of presenting a precise-looking estimate.

Outcome and Learning Points

The case illustrates that model selection is not a contest based on one metric. The chosen solution balances accuracy, leakage prevention, interpretability, uncertainty and operational usefulness. It also shows why chronological validation, residual diagnosis and explicit treatment of correlated predictors matter in deployment.

Challenges, Risks and Common Mistakes

A frequent error is interpreting association as causation. Another is fitting transformations or selecting variables using the entire dataset before cross-validation, which leaks information from validation folds. Random train-test splitting can also exaggerate performance when observations are temporally or spatially dependent.

Outliers require investigation rather than automatic deletion. They may reflect measurement errors, rare but important events or a different population. Extrapolation is dangerous because a fitted relationship may not continue beyond the observed predictor range. High-degree polynomials, excessive feature engineering and repeated test-set inspection increase overfitting risk.

Missing data can introduce bias when the missingness mechanism is ignored. Imputation should respect the data-generating and deployment process. Distribution shift may occur when customer behaviour, prices, regulations or technology changes. Production models therefore require monitoring of input distributions, residual errors, subgroup performance and calibration.

Ethical, Governance and Professional Considerations

Continuous predictions can influence prices, access to services, credit limits, staffing and medical prioritization. Analysts should document the purpose of the model, affected populations and consequences of error. Fairness cannot be assessed only by aggregate MAE; error distributions should be examined across relevant groups, while respecting privacy and legal requirements.

Sensitive attributes may be excluded yet reconstructed through proxy variables such as location or purchasing patterns. Data minimization, access controls, retention limits and audit trails are important. Predictions should not be presented with unjustified precision. Where decisions materially affect individuals, human review, appeal mechanisms and clear communication of uncertainty may be necessary.

Current Developments and Best Practice

Modern regression systems increasingly combine interpretable statistical models with automated machine-learning pipelines. Gradient-boosted trees and neural networks can capture complex relationships, but they should be compared with well-designed linear and regularized baselines. Explainability tools, conformal prediction and distribution-free uncertainty methods are becoming more common, especially where prediction intervals are needed without strong distributional assumptions.

Best practice is to begin with a transparent baseline, use leakage-resistant validation, inspect residuals, quantify uncertainty, test subgroup performance and document assumptions. A model should be retrained only when monitoring indicates that new data improve expected performance, not merely because automation makes retraining easy.

Chapter Summary

Regression models estimate or predict continuous outcomes from explanatory variables. OLS provides a foundation, but its usefulness depends on functional form, independence, variance and residual behaviour. Polynomial terms, transformations, splines and generalized linear models extend the range of relationships that can be represented. Ridge, lasso and elastic net manage complexity, multicollinearity and high-dimensional predictors through regularization.

Reliable evaluation requires separation of training and test data, appropriate cross-validation, leakage prevention and metrics suited to the decision context. MAE and RMSE measure different types of error, while confidence and prediction intervals express different forms of uncertainty. Residual analysis, VIF diagnostics and influence assessment help identify weaknesses that a single performance score cannot reveal. In professional use, fairness, privacy, governance, monitoring and clear communication are integral parts of regression practice.

Key Terms

Regression: A method for modelling the relationship between an outcome and one or more predictors.

Continuous outcome: A numerical response that can take values on a scale, such as income or temperature.

Predictor: An input variable used to estimate or predict an outcome.

Residual: The observed outcome minus its fitted value.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Intercept: The model’s expected outcome when predictors equal zero.

Coefficient: An estimated parameter describing a predictor’s conditional relationship with the outcome.

R-squared: The proportion of sample outcome variation represented by a fitted model.

Homoscedasticity: Constant error variance across levels of fitted values or predictors.

Heteroscedasticity: Non-constant error variance.

Multicollinearity: Strong dependence or overlap among predictors.

Variance inflation factor: A diagnostic for the inflation of coefficient variance due to predictor dependence.

Polynomial regression: Regression using powers of predictors to represent curvature.

Regularization: Penalization used to control model complexity.

Ridge regression: Regression using an L2, or squared-coefficient, penalty.

Lasso regression: Regression using an L1, or absolute-coefficient, penalty.

Elastic net: A model combining ridge and lasso penalties.

Cross-validation: Repeated partitioning used to estimate generalization performance or tune models.

Prediction interval: A range intended to contain a future individual outcome.

Target leakage: Use of information unavailable at prediction time.

Review Questions

Short Answer Questions

1. Distinguish a residual from a fitted value.

2. What does a multiple-regression coefficient mean when other predictors are held constant?

3. Why are predictors commonly standardized before ridge or lasso regression?

4. State one difference between MAE and RMSE.

5. What does a funnel pattern in a residual plot suggest?

Long Answer Questions

1. Explain the main assumptions of linear regression and discuss how residual diagnostics can reveal their violation.

2. Compare OLS, ridge, lasso and elastic net in terms of objective, coefficient behaviour and practical use.

3. Explain how polynomial regression, transformations, splines and generalized linear models address non-linear or non-normal outcomes.

4. Describe a leakage-resistant workflow for developing and evaluating a regression model.

5. Discuss why multicollinearity affects inference more severely than prediction in some applications.

Analytical / Application Questions

1. A model has excellent training RMSE but poor test RMSE. Diagnose the likely problem and propose a revised modelling procedure.

2. A delivery-time model performs well overall but has much larger errors for remote locations. What analyses and operational changes should be undertaken?

3. Two highly correlated predictors have unstable coefficients, but the model’s predictions remain reliable. Explain this result and recommend an appropriate response.

22. Multiple Choice Questions with Answers

Question 1

In ordinary least squares regression, the fitted coefficients are chosen to minimize which quantity?

A. The sum of absolute prediction errors

B. The sum of squared residuals

C. The variance of the predictors

D. The number of included variables

Correct Answer: B. The sum of squared residuals

Question 2

A funnel-shaped pattern in a residual-versus-fitted plot most strongly suggests:

A. Constant error variance

B. Perfect multicollinearity

C. Heteroscedasticity

D. Absence of influential observations

Correct Answer: C. Heteroscedasticity

Question 3

Which statement best distinguishes a prediction interval from a confidence interval?

A. A prediction interval concerns an individual future outcome and is usually wider

B. A prediction interval is always narrower than a confidence interval

C. A confidence interval measures only residual variance

D. They describe exactly the same uncertainty

Correct Answer: A. A prediction interval concerns an individual future outcome and is usually wider

Question 4

Why can ridge regression be useful when predictors are strongly correlated?

A. It guarantees that all coefficients become zero

B. It removes the need for a test set

C. It tends to stabilize coefficients through shrinkage

D. It converts a continuous outcome into a category

Correct Answer: C. It tends to stabilize coefficients through shrinkage

Question 5

Which model is most directly associated with coefficients becoming exactly zero?

A. Ordinary least squares

B. Lasso regression

C. Ridge regression

D. Unpenalized polynomial regression

Correct Answer: B. Lasso regression

Question 6

A delivery prediction model uses dispatch delay, although dispatch delay is known only after the order is placed. This is an example of:

A. Standardization

B. Target leakage

C. Homoscedasticity

D. Cross-validation

Correct Answer: B. Target leakage

Question 7

What is the principal concern with a high-degree polynomial near the edges of the observed predictor range?

A. It cannot represent curvature

B. It always has lower training fit

C. It may behave unstably and extrapolate poorly

D. It prevents the use of categorical variables

Correct Answer: C. It may behave unstably and extrapolate poorly

Question 8

Which metric gives greater emphasis to large errors?

A. RMSE

B. MAE

C. Median absolute error only

D. The number of predictors

Correct Answer: A. RMSE

Question 9

A high variance inflation factor indicates that:

A. The response is necessarily non-normal

B. A predictor has overlapping information with other predictors

C. The model has no predictive value

D. The test set is too large

Correct Answer: B. A predictor has overlapping information with other predictors

Question 10

For time-dependent data, why is a chronological train-test split often preferable to random shuffling?

A. It guarantees a higher R-squared

B. It prevents the model from using future information to predict the past

C. It removes all missing values

D. It makes residuals normally distributed

Correct Answer: B. It prevents the model from using future information to predict the past