Regression and Continuous-Outcome Prediction

Introduction

Regression is one of the foundational methods of predictive analytics. It estimates how a numerical outcome changes in relation to one or more explanatory variables and uses that relationship to produce predictions. Organizations use regression to forecast sales, estimate customer lifetime value, predict delivery time, assess credit losses, model energy consumption, and quantify the effects of pricing, advertising, staffing, or operational conditions.

Regression is valuable because it combines prediction with interpretation. A well-designed model can answer both “What is likely to happen?” and “Which variables are associated with the outcome?” These are different questions. A model may predict accurately without supporting a causal conclusion. Consequently, regression analysis requires statistical reasoning, domain knowledge, careful validation, and awareness of ethical limitations.

This chapter develops simple linear, multiple, polynomial, and regularized regression. It explains estimation, interpretation, assumptions, prediction errors, model evaluation, implementation choices, and professional applications.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and structure of regression models.
2. Fit and interpret simple and multiple linear regression.
3. Distinguish correlation, prediction, and causation.
4. Use polynomial terms to represent nonlinear relationships.
5. Explain the purpose of ridge and lasso regularization.
6. Test important regression assumptions using numerical and graphical evidence.
7. Calculate and interpret MAE, MSE, RMSE, R², and adjusted R².
8. Develop a defensible regression workflow for a business problem.
9. Identify ethical, operational, and governance risks in continuous-outcome prediction.

1. The Regression Framework

Suppose an analyst wants to predict monthly sales from advertising expenditure. Sales is the response, target, or dependent variable. Advertising expenditure is a predictor, feature, or independent variable. Regression estimates a function linking predictors to the conditional mean of the response.

In simple linear regression, the population model is:

Here, Y_i is the outcome for observation i, X_i is the predictor, β_0 is the intercept, β_1 is the slope, and ε_i is the error term representing influences not included in the model. The equation describes a population relationship; the fitted sample equation is:

where ŷ_i is the predicted outcome and b_0 and b_1 are estimated coefficients. The residual is:

The slope has units of outcome per unit of predictor. For example, if sales is measured in thousands of dollars and advertising in thousands of dollars, a slope of 2.4 means that predicted sales increase by 2.4 thousand dollars for each additional thousand dollars of advertising, within the observed range and subject to the model’s assumptions.

Regression does not automatically establish causality. If advertising spending is higher in large markets, an observed association between advertising and sales may partly reflect market size. Causal interpretation normally requires experimental or carefully designed observational evidence.

Estimating the line by least squares

Ordinary least squares (OLS) chooses coefficients that minimize the sum of squared residuals:

For simple linear regression, the estimated slope and intercept are:

where x̄ and ȳ are sample means. These formulas apply when the denominator is nonzero; there must be variation in X. OLS minimizes vertical prediction errors in the response scale. It does not minimize horizontal distances and is not automatically appropriate when both variables are measured with substantial error.

A small worked example

Assume three observations have predictor values X = 1, 2, 3 and outcomes Y = 2, 3, 5. The means are x̄ = 2 and ȳ = 10/3. The numerator for b_1 is:

(1 − 2)(2 − 10/3) + (2 − 2)(3 − 10/3) + (3 − 2)(5 − 10/3) = 3.

The denominator is (1 − 2)² + 0² + (3 − 2)² = 2. Thus b_1 = 1.5 and b_0 = 10/3 − 1.5(2) = 1/3. The fitted model is ŷ = 1/3 + 1.5X. At X = 4, the extrapolated prediction is 6.33. Because 4 lies outside the observed range 1 to 3, this prediction should be treated cautiously.

2. Multiple Linear Regression

A multiple regression model uses several predictors:

The coefficient β_j represents the expected change in Y associated with a one-unit increase in X_j, holding the other included predictors constant. The phrase “holding other predictors constant” is essential. It distinguishes a multiple-regression coefficient from a simple bivariate association.

For example:

If β_1 = 0.18 and price is measured in thousands of dollars while size is measured in hundreds of square metres, the model associates one additional hundred square metres with an estimated 0.18 thousand-dollar increase in price, conditional on bedrooms and age. The interpretation is meaningful only if the units and coding are understood.

Categorical variables can be incorporated through indicator variables. If region has categories North, South, and West, one category is usually selected as the reference group. For example, a South indicator equals 1 for South observations and 0 otherwise. Its coefficient compares South with the reference region, conditional on the other predictors. Including an indicator for every category along with an intercept creates perfect multicollinearity, known as the dummy-variable trap.

Matrix representation

Multiple regression is compactly represented as:

The OLS estimator is:

This expression requires XᵀX to be invertible. Exact or severe multicollinearity can make the inverse unavailable or make estimates unstable. The vector b contains the intercept and slope estimates, X is the design matrix, y is the outcome vector, and ε is the error vector. In practical software, numerically stable decompositions are preferred to explicitly calculating a matrix inverse.

3. Nonlinearity and Polynomial Regression

A straight line may be inadequate when the outcome rises rapidly at first and then levels off, or when it falls and later rises. Polynomial regression adds powers of a predictor:

Although the curve is nonlinear in X, the model remains linear in its coefficients β. A quadratic model has the form:

Its instantaneous rate of change is:

Thus b_1 alone is not the overall effect of X. If b_2 is negative, the curve is concave downward; if positive, it is concave upward. Polynomial degree should be chosen using validation performance, domain reasoning, and residual diagnostics—not by maximizing training fit.

Raw powers can be highly correlated, especially at high degree. Centering X before creating powers, such as z = X − x̄, often improves interpretability and numerical stability. Polynomial models also become unreliable when extrapolated beyond the training range because their tails can grow rapidly.

4. Regularized Regression

When predictors are numerous, correlated, or more numerous than observations, OLS can overfit or produce unstable coefficients. Regularization adds a penalty to the loss function.

Ridge regression

Ridge regression minimizes:

The penalty parameter λ ≥ 0 controls shrinkage. The intercept is generally not penalized. When λ = 0, ridge is OLS; as λ increases, coefficients are pulled toward zero but usually not exactly to zero. Ridge is useful when many correlated predictors each contribute modest information.

Lasso regression

Lasso minimizes:

The absolute-value penalty can set some coefficients exactly to zero, providing a form of embedded feature selection. With strongly correlated predictors, lasso may select one variable and discard another similar variable somewhat arbitrarily. Elastic net combines ridge and lasso penalties and can be useful when both shrinkage and grouped selection are desired.

Predictors should normally be standardized before regularization so that the penalty treats variables comparably. Standardization commonly uses:

where s_j is the sample standard deviation. Scaling must be learned from the training data only; otherwise information from the test set leaks into model development. The value of λ should be selected using cross-validation within the training data.

5. Evaluating Continuous Predictions

A model should be judged on observations not used to fit it. A basic train–test split separates data into training and test sets. Cross-validation repeatedly divides the training data into folds, fits the model on some folds, and evaluates it on the remaining fold. The final test set should be used sparingly for an approximately unbiased estimate of final generalization performance.

Let y_i be the actual value, ŷ_i the prediction, and n the number of evaluated observations.

Mean absolute error

MAE is in the same units as the outcome and gives equal linear weight to errors. It is easy to communicate and relatively less sensitive to extreme errors than squared-error measures.

Mean squared error

MSE is in squared outcome units. Because errors are squared, large mistakes receive disproportionate weight.

Root mean squared error

RMSE returns to the outcome’s units while retaining greater sensitivity to large errors. MAE and RMSE answer different managerial questions: MAE describes typical absolute error, whereas RMSE emphasizes costly large misses.

R-squared

where SSE = ∑(y_i − ŷ_i)² and SST = ∑(y_i − ȳ)². In models with an intercept, R² is the proportion of sample variation around ȳ explained by the fitted model. On unseen data, R² can be negative if predictions are worse than the baseline mean predictor. A high R² does not prove causality or guarantee useful business accuracy.

Adjusted R² accounts for the number of predictors:

where p is the number of predictors excluding the intercept. It is mainly a descriptive in-sample measure and should not replace out-of-sample evaluation.

Worked metric example

Suppose actual outcomes are 10, 12, and 15, while predictions are 11, 10, and 14. Errors are −1, 2, and 1. Absolute errors sum to 4, so MAE = 4/3 = 1.33. Squared errors sum to 1 + 4 + 1 = 6, so MSE = 2 and RMSE = √2 ≈ 1.41. The units of MAE and RMSE are outcome units; MSE has squared units.

6. Assumptions and Diagnostics

Assumptions concern the errors, not merely the raw variables.

Linearity. The conditional mean should be adequately represented by the chosen functional form. A residual-versus-fitted plot with systematic curvature suggests missing nonlinear structure.

Independence. Errors should not be systematically related across observations. Time series, repeated customers, stores, or geographic units may violate this assumption. Random splitting can be inappropriate when observations are temporal or grouped.

Constant variance. Homoscedasticity means that error variance is approximately constant across fitted values or predictor levels. A funnel-shaped residual plot indicates heteroscedasticity. Transforming the outcome, using weighted least squares, or using robust standard errors may help, depending on the purpose.

Normality of errors. Normally distributed errors are especially relevant for small-sample confidence intervals and hypothesis tests. Normality is not required for producing a point prediction, although extreme non-normality may indicate misspecification or outliers.

Limited influence of outliers. An outlier has an unusual outcome; a high-leverage point has unusual predictor values; an influential point materially changes the fitted model. Cook’s distance and leverage statistics can identify observations for investigation. They should not be deleted automatically: an unusual observation may be a valid and important business event.

No harmful multicollinearity. Correlated predictors can inflate standard errors and make coefficients unstable. Variance inflation factors are often used diagnostically. A common rule of thumb is that large values, frequently above 5 or 10 depending on context, warrant investigation rather than automatic removal. Correlation does not necessarily harm prediction, but it complicates interpretation.

Residual diagnostics should combine plots, domain knowledge, and quantitative checks. Statistical tests are not substitutes for substantive reasoning, particularly in large samples where trivial departures may become statistically significant.

7. A Professional Modeling Workflow

A defensible workflow is iterative:

1. Define the decision. Specify the outcome, prediction horizon, unit of analysis, acceptable error, and action triggered by a prediction.
2. Audit the data. Check missingness, duplicates, units, impossible values, timestamps, and target leakage.
3. Create a valid split. Use temporal or grouped splitting when deployment involves future periods or new entities.
4. Establish a baseline. For many problems, the baseline is the training-set mean or a domain-specific forecast.
5. Explore relationships. Use scatterplots, summaries, grouped comparisons, and domain hypotheses.
6. Fit a transparent benchmark. Start with linear regression before adding complexity.
7. Engineer features carefully. Encode categories, transform skewed variables, and construct polynomial or interaction terms when justified.
8. Tune with cross-validation. Select polynomial degree, regularization strength, and other choices without using the final test set.
9. Diagnose errors. Examine residual patterns, subgroup performance, calibration, and influential records.
10. Communicate uncertainty and limitations. Report performance in operational units and explain where the model should not be used.
11. Monitor after deployment. Track data drift, outcome drift, error metrics, fairness indicators, and retraining triggers.

A pipeline should fit preprocessing steps only on training data. For example, imputation, scaling, and category encoding must be learned inside each cross-validation fold. This prevents leakage and produces realistic validation results.

8. Illustrative Case: Retail Delivery-Time Prediction

Background. A fictional online retailer wants to predict delivery time in hours so that it can communicate reliable arrival estimates and identify orders at risk of delay.

Problem. Historical data contain distance, package weight, number of items, warehouse workload, carrier, order hour, weather indicator, and actual delivery time. The data include repeated orders from the same warehouses and several months of records.

Analysis. The team defines the target as hours from dispatch to delivery. Because the model will predict future orders, the last month is reserved as a chronological test set. Earlier data are divided into training and validation periods. The team discovers that a “delivery status” field recorded after dispatch would reveal the outcome and removes it as target leakage.

A baseline predicts the mean delivery time. A multiple linear model provides a transparent benchmark. Scatterplots show that distance has a nonlinear relationship with time: very short routes take a minimum handling time, while long routes increase more rapidly. The team therefore tests a centered quadratic distance term. Workload and weather are also included. Carrier is represented with indicator variables, using the largest carrier as the reference.

The team standardizes numeric predictors and compares OLS, ridge, and lasso using time-aware cross-validation. Ridge performs best on validation RMSE because distance, workload, and package characteristics are correlated. Lasso removes several weak variables, but its validation error is slightly higher. The final model is selected based on test performance, interpretability, and operational cost—not on training R².

Outcome. Suppose the selected model has test MAE of 2.1 hours and RMSE of 3.8 hours. The difference indicates that a minority of orders have substantially larger errors. Residual analysis shows that severe storms produce underpredictions, suggesting a missing interaction between weather and carrier or a need for a separate disruption model. Performance is also worse for rural destinations, so the company reports subgroup metrics rather than only an overall average.

The model is used to set customer-facing delivery windows, but not to deny service or penalize drivers. Predictions are accompanied by an uncertainty band or conservative buffer. Monitoring checks whether route patterns, carrier mix, and weather conditions change over time.

Learning points. The case demonstrates that model quality depends on temporal validation, leakage prevention, nonlinear specification, regularization, subgroup analysis, and alignment with the decision. A lower average error is not enough if the model fails systematically for a group or under important operating conditions.

9. Interpretation, Governance, and Ethical Issues

Regression coefficients can be misunderstood as causal effects, guarantees, or universal laws. Reports should distinguish association from intervention impact and should state the population, time period, and feature range for which conclusions are supported.

Fairness concerns arise when predictions affect access to prices, credit, insurance, employment, or public services. A continuous prediction may be unbiased on average while still producing systematically larger errors for a subgroup. Analysts should examine group-specific MAE, RMSE, calibration, missingness, and error direction. Sensitive attributes should not be removed reflexively: excluding them may prevent fairness auditing and may leave proxy variables untouched.

Privacy, data minimization, access control, retention limits, and appropriate consent are part of responsible modeling. Predictions should be explainable enough for affected users and decision-makers to understand the principal drivers, limitations, and avenues for review. Governance should document data sources, transformations, model versions, training dates, validation design, owners, and permitted uses.

Contemporary tools can automate feature engineering, hyperparameter tuning, and monitoring, but automation does not remove the need for statistical judgment. More complex machine-learning models may outperform regression, yet regression remains valuable as a benchmark, an interpretable model, and a diagnostic instrument. A responsible analyst compares alternatives using the same leakage-free evaluation design and considers operational consequences, not only a single score.

Challenges and Limitations

Regression can fail when important variables are omitted, relationships change over time, measurement error is substantial, or training data do not represent deployment conditions. Extrapolation is particularly dangerous. Multicollinearity may make individual coefficients unstable even when predictions are adequate. A statistically significant coefficient may be too small to matter commercially, while a practically important effect may be estimated imprecisely in a small sample. Finally, optimizing RMSE may encourage behavior that reduces large errors but worsens the typical customer experience. Metric selection must reflect the decision’s cost structure.

Best Practices

Use a clearly defined target and prediction time; prevent leakage; preserve units and document transformations; begin with a baseline and transparent model; choose splits that mimic deployment; standardize variables before ridge or lasso; tune hyperparameters within training data; inspect residuals and subgroup errors; avoid unjustified extrapolation; report MAE and RMSE together when appropriate; distinguish prediction from causation; and establish monitoring, review, and retirement procedures before deployment.

Chapter Summary

Regression models estimate continuous outcomes from one or more predictors. Simple linear regression uses a line, while multiple regression adjusts each coefficient for the other included variables. Polynomial terms represent curvature but require careful validation and should not be trusted for unbounded extrapolation. Ridge and lasso add penalties that reduce overfitting; ridge generally retains correlated variables, whereas lasso can set coefficients to zero.

MAE measures average absolute error in outcome units. MSE squares errors, and RMSE expresses that squared-error emphasis in the original units. R² describes explained variation in a specified sample, while adjusted R² accounts for model size. None of these measures alone proves that a model is useful or causal.

Reliable regression practice includes appropriate data splitting, leakage prevention, cross-validation, residual diagnostics, attention to outliers and multicollinearity, subgroup evaluation, and monitoring after deployment. Ethical use requires privacy protection, transparency, fairness assessment, governance, and careful limitation of decisions influenced by predictions.

Key Terms

Regression: A method for modeling or predicting a numerical outcome from explanatory variables.

Response variable: The outcome a model predicts or explains.

Predictor: An input variable used to estimate the response.

Intercept: The predicted outcome when all numeric predictors equal zero, subject to the model’s coding and applicability.

Slope: The estimated change in the response associated with a one-unit predictor change.

Residual: The observed outcome minus its fitted prediction.

Ordinary least squares: Estimation that minimizes the sum of squared residuals.

Multiple regression: Regression using two or more predictors.

Polynomial regression: Regression containing powers of a predictor to represent curvature.

Multicollinearity: Strong dependence among predictors that can destabilize coefficient estimates.

Regularization: Penalization used to control model complexity and overfitting.

Ridge regression: L2-penalized regression that shrinks coefficients toward zero.

Lasso regression: L1-penalized regression that may set coefficients exactly to zero.

MAE: Mean absolute error, the average absolute prediction error.

MSE: Mean squared error, the average squared prediction error.

RMSE: The square root of MSE, expressed in outcome units.

R²: A measure based on the reduction in squared error relative to the mean baseline.

Heteroscedasticity: Nonconstant variance of regression errors.

Data leakage: Use of information unavailable at prediction time or improperly shared across validation boundaries.

Extrapolation: Prediction outside the range of predictor values used for model fitting.

Review Questions

Short Answer Questions

1. What is the difference between a response variable and a predictor?
2. Why does a multiple-regression coefficient include the phrase “holding other predictors constant”?
3. State one important difference between MAE and RMSE.
4. What problem does regularization address?
5. Why can extrapolation be dangerous in regression?

Long Answer Questions

1. Explain OLS estimation in simple linear regression and interpret the slope in business terms.
2. Compare ridge and lasso regression, including their penalties, coefficient behavior, and suitable applications.
3. Discuss the major regression assumptions and explain how residual diagnostics can reveal violations.
4. Design a complete regression workflow for a business forecasting problem, from target definition through deployment monitoring.
5. Explain why predictive accuracy does not establish causality, using a suitable business example.

Analytical / Application Questions

1. A model has MAE of 4 units and RMSE of 9 units. What does the difference suggest about its errors, and what additional diagnostic would you perform?
2. A retailer’s training R² rises whenever more polynomial terms are added, but test RMSE initially falls and then rises. Explain the pattern and recommend a model-selection approach.
3. A delivery model performs well overall but has twice the MAE for rural customers. Propose an investigation and discuss possible modeling, data, and governance responses.

Multiple Choice Questions with Answers

Question 1

In a simple regression model, what does the slope represent?

A. The average outcome when the predictor equals zero

B. The expected change in outcome for a one-unit predictor increase

C. The proportion of unexplained variation

D. The largest observed prediction error

Correct Answer: B. The expected change in outcome for a one-unit predictor increase

Question 2

Which objective is minimized by ordinary least squares?

A. The sum of absolute residuals

B. The maximum residual

C. The sum of squared residuals

D. The correlation between predictors

Correct Answer: C. The sum of squared residuals

Question 3

Actual outcomes are 10, 12, and 15; predictions are 11, 10, and 14. What is the RMSE?

A. 1.00

B. 1.33

C. 1.41

D. 2.00

Correct Answer: C. 1.41

Question 4

Which metric is expressed in the original units of the response and gives linear weight to absolute errors?

A. MAE

B. MSE

C. R²

D. Adjusted R²

Correct Answer: A. MAE

Question 5

What is a typical effect of increasing the ridge penalty λ?

A. Coefficients are generally shrunk toward zero

B. All coefficients become exactly zero immediately

C. The intercept is always removed

D. The training sample size increases

Correct Answer: A. Coefficients are generally shrunk toward zero

Question 6

Which regularization method can set some coefficients exactly to zero?

A. Ordinary least squares

B. Ridge regression

C. Lasso regression

D. Simple averaging

Correct Answer: C. Lasso regression

Question 7

A funnel-shaped residual-versus-fitted plot most directly suggests:

A. Constant variance

B. Heteroscedasticity

C. Perfect multicollinearity

D. A categorical response

Correct Answer: B. Heteroscedasticity

Question 8

Why should scaling be fitted within the training data during cross-validation?

A. To increase the number of predictors

B. To prevent information leakage from validation data

C. To guarantee a higher R²

D. To remove the intercept

Correct Answer: B. To prevent information leakage from validation data

Question 9

In a multiple regression model, a coefficient is usually interpreted as:

A. A causal effect in every situation

B. The change associated with a predictor while other included predictors are held constant

C. The response variance divided by sample size

D. The model’s total number of observations

Correct Answer: B. The change associated with a predictor while other included predictors are held constant

Question 10

Why is a chronological test split appropriate for a model predicting future delivery times?

A. It ensures future information is not used to predict the past

B. It always produces the lowest RMSE

C. It removes all outliers

D. It makes polynomial terms unnecessary

Correct Answer: A. It ensures future information is not used to predict the past