Regression Models

Introduction

Regression is a supervised machine-learning method used to estimate a numerical outcome from one or more explanatory variables. Examples include predicting house prices, electricity demand, delivery time, customer spending, and crop yield. Unlike classification, whose output is a category, regression produces a continuous value.

Regression is important in data science for two related reasons. First, it supports prediction. Second, it provides an interpretable description of how predictors are associated with an outcome. A useful regression model must therefore be judged not only by predictive accuracy but also by the quality of its assumptions, the reliability of its evaluation, and the appropriateness of its interpretation.

This chapter develops simple linear regression, multiple linear regression, polynomial regression, and regularized regression. It also explains model assumptions, residual analysis, prediction metrics, practical implementation, and professional limitations.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish a response variable from explanatory variables;
• fit and interpret simple and multiple linear regression;
• explain polynomial features and their risks;
• identify important regression assumptions using residual diagnostics;
• calculate and interpret MAE, MSE, RMSE, and R-squared;
• explain ridge and lasso regularization;
• evaluate regression models using suitable training and test procedures; and
• communicate predictions without overstating causality or certainty.

1. Regression as a predictive model

Let yᵢ denote the observed response for observation i, and let xᵢ₁, xᵢ₂, …, xᵢₚ denote p predictors. A regression model describes the conditional mean of the response as a function of the predictors:

Here, E(·) is the expected or average value. In practice, the fitted model produces a prediction ŷᵢ, while the difference between the observed and predicted values is the residual:

The response must be numerical. Predictors may be numerical, binary, or categorical after suitable encoding. A categorical predictor with k categories is commonly represented by k − 1 dummy variables when an intercept is included. The omitted category becomes the reference group.

Regression can be used for interpolation, prediction within the range of observed data, and—in carefully designed studies—estimation of associations. Prediction outside the observed range is extrapolation and is usually less reliable.

2. Simple linear regression

Simple linear regression uses one predictor x to model a response y:

The fitted equation is:

where β₀ and β₁ are population parameters, b₀ and b₁ are estimates, and εᵢ is the random error. The intercept b₀ is the predicted response when x = 0. It may have no practical interpretation if x = 0 is impossible or outside the data range. The slope b₁ is the estimated average change in y for a one-unit increase in x.

The ordinary least-squares method chooses b₀ and b₁ to minimize the residual sum of squares:

For simple regression, the least-squares estimates are:

where x̄ and ȳ are the sample means. These formulas require variation in x; if all x values are identical, the denominator is zero and the slope cannot be estimated.

Worked example: advertising and sales

Suppose weekly advertising expenditure x, measured in thousands of dollars, and sales y, measured in thousands of units, produce the fitted model:

For an advertising expenditure of 4 thousand dollars:

The predicted sales are 26 thousand units. The slope means that, within the observed data and under the model, an additional thousand dollars of advertising is associated with an average increase of 3.5 thousand units in predicted sales. It does not, by itself, prove that advertising causes the increase.

3. Multiple linear regression

Multiple linear regression uses several predictors:

The fitted version is:

The coefficient bⱼ represents the estimated change in the response for a one-unit increase in xⱼ, holding the other included predictors constant. The phrase “holding other predictors constant” is essential. It distinguishes a multiple-regression coefficient from a simple correlation or an uncontrolled comparison.

In matrix notation, let y be the n × 1 response vector, X the n × (p + 1) design matrix including a column of ones, and β the coefficient vector. Ordinary least squares estimates β by minimizing:

When XᵀX is invertible, the least-squares estimate is:

The inverse exists only when the columns of X are linearly independent. Strong or exact relationships among predictors create multicollinearity. In practice, numerical software often uses more stable matrix decompositions rather than explicitly calculating the inverse.

Interpretation of categorical predictors

Suppose a model predicts salary using years of experience and department, with sales as the reference department:

The coefficient b₂ compares engineering with sales for employees with the same experience, assuming the model is appropriate. It is not a comparison with all departments simultaneously.

Interaction terms

Sometimes the effect of one predictor depends on another. An interaction model is:

The marginal association of x₁ is β₁ + β₃x₂. Therefore, β₁ alone is the effect of x₁ only when x₂ = 0. Centering variables can make the reference point more meaningful, although centering does not remove multicollinearity between genuinely related variables.

4. Polynomial regression

A straight line may be too restrictive when the relationship bends. Polynomial regression adds powers of a predictor:

Although powers are included, the model is linear in its coefficients β₀, β₁, …, βₖ, so ordinary least squares can still be used. A quadratic model has k = 2. Its slope at a particular x is:

Thus, the effect of x is not constant. For example, in a demand model, a negative coefficient on x² may indicate that demand rises at first but eventually declines.

Polynomial regression should be used cautiously. High-degree polynomials can fit random fluctuations, produce unstable edge behaviour, and extrapolate implausibly. Scaling or centering x often improves numerical stability. A polynomial feature should be generated using training data only within a proper machine-learning pipeline, preventing leakage from the test set.

5. Regularization

Ordinary least squares minimizes squared errors, but a model with many correlated or weak predictors may have large, unstable coefficients. Regularization adds a penalty for coefficient size. The intercept is usually not penalized.

Ridge regression

Ridge regression minimizes:

where λ ≥ 0 controls the penalty. When λ = 0, ridge regression is ordinary least squares. As λ increases, coefficients are shrunk toward zero, but generally not exactly to zero. Ridge is useful when predictors are numerous and correlated. Predictors should normally be standardized because the penalty depends on coefficient magnitude and therefore on measurement scale.

Lasso regression

Lasso regression minimizes:

The absolute-value penalty can set some coefficients exactly to zero, making lasso useful for embedded feature selection. However, with highly correlated predictors, lasso may select one variable and discard another similar variable somewhat arbitrarily.

Choosing λ

The regularization parameter is not normally selected by looking at test performance repeatedly. Instead, candidate values are evaluated using cross-validation on the training data. The final model is then fitted using the chosen value and assessed once on an untouched test set. Ridge and lasso are sensitive to scaling, so standardization must be fitted on training data and applied unchanged to validation and test data.

6. Assumptions and diagnostics

Assumptions concern the data-generating process and the validity of particular inferences. They are not requirements that every scatterplot be perfectly linear.

Linearity

The conditional mean should be adequately represented by the selected predictors. A residual-versus-fitted plot should show no systematic curve. Curvature suggests transformations, polynomial terms, interactions, or a different model.

Independence

Observations or errors should be independent for standard errors and many inferential procedures. Time-series observations, repeated measurements, and clustered data often violate this assumption. Random splitting is also inappropriate when future observations must be predicted from past observations; a time-based split is preferable.

Constant variance

Homoscedasticity means that the error variance is approximately constant across fitted values or predictor levels:

A funnel-shaped residual plot indicates heteroscedasticity. Predictions may still be useful, but conventional standard errors and confidence intervals may be unreliable. Possible responses include transforming the response, using weighted least squares, or using robust standard errors.

Error distribution

For least-squares prediction, normally distributed errors are not required for the coefficients to be computed. Approximate normality is more relevant to small-sample t-tests, confidence intervals, and prediction intervals. A Q–Q plot can reveal skewness or heavy tails.

Multicollinearity

Multicollinearity occurs when predictors contain overlapping information. It can make coefficients unstable and inflate standard errors. Correlation matrices, coefficient sensitivity, and the variance inflation factor are useful diagnostics. For predictor j:

Here Rⱼ² is obtained by regressing predictor j on the other predictors. A VIF of 1 indicates no linear relationship with the other predictors. There is no universal cutoff, but very large values warrant investigation rather than automatic deletion.

Outliers and influential observations

An outlier has an unusual response or predictor value. A high-leverage observation has an unusual predictor combination. An influential observation substantially changes the fitted model when removed. Residuals, leverage, studentized residuals, and Cook’s distance can help identify such cases. An observation should not be deleted solely because it is inconvenient; its measurement quality, context, and population relevance must be investigated.

7. Evaluating regression predictions

Use predictions from a validation or test set, not only training data. Training performance is usually optimistic because the model has already seen those observations.

Mean absolute error

MAE is in the same units as y and represents average absolute error. It is relatively less sensitive to extreme errors than squared-error measures.

Mean squared error

MSE is in squared response units. It penalizes large errors more strongly than MAE and is the objective minimized by ordinary least squares.

Root mean squared error

RMSE returns to the units of y and gives greater influence to large errors. It is often useful when serious large mistakes are especially costly.

R-squared

For a dataset containing an intercept, the coefficient of determination is:

where:

and

R² measures the proportion of variation around the sample mean reduced by the model on the evaluated data. It can be negative on test data when predictions are worse than simply predicting the training mean. A high R² does not prove causation, fairness, or useful future performance.

Small numerical illustration

Suppose actual values are 10, 12, and 14, while predictions are 11, 10, and 15. The errors are −1, 2, and −1, so:

The units of MAE and RMSE are the units of the response; MSE has squared units.

8. A professional modelling workflow

A defensible regression project commonly follows these stages:

1. Define the prediction target, decision context, time horizon, and acceptable error.
2. Inspect data types, missing values, duplicates, distributions, and potential leakage.
3. Separate predictors from the response before preprocessing.
4. Split data into training and test sets, using a time-aware or grouped split when required.
5. Establish a baseline, such as predicting the training mean.
6. Build a simple interpretable model before adding complexity.
7. Encode categorical variables and scale features within a pipeline.
8. Diagnose residuals, nonlinear patterns, collinearity, and influential observations.
9. Tune polynomial degree or regularization using cross-validation on training data.
10. Evaluate once on the untouched test set using several relevant metrics.
11. Report uncertainty, limitations, subgroup performance, and the intended use of the model.
12. Monitor performance after deployment because data distributions and relationships can change.

9. Applied case: predicting residential energy use

Background

The following is an illustrative case. A utility company wants to predict monthly electricity consumption for residential customers so that it can plan capacity and identify unusually high expected demand.

Problem

The available variables include floor area, average outdoor temperature, number of occupants, heating-system type, and previous-month consumption. The company wants accurate predictions, but it also needs explanations that customer-service staff can understand.

Analysis

A scatterplot shows that consumption increases with floor area but that the relationship is not perfectly linear. Temperature has a curved relationship: consumption is high during very cold and very hot months and lower near a moderate temperature. Previous-month consumption is strongly predictive, but it may be unavailable when predicting a new customer.

A baseline model predicts the training mean. A multiple linear model is then fitted using floor area, occupants, heating type, and temperature. A quadratic temperature term is added because residual plots from the linear model show systematic curvature. All preprocessing is performed inside a cross-validation pipeline.

Because floor area and the number of rooms are strongly correlated, including both creates unstable coefficients. The team either removes one variable for interpretability or uses ridge regression. The regularization value is selected through five-fold cross-validation. Evaluation is performed on a later-month test set to reflect the deployment situation.

Approach and outcome

The final model is selected using test-set MAE, RMSE, residual diagnostics, and operational requirements. Suppose the test MAE is 42 kWh and RMSE is 68 kWh. The difference indicates that a small number of homes have substantially larger errors. Those cases are investigated rather than hidden: some involve electric-vehicle charging and others contain faulty meter readings.

The model is documented as predictive, not causal. For example, a positive coefficient for occupants does not prove that adding an occupant causes a fixed increase in consumption under all circumstances. Staff receive prediction intervals or error ranges where possible, and performance is checked across housing types to detect unequal accuracy.

Learning points

The case demonstrates that model selection is not only a contest for the largest R². Data availability at prediction time, leakage prevention, nonlinear relationships, correlated predictors, error costs, data quality, and subgroup performance all affect whether a regression system is professionally suitable.

10. Limitations, ethics, and governance

Regression models can encode historical inequalities. If salary, credit, healthcare, or service-access data reflect discriminatory decisions, a highly accurate model may reproduce those patterns. Practitioners should examine whether predictors are justified, whether sensitive attributes are being used directly or indirectly, and whether errors differ among relevant groups.

Privacy must be protected through lawful collection, purpose limitation, access controls, retention policies, and appropriate de-identification. De-identification does not guarantee anonymity when datasets can be linked. Models used in high-impact decisions require documentation, human oversight, contestability, and monitoring.

Interpretability also has limits. A coefficient describes a model association under its assumptions; it is not automatically a causal effect. Causal claims require an appropriate research design, such as randomization or a credible quasi-experimental method. Explainability tools may describe model behaviour but cannot repair biased data or prove causality.

Contemporary practice increasingly combines regularized linear models with automated pipelines, drift monitoring, uncertainty estimation, and responsible-AI review. More complex models may improve accuracy, but a transparent linear model can be preferable when auditability, stability, and communication are central requirements.

Best practices

• Begin with a meaningful baseline and a transparent model.
• Keep preprocessing, feature generation, and scaling inside a reproducible pipeline.
• Fit preprocessing only on training data.
• Use MAE and RMSE together when both typical and large errors matter.
• Interpret R² as explained sample variation, not as proof of usefulness or causation.
• Inspect residuals rather than relying on a single score.
• Standardize predictors before ridge or lasso.
• Use cross-validation for model selection and reserve the test set for final assessment.
• Investigate unusual observations using domain knowledge.
• Report units, data period, sample definition, uncertainty, and limitations.
• Check model performance across important subgroups and over time.

Chapter Summary

Regression models estimate numerical outcomes from explanatory variables. Simple linear regression uses one predictor and a straight-line relationship; multiple regression extends the model to several predictors and interprets each coefficient conditional on the others. Polynomial regression represents curvature by adding powers of predictors while remaining linear in its coefficients. Ridge and lasso add penalties to control coefficient size, with lasso also capable of setting coefficients to zero.

Ordinary least squares minimizes squared residuals. Reliable modelling requires attention to linearity, independence, constant variance, error behaviour, multicollinearity, outliers, leverage, and data leakage. MAE measures average absolute error, MSE emphasizes large errors, RMSE expresses squared-error performance in response units, and R² compares model variation with total variation. No metric is sufficient alone. Professional regression work combines sound validation, diagnostic analysis, domain knowledge, ethical governance, and clear communication of uncertainty.

Key Terms

Response variable: The numerical outcome being predicted or explained.

Predictor: An input variable used to estimate the response.

Residual: The observed value minus its fitted prediction.

Intercept: The fitted response when all predictors equal zero.

Slope: The coefficient describing the change associated with a one-unit predictor increase.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Simple linear regression: Regression with one predictor.

Multiple linear regression: Regression with two or more predictors.

Polynomial regression: Regression containing powers of one or more predictors.

Interaction: A term representing dependence of one predictor’s effect on another predictor.

Multicollinearity: Strong linear dependence among predictors.

Regularization: Penalization used to reduce model complexity or coefficient instability.

Ridge regression: L2-penalized regression that shrinks coefficients toward zero.

Lasso regression: L1-penalized regression that can produce zero coefficients.

Cross-validation: Repeated training-validation assessment used for model selection.

MAE: Mean absolute prediction error.

MSE: Mean squared prediction error.

RMSE: Square root of mean squared error.

R-squared: The proportion of sample response variation represented by a fitted model.

Data leakage: Use of information during training that would not be available at prediction time.

Review Questions

Short Answer Questions

1. What is the difference between a response variable and a predictor?
2. Interpret the slope in a simple linear regression model.
3. Why should predictors usually be standardized before ridge regression?
4. What does a funnel-shaped residual plot suggest?
5. Why is a test set kept separate from model selection?

Long Answer Questions

1. Explain ordinary least squares and describe how it estimates the coefficients of a multiple regression model.
2. Compare simple linear, multiple linear, and polynomial regression, including a suitable use for each.
3. Explain the assumptions of linear regression and describe diagnostic methods for assessing them.
4. Compare MAE, MSE, RMSE, and R², including the strengths and limitations of each.
5. Explain ridge and lasso regression and discuss situations in which each may be preferred.

Analytical / Application Questions

1. A model has excellent training RMSE but substantially worse test RMSE. Diagnose two likely causes and propose a revised workflow.
2. A house-price model includes floor area, number of rooms, and property volume. The coefficients change substantially when one predictor is removed. What problem may be present, and how could it be investigated or addressed?
3. A public agency uses a regression model to allocate inspections. Design an evaluation plan that includes prediction metrics, subgroup fairness checks, residual diagnostics, data governance, and monitoring after deployment.

Multiple Choice Questions with Answers

Question 1

In a simple linear regression model, what does the slope represent?

A. The predicted response when the predictor is zero

B. The average change in the response for a one-unit increase in the predictor

C. The total unexplained variation in the response

D. The proportion of observations classified correctly

Correct Answer: B. The average change in the response for a one-unit increase in the predictor

Question 2

Which metric is expressed in the same units as the response and gives greater emphasis to large errors?

A. MAE

B. MSE

C. RMSE

D. R-squared

Correct Answer: C. RMSE

Question 3

What is the main effect of ridge regularization?

A. It removes the intercept from the model

B. It shrinks coefficients toward zero, usually without making them exactly zero

C. It guarantees a causal interpretation of coefficients

D. It converts a regression problem into classification

Correct Answer: B. It shrinks coefficients toward zero, usually without making them exactly zero

Question 4

A curved pattern in a residual-versus-fitted plot most directly suggests that:

A. The response has no measurement units

B. The model may be missing nonlinear structure

C. The test set is necessarily too large

D. The intercept must be removed

Correct Answer: B. The model may be missing nonlinear structure

Question 5

Which expression defines R-squared for a model containing an intercept?

A. SSE ÷ SST

B. 1 − SSE ÷ SST

C. SST ÷ SSE

D. √(SSE ÷ n)

Correct Answer: B. 1 − SSE ÷ SST

Question 6

If actual values are 10, 12, and 14 and predictions are 11, 10, and 15, what is the MSE?

A. 1.33

B. 1.41

C. 2

D. 4

Correct Answer: C. 2

Question 7

What is a key characteristic of lasso regression?

A. It always increases every coefficient

B. It can set some coefficients exactly to zero

C. It requires the response to be categorical

D. It eliminates the need for a test set

Correct Answer: B. It can set some coefficients exactly to zero

Question 8

Why is random train-test splitting potentially inappropriate for forecasting future values?

A. It makes all predictors categorical

B. It can allow information from the future to influence training

C. It guarantees a negative R-squared

D. It prevents calculation of MAE

Correct Answer: B. It can allow information from the future to influence training

Question 9

In multiple regression, a coefficient is generally interpreted as the association between its predictor and the response:

A. With all other included predictors held constant

B. After removing the response variable

C. Only when the predictor has zero variance

D. Without regard to the measurement scale

Correct Answer: A. With all other included predictors held constant

Question 10

Which issue is indicated by strong linear relationships among predictors?

A. Homoscedasticity

B. Multicollinearity

C. Data normalization

D. Perfect independence

Correct Answer: B. Multicollinearity