Regression Models

Introduction and Context

Regression is a family of supervised learning methods used to estimate a numerical response from one or more explanatory variables. Examples include predicting house prices from property characteristics, estimating energy demand from weather conditions, forecasting delivery time, and modelling a machine’s output from operating conditions.

A regression model does more than produce predictions. It can quantify associations, support explanation, identify influential variables, and provide a basis for decisions. However, predictive usefulness and causal interpretation are different goals. A regression coefficient may describe an adjusted association without proving that changing the corresponding variable causes the response to change.

This chapter develops ordinary least squares (OLS) regression, multiple regression, polynomial regression, residual analysis, and regularization using Ridge and Lasso regression. It also explains loss functions, assumptions, model evaluation, and the bias–variance trade-off.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the structure and purpose of regression models.

2. Derive the OLS estimator for simple and multiple linear regression.

3. Interpret coefficients, fitted values, residuals, and common evaluation metrics.

4. Distinguish linearity in parameters from assumptions about the conditional mean.

5. Use polynomial features and recognize their risks.

6. Explain residual diagnostics and important model assumptions.

7. Compare Ridge and Lasso regression and select suitable regularization strategies.

8. Analyze the bias–variance trade-off and apply regression responsibly.

1. Regression as a Predictive Model

Let the response variable be $y$ and let the predictors be $x_1,\ldots,x_p$. A general regression relationship can be written as

$$y_i=f(x_{i1},\ldots,x_{ip})+\varepsilon_i,$$

where $f$ is an unknown systematic component and $\varepsilon_i$ represents unexplained variation. A fitted model, denoted $\hat f$, estimates the systematic component from training data. For a new input $x_0$, the prediction is $\hat y_0=\hat f(x_0)$.

In a probabilistic interpretation, the ideal prediction under squared-error loss is the conditional mean:

$$f(x)=E(Y\mid X=x).$$

In practice, a model may be deliberately simpler than the true conditional mean because of limited data, computational constraints, interpretability requirements, or concern about overfitting.

Important terminology includes:

• Response or target: the numerical variable being predicted.
• Predictor, feature, or explanatory variable: an input used by the model.
• Parameter: an unknown population quantity, such as a slope.
• Estimate: a value calculated from a sample.
• Fitted value: the prediction for an observed training input.
• Residual: the observed response minus its fitted value.
• Error: the unobserved difference between an observation and the underlying conditional mean.

Residuals and errors are not interchangeable: residuals can be calculated, whereas errors generally cannot be observed directly.

2. Simple Linear Regression

Simple linear regression models one predictor as

$$Y_i=\beta_0+\beta_1x_i+\varepsilon_i,$$

where $\beta_0$ is the intercept, $\beta_1$ is the slope, and $\varepsilon_i$ is the random error. The conditional mean specification is

$$E(Y_i\mid X_i=x_i)=\beta_0+\beta_1x_i.$$

The fitted equation is

$$\hat y_i=b_0+b_1x_i.$$

The slope describes the estimated change in the conditional mean of $Y$ associated with a one-unit increase in $X$, within the observed or otherwise scientifically justified range. The intercept is the predicted response at $x=0$; it may have no practical interpretation if zero is outside the relevant domain.

2.1 Derivation of the OLS Estimator

OLS chooses $b_0$ and $b_1$ to minimize the residual sum of squares (RSS):

$$S(b_0,b_1)=\sum_{i=1}^{n}[y_i-b_0-b_1x_i]^2.$$

Define $e_i=y_i-b_0-b_1x_i$. Differentiating with respect to $b_0$ and setting the derivative to zero gives

$$\frac{\partial S}{\partial b_0}=-2\sum_{i=1}^{n}(y_i-b_0-b_1x_i)=0,$$

so

$$\sum y_i-nb_0-b_1\sum x_i=0.$$

Dividing by $n$ and using $\bar x=\frac{1}{n}\sum x_i$ and $\bar y=\frac{1}{n}\sum y_i$ gives

$$b_0=\bar y-b_1\bar x.$$

Differentiation with respect to $b_1$ gives

$$\frac{\partial S}{\partial b_1}=-2\sum_{i=1}^{n}x_i(y_i-b_0-b_1x_i)=0.$$

Substitute $b_0=\bar y-b_1\bar x$:

$$0=\sum x_i[(y_i-\bar y)-b_1(x_i-\bar x)].$$

Using $\sum(y_i-\bar y)=0$ and rearranging in centered form yields

$$b_1=\frac{\sum_{i=1}^{n}(x_i-\bar x)(y_i-\bar y)}{\sum_{i=1}^{n}(x_i-\bar x)^2}.$$

Thus,

$$\boxed{b_1=\frac{S_{xy}}{S_{xx}},\qquad b_0=\bar y-b_1\bar x.}$$

The denominator is positive provided the predictor is not constant. These equations identify the unique OLS minimizer in the ordinary nondegenerate case because the objective is a convex quadratic and $S_{xx}>0$.

Worked Example 1: Simple OLS

Suppose training data are $(x,y)=(1,2),(2,3),(3,5)$. Then $\bar x=2$ and $\bar y=10/3$. The centered products are

$$(-1)(-4/3)+0(-1/3)+(1)(5/3)=3,$$

and $S_{xx}=(-1)^2+0^2+1^2=2$. Therefore,

$$b_1=3/2=1.5,\qquad b_0=10/3-1.5(2)=1/3.$$

The fitted model is $\hat y=1/3+1.5x$. Predictions are $11/6,10/3,29/6$, and residuals are respectively $1/6,-1/3,1/6$. Their sum is zero, as required when an intercept is included.

3. Multiple Linear Regression

With $p$ predictors,

$$Y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\varepsilon_i.$$

In matrix notation,

$$\mathbf y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon,$$

where $\mathbf X$ contains a column of ones and the predictor columns. OLS minimizes

$$S(\boldsymbol\beta)=\|\mathbf y-\mathbf X\boldsymbol\beta\|_2^2.

$$

Expanding and differentiating,

$$S=\mathbf y^T\mathbf y-2\boldsymbol\beta^T\mathbf X^T\mathbf y+\boldsymbol\beta^T\mathbf X^T\mathbf X\boldsymbol\beta,$$

$$\nabla S=-2\mathbf X^T\mathbf y+2\mathbf X^T\mathbf X\boldsymbol\beta.$$

Setting the gradient to zero produces the normal equations:

$$\mathbf X^T\mathbf X\hat{\boldsymbol\beta}=\mathbf X^T\mathbf y.$$

If the columns of $\mathbf X$ are linearly independent, $\mathbf X^T\mathbf X$ is invertible and

$$\boxed{\hat{\boldsymbol\beta}=(\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y.}$$

In numerical software, QR or singular-value decomposition is generally preferred to explicitly computing an inverse because it is more stable. If predictors are perfectly collinear, the coefficient vector is not uniquely identified, although predictions may still be identifiable through a suitable generalized solution.

A multiple-regression coefficient is a partial association: $b_j$ is the predicted change in $Y$ for a one-unit increase in $X_j$, holding the other included predictors fixed. This interpretation can be unrealistic when predictors cannot vary independently or can be misleading when important confounders are omitted.

4. Loss Functions and Model Evaluation

For observations $(y_i,\hat y_i)$, common losses include:

• Squared error: $(y_i-\hat y_i)^2$. It strongly penalizes large errors and leads to OLS when summed over observations.
• Mean squared error (MSE): $\frac{1}{n}\sum(y_i-\hat y_i)^2$.
• Root mean squared error (RMSE): $\sqrt{\mathrm{MSE}}$, expressed in the response’s units.
• Mean absolute error (MAE): $\frac{1}{n}\sum|y_i-\hat y_i|$, less sensitive to extreme errors than MSE.

The coefficient of determination is

$$R^2=1-\frac{\mathrm{SSE}}{\mathrm{SST}},$$

where, for a specified evaluation set,

$$\mathrm{SSE}=\sum(y_i-\hat y_i)^2,\qquad \mathrm{SST}=\sum(y_i-\bar y_{\mathrm{eval}})^2.$$

For a test set, the standard test-set $R^2$ uses the test-set mean $\bar y_{\mathrm{test}}$ in SST. It compares the model with predicting the test-set mean, not the training-set mean. A negative test $R^2$ means the model has larger squared error than that test-set-mean predictor. It does not, by itself, establish comparison with a training-mean baseline.

Training $R^2$ is often optimistic, especially for flexible models. Test data must remain separate from model selection; repeated tuning against the test set causes leakage. Cross-validation provides a more reliable estimate during development.

5. Assumptions: Two Distinct Ideas

The word “linearity” can refer to two different concepts.

Linearity in parameters means the model is linear in unknown coefficients. For example,

$$E(Y\mid X=x)=\beta_0+\beta_1x+\beta_2x^2$$

is linear in $\beta_0,\beta_1,\beta_2$, even though it is nonlinear in $x$. Likewise, $\beta_1\log x$ is linear in its parameter.

Correct specification of the conditional mean means the chosen predictors, transformations, interactions, and functional form adequately represent $E(Y\mid X)$. A model may be linear in parameters but miss a necessary nonlinear term or interaction. Conversely, nonlinear-in-parameter models such as $y=ae^{bx}+\varepsilon$ are not linear regression models in the OLS sense.

Common assumptions for classical inference are:

1. Correct conditional-mean specification: the systematic mean is represented adequately.

2. Exogeneity: $E(\varepsilon\mid X)=0$. This is central for unbiased coefficient interpretation.

3. Independence: observations or errors are independent when the sampling design requires it; time series often violate this.

4. Constant conditional variance (homoscedasticity): $\operatorname{Var}(\varepsilon\mid X)=\sigma^2$. Heteroscedasticity does not necessarily bias OLS coefficients but affects conventional standard errors.

5. No perfect multicollinearity: predictors are not exact linear combinations.

6. Normal errors: not required for OLS prediction or unbiasedness, but useful for exact small-sample t and F procedures under the classical model.

Diagnostics support, rather than prove, assumptions. A residual-versus-fitted plot can reveal curvature or changing spread; a Q–Q plot assesses approximate normality; residuals ordered by time can reveal dependence. Domain knowledge and data collection design are as important as plots.

6. Polynomial Regression and Feature Engineering

Polynomial regression extends a linear model with transformed features:

$$E(Y\mid X=x)=\beta_0+\beta_1x+\beta_2x^2+\cdots+\beta_dx^d.$$

It remains linear in parameters and can therefore be fitted by least squares after constructing the features. A coefficient such as $\beta_1$ should not automatically be interpreted as the overall slope because the marginal effect is

$$\frac{dE(Y\mid x)}{dx}=\beta_1+2\beta_2x+\cdots+d\beta_dx^{d-1}.$$

High-degree polynomials may oscillate and extrapolate dangerously. Centering and scaling $x$ can improve numerical conditioning. Splines or tree-based models may provide more stable local flexibility. Any transformation must be learned using training data only when it involves estimated quantities such as scaling parameters.

7. Regularization and the Bias–Variance Trade-off

Flexible models can fit random noise in the training sample. This is overfitting: training error is low but generalization error is high. A very restricted model may underfit, producing high bias. Expected prediction error can be conceptualized as

$$\text{Error}\approx\text{Bias}^2+\text{Variance}+\text{Irreducible noise}.$$

The bias–variance trade-off is not a literal identity for every loss and setting, but under squared prediction loss it provides a useful decomposition framework.

7.1 Ridge Regression

Ridge minimizes

$$\sum_{i=1}^{n}(y_i-\beta_0-\mathbf x_i^T\boldsymbol\beta)^2+\lambda\sum_{j=1}^{p}\beta_j^2,$$

where $\lambda\geq0$. Usually the intercept is not penalized and predictors are standardized first. In matrix form, with a separately handled intercept,

$$\hat{\boldsymbol\beta}_{R}=(\mathbf X^T\mathbf X+\lambda\mathbf I)^{-1}\mathbf X^T\mathbf y,$$

when $\mathbf X$ is centered and $\mathbf I$ applies to penalized coefficients. For $\lambda>0$, the matrix is better conditioned in many collinear settings. Ridge generally shrinks coefficients toward zero but does not usually make them exactly zero.

7.2 Lasso Regression

Lasso minimizes

$$\sum_{i=1}^{n}(y_i-\beta_0-\mathbf x_i^T\boldsymbol\beta)^2+\lambda\sum_{j=1}^{p}|\beta_j|.$$

The absolute-value penalty can set some coefficients exactly to zero, producing a sparse model and possible feature selection. When predictors are highly correlated, which variable Lasso retains can be unstable; it may select one of several similar predictors. Elastic Net combines L1 and L2 penalties and can be useful when both sparsity and grouped correlated predictors matter.

The penalty parameter $\lambda$ is normally selected using cross-validation. Standardization is essential when predictor units differ, because otherwise the penalty acts unevenly across coefficients. Regularization changes coefficient interpretation: estimates are intentionally biased toward zero in exchange for potentially lower prediction variance.

8. Residual Analysis and Influential Observations

A residual is $e_i=y_i-\hat y_i$. Useful diagnostics include:

• Residual-versus-fitted plot: curvature suggests misspecification; a funnel suggests nonconstant variance.
• Residual-versus-predictor plots: reveal omitted nonlinear structure.
• Q–Q plot: assesses whether residuals approximately follow a reference normal distribution.
• Leverage: measures how unusual an observation’s predictor values are.
• Influence: measures how much a fitted model changes when an observation is removed; Cook’s distance is one diagnostic.

An outlier is unusual in response, leverage is unusual in predictor space, and an influential point substantially changes the fitted model. These are distinct categories. An observation should not be deleted merely because it is inconvenient. Investigate measurement error, data-generating conditions, subgroup membership, and whether the case is important for the deployment population.

9. Applied Case: Predicting Residential Energy Demand

This is an illustrative case. A utility wants to predict hourly electricity demand for small commercial buildings. Available predictors include outdoor temperature, hour of day, day type, floor area, and recent demand.

Problem. A baseline linear model underpredicts demand during very hot afternoons and has unstable coefficients because temperature and recent demand are correlated.

Analysis. Residual-versus-fitted plots show curvature, while time-ordered residuals show dependence. A random train–test split would place highly similar adjacent hours in both sets and could overstate generalization.

Approach. The team creates temperature-squared and hour-of-day features, encodes day type, and uses a chronological validation split. Continuous predictors are standardized. Ridge regression is evaluated alongside OLS; $\lambda$ is selected using time-aware cross-validation. MAE is reported because operational planning values errors in demand units, while RMSE is also reported to expose large misses. Prediction intervals or calibrated uncertainty estimates are provided for capacity decisions.

Outcome. Ridge reduces variance caused by correlated predictors, while the quadratic temperature term captures the nonlinear demand response. Residual checks are repeated on the final holdout period. The model is monitored for seasonal drift and unusual weather conditions.

Learning points. Feature transformations can preserve linearity in parameters; validation must match deployment chronology; regularization is not a substitute for correcting leakage; and operational decisions require uncertainty, monitoring, and a clear cost function rather than a single accuracy score.

10. Limitations, Governance, and Best Practice

Regression can encode historical discrimination, amplify measurement errors, and produce unreliable predictions for groups or conditions underrepresented in training data. Sensitive attributes and their proxies require governance consistent with applicable privacy, equality, and sector regulations. A model used for credit, employment, healthcare, or public services may require documentation, explainability, auditability, human oversight, and impact assessment.

A professional workflow should:

1. Define the prediction target, time horizon, unit of analysis, and decision cost.

2. Separate training, validation, and final test data according to deployment conditions.

3. Inspect missingness, measurement quality, outliers, leakage, and sampling bias.

4. Establish a meaningful baseline.

5. Fit preprocessing and feature engineering only on training data.

6. Compare OLS, regularized, and appropriate nonlinear alternatives.

7. Use cross-validation suited to independent, grouped, spatial, or temporal data.

8. Evaluate calibration, subgroup performance, uncertainty, and operational consequences.

9. Document variables, assumptions, versioning, and limitations.

10. Monitor drift and periodically reassess whether the model remains fit for purpose.

Chapter Summary

Regression models estimate numerical responses from explanatory variables. Simple OLS minimizes squared residuals and has closed-form slope and intercept estimators derived from the normal equations. Multiple OLS uses matrix notation and requires attention to rank and numerical stability. MSE, RMSE, MAE, and $R^2$ answer different evaluation questions; standard test-set $R^2$ compares against the test-set mean predictor. Linearity in parameters is distinct from correct specification of the conditional mean. Polynomial features allow curvature while remaining linear in coefficients, but can create instability and poor extrapolation. Residual diagnostics assess, but do not prove, model adequacy. Ridge shrinks correlated coefficients, whereas Lasso can produce sparse solutions. Regularization manages the bias–variance trade-off and must be combined with sound validation, governance, uncertainty analysis, and monitoring.

Key Terms

Conditional mean: The expected response given specified predictor values.

Regression: Supervised learning for modelling or predicting a numerical response.

Intercept: Expected or fitted response when all predictors equal zero.

Slope: Change in a conditional mean associated with a one-unit predictor change, under the model.

OLS: Estimation method minimizing the sum of squared residuals.

Residual: Observed response minus fitted response.

Error: Unobserved deviation from the conditional mean.

RSS/SSE: Sum of squared residuals or prediction errors.

MSE: Average squared prediction error.

RMSE: Square root of MSE, in response units.

MAE: Average absolute prediction error.

$R^2$: Proportion-like measure comparing model squared error with mean-baseline variation.

Multicollinearity: Strong linear dependence among predictors.

Homoscedasticity: Constant conditional error variance.

Leverage: Unusualness of an observation’s predictor values.

Influence: Degree to which an observation changes fitted results.

Ridge regression: Regression with an L2 coefficient penalty.

Lasso regression: Regression with an L1 coefficient penalty.

Regularization: Penalization used to control model complexity.

Bias–variance trade-off: Balance between systematic error and sensitivity to sampling variation.

Review Questions

Short Answer Questions

1. Distinguish a residual from an error.

2. State the OLS slope estimator in simple linear regression.

3. What does a multiple-regression coefficient mean when other predictors are held fixed?

4. Why are predictors usually standardized before Ridge or Lasso regression?

5. State one purpose of a residual-versus-fitted plot.

Long Answer Questions

1. Derive the simple linear regression OLS estimators from the residual sum of squares.

2. Derive the multiple-regression normal equations and explain the full-rank condition.

3. Explain the difference between linearity in parameters and correct conditional-mean specification.

4. Compare Ridge and Lasso in terms of objective function, coefficient behavior, and applications.

5. Explain the bias–variance trade-off and how cross-validation helps manage it.

Analytical / Application Questions

1. A model has low training RMSE but much higher validation RMSE. Diagnose possible causes and propose a testing and regularization strategy.

2. A residual plot shows a curved pattern and increasing spread as fitted values increase. What assumptions or modelling choices are implicated, and what remedies would you investigate?

3. Design a validation plan for an hourly energy-demand model where future observations must be predicted from past data. Justify the split strategy, metrics, features, and monitoring plan.

Multiple Choice Questions with Answers

Question 1

In simple OLS regression, which quantity appears in the denominator of the slope estimator?

A. The sum of the response values

B. The sum of squared deviations of predictors from their mean

C. The residual sum of squares

D. The number of predictors

Correct Answer: B. The sum of squared deviations of predictors from their mean

Question 2

Which statement correctly distinguishes linearity in parameters from conditional-mean specification?

A. Any nonlinear function of a predictor violates linear regression

B. A model with x² cannot be linear in parameters

C. A model can be linear in coefficients yet incorrectly specify the conditional mean

D. Correct conditional-mean specification guarantees normally distributed errors

Correct Answer: C. A model can be linear in coefficients yet incorrectly specify the conditional mean

Question 3

For a standard test-set R², the SST baseline is calculated using:

A. The training-set mean response

B. The test-set mean response

C. The model’s fitted mean response

D. The population mean, which is always known

Correct Answer: B. The test-set mean response

Question 4

What is a typical effect of increasing the Ridge penalty parameter?

A. It generally shrinks coefficients toward zero

B. It always sets all coefficients exactly to zero

C. It removes the intercept regardless of implementation

D. It guarantees lower test error for every dataset

Correct Answer: A. It generally shrinks coefficients toward zero

Question 5

Which penalty can produce coefficients exactly equal to zero?

A. L1 penalty used by Lasso

B. L2 penalty used by Ridge

C. Squared-error loss alone

D. A test-set R² calculation

Correct Answer: A. L1 penalty used by Lasso

Question 6

For the data (1,2), (2,3), and (3,5), what is the OLS slope?

A. 0.50

B. 1.00

C. 1.50

D. 2.00

Correct Answer: C. 1.50

Question 7

Which pattern in a residual-versus-fitted plot most directly suggests nonconstant variance?

A. A random horizontal cloud with stable spread

B. A funnel-shaped increase in spread

C. All residuals summing to zero

D. A single predictor having a positive coefficient

Correct Answer: B. A funnel-shaped increase in spread

Question 8

Why can a high-degree polynomial be risky for prediction?

A. It cannot be fitted using least squares

B. It is never linear in its parameters

C. It may oscillate and extrapolate poorly

D. It always has higher bias than a straight line

Correct Answer: C. It may oscillate and extrapolate poorly

Question 9

Under the exogeneity condition E(ε|X)=0, what is its principal relevance to OLS?

A. It ensures every residual is zero

B. It supports unbiased coefficient estimation under the model conditions

C. It guarantees homoscedasticity

D. It guarantees normally distributed responses

Correct Answer: B. It supports unbiased coefficient estimation under the model conditions

Question 10

A negative standard test-set R² means that the fitted model has:

A. Higher squared error than predicting the test-set mean

B. Higher squared error than predicting the training-set mean necessarily

C. Zero predictive value under every possible loss function

D. A mathematically impossible evaluation

Correct Answer: A. Higher squared error than predicting the test-set mean