Supervised Learning: Regression

Introduction and Context

Regression is a central supervised-learning task in which a model learns a relationship between input variables and a numerical target. Examples include predicting house prices, electricity demand, delivery time, crop yield, and customer revenue. Unlike classification, whose outputs are categories, regression produces quantitative values.

In supervised learning, a dataset contains observations with known targets. Let the input for observation i be xᵢ and its observed target be yᵢ. A regression algorithm estimates a function that maps inputs to predictions:

Here, ŷᵢ is the predicted target, f is the model, and θ represents learned parameters. The model is trained by comparing predictions with observed values through a loss function. It is then evaluated on data that was not used for fitting.

Regression is important because it introduces several ideas used throughout machine learning: optimization, generalization, bias and variance, feature engineering, regularization, residual analysis, and responsible interpretation.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish regression from classification and explain common regression applications;
• formulate and interpret simple and multiple linear regression models;
• explain least-squares estimation and common loss functions;
• recognize the assumptions and limitations of linear regression;
• describe nonlinear regression using transformations and polynomial features;
• explain ridge and lasso regularization;
• calculate and interpret MAE, MSE, RMSE, and R²;
• use residuals to diagnose model problems;
• apply a disciplined regression workflow and communicate results responsibly.

1. Regression Fundamentals

A regression dataset usually has n rows and p predictor variables. The predictors may be numerical, such as temperature, or categorical, such as region, after suitable encoding. The target must be numerical, although it may be continuous or a count depending on the problem.

A useful distinction is between:

• descriptive modelling, which explains associations in observed data; and
• predictive modelling, which aims to estimate targets for new observations.

A strong prediction does not automatically establish causation. For example, a model may predict house prices using both floor area and location without proving that changing one variable causes a particular price change.

Simple linear regression

With one predictor, the population relationship is often written:

The fitted model is:

where β₀ and β₁ are population parameters, b₀ and b₁ are estimates, and εᵢ is an error term. The intercept b₀ is the predicted target when x = 0, provided that this value is meaningful and within the relevant data range. The slope b₁ is the expected change in the target for a one-unit increase in x.

For example, if a delivery-time model is ŷ = 18 + 2.5d, where d is distance in kilometres, the estimated time is 18 minutes at zero kilometres and increases by 2.5 minutes per additional kilometre. The intercept may be mathematically necessary but operationally unrealistic if deliveries never occur at zero distance.

Multiple linear regression

With several predictors:

Each coefficient is interpreted while holding the other included predictors constant. This conditional interpretation is useful but can be unstable when predictors are strongly correlated. Such correlation is called multicollinearity.

Categorical predictors are commonly represented with dummy variables. If a variable has k categories, a model generally uses k − 1 indicator columns and treats the omitted category as the reference. The coefficient of an indicator describes the predicted difference from that reference, conditional on the other predictors.

2. Least Squares and Loss Functions

A residual is the difference between an observed and predicted value:

Ordinary least squares (OLS) chooses coefficients that minimize the sum of squared residuals:

For a simple linear model, the slope and intercept have closed-form estimates:

Here, x̄ and ȳ are sample means. The denominator must be nonzero, so the predictor must vary. These expressions apply to simple OLS with an intercept. Multiple regression is commonly solved using linear algebra or numerical optimization.

The mean squared error used as a training objective is:

MSE has squared target units, which can make it difficult to interpret. Squaring gives large errors disproportionately high influence and makes the objective differentiable, a useful property for optimization.

The mean absolute error is:

MAE has the same units as the target and is less sensitive to extreme errors than MSE. However, its absolute-value function is not differentiable at zero; modern optimizers can still handle it using suitable methods.

The root mean squared error is:

RMSE returns to the target’s units while retaining MSE’s sensitivity to large errors. MAE, MSE, and RMSE are meaningful only when calculated on the same target scale and dataset.

Worked example: calculating errors

Suppose actual values are 10, 12, and 15, while predictions are 12, 11, and 14. Errors are −2, 1, and 1; absolute errors are 2, 1, and 1; squared errors are 4, 1, and 1.

The units of MAE and RMSE are target units, whereas MSE has squared target units.

3. Measuring Explained Variation: R²

The coefficient of determination compares a model with a baseline that always predicts the training-set mean:

where:

SSE is the residual sum of squares and SST is the total sum of squares. In an OLS model with an intercept evaluated on the same data used for fitting, R² is between 0 and 1. On unseen data, R² can be negative because the model may perform worse than the mean baseline. R² is dimensionless and expresses relative reduction in squared error, not the percentage of individual predictions that are correct.

An R² of 0.72 means that, relative to the mean baseline and under the stated evaluation conditions, the model reduces squared error by 72%. It does not mean that every prediction is 72% accurate, nor does it demonstrate causation. Adjusted R² may be used in explanatory multiple regression to account for the number of predictors, but it is not a substitute for proper test-set evaluation.

4. Assumptions and Their Practical Meaning

OLS can produce predictions without all classical assumptions being satisfied, but assumptions affect interpretation, uncertainty estimates, and reliability.

Linearity

The conditional mean of the target should be adequately represented by a linear combination of predictors. A curved relationship can cause systematic residual patterns. Transformations or nonlinear models may be appropriate.

Independence

Observations or errors should be independent for standard inferential calculations. Repeated measurements from the same person, time-series observations, or spatially related observations may violate this assumption. Random splitting can also cause leakage when related records appear in both training and test sets.

Constant variance

Homoscedasticity means that residual spread is approximately constant across fitted values or predictor levels. Increasing spread, called heteroscedasticity, can make ordinary standard errors unreliable and may require transformations, robust methods, or weighted regression.

Error distribution

Normal residuals are mainly important for small-sample confidence intervals and hypothesis tests, not as a strict requirement for obtaining predictions. Large samples often make estimates more robust, but systematic non-normality and outliers still require investigation.

No perfect multicollinearity

No predictor should be an exact linear combination of other predictors. High, though not perfect, multicollinearity increases coefficient uncertainty and makes individual coefficients difficult to interpret. It may have less effect on overall prediction than on interpretation.

Measurement quality and representative data

The target should be measured consistently, predictors should be available at prediction time, and the training data should represent the deployment population. These practical conditions are often more consequential than a minor departure from normality.

5. Residual Analysis

Residual analysis asks whether prediction errors show structure. A useful diagnostic set includes:

• residuals versus fitted values, to detect curvature and changing variance;
• residuals versus each important predictor, to identify omitted nonlinear effects;
• a histogram or quantile plot, to inspect extreme asymmetry and tails;
• influence diagnostics, to identify observations that strongly affect the fitted model.

A random cloud around zero is generally desirable. A U-shaped pattern suggests missing curvature; a funnel shape suggests non-constant variance; clusters may indicate groups, omitted variables, or data-process changes. A large residual is not automatically a bad observation. It may represent a genuine rare case, a measurement error, or a population segment requiring a different model.

A model should not be repeatedly modified until a preferred test score appears. The test set must remain untouched until final evaluation. Diagnostic decisions should be based on training data or validation procedures, followed by one final test evaluation.

6. Nonlinear Regression

A relationship can be nonlinear even when the model is linear in its parameters. Polynomial regression is one introductory approach:

This is linear in the coefficients b₀, b₁, …, bₖ, so least squares can be used. The curve is nonlinear in x. Polynomial terms should be selected using validation performance, residual evidence, and domain knowledge. High-degree polynomials may oscillate and extrapolate dangerously.

Other approaches include logarithmic or reciprocal transformations, splines, regression trees, random forests, and gradient-boosted trees. A logarithmic transformation can be useful for positively skewed targets or multiplicative relationships, but predictions must be transformed back carefully. If a model is fitted to log(y), simply exponentiating a predicted log value may produce a biased estimate of the mean of y because exponentiation is nonlinear.

Extrapolation—predicting outside the range of training data—is especially risky for both linear and nonlinear models. A model that fits observed data well may behave implausibly beyond that range.

7. Regularization and Generalization

Flexible models can overfit: they fit random noise in training data and perform poorly on new data. Regularization adds a penalty for large coefficients to the training objective.

Ridge regression uses an L2 penalty:

Lasso regression uses an L1 penalty:

Here, λ ≥ 0 controls penalty strength and the sums exclude the intercept, which is normally not penalized. At λ = 0, these objectives reduce to OLS. Larger λ generally increases bias but can reduce variance. Lasso can shrink some coefficients exactly to zero, creating a form of feature selection; this is not guaranteed to identify causal or scientifically important variables. Ridge tends to distribute effects among correlated predictors.

Predictors should generally be standardized before regularization so that the penalty treats coefficients comparably. Scaling must be fitted on the training set only and then applied to validation and test data. λ should be selected using cross-validation within the training data.

8. A Professional Regression Workflow

A reliable workflow is more than calling a regression function:

1. Define the prediction target, decision context, acceptable error, and prediction time.
2. Inspect data types, missingness, duplicates, unusual values, and target distribution.
3. Split data into training and test sets before learning preprocessing parameters. Use time-based splits for forecasting when appropriate.
4. Build a simple baseline, such as predicting the training mean.
5. Establish a transparent linear model as a benchmark.
6. Construct preprocessing and modelling pipelines to prevent leakage.
7. Train candidate models and tune hyperparameters with cross-validation on the training set.
8. Compare MAE, RMSE, and R², while considering business costs and subgroup performance.
9. Inspect residuals, outliers, calibration, and errors across relevant segments.
10. Evaluate once on the untouched test set and document data, features, metrics, assumptions, and limitations.
11. Monitor performance after deployment because relationships, populations, and measurement systems can change.

A model should not be selected solely because it has the highest R². If large errors are especially costly, RMSE may be important; if interpretability and typical error matter, MAE may be preferable. The choice must reflect the application.

9. Illustrative Case: Predicting Apartment Rental Prices

Background

A property-management company wants to estimate monthly apartment rent before listing units. It has historical records containing floor area, number of bedrooms, distance to a transport station, building age, neighbourhood, and observed rent.

Problem

The company needs a model that supports initial pricing while avoiding unfair or unstable recommendations. It also wants to understand the main patterns rather than rely on an unexplained score.

Analysis

The target is monthly rent. The data is divided by property and time so that near-duplicate listings do not appear in both training and test sets. Missing values are handled inside a preprocessing pipeline. Neighbourhood is one-hot encoded, and numerical features are standardized for regularized models. A mean-prediction baseline and OLS model are established first.

Residual plots show that error increases for luxury properties and that the area–rent relationship is slightly curved. A polynomial area term or a tree-based model is tested using cross-validation. Ridge regression is also tested because floor area and number of bedrooms are correlated.

Approach and solution

The final model is selected using validation MAE, RMSE, and subgroup checks. Suppose the test results show MAE = 85 currency units, RMSE = 140 currency units, and R² = 0.78. These values mean that the typical absolute error is about 85 units, larger errors raise the RMSE to 140, and the model reduces squared error by 78% relative to the test-set mean baseline, subject to the exact evaluation design.

The team examines errors by neighbourhood and price band. If the model performs poorly in a low-data neighbourhood, the result is reported rather than hidden. Predictions are presented with an explanation of important features, an indication of training-data coverage, and a warning against using the model as an automatic decision-maker.

Outcome and learning points

The model supports consistent initial estimates but does not replace human review. The case demonstrates that data splitting, leakage prevention, residual analysis, multiple metrics, subgroup evaluation, and transparent communication are as important as the choice of algorithm.

10. Ethical, Governance, and Operational Considerations

Regression models can reproduce historical inequities. A price model may encode unequal access to services; a wage model may reflect discriminatory historical decisions. Sensitive variables should not be included casually, but removing them does not guarantee fairness because proxy variables may remain.

Important controls include data provenance, access management, documentation, versioning, audit trails, privacy protection, and monitoring for drift. Organisations should define who may use predictions, how errors are appealed, and when human review is mandatory. Personal data should be collected and processed lawfully, minimally, and for a specified purpose. Regulatory requirements vary by jurisdiction and application, so technical teams should consult applicable privacy, consumer-protection, and AI-governance rules.

Challenges and Limitations

Regression is limited by incomplete variables, noisy targets, measurement error, distribution shift, outliers, leakage, and inappropriate extrapolation. A high score may result from a target proxy, duplicated records, or a split that does not reflect deployment. Coefficients are not automatically causal effects. Regularization improves generalization but does not cure biased data. Finally, metrics aggregate performance and can conceal serious errors for particular groups or high-risk cases.

Best Practices

Use a meaningful baseline, preserve a genuine test set, fit preprocessing only on training data, choose metrics according to consequences, inspect residuals, validate across important subgroups, prefer simpler models when performance is comparable, document assumptions, and monitor the model after deployment. Communicate uncertainty and limitations rather than presenting a prediction as a fact.

Chapter Summary

Regression learns mappings from inputs to numerical targets. Linear regression represents the conditional mean as a weighted combination of predictors, while polynomial terms, transformations, trees, and other models represent nonlinear patterns. OLS minimizes squared residuals. MAE measures average absolute error, MSE emphasizes large errors, RMSE expresses that emphasis in target units, and R² compares squared error with a mean-prediction baseline.

Reliable regression requires attention to linearity, independence, variance, multicollinearity, data quality, and representativeness. Residual analysis reveals patterns that aggregate metrics can hide. Regularization controls complexity: ridge shrinks coefficients through an L2 penalty, while lasso uses an L1 penalty and may set coefficients to zero. A professional workflow uses leakage-safe splitting, validation, multiple metrics, subgroup analysis, documentation, and monitoring. Ethical deployment requires privacy, fairness assessment, transparency, human oversight, and recognition that predictive association is not causation.

Key Terms

Regression: Supervised learning for predicting a numerical target.

Predictor: An input variable used to generate a prediction.

Target: The numerical outcome the model attempts to predict.

Residual: Observed value minus predicted value.

Intercept: Predicted target when all predictors equal zero, if meaningful.

Slope: Change in predicted target for a one-unit predictor change, holding other variables constant.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Loss function: A numerical measure of prediction error used for training or evaluation.

MAE: Mean absolute prediction error.

MSE: Mean squared prediction error.

RMSE: Square root of MSE, expressed in target units.

R²: Relative reduction in squared error compared with a mean baseline.

Homoscedasticity: Approximately constant residual variance.

Multicollinearity: Strong correlation or linear dependence among predictors.

Regularization: Penalization used to control model complexity.

Ridge regression: Regression with an L2 coefficient penalty.

Lasso regression: Regression with an L1 coefficient penalty.

Overfitting: Strong training performance combined with weak generalization.

Data leakage: Use of information unavailable at prediction time or improperly shared across splits.

Extrapolation: Prediction outside the range represented in training data.

Review Questions

Short Answer Questions

1. What is the difference between a regression target and a classification target?
2. Interpret the slope in a multiple linear regression model.
3. Why is RMSE expressed in the same units as the target?
4. State two consequences of multicollinearity.
5. What does a funnel-shaped residual plot commonly suggest?

Long Answer Questions

1. Explain OLS, including its objective, residual definition, and important assumptions.
2. Compare MAE, MSE, RMSE, and R², including situations in which each is useful.
3. Explain how polynomial regression represents nonlinear relationships and why high-degree polynomials can be risky.
4. Compare ridge and lasso regression in terms of penalties, coefficient behaviour, scaling, and feature selection.
5. Describe a complete leakage-safe workflow for developing and evaluating a regression model.

Analytical / Application Questions

1. A delivery model has excellent training R² but much lower test R² and a funnel-shaped residual plot. Diagnose two likely problems and propose remedies.
2. A model predicts salaries using education, experience, and job role. Explain why a large coefficient for education should not automatically be interpreted as a causal effect.
3. A hospital values avoiding very large prediction errors more than minimizing typical small errors. Which evaluation metric would you emphasize, and what additional analyses would you perform?

Multiple Choice Questions with Answers

Question 1

Which task is a regression problem?

A. Predicting whether an email is spam or not spam

B. Predicting the monthly electricity consumption of a building

C. Grouping customers without target labels

D. Identifying the topic of a document

Correct Answer: B. Predicting the monthly electricity consumption of a building

Question 2

In a simple linear model ŷ = 5 + 3x, what does 3 represent?

A. The predicted value when x is zero

B. The number of observations

C. The predicted increase for a one-unit increase in x

D. The model's R² value

Correct Answer: C. The predicted increase for a one-unit increase in x

Question 3

For actual values 10 and 14 and predictions 8 and 13, what is the MAE?

A. 1.5

B. 2.5

C. 3.5

D. 5.0

Correct Answer: A. 1.5

Question 4

Why does MSE give relatively greater influence to large errors?

A. It divides errors by the target mean

B. It squares each error

C. It removes negative errors before averaging

D. It uses categorical encoding

Correct Answer: B. It squares each error

Question 5

What does a negative R² on a test set indicate?

A. The target has no units

B. The model is necessarily causal

C. The model performs worse than the mean baseline under that evaluation

D. The residuals are all positive

Correct Answer: C. The model performs worse than the mean baseline under that evaluation

Question 6

Which residual pattern most strongly suggests non-constant variance?

A. A random horizontal cloud with equal spread

B. A funnel-shaped increase in spread

C. A single intercept value

D. A perfectly horizontal predictor scale

Correct Answer: B. A funnel-shaped increase in spread

Question 7

Which statement about ridge regression is correct?

A. It uses an L2 penalty on coefficients

B. It always removes every correlated predictor

C. It cannot be used with numerical predictors

D. It maximizes the unregularized SSE

Correct Answer: A. It uses an L2 penalty on coefficients

Question 8

Why should regularized predictors usually be standardized?

A. To make the target categorical

B. To ensure penalty effects are comparable across predictor scales

C. To guarantee R² equals one

D. To eliminate the need for a test set

Correct Answer: B. To ensure penalty effects are comparable across predictor scales

Question 9

Which practice helps prevent data leakage?

A. Fitting a scaler on the complete dataset before splitting

B. Using test-set results to select hyperparameters

C. Fitting preprocessing parameters only on training data

D. Duplicating records across training and test sets

Correct Answer: C. Fitting preprocessing parameters only on training data

Question 10

What is a major danger of extrapolating with a regression model?

A. Predictions are made within the observed data range

B. The model may behave implausibly outside the training range

C. Residuals become definitions of predictors

D. MAE automatically becomes zero

Correct Answer: B. The model may behave implausibly outside the training range