Regression and Forecasting Models

Introduction

Business analytics is concerned not only with describing what has happened, but also with estimating what is likely to happen and identifying the factors associated with that outcome. Regression models support both purposes. They quantify relationships between a response variable and one or more explanatory variables, while forecasting models use patterns in time-ordered data to predict future values.

For example, a retailer may estimate weekly sales from price, advertising expenditure and store traffic. A bank may estimate loan risk from income, repayment history and debt. A manufacturer may forecast next month’s demand from historical orders, promotions and seasonal patterns. In each case, the model is useful only when its assumptions, uncertainty and business meaning are understood.

This chapter develops linear and multiple regression, examines model assumptions and interactions, introduces regularisation, and then applies time-series concepts to forecasting and validation.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the purpose and structure of regression models;
• estimate and interpret simple and multiple linear regression;
• distinguish association from causation and statistical significance from business importance;
• diagnose major regression assumptions and recognise problematic observations;
• interpret categorical variables, interactions and transformations;
• explain ridge and lasso regularisation;
• identify trend, seasonality, cycles and noise in time-series data;
• compare forecasting methods using appropriate validation procedures; and
• communicate model results responsibly for data-driven decisions.

1. Regression as a Decision-Support Model

Regression estimates the expected value of a response variable, often called the dependent variable, given one or more predictors, also called explanatory or independent variables. A simple linear regression model is:

Here, y_i is the response for observation i, x_i is the predictor, β₀ is the intercept, β₁ is the population slope, and ε_i is the error term. The error represents influences on y_i not captured by the model and random measurement variation.

The fitted or predicted value is:

Lowercase coefficients b₀ and b₁ are estimates calculated from sample data. The slope is measured in response units per predictor unit. If sales are measured in dollars and advertising in dollars, b₁ has units of dollars of sales per dollar of advertising.

The least-squares estimates minimise the sum of squared residuals:

A residual is e_i = y_i − ŷ_i. Squaring gives greater weight to large errors and prevents positive and negative errors from cancelling. Least squares is appropriate when the conditional mean is approximately linear and the observations are sufficiently independent; it is not automatically proof that the relationship is causal.

Interpreting the intercept and slope

The intercept is the predicted response when every predictor equals zero. It may have little practical meaning if zero is outside the observed range. The slope describes the expected change in the response associated with a one-unit increase in the predictor, within the relevant data range.

Suppose a fitted sales model is:

If both variables are measured in dollars, an additional dollar of advertising is associated with an estimated $3.50 increase in sales, on average. This interpretation does not establish that advertising caused the entire increase: seasonality, store quality and marketing targeting may also matter.

2. Multiple Linear Regression

Most business outcomes have several relevant drivers. Multiple regression extends the model as follows:

The fitted model is:

The coefficient b_j estimates the change in the expected response for a one-unit increase in predictor j, holding the other included predictors constant. This conditional interpretation is central. If price and advertising are correlated, the advertising coefficient is not simply the raw association between advertising and sales; it estimates the association remaining after accounting for price and the other variables.

Worked example: sales estimation

An online retailer fits the illustrative model below, where sales and advertising are measured in thousands of dollars and site visits in thousands:

For a week with advertising of 10, visits of 40 and a price index of 1.10:

Predicted sales are therefore $43.55 thousand. The advertising coefficient means that, holding visits and price index constant, an additional $1,000 of advertising is associated with an estimated $2,400 increase in sales. The coefficient should not be interpreted outside the range of data used to estimate it.

Goodness of fit

The total variation in the response can be partitioned into explained and unexplained variation:

where SST = ∑(y_i − ȳ)² is total sum of squares, SSR = ∑(ŷ_i − ȳ)² is regression sum of squares, and SSE = ∑(y_i − ŷ_i)² is residual sum of squares. The coefficient of determination is:

R² is the proportion of sample variation explained by the fitted model. It is between 0 and 1 when an intercept is included and ordinary definitions apply. Adding predictors cannot decrease ordinary R², even if they are unhelpful. Adjusted R² penalises unnecessary predictors:

Here n is sample size and p is the number of predictors, excluding the intercept. Adjusted R² is useful for comparing models with different numbers of predictors, but neither measure proves predictive performance on new data.

For prediction, the root mean squared error is often useful:

RMSE has the same units as the response and is sensitive to large errors. Mean absolute error is:

MAE is easier to interpret as average absolute error and is less influenced by extreme errors than RMSE.

3. Statistical Inference and Business Interpretation

A coefficient estimate is uncertain because it is based on a sample. A confidence interval gives a range of plausible population values under the model assumptions. A common two-sided interval is:

where SE(b_j) is the standard error and t* is a critical value from the t distribution with appropriate degrees of freedom.

A hypothesis test may assess H₀: β_j = 0. The t statistic is:

A small p-value indicates that the observed estimate would be relatively unusual if the null hypothesis were true, given the assumptions. It does not measure the probability that the model is correct, nor does it establish practical importance. A small effect can be statistically significant in a large sample, while a commercially important effect may be uncertain in a small sample.

Prediction intervals are wider than confidence intervals for the mean because they include both uncertainty in the estimated mean and the variability of a new individual outcome. Managers should receive ranges, not only point predictions.

4. Regression Assumptions and Diagnostics

Regression inference and interpretation depend on assumptions. The assumptions need not be perfectly true, but serious violations can make conclusions unreliable.

Normality is often misunderstood. A large sample can support approximate inference even when residuals are not exactly normal, but extreme skewness, outliers or dependence may still harm estimates and prediction.

Outliers and influential observations

An outlier has an unusual response or predictor value. A high-leverage observation has unusual predictor values. An influential observation substantially changes fitted coefficients when removed. Analysts should investigate data quality, process changes and legitimate unusual cases rather than automatically deleting them. Useful measures include studentised residuals, leverage and Cook’s distance, but no single threshold should replace professional judgement.

Multicollinearity

Multicollinearity occurs when predictors contain overlapping information. It can produce unstable coefficients and large standard errors, even when the model predicts well. A variance inflation factor for predictor j is:

Here R_j² is obtained by regressing predictor j on the other predictors. High values signal potential concern, although acceptable levels depend on context. Remedies include removing redundant variables, combining measures, collecting more informative data or using regularisation. Removing a variable solely because its p-value is large can damage the model if that variable is an important confounder.

5. Categorical Predictors, Interactions and Nonlinearity

A categorical variable is represented using indicator variables. If customer type is either business or consumer, one category is selected as the reference category. The model might be:

For consumers, predicted sales at a given visit level are based on b₀ + b₂Visits. For businesses, the prediction differs by b₁, holding visits constant. With k categories, the usual intercept model uses k − 1 indicators to avoid perfect linear dependence.

An interaction allows the effect of one predictor to depend on another:

The marginal effect of x is:

Thus β₃ measures how the slope for x changes as z increases by one unit. Main effects remain important even when an interaction is present. Centre variables, such as replacing x by x − x̄, when useful for making the intercept and coefficients more interpretable and reducing nonessential correlation between main effects and interaction terms.

Linear regression can represent curvature by including transformations such as x², logarithms or piecewise terms. A log transformation often changes interpretation. If:

then a one-unit increase in x is associated approximately with a 100β₁ percent change in y when β₁ is small, subject to the model assumptions. Transformations must be chosen for substantive and statistical reasons, not merely to improve a fit statistic.

6. Regularisation and Model Selection

When there are many predictors, correlated variables or limited observations, ordinary least squares can overfit. Regularisation adds a penalty to large coefficients. Ridge regression solves:

Lasso regression solves:

The intercept is generally not penalised, and predictors should usually be standardised so penalty strength is comparable across units. λ ≥ 0 controls the penalty. At λ = 0, the methods reduce to ordinary least squares. Larger values shrink coefficients; lasso can set some exactly to zero, providing a form of variable selection. Ridge generally retains correlated predictors and stabilises estimates.

Regularisation introduces bias deliberately to reduce variance and improve out-of-sample prediction. The value of λ should be selected using cross-validation performed inside the training data. A final untouched test set should be used only for final evaluation. Feature selection, scaling and imputation must also be fitted within training folds to avoid data leakage.

7. Time-Series Foundations

A time series is a sequence of observations indexed in time: daily transactions, monthly revenue or quarterly employment. Time order matters because observations may be dependent. A useful conceptual decomposition is:

for an additive model, or:

for a multiplicative model. Here T_t is trend, S_t is seasonal movement that repeats at a known frequency, and R_t is the remaining irregular component. The additive form is suitable when seasonal fluctuations have roughly constant absolute size; the multiplicative form is suitable when seasonal size grows with the series level.

Trend is a persistent long-term direction. Seasonality repeats at a fixed calendar frequency, such as higher retail sales every December. A cycle is a longer, often less regular fluctuation associated with economic or business conditions. Noise is unpredictable variation. Confusing a cycle with seasonality can lead to poor forecasts.

Baseline forecasting methods

A naive forecast uses the latest observed value:

A seasonal-naive forecast uses the value from the corresponding previous season:

where m is the seasonal period and h is the forecast horizon. A moving average of width k is:

Moving averages smooth noise but lag when a trend changes. Exponential smoothing gives greater weight to recent observations:

where 0 < α < 1 and ℓ_t is the level. Simple exponential smoothing is suitable for a series without systematic trend or seasonality. Trend-adjusted and seasonal exponential-smoothing models extend this framework.

Forecast error measures

For actual value y_t and forecast ŷ_t, error is often defined as e_t = y_t − ŷ_t. Common measures include:

Mean absolute percentage error is:

MAPE is undefined when y_t = 0 and unstable near zero. It can also favour models that underforecast high-volume observations. Weighted measures or scale-dependent measures may be preferable. Forecast evaluation should reflect the cost of errors: underforecasting a hospital bed requirement may be more serious than overforecasting it.

8. Forecast Validation and Leakage Prevention

Randomly splitting time-series observations into training and test sets can leak future information into the past. Instead, use a chronological holdout or rolling-origin evaluation. In a rolling-origin design, a model is trained through time t, forecasts later periods, then the origin advances and the process repeats.

A robust workflow is:

1. define the forecasting horizon and decision context;
2. reserve the latest period as a final test set;
3. use earlier data for training and rolling validation;
4. compare against naive and seasonal-naive baselines;
5. select models using a decision-relevant error measure;
6. refit the selected specification on permitted historical data; and
7. monitor errors and recalibrate after deployment.

Predictors must be available at the time the forecast is made. Using next month’s actual promotion, a revised economic statistic or a data field created after the outcome occurs is leakage. Forecast intervals should be reported where uncertainty matters.

Regression with time-related predictors can be useful, for example:

However, residual autocorrelation may remain. Time-series regression should therefore inspect residuals over time and include lagged variables or specialised time-series error structures where justified.

9. Illustrative Case: Forecasting Fresh-Meal Demand

Background and problem

This illustrative case concerns FreshBite, a chain selling prepared meals. Stockouts reduce customer satisfaction, while excess preparation creates food waste. Managers need a seven-day-ahead forecast for each store and product category.

Analysis

The analyst examines two years of daily sales. Plots reveal a gradual upward trend, higher demand on Fridays and Saturdays, and a sharp fall on public holidays. Variance increases with sales volume. Promotional flags are known in advance for some campaigns but not for others.

The team establishes a seasonal-naive baseline: each day’s forecast initially equals sales on the same weekday from the previous week. A regression model then includes trend, day-of-week indicators, planned promotion, store traffic forecast and temperature forecast. A logarithmic response is considered because variability grows with the level.

The data are split chronologically. The most recent six weeks are held out for final testing; rolling-origin validation is conducted on earlier observations. All transformations and variable selection occur inside each training window. The team compares MAE, RMSE and the cost of stockouts versus waste.

Approach and solution

The seasonal-naive baseline performs well during stable periods but misses promotional spikes. The regression model improves accuracy when planned promotions and traffic forecasts are reliable. During an unexpected local event, both models perform poorly, demonstrating that no historical model can fully anticipate unrecorded shocks.

The final operational design combines a point forecast with an uncertainty range and an exception dashboard. Store managers can override forecasts only by recording a reason, such as a local event or supply disruption. Overrides are later evaluated rather than treated as unquestionable expertise.

Outcome and learning points

The case illustrates that model selection is not based on the highest in-sample fit. A simple baseline provides a meaningful benchmark; chronological validation reflects actual deployment; business costs determine the preferred error measure; and operational knowledge improves interpretation. Forecasts support decisions but do not eliminate uncertainty.

10. Governance, Ethics and Contemporary Practice

Regression and forecasting can affect prices, credit, staffing and access to services. Analysts should document data sources, modelling choices, exclusions, validation results and known limitations. Personal data should be minimised, protected and used for a legitimate purpose. Sensitive characteristics and their proxies require particular care because a model can reproduce historical discrimination even when protected attributes are omitted.

Performance should be examined across relevant groups where legally and ethically appropriate. Accuracy alone is not sufficient: fairness, explainability, contestability and operational impact matter. Automated decisions should have human oversight and an appeal process when consequences are significant.

Modern tools include automated forecasting systems, cloud machine-learning platforms and explainability methods. Automation does not remove the need for assumptions, leakage checks or monitoring. A transparent, slightly less accurate model may be preferable when decisions require explanation. After deployment, analysts should monitor drift, forecast errors, missingness, changes in customer behaviour and changes in the data-generating process.

Best-Practice Checklist

• Define the decision, response, horizon and cost of errors before modelling.
• Inspect data visually and substantively before fitting a model.
• Use a baseline and compare out-of-sample performance.
• Interpret coefficients in units meaningful to managers.
• Check residuals, dependence, nonlinearity, multicollinearity and influence.
• Avoid causal claims from observational regression without an appropriate design.
• Use interactions only when theory or evidence supports them, and interpret them correctly.
• Standardise predictors before regularisation and tune penalties without leakage.
• Preserve time order during forecasting validation.
• Report uncertainty, limitations and monitoring responsibilities.

Chapter Summary

Regression models estimate conditional relationships between a response and explanatory variables. Simple regression uses one predictor, while multiple regression estimates each predictor’s association while holding other included predictors constant. Least squares, residuals, sums of squares, R², RMSE and MAE provide complementary views of fit and prediction.

Reliable interpretation requires attention to linearity, independence, constant variance, influential observations, multicollinearity and specification. Categorical variables require reference coding; interactions allow effects to vary; transformations can represent curvature or alter interpretation. Ridge and lasso regularisation reduce overfitting, with lasso capable of setting coefficients to zero.

Time-series forecasting recognises trend, seasonality, cycles and noise. Naive, seasonal-naive, moving-average and exponential-smoothing methods provide practical baselines. Forecasts must be validated chronologically, assessed with decision-relevant metrics and protected against leakage. Ethical deployment requires documentation, privacy, fairness assessment, uncertainty communication and ongoing monitoring.

Key Terms

Regression: A method for estimating the relationship between a response and one or more predictors.

Response variable: The outcome a model seeks to explain or predict.

Predictor: An explanatory variable used to estimate the response.

Residual: The difference between an observed and fitted response.

Least squares: Estimation by minimising the sum of squared residuals.

Intercept: The fitted response when predictors equal zero.

Coefficient: An estimated effect or association parameter in a model.

R²: The proportion of sample response variation explained by a fitted model.

RMSE: A square-root average of squared prediction errors.

Multicollinearity: Strong overlap among predictors.

Interaction: A term indicating that one predictor’s effect depends on another.

Indicator variable: A numerical variable representing category membership.

Regularisation: Penalisation used to control model complexity.

Ridge regression: Regularised regression using squared coefficient penalties.

Lasso regression: Regularised regression using absolute-value penalties.

Time series: Observations recorded sequentially over time.

Trend: Persistent long-term movement in a series.

Seasonality: A repeating pattern at a known calendar frequency.

Forecast horizon: The future period or distance being predicted.

Data leakage: Use of information unavailable at the time a prediction would be made.

Review Questions

5 Short Answer Questions

1. What is the difference between a residual and a forecast error?
2. Interpret a regression slope while holding other predictors constant.
3. Why can a high R² fail to guarantee good predictions?
4. State two consequences of multicollinearity.
5. What distinguishes seasonality from a general trend?

5 Long Answer Questions

1. Explain the principal assumptions of multiple linear regression and describe a diagnostic for each.
2. Compare ridge and lasso regression, including their penalties and practical uses.
3. Explain how categorical variables and interaction terms are incorporated into regression.
4. Compare naive, moving-average and exponential-smoothing forecasts.
5. Explain why chronological validation is necessary for time-series forecasting.

3 Analytical / Application Questions

1. A retailer’s model has strong in-sample fit but performs poorly on the latest three months. Analyse possible causes and propose a validation and monitoring plan.
2. A promotion variable is strongly related to sales, but promotions are targeted mainly at stores already expected to grow. Explain why the coefficient may not represent a causal promotion effect and suggest a better study design.
3. A hospital must forecast daily admissions and faces much higher costs from understaffing than overstaffing. Recommend an evaluation metric, validation approach and communication method for forecast uncertainty.

Multiple Choice Questions with Answers

Question 1

In a multiple regression model, what does a coefficient usually represent?

A. The response value when all predictors are at their means

B. The expected change in the response for a one-unit predictor increase, holding other included predictors constant

C. The percentage of observations correctly classified

D. The total unexplained variation in the response

Correct Answer: B. The expected change in the response for a one-unit predictor increase, holding other included predictors constant

Question 2

Which measure is expressed in the same units as the response variable?

A. R²

B. MAPE

C. RMSE

D. VIF

Correct Answer: C. RMSE

Question 3

What is a common consequence of severe multicollinearity?

A. Coefficients become easier to interpret causally

B. Standard errors and coefficient instability may increase

C. Residuals necessarily become normally distributed

D. The sample size automatically increases

Correct Answer: B. Standard errors and coefficient instability may increase

Question 4

In an interaction model y = β₀ + β₁x + β₂z + β₃xz + ε, what does β₃ describe?

A. The intercept when x and z are zero

B. The average residual

C. How the effect of x changes as z changes

D. The total sample size

Correct Answer: C. How the effect of x changes as z changes

Question 5

Which regularisation method can set some coefficients exactly to zero?

A. Ridge regression

B. Lasso regression

C. Naive forecasting

D. Simple moving average

Correct Answer: B. Lasso regression

Question 6

A seasonal-naive forecast for a monthly series usually uses which value?

A. The overall sample mean only

B. The immediately previous observation regardless of season

C. The observation from the corresponding month in a previous season

D. The largest historical observation

Correct Answer: C. The observation from the corresponding month in a previous season

Question 7

Why should random train-test splitting usually be avoided for ordinary time-series forecasting?

A. It always reduces the training sample to zero

B. It can allow future information to influence model development

C. It prevents calculation of MAE

D. It makes seasonality impossible to measure

Correct Answer: B. It can allow future information to influence model development

Question 8

If actual demand is 120 and the forecast is 100, using e = y − ŷ, what is the forecast error?

A. −20

B. 0.83

C. 20

D. 220

Correct Answer: C. 20

Question 9

Which statement about ordinary R² is correct when an intercept is included?

A. Adding a predictor cannot decrease it in the training sample

B. It is always equal to out-of-sample accuracy

C. It directly measures causality

D. It cannot exceed zero

Correct Answer: A. Adding a predictor cannot decrease it in the training sample

Question 10

Which practice best prevents leakage during rolling forecast validation?

A. Using revised future data in every training window

B. Fitting preprocessing steps only on the relevant training window

C. Randomly shuffling all dates before splitting

D. Selecting the model using the final test set repeatedly

Correct Answer: B. Fitting preprocessing steps only on the relevant training window