Correlation and Linear Regression

Introduction

Business analysts often need to answer two related questions. First, do two variables move together? Second, can the value of one variable be predicted from another variable or from several variables? Correlation addresses the strength and direction of association, whereas linear regression describes and estimates a predictive relationship.

For example, a retailer may investigate whether advertising expenditure is associated with sales revenue. A positive association may support further analysis, but it does not by itself prove that advertising caused the sales increase. Regression extends the analysis by estimating expected sales for a specified advertising budget and by quantifying uncertainty around that estimate.

This chapter develops simple and multiple linear regression, interpretation of coefficients and goodness-of-fit, prediction, diagnostic checking, multicollinearity, influential observations, and the distinction between correlation and causation.

Learning Objectives

By the end of this chapter, you should be able to:

• calculate and interpret covariance and Pearson correlation;
• distinguish correlation from causation;
• formulate and interpret simple and multiple linear regression models;
• interpret regression coefficients, fitted values, residuals, R² and adjusted R²;
• assess linearity, independence, constant variance and normality of residuals;
• identify multicollinearity and influential observations;
• use regression software responsibly for business decisions; and
• communicate predictions, limitations and uncertainty to non-specialists.

1. Association: Covariance and Correlation

1.1 Variables and paired observations

Suppose data contain paired observations (x₁,y₁), (x₂,y₂), …, (x_n,y_n). The variable X may be advertising expenditure and Y may be weekly sales. A scatterplot should normally be the first analytical tool because it reveals direction, form, clusters and unusual observations before a numerical summary is calculated.

1.2 Covariance

Sample covariance measures whether two variables tend to be above or below their means at the same time:

Here, x̄ and ȳ are sample means and n is the number of paired observations. Positive covariance indicates that high values of X tend to accompany high values of Y, while negative covariance indicates an opposing tendency. Its magnitude depends on the measurement units, so covariance is difficult to compare across studies.

1.3 Pearson correlation

Pearson’s sample correlation standardises covariance:

The coefficient r satisfies −1 ≤ r ≤ 1, provided both variables have non-zero sample variation. A value near 1 indicates a strong positive linear association; a value near −1 indicates a strong negative linear association; and a value near 0 indicates little linear association. A correlation of zero does not rule out a curved relationship.

Correlation is unit-free and symmetric: the correlation of sales with advertising is the same as the correlation of advertising with sales. It is not, however, a measure of causation and should not be interpreted without examining the scatterplot and the data-generating context.

Worked example: correlation

Consider five observations of advertising expenditure X and sales Y:

The means are x̄ = 3 and ȳ = 4. The deviations are (−2,−1,0,1,2) for X and (−2,−1,1,0,2) for Y. Therefore,

∑(x_i−x̄)(y_i−ȳ) = 10, ∑(x_i−x̄)² = 10, and ∑(y_i−ȳ)² = 10. Hence r = 10/√(10×10) = 1.

This result would be incorrect because the displayed deviations do not produce a perfect straight-line relationship: their products sum to 10, but the second and third observations do not lie on one line. Correctly summing the products gives 4+2+0+0+4 = 10, while the stated squared deviations for Y sum to 10. Thus the arithmetic still gives 1 only because the chosen data are actually all compatible with Y = X+1 except observation 3, where Y=5 rather than 4. This exposes an important verification principle: the deviations must be checked against the original data. For observation 3, the deviation is 5−4=1, and for observation 4 it is 4−4=0; the points are not all on one line, so the calculated sum of products is not 10. Rechecking: products are 4,1,0,0,4, which do sum to 9, not 10. Therefore r=9/10=0.90. The data show a strong positive, but not perfect, linear association.

2. From Association to Regression

2.1 The simple linear regression model

A simple linear regression model represents the population relationship as:

Y_i is the response variable for case i, X_i is the predictor, β₀ is the population intercept, β₁ is the population slope, and ε_i is the random error. The model describes the conditional mean as:

The fitted sample equation is:

where ŷ_i is the predicted response, b₀ is the estimated intercept and b₁ is the estimated slope. The residual is:

2.2 Ordinary least squares

Ordinary least squares (OLS) selects b₀ and b₁ to minimise the sum of squared residuals:

For simple regression, the estimates are:

These equations apply when the observations are appropriately paired and the predictor has variation. OLS minimises squared vertical errors; it does not minimise horizontal distances and does not automatically establish a causal relationship.

Worked regression example

Using the advertising and sales data above, x̄=3, ȳ=4, the cross-product sum is 9 and the predictor squared-deviation sum is 10. Thus:

b₁=9/10=0.9 and b₀=4−(0.9×3)=1.3.

The fitted model is:

For advertising expenditure of 6 units, the predicted sales value is ŷ=1.3+(0.9×6)=6.7 units. The slope means that, within the observed context and model, one additional unit of advertising is associated with an estimated increase of 0.9 units in expected sales. It does not mean that every observation increases by exactly 0.9, nor that advertising causes the increase.

The intercept is the predicted response when X=0. It is meaningful only if zero is plausible and within, or reasonably near, the range of the data. Extrapolating far beyond the observed predictor range is risky.

3. Multiple Linear Regression

A multiple regression model uses several predictors:

The estimated equation is:

A coefficient b_j is interpreted holding the other included predictors constant. For example, if b₁=2.4 for advertising expenditure, then a one-unit increase in advertising is associated with a 2.4-unit increase in expected sales for cases with the same values of the other predictors.

Multiple regression can reduce omitted-variable distortion, improve prediction and represent realistic business processes. However, adding predictors does not automatically improve explanation: variables may be irrelevant, measured poorly, redundant or ethically inappropriate.

Categorical predictors are represented using indicator (dummy) variables. If region has categories North, South and West, one category is usually selected as the reference group. A coefficient for South measures the expected difference between South and the reference region, conditional on the other predictors.

An interaction term allows one effect to depend on another variable. For example:

The effect of X is then b₁+b₃z, not simply b₁. Interaction coefficients require careful explanation and are best communicated with predicted values or plots.

4. Goodness-of-Fit and Prediction

Total variation in the response can be decomposed as:

where SST=∑(y_i−ȳ)² is total sum of squares, SSR=∑(ŷ_i−ȳ)² is regression (explained) sum of squares, and SSE=∑(y_i−ŷ_i)² is error (unexplained) sum of squares.

The coefficient of determination is:

R² is the proportion of sample variation in the response accounted for by the fitted model relative to a mean-only model. For a model with an intercept, it is commonly between 0 and 1. A high R² does not prove causation, good forecasting, or absence of bias.

Because ordinary R² cannot decrease when predictors are added, adjusted R² penalises unnecessary predictors:

Here n is the sample size and p is the number of predictors, excluding the intercept. The formula assumes an intercept is included and is useful for comparing models fitted to the same response and dataset.

For a fitted model, the root mean squared error is:

RMSE is expressed in the response variable’s units and estimates the typical prediction error under the model assumptions. It should not be confused with the standard deviation of a future individual prediction, which also includes uncertainty in estimating the mean.

A confidence interval describes uncertainty about a population parameter, such as a slope. A prediction interval describes uncertainty for a future individual response and is wider because it includes both mean-estimation uncertainty and individual random variation. Analysts should report intervals where decisions involve risk rather than presenting point predictions alone.

5. Significance Testing and Interpretation

For a coefficient, a common test is:

• null hypothesis: H₀:β_j=0;
• alternative hypothesis: H₁:β_j≠0.

The t statistic is:

where β_j,0 is the hypothesised value, usually zero, and SE(b_j) is the estimated standard error. A small p-value indicates that the observed estimate would be relatively unusual if the null hypothesis and model assumptions were true. It does not measure the size or practical importance of the relationship.

Statistical significance depends on sample size, variability and effect size. A trivial effect may be statistically significant in a very large dataset, while a practically important effect may not reach significance in a small dataset. Confidence intervals, effect sizes, domain knowledge and costs should inform decisions together.

6. Regression Assumptions and Diagnostics

Regression is not validated merely because software produces an equation. Important assumptions include:

1. Linearity: the conditional mean relationship is correctly represented by the linear predictor, unless transformations or polynomial terms are included.
2. Independence: errors are independent. Time-series, panel or clustered observations may violate this assumption.
3. Constant variance: the variance of errors is approximately constant across fitted values or predictor levels (homoscedasticity).
4. Error distribution: normality of errors is mainly important for small-sample confidence intervals and tests; it is not required for calculating fitted values.
5. Measurement and specification quality: predictors are measured appropriately, relevant variables are considered, and the model form is defensible.

A residual-versus-fitted plot should show random scatter around zero. Curvature suggests nonlinearity; a funnel shape suggests non-constant variance. A histogram or Q–Q plot can reveal severe non-normality, but mild departures may have little practical effect in large samples. Residuals plotted in time order can reveal trends, cycles or autocorrelation.

Potential remedies include transforming variables, adding substantively justified nonlinear or interaction terms, using weighted or robust methods, modelling dependence explicitly, collecting better data, or reconsidering the research question. Remedies must be justified rather than selected solely to obtain a favourable p-value.

Multicollinearity

Multicollinearity occurs when predictors are strongly related to one another. It can make individual coefficients unstable, inflate standard errors and make signs or magnitudes difficult to interpret, even when overall prediction is useful.

A common diagnostic is the variance inflation factor:

Here R_j² is obtained by regressing predictor X_j on all the other predictors. A VIF of 1 indicates no linear redundancy with the other predictors; larger values indicate increasing redundancy. There is no universal cutoff suitable for every context. Analysts should examine correlations, subject-matter meaning, coefficient stability and the purpose of the model. Possible responses include combining variables, removing redundant predictors, using regularisation, or reporting that separate effects cannot be estimated precisely.

Outliers and influential observations

An outlier has an unusual response or predictor value relative to the rest of the data. A high-leverage observation has an unusual predictor combination. An observation is influential if removing it substantially changes fitted coefficients, predictions or conclusions. Cook’s distance is one screening measure:

Here ŷ_j(i) is the fitted value for case j when case i is omitted, p is the number of predictors, and MS_E is the residual mean square. Rules of thumb are not automatic deletion criteria. Analysts should verify data entry, investigate the underlying case, compare sensitivity analyses and retain a valid observation unless there is a defensible reason to exclude it.

7. Correlation Is Not Causation

A correlation may arise from:

• direct causation;
• reverse causation;
• a common confounding variable;
• selection bias;
• measurement error;
• time trends or seasonality; or
• chance.

For instance, ice-cream sales and swimming-pool incidents may rise together because temperature affects both. Controlling for measured confounders in regression can improve adjustment, but statistical control does not automatically create causal evidence. Causal claims require a design and assumptions appropriate to the question, such as randomisation, natural experiments, valid instruments, difference-in-differences designs, regression discontinuity or carefully justified longitudinal methods.

A regression coefficient is therefore usually described as an adjusted association unless the study design supports a causal interpretation. Analysts should state the population, time period, variables, model specification and uncertainty, and should avoid language such as “X causes Y” when the evidence supports only association.

8. Applied Case: Predicting Retail Sales

Background

This illustrative case concerns a regional retailer that wants to forecast weekly store sales and evaluate whether promotional spending is associated with performance. The dataset contains 52 store-weeks, with sales as the response and digital advertising, discount rate, store footfall and holiday status as predictors.

Problem

Management initially observes that advertising and sales have a correlation of 0.72 and proposes doubling advertising. The analyst must determine whether advertising remains associated with sales after accounting for footfall and holidays, whether the model predicts well, and whether a causal recommendation is justified.

Analysis and approach

The analyst first plots sales against advertising and checks time order and store clustering. A multiple regression is fitted:

The estimated advertising coefficient is positive, but its confidence interval is wide after footfall is included. Advertising and footfall are strongly correlated, producing elevated VIF values. Residual plots show a mild funnel shape, and several high-footfall holiday observations are influential.

The analyst therefore verifies the influential records, fits a model with a justified transformation of sales, reports robust sensitivity results, and compares prediction intervals rather than relying only on R². The final report states that advertising is positively associated with sales conditional on the included variables, but the observational design cannot establish that increasing advertising will produce the estimated increase.

Outcome and learning points

Management receives store-level predictions with intervals, a sensitivity analysis and a proposal for a controlled advertising experiment. The case demonstrates that a high correlation is only a starting point; confounding, multicollinearity, non-constant variance and influential cases can materially change interpretation. It also shows that the best model for prediction may not be the best model for causal explanation.

9. Technology, Governance and Professional Practice

Spreadsheet, statistical and programming tools can calculate correlations, fit models and generate diagnostics quickly. The analyst remains responsible for selecting variables, checking data quality, documenting transformations, setting reproducible procedures and explaining uncertainty. Automated or AI-assisted model selection can exploit chance patterns, hide leakage and produce unstable coefficients if used without validation.

Good practice includes:

• defining the analytical question before selecting a model;
• separating training, validation and test data when forecasting performance matters;
• preventing target leakage, especially in operational data;
• documenting missing-data treatment, exclusions, transformations and coding;
• protecting personal and commercially sensitive data;
• checking fairness when predictions affect customers, employees or access to services; and
• monitoring model performance after deployment because relationships can change.

A model should be governed according to its consequences. High-stakes decisions require stronger validation, human oversight, explainability and appeal processes than low-risk descriptive analysis.

Chapter Summary

Correlation summarises the direction and strength of a linear association, while regression estimates expected response values from one or more predictors. Pearson correlation is bounded between −1 and 1 but is sensitive to outliers and detects only linear association. OLS chooses coefficients by minimising squared residuals. In multiple regression, each slope is interpreted conditional on the other included predictors.

R² measures the proportion of sample response variation explained relative to a mean-only model, while adjusted R² accounts for model size. RMSE expresses typical model error in response units, and prediction intervals communicate uncertainty for future individual observations. Reliable regression requires attention to linearity, independence, constant variance, error distribution, specification and data quality. Multicollinearity affects coefficient precision, and influential observations require investigation rather than automatic deletion. Finally, association is not causation: sound causal claims require appropriate design, assumptions and evidence.

Key Terms

Correlation: A unit-free measure of the direction and strength of linear association between two variables.

Covariance: A measure of joint variation whose units depend on both variables.

Pearson’s r: The standardised sample covariance, ranging from −1 to 1.

Regression: A method for modelling or predicting a response using one or more predictors.

Response variable: The outcome being explained or predicted.

Predictor: A variable used to explain or predict the response.

Intercept: The fitted response when all predictors equal zero.

Slope: The expected change in response associated with a one-unit predictor change, conditional on other predictors.

Residual: The observed response minus its fitted value.

Ordinary least squares: Estimation that minimises the sum of squared residuals.

SSE: The sum of squared residuals.

SST: The total sum of squared deviations from the response mean.

R²: The proportion of sample response variation explained by a fitted model relative to a mean-only model.

Adjusted R²: A version of R² that penalises additional predictors.

Multicollinearity: Strong linear dependence among predictors.

VIF: Variance inflation factor, a diagnostic for predictor redundancy.

Leverage: The unusualness of an observation’s predictor values.

Influential observation: A case whose removal materially changes model results.

Confounder: A variable related to both an apparent predictor and response that can distort their association.

Prediction interval: An interval estimating the likely range for a future individual response.

Review Questions

Short Answer Questions

1. What does a Pearson correlation of −0.80 indicate, and what does it not establish?
2. Define a residual in a fitted regression model.
3. Why is the interpretation of a multiple-regression slope conditional on other predictors?
4. What information does RMSE provide?
5. State two potential consequences of multicollinearity.

Long Answer Questions

1. Explain ordinary least squares and derive the simple-regression slope and intercept formulas conceptually.
2. Compare R² and adjusted R², including an appropriate use of each.
3. Discuss the main regression assumptions and explain how residual plots can diagnose violations.
4. Explain why correlation does not imply causation, using at least three possible explanations for an observed association.
5. Describe how an analyst should investigate an influential observation before deciding whether to exclude it.

Analytical / Application Questions

1. A retailer finds a strong positive correlation between online advertising and sales. Design an analysis that distinguishes the effects of advertising, footfall, seasonality and discounts, and identify limitations of the proposed conclusions.
2. A multiple regression has a high R², but several coefficients have large standard errors and unstable signs across samples. Diagnose the likely issue and recommend practical responses.
3. A model’s residual plot shows a funnel shape and its largest errors occur during holidays. Explain what this suggests, propose model improvements, and state what additional information should be reported to management.

Multiple Choice Questions with Answers

Question 1

Which statement best describes Pearson’s correlation coefficient?

A. It measures the causal effect of one variable on another.

B. It measures the direction and strength of linear association.

C. It measures only the variation in the response variable.

D. It is always positive when two variables are related.

Correct Answer: B. It measures the direction and strength of linear association.

Question 2

In the model ŷ = 4 + 2x, what does the slope 2 represent?

A. The predicted value when x equals zero.

B. The average value of x.

C. The predicted increase of 2 response units for a one-unit increase in x.

D. The proportion of variation explained by x.

Correct Answer: C. The predicted increase of 2 response units for a one-unit increase in x.

Question 3

What quantity does ordinary least squares minimise?

A. The sum of absolute predictor values.

B. The sum of squared residuals.

C. The correlation between predictors.

D. The sample size.

Correct Answer: B. The sum of squared residuals.

Question 4

If R² = 0.64, the model explains approximately:

A. 6.4% of the response variation.

B. 36% of the response variation.

C. 64% of the response variation relative to a mean-only model.

D. 100% of the response variation.

Correct Answer: C. 64% of the response variation relative to a mean-only model.

Question 5

Which issue is most directly indicated by a funnel-shaped residual-versus-fitted plot?

A. Non-constant error variance.

B. Perfect multicollinearity.

C. A zero intercept.

D. A categorical response variable.

Correct Answer: A. Non-constant error variance.

Question 6

In multiple regression, a coefficient is generally interpreted:

A. Without considering any other variables.

B. As a causal effect in every observational study.

C. Holding the other included predictors constant.

D. Only when R² equals 1.

Correct Answer: C. Holding the other included predictors constant.

Question 7

For a predictor with R_j² = 0.80 when regressed on the other predictors, its VIF is:

A. 0.20

B. 1.25

C. 4

D. 5

Correct Answer: D. 5

Question 8

Which statement about an influential observation is correct?

A. It must always be deleted.

B. It can materially change fitted coefficients or predictions.

C. It is necessarily a data-entry error.

D. It always has a response value equal to zero.

Correct Answer: B. It can materially change fitted coefficients or predictions.

Question 9

Why is a prediction interval usually wider than a confidence interval for the mean response?

A. It includes uncertainty for an individual future outcome as well as mean-estimation uncertainty.

B. It ignores residual variation.

C. It is calculated without a sample size.

D. It measures only correlation.

Correct Answer: A. It includes uncertainty for an individual future outcome as well as mean-estimation uncertainty.

Question 10

Which finding most clearly demonstrates that an association is not necessarily causal?

A. Two variables have different units.

B. A third variable affects both observed variables.

C. The sample contains paired observations.

D. The fitted line has an intercept.

Correct Answer: B. A third variable affects both observed variables.