Correlation and Regression Modelling

Introduction and context

Correlation and regression are central tools in statistical and predictive analytics. Correlation describes the strength and direction of association between variables, whereas regression develops an equation for estimating or predicting a response variable from one or more explanatory variables. In business, these methods support demand forecasting, pricing analysis, customer-value modelling, workforce planning, credit assessment and performance evaluation.

A regression model is not automatically a causal model. A strong association may arise because variables share a common cause, because of reverse causality, coincidence, measurement error or biased sampling. Sound analysis therefore combines statistical computation with subject-matter knowledge, appropriate design, diagnostics and cautious communication.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish correlation from regression and causation;
• estimate and interpret simple and multiple linear regression models;
• use categorical predictors, dummy variables and interaction terms;
• assess model fit and test regression coefficients;
• diagnose nonlinearity, non-normality, heteroscedasticity, multicollinearity and influential observations;
• use regression responsibly for prediction and managerial decision-making.

1. Correlation: measuring linear association

The Pearson correlation coefficient, r, measures the direction and strength of a linear relationship between two quantitative variables X and Y. It is defined by:

Here xᵢ and yᵢ are observations, x̄ and ȳ are sample means, and the summation is over n paired observations. The coefficient has no units and lies between −1 and +1. A positive value indicates that larger values of X tend to accompany larger values of Y; a negative value indicates the opposite. Values near zero indicate little linear association, not necessarily no relationship.

Pearson correlation is most informative when observations are independent, variables are measured quantitatively, the relationship is approximately linear, and extreme outliers are absent or understood. It is sensitive to outliers and does not adequately describe curved relationships. Rank correlation, such as Spearman’s coefficient, may be preferable for ordinal data or monotonic but non-linear relationships.

Correlation is symmetric: the correlation of X with Y equals the correlation of Y with X. Regression is asymmetric because it identifies a response variable and explanatory variable(s). Neither correlation nor regression alone proves causation.

2. Simple linear regression

Simple linear regression models the conditional mean of a response variable Y using one predictor X:

Yᵢ is the response for observation i, Xᵢ is the predictor, β₀ is the population intercept, β₁ is the population slope, and εᵢ is the random error. The fitted sample equation is:

where ŷᵢ is the predicted response and b₀ and b₁ are estimated from data. The residual is eᵢ = yᵢ − ŷᵢ.

The ordinary least squares method chooses b₀ and b₁ to minimize the sum of squared residuals:

The estimates are:

The slope has response-units per predictor-unit. The intercept is the predicted value when X equals zero, but it is meaningful only when X = 0 is plausible and within, or reasonably close to, the observed range.

Worked example: advertising and sales

Suppose a retailer records weekly advertising expenditure X, measured in thousands of dollars, and sales Y, measured in thousands of dollars. The fitted model is:

The slope means that an additional $1,000 of advertising is associated with an average increase of $3,600 in weekly sales, within the observed range and subject to the model assumptions. At X = 8, predicted sales are:

Thus predicted sales are $70,800. This is a prediction, not a guaranteed outcome and not necessarily a causal estimate of advertising effectiveness.

3. Measuring model fit and uncertainty

The total variation in Y can be partitioned into explained and unexplained variation:

where SST is total sum of squares, SSR is regression sum of squares, and SSE is error sum of squares. The coefficient of determination is:

R² is the proportion of sample variation in Y explained by the fitted model. It ranges from 0 to 1 when an intercept is included. A high R² does not establish causation, good forecasting, or an appropriate model. A low R² may still be useful when individual outcomes are inherently variable but average effects are important.

For a model with p predictors and n observations, adjusted R² is:

It penalizes the addition of predictors that contribute little explanatory value. It should not be used as the sole criterion for model selection.

Under the classical linear model assumptions, an estimated coefficient bⱼ has standard error SE(bⱼ), and its t statistic for testing H₀: βⱼ = βⱼ⁰ is:

Usually βⱼ⁰ = 0. The test uses n − p − 1 degrees of freedom when an intercept and p predictors are estimated. A confidence interval is:

where t* is the relevant critical value. A p-value measures compatibility with the null hypothesis under the model; it does not measure practical importance or the probability that the null hypothesis is true.

A confidence interval for the mean response at a specified X is narrower than a prediction interval for one future observation, because the latter includes both uncertainty in the estimated mean and individual random variation.

4. Multiple linear regression

Multiple regression estimates the relationship between a response and several predictors:

The coefficient βⱼ represents the expected change in Y associated with a one-unit increase in Xⱼ, holding the other included predictors constant. This “holding constant” interpretation is conditional and may be unrealistic if predictors are strongly related or if important variables are omitted.

For example:

If Experience is measured in months, Certification is coded 1 for possession and 0 otherwise, and Workload is measured in hours per week, the certification coefficient compares otherwise similar employees and predicts an average 2.8-unit difference. It should not automatically be described as the causal effect of certification.

The overall F test evaluates:

• H₀: β₁ = β₂ = … = βₚ = 0;
• H₁: at least one slope differs from zero.

The statistic is:

This test is useful for assessing whether the predictor set contributes collectively, but it does not identify which predictor matters or whether the model is substantively useful.

5. Categorical predictors and dummy variables

A categorical variable must be represented numerically before inclusion in a linear model. For a binary variable, a dummy variable takes values 0 and 1. If Region is coded 1 for urban and 0 for rural, its coefficient is the predicted urban–rural difference after controlling for other variables.

For a variable with k categories, use k − 1 dummy variables when an intercept is included. One category is the reference category. For four customer segments, the model might include Premium, Standard and Student indicators, with Basic as reference. Each coefficient compares its category with Basic, holding other predictors constant.

Including all k dummies together with an intercept creates perfect multicollinearity because the dummies sum to one. Alternatives include removing the intercept, but this changes interpretation and is usually less transparent.

6. Interactions and moderation

An interaction allows the effect of one predictor to depend on another. A model involving advertising expenditure A and price P is:

The marginal effect of advertising is:

Thus β₃ measures how the advertising slope changes as price changes. Main effects in an interaction model are conditional: β₁ is the advertising effect when P = 0, and β₂ is the price effect when A = 0. Centre continuous predictors, often by subtracting their means, to make these main effects more meaningful and reduce nonessential correlation between main-effect and interaction terms. Centring does not change fitted values or the substantive interaction pattern.

Interactions should be interpreted with predicted values, marginal-effects tables or plots rather than by reading β₃ alone. A statistically significant interaction may be too small to matter commercially, while a practically important interaction may be estimated imprecisely in a small sample.

7. Regression assumptions and diagnostic practice

A useful diagnostic process examines the data, the fitted model and the business context. Important assumptions are:

1. Linearity: the conditional mean of Y is correctly represented by the predictors and their transformations.
2. Independence: observations or errors are independent, unless dependence is explicitly modelled.
3. Exogeneity: the error has conditional mean zero, E(ε|X) = 0. Violations can produce biased coefficients, often through omitted variables, simultaneity or selection.
4. Constant variance: Var(ε|X) is constant when conventional standard errors are used.
5. Normality of errors: mainly important for small-sample t and F inference, not necessary for unbiased least-squares coefficients or large-sample prediction.
6. No perfect multicollinearity: no predictor is an exact linear combination of others.

Residual-versus-fitted plots help detect curvature and unequal spread. A funnel shape suggests heteroscedasticity. A Q–Q plot assesses whether residuals are approximately normal. Residuals should be plotted against time, geography or operational sequence when dependence is plausible.

Heteroscedasticity does not necessarily bias ordinary least-squares coefficients, but conventional standard errors, confidence intervals and tests may be unreliable. Remedies include transforming the response, modelling the variance, using weighted least squares when a variance structure is known, or using heteroscedasticity-robust standard errors. Robust standard errors address inference; they do not repair nonlinearity, omitted-variable bias or poor predictions.

8. Multicollinearity

Multicollinearity occurs when predictors are strongly related. It does not usually bias least-squares estimates, but it inflates standard errors, makes coefficients unstable, complicates interpretation and can produce unexpected signs. A model may have good overall fit while individual coefficients are imprecise.

The variance inflation factor for predictor j is:

Rⱼ² is obtained by regressing predictor j on all the other predictors. VIF = 1 indicates no linear relationship with the other predictors. Rules such as 5 or 10 are screening conventions, not universal laws; context, coefficient stability and purpose should guide judgement.

Possible responses include removing redundant variables, combining conceptually related measures, collecting more informative data, using regularization such as ridge regression, or changing the research question. Do not delete a theoretically essential variable solely because its p-value is large.

9. Outliers, leverage and influence

An outlier has an unusual response or predictor value relative to the model. A high-leverage observation has unusual predictor values. An influential observation substantially changes fitted results when included or removed. These are different concepts: a point can have high leverage but little influence if it follows the general pattern.

Studentized residuals, leverage values and Cook’s distance are useful diagnostic measures. Cook’s distance compares the fitted model with and without an observation; a common screening rule is to investigate values around 4/n or larger, but no single cutoff proves that an observation is erroneous.

Investigate the source, measurement process and business meaning of unusual observations. Correct data errors, use robust or transformed models when justified, and report sensitivity analyses. Deleting inconvenient observations merely to improve significance is unethical and undermines reproducibility.

10. Model development and validation

A credible workflow is:

1. define the decision and response variable;
2. understand the data-generating process and measurement units;
3. inspect distributions, missingness and unusual values;
4. specify a model based on theory and operational knowledge;
5. fit the model using a training sample or cross-validation strategy;
6. examine residuals, collinearity and influence;
7. validate predictions on data not used for fitting;
8. communicate uncertainty, assumptions and limitations.

For prediction, mean absolute error is:

and root mean squared error is:

Both are in the units of Y. RMSE penalizes large errors more heavily than MAE. Mean absolute percentage error can be misleading when actual values are zero or close to zero. Validation must respect time order in forecasting and group structure where observations are clustered.

Linear regression remains valuable because it is transparent, computationally efficient and interpretable. Regularized regression can improve prediction when many correlated predictors exist. Ridge regression minimizes:

and lasso regression minimizes:

where λ ≥ 0 controls penalization and sums generally exclude the intercept. These methods trade some bias for lower variance; coefficients should not be interpreted as ordinary least-squares estimates without care.

Applied case: an illustrative customer-retention model

Background. A subscription company wants to identify factors associated with monthly customer spending and improve retention planning. It has records for 1,200 customers, including monthly spend, tenure, service incidents, customer-support contacts, plan type and satisfaction score.

Problem. Managers initially propose increasing support contacts because customers with more contacts have higher spending. Analysts warn that high-value customers may contact support more often, so the association does not establish that contacts cause spending.

Analysis. The team first plots spending against tenure and satisfaction, checks missing values and defines a pre-specified validation sample. Plan type is represented with reference-category dummies. A multiple model includes tenure, satisfaction, service incidents, plan indicators and a satisfaction-by-plan interaction. Residual plots reveal increasing variance for high-spending customers; the team reports heteroscedasticity-robust standard errors and evaluates a log-spending sensitivity model. VIF values are inspected because satisfaction and service incidents are related.

Approach and outcome. The final report presents coefficient estimates with confidence intervals, validation MAE and predicted spending for representative customer profiles. The interaction indicates that satisfaction is more strongly associated with spending in the premium plan, but the confidence interval is wide. Managers use the model to prioritise customer research and targeted service improvements, not to claim that a support contact will mechanically increase spending.

Learning points. The case demonstrates the importance of reference categories, conditional interpretation, interactions, diagnostics, out-of-sample validation and the distinction between prediction and causal explanation. It also shows why a statistically significant relationship should be translated into operational value before action is taken.

Managerial communication, ethics and governance

A good managerial report states the decision purpose, unit of analysis, sample period, variable definitions, model specification, validation method and important limitations. Report effect sizes in operational units, not only p-values. Explain whether conclusions concern association, prediction or a defensible causal comparison.

Regression used in lending, recruitment, insurance or customer eligibility may affect people unequally. Analysts should examine subgroup performance, disparate error rates, proxy variables, privacy risks and data provenance. Fairness cannot be guaranteed by removing protected attributes because correlated variables may preserve their information. Governance should include documentation, access controls, reproducible code, versioned data, human review and monitoring for drift.

Automated or AI-assisted model selection does not remove the need for assumptions, domain knowledge or accountability. A model that predicts well historically may fail after a policy change, economic shock or change in customer behaviour. Monitor performance and recalibrate or retrain only through a documented process.

Chapter Summary

Correlation measures the direction and strength of linear association, while regression models a conditional response and supports estimation or prediction. Simple regression uses one predictor; multiple regression controls for several predictors, although “holding constant” does not automatically imply causation. Least squares estimates coefficients by minimising squared residuals. R², adjusted R², coefficient tests and prediction errors answer different questions.

Categorical predictors require carefully chosen dummy variables and a reference category. Interaction terms allow effects to vary across conditions and should be interpreted through conditional predictions. Reliable regression requires attention to linearity, independence, exogeneity, variance, normality for small-sample inference and multicollinearity. Residual plots, VIF, leverage and influence diagnostics reveal weaknesses that summary fit statistics can conceal. Robust standard errors, transformations, alternative specifications and validation can improve analysis, but they cannot substitute for sound design.

The managerial value of regression depends on transparent communication, practical effect sizes, out-of-sample validation, ethical governance and continuous monitoring. Above all, analysts must distinguish association from causation and statistical significance from business importance.

Key Terms

Correlation: A numerical measure of the direction and strength of association between two variables.

Pearson correlation: A measure of linear association based on centred products and standard deviations.

Regression: A statistical method for modelling a response using one or more predictors.

Response variable: The outcome being explained or predicted.

Predictor: A variable used to explain or predict the response.

Intercept: The predicted response when all predictors equal zero.

Slope: The expected change in the response for a one-unit predictor change, conditional on other predictors.

Residual: The observed response minus its fitted value.

Ordinary least squares: Estimation by minimising the sum of squared residuals.

R²: The proportion of sample response variation explained by a fitted model.

Adjusted R²: An R² measure adjusted for sample size and number of predictors.

Dummy variable: A zero–one indicator representing a category.

Reference category: The category against which other dummy-coded categories are compared.

Interaction: A model term showing that one predictor’s effect depends on another predictor.

Heteroscedasticity: Non-constant error variance across observations or fitted values.

Multicollinearity: Strong linear association among predictors.

Leverage: The unusualness of an observation’s predictor values.

Influential observation: An observation that materially changes model results.

Validation: Assessment of performance on data not used to fit the model.

Exogeneity: The condition that the error has conditional mean zero given the predictors.

Review Questions

Short Answer Questions

1. Distinguish correlation from regression.
2. What does a regression slope represent when other predictors are held constant?
3. Why is one category omitted when dummy variables are used with an intercept?
4. What does a funnel pattern in a residual plot suggest?
5. Why does correlation not establish causation?

Long Answer Questions

1. Explain the assumptions of the classical linear regression model and describe how each can be assessed.
2. Explain R², adjusted R², coefficient tests and prediction-error measures, including the question each addresses.
3. Describe how dummy variables and interaction terms are specified and interpreted in a business model.
4. Explain multicollinearity, its consequences and defensible responses.
5. Discuss how influential observations should be investigated and reported.

Analytical / Application Questions

1. A retailer obtains a high R² model of sales using price, advertising and store size, but advertising has a negative coefficient. Propose a diagnostic and substantive investigation before recommending a decision.
2. A bank’s credit model performs well overall but has higher false-negative rates for a protected subgroup. Explain the statistical and governance analyses required before deployment.
3. Design a regression workflow for forecasting monthly demand when observations are ordered over time and promotional variables include several categorical campaign types.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes correlation from regression?

A. Correlation identifies a response variable, while regression does not.

B. Regression is symmetric, while correlation is asymmetric.

C. Correlation measures linear association, while regression models a response conditional on predictors.

D. Correlation proves causation, while regression only measures association.

Correct Answer: C. Correlation measures linear association, while regression models a response conditional on predictors.

Question 2

In the model ŷ = 12 + 2.5X, what is the predicted value when X = 4?

A. 14.5

B. 18.0

C. 20.0

D. 22.0

Correct Answer: D. 22.0

Question 3

If a binary variable is coded 1 for urban and 0 for rural, its coefficient is interpreted as:

A. The predicted urban–rural difference, conditional on other predictors.

B. The percentage of urban observations in the sample.

C. The correlation between region and the response.

D. The predicted response when every predictor equals one.

Correct Answer: A. The predicted urban–rural difference, conditional on other predictors.

Question 4

What problem occurs when all categories are dummy-coded along with an intercept?

A. Heteroscedasticity

B. Perfect multicollinearity

C. Autocorrelation

D. Non-normal residuals

Correct Answer: B. Perfect multicollinearity

Question 5

A funnel-shaped residual-versus-fitted plot most commonly indicates:

A. Constant variance

B. Perfect prediction

C. Heteroscedasticity

D. A categorical predictor

Correct Answer: C. Heteroscedasticity

Question 6

A VIF of 1 for a predictor indicates:

A. The predictor has no linear relationship with the other predictors.

B. The predictor is statistically significant.

C. The model has an R² of 1.

D. The residuals are normally distributed.

Correct Answer: A. The predictor has no linear relationship with the other predictors.

Question 7

In a model containing an interaction between advertising and price, the interaction coefficient describes:

A. The average advertising expenditure.

B. How the advertising effect changes as price changes.

C. The total number of observations.

D. The model’s unexplained variance.

Correct Answer: B. How the advertising effect changes as price changes.

Question 8

Which measure is expressed in the same units as the response variable?

A. R²

B. VIF

C. MAE

D. The correlation coefficient

Correct Answer: C. MAE

Question 9

What is a high-leverage observation?

A. An observation with an unusual predictor pattern

B. An observation with a missing response

C. An observation with a guaranteed causal effect

D. An observation whose residual is exactly zero

Correct Answer: A. An observation with an unusual predictor pattern

Question 10

Which statement about a statistically significant regression coefficient is correct?

A. It proves the predictor causes the response.

B. It guarantees the effect is commercially important.

C. It indicates evidence against the null value under the model assumptions.

D. It means the model will predict accurately for all future cases.

Correct Answer: C. It indicates evidence against the null value under the model assumptions.