Association, Correlation and Regression

Introduction

Business analytics frequently asks whether two variables move together and whether one variable can be used to predict another. Examples include advertising expenditure and sales, delivery time and customer satisfaction, employee training and productivity, and website visits and online purchases. Association, correlation and regression provide related but distinct tools for answering these questions.

Association describes a general relationship between variables. Correlation measures the strength and direction of a linear relationship between two quantitative variables. Regression develops an equation for estimating or predicting a response variable from one or more explanatory variables. None of these methods, by themselves, proves that one variable causes another.

The central analytical discipline is to combine numerical results with sound reasoning about data quality, study design, assumptions and the business context.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish association, correlation, prediction and causation;
• calculate and interpret covariance and Pearson’s correlation coefficient;
• fit and interpret simple and multiple linear regression models;
• explain least-squares estimation, residuals and goodness of fit;
• assess important regression assumptions using graphs and diagnostics;
• use categorical predictors through dummy variables;
• identify common sources of misleading regression results; and
• communicate regression findings responsibly in an organisational setting.

1. Association and the language of variables

A variable is an attribute measured for each observational unit, such as a customer, transaction, employee or store. In regression terminology, the response variable, commonly denoted by Y, is the outcome to be explained or predicted. An explanatory or predictor variable, commonly denoted by X, is used to account for variation in Y.

An association exists when the distribution or typical value of one variable differs according to the value of another. Association may be:

• positive: larger values of X tend to accompany larger values of Y;
• negative: larger values of X tend to accompany smaller values of Y;
• nonlinear: Y changes with X but not at a constant linear rate; or
• absent or weak: no systematic pattern is evident.

Association does not necessarily mean that every observation follows the same pattern. A scatterplot is therefore an essential first step. It can reveal curvature, clusters, outliers, unequal spread and data-entry errors that a single numerical coefficient may conceal.

Figure: A positive association with observations scattered around a fitted line.

2. Covariance and correlation

Covariance

Covariance indicates whether two variables tend to deviate from their means in the same direction. For a sample of n paired observations, sample covariance is:

Here x_i and y_i are the values for observation i, x̄ and ȳ are sample means, and n is the number of pairs. The denominator n − 1 provides the usual unbiased sample estimate of population covariance. Positive covariance indicates generally same-direction deviations; negative covariance indicates opposite-direction deviations.

Covariance is difficult to compare across datasets because its units are the product of the units of X and Y. For example, covariance between advertising dollars and sales dollars has dollar-squared units. Its magnitude also changes when either variable is rescaled.

Pearson’s correlation coefficient

Pearson’s sample correlation standardises covariance:

where s_X and s_Y are the sample standard deviations. An equivalent computational form is:

Correlation has no units and lies between −1 and +1. Values near +1 indicate a strong positive linear relationship, values near −1 a strong negative linear relationship, and values near 0 little linear relationship. These descriptions are contextual rather than universal cut-offs: a correlation of 0.60 may be valuable in one field and inadequate in another.

Pearson correlation measures linear association. A correlation near zero does not prove that variables are unrelated; a strong curved relationship can have a small Pearson correlation. Correlation is also sensitive to outliers and can be distorted by restricted ranges, subgroup mixtures or time trends.

Worked example: covariance and correlation

Suppose weekly training hours X and productivity scores Y for four employees are:

The means are x̄ = 2.5 and ȳ = 60. The deviations are (−1.5, −1), (−0.5, −5), (0.5, 5), and (1.5, 10). The sum of cross-products is 27.5, so:

The sums of squared deviations are 5 and 150. Thus:

Correcting the arithmetic, √750 ≈ 27.386, so r ≈ 1.005 is impossible because the listed deviations for Y are incorrect: with mean 60, the Y deviations are −10, −5, 5, 10, whose cross-products are 15, 2.5, 2.5 and 15, summing to 35; the sum of Y squares is 250. Therefore:

and

The points lie exactly on a straight line, so the correlation is +1. This example also illustrates why every numerical result should be checked against the data and against the logical bounds of the statistic.

3. Simple linear regression

Simple linear regression models the conditional mean of Y using one predictor X:

Y_i is the observed response, β_0 is the population intercept, β_1 is the population slope, and ε_i is the random error for observation i. The fitted sample equation is:

where ŷ_i is the predicted response, b_0 is the estimated intercept and b_1 is the estimated slope. The slope is estimated by:

and the intercept by:

The slope has units of Y per unit of X. The intercept is the predicted mean Y when X = 0; it should not be interpreted substantively if X = 0 is outside the observed or realistic range.

Least squares and residuals

A residual is the observed minus fitted value:

The least-squares method chooses b_0 and b_1 to minimise the residual sum of squares:

With an intercept in the model, the residuals sum to approximately zero and the fitted line passes through (x̄, ȳ). Least squares is not automatically the best method for every dataset; it is most useful when a linear conditional mean and reasonably well-behaved errors are plausible.

Worked regression example

Using the training data above, the slope is:

The intercept is:

The fitted model is therefore:

An employee completing 3 hours of training has a predicted productivity score of 63.5. The slope means that, within this small dataset and its assumptions, one additional training hour is associated with a 7-point increase in the predicted score. It does not establish that training causes the increase.

4. Measuring model fit

The coefficient of determination is:

where total sum of squares is:

With an intercept, SST = SSR + SSE, where SSR is the regression sum of squares. R² is the proportion of sample variation in Y explained by the fitted linear relationship with the predictors. It ranges from 0 to 1 in the usual intercept model. An R² of 0.64 means that 64% of observed variation in the sample response is accounted for by the fitted predictors, not that the model is 64% accurate and not that 64% of individual outcomes are caused by the predictors.

In multiple regression, adding predictors cannot reduce ordinary R², even if they contribute little practical value. Adjusted R² applies a penalty for model size:

where n is sample size and p is the number of predictors, excluding the intercept. Adjusted R² is useful for comparison but is not a substitute for theory, validation or residual analysis.

Prediction error should also be reported in the response variable's units. A common measure is the root mean squared error:

This estimates the typical residual scale under the model assumptions. For out-of-sample performance, calculate RMSE or mean absolute error on a validation or test dataset rather than only on training data.

5. Multiple linear regression

Multiple regression extends the model to p predictors:

The estimated equation is:

Each slope b_j is a partial association: the expected change in Y for a one-unit increase in X_j while holding the other included predictors constant. This “holding constant” interpretation is mathematical and conditional; it may not represent a realistic intervention if predictors naturally change together.

For example, a delivery-time model might be:

If congestion is measured on a 1–5 scale and Weekend is coded 1 for weekend and 0 otherwise, the congestion coefficient means a one-point increase is associated with 1.6 additional predicted minutes, conditional on driver experience and weekend status. The weekend coefficient compares otherwise similar weekend and non-weekend cases.

Multiple regression is often more useful than bivariate correlation because business outcomes have several simultaneous influences. However, adding variables can introduce multicollinearity, overfitting or adjustment for a variable that lies on the causal pathway.

6. Categorical predictors and dummy variables

A nominal category cannot be inserted into a regression equation as arbitrary numbers because the numbers would imply a false order and spacing. Instead, use indicator, or dummy, variables. For a binary variable:

The category coded 0 is the reference category. In a model with an intercept, k categories require k − 1 dummy variables. Including all k dummies alongside an intercept creates perfect multicollinearity, known as the dummy-variable trap.

Suppose service channel has categories Store, Web and Mobile. Choose Store as reference and define Web = 1 for web transactions and Mobile = 1 for mobile transactions. In:

b_0 is the predicted mean for Store, b_1 is the Web-versus-Store difference, and b_2 is the Mobile-versus-Store difference, conditional on other predictors.

Interactions allow an effect to differ across groups. For example:

For D = 0, the slope of X is b_1. For D = 1, the slope is b_1 + b_3. Interaction terms should be motivated by a plausible business mechanism and interpreted jointly, rather than judged only by an isolated p-value.

7. Regression assumptions and diagnostics

Regression conclusions depend on assumptions. They should be assessed rather than merely stated.

Linearity. The conditional mean of Y should be reasonably linear in the predictors. A residual-versus-fitted plot with systematic curves suggests missing nonlinear terms or transformations.

Independence. Errors should not be correlated. Time-series observations, repeated customer records and clustered employees often violate this assumption. Random sampling, suitable study design or time-series/multilevel methods may be required.

Constant variance. The spread of errors should be approximately stable across fitted values. A funnel shape suggests heteroscedasticity. Robust standard errors, transformation or a model with nonconstant variance may be appropriate.

Normality of errors. Normally distributed errors are mainly important for small-sample confidence intervals and hypothesis tests, not for calculating predictions themselves. A Q–Q plot is more informative than relying only on a formal test.

No severe multicollinearity. Predictors should not be nearly redundant. Correlation matrices, variance inflation factors and subject-matter review can help detect the problem. Multicollinearity inflates standard errors and makes individual coefficients unstable, although overall prediction may remain reasonable.

No influential observations. An outlier has an unusual response; a high-leverage point has unusual predictor values; an influential point materially changes the fitted model. Studentised residuals, leverage and Cook’s distance are useful diagnostics, but deleting observations should require a documented substantive reason.

Extrapolation is another danger. A regression equation estimated for customers aged 18–65 may be unreliable for age 90, even if the arithmetic produces a number. Predictions should remain within a defensible domain and should include uncertainty where decisions warrant it.

8. Inference and communication

A slope estimate is uncertain because it is based on a sample. A confidence interval for β_j is generally reported as:

where SE(b_j) is the estimated standard error and t* is a critical value from the appropriate t distribution, under the model assumptions. A 95% confidence interval describes a procedure that captures the fixed population parameter in 95% of repeated samples under repeated sampling; it is not correctly interpreted as a 95% probability that the particular fixed parameter lies in this interval.

A p-value tests a specified null hypothesis, commonly H₀: β_j = 0. Statistical significance does not establish practical importance, causation or a useful predictive model. Report the estimate, units, interval, sample size, validation performance and relevant limitations.

9. Association, prediction and causation

These ideas must be separated:

A confounding variable influences both X and Y. For example, ice-cream sales and emergency hospital visits may rise together because temperature affects both. Time trends can create spurious correlations, and aggregation can reverse relationships between groups, a phenomenon related to Simpson’s paradox. Regression adjustment can reduce confounding only when relevant variables are measured and the causal structure is appropriate; it is not a universal cure.

Randomised experiments support causal interpretation when treatment is properly assigned and implementation is sound. In observational analytics, causal claims require explicit assumptions, a defensible design and often methods such as matching, difference-in-differences or instrumental variables. These methods require advanced justification and should not be presented as automatic replacements for randomisation.

10. Illustrative case: predicting retail demand

Background. A fictional supermarket chain wants to forecast weekly sales for each store and understand whether promotional spending is associated with demand.

Problem. Managers observe that stores with larger advertising budgets often have higher sales. However, large stores also have more floor space and are more likely to be located in high-income areas. A simple correlation could therefore exaggerate the role of advertising.

Analysis. The analytics team defines weekly sales as Y and includes advertising expenditure, store floor area, average local income, holiday status and store format. Store format is represented with dummy variables, using standard store as the reference. A training dataset is used to estimate the model, and later weeks are reserved for validation.

Approach. The team first examines scatterplots and correlations, then fits a multiple regression. Residual-versus-fitted plots show increasing variance for very large stores, so prediction intervals and robust standard errors are considered. Variance inflation factors reveal strong overlap between floor area and advertising budget. Managers decide to retain both because they represent distinct decisions, but they interpret individual coefficients cautiously. Performance is reported using validation RMSE and compared with a simple historical-average benchmark.

Outcome. The model improves forecast accuracy over the benchmark, but the advertising coefficient is described as a conditional association, not a guaranteed incremental sales effect. To estimate causal return on advertising, the chain plans a geographically staggered experiment in which comparable stores receive different campaign intensities.

Learning points. A useful model can support prediction without proving causation. Feature definition, reference categories, validation, residual checks and experimental follow-up are all necessary for responsible business action.

11. Technology, governance and best practice

Spreadsheet software, statistical packages and Python or R can calculate coefficients, diagnostics and validation metrics efficiently. Automation does not remove the need to inspect data, define variables, check units or explain assumptions. Machine-learning systems may use nonlinear models and regularisation for prediction, but ordinary linear regression remains valuable for transparency and interpretable conditional effects.

Good practice includes:

1. define the decision and the target population before modelling;
2. inspect missing values, measurement units, duplicates and outliers;
3. visualise relationships before fitting a line;
4. avoid selecting variables solely because they improve in-sample fit;
5. separate training, validation and test information where prediction is intended;
6. document coding choices, reference categories and transformations;
7. report uncertainty and practical effect size, not only p-values; and
8. protect privacy, avoid discriminatory proxies and monitor model performance after deployment.

Ethically, a regression model can reproduce historical discrimination if predictors encode unequal access or biased decisions. Sensitive variables may be required for fairness auditing even when they are excluded from prediction. Governance should include version control, reproducibility, access controls, model documentation and a process for challenging automated decisions.

Challenges and limitations

Regression results can be undermined by nonrepresentative samples, missing-not-at-random data, measurement error, omitted variables, reverse causality, selection bias and data leakage. A high R² can coexist with poor future performance, while a modest R² may still support valuable decisions in a noisy environment. Statistical significance becomes easier to obtain with very large samples and harder with small samples; neither alone determines business value.

The analyst should therefore treat regression as a structured way to summarise patterns and generate predictions, not as a substitute for research design or professional judgement.

Chapter Summary

Association describes a relationship between variables, while covariance records joint deviations in the original measurement units and correlation standardises covariance to a unit-free measure between −1 and +1. Pearson correlation measures linear association and can be distorted by outliers, restricted ranges and nonlinear patterns.

Simple linear regression estimates a response from one predictor; multiple regression estimates a response from several predictors while interpreting each slope conditionally on the others. Least squares minimises squared residuals. R² summarises in-sample explained variation, whereas RMSE and validation errors describe prediction performance. Dummy variables represent categorical predictors relative to a reference category, and interaction terms allow slopes to differ between groups.

Reliable regression requires attention to linearity, independence, constant variance, error distributions, multicollinearity, influential observations and extrapolation. Most importantly, association and prediction do not by themselves establish causation. Sound analysis combines statistical computation, graphical diagnostics, study design, ethical governance and clear communication.

Key Terms

Association: A systematic relationship or dependence between variables.

Covariance: A measure of joint deviation from two variables’ means, expressed in product units.

Correlation: A standardised measure of linear association.

Response variable: The outcome being explained or predicted.

Explanatory variable: A variable used to explain or predict the response.

Simple linear regression: Regression with one explanatory variable.

Multiple linear regression: Regression with two or more explanatory variables.

Intercept: Predicted response when all predictors equal zero.

Slope: Expected change in the response for a one-unit predictor change, conditional on other predictors.

Residual: Observed response minus fitted response.

Least squares: Estimation by minimising the sum of squared residuals.

R²: Proportion of sample response variation explained by a fitted model.

RMSE: Root mean squared prediction or residual error in response units.

Dummy variable: A zero–one indicator representing a category.

Reference category: The category against which other dummy-coded categories are compared.

Multicollinearity: Strong linear dependence among predictors.

Heteroscedasticity: Nonconstant error variance.

Leverage: Unusualness of an observation’s predictor values.

Confounding: A third-variable influence that distorts an observed relationship.

Extrapolation: Prediction beyond the range or domain of the observed data.

Review Questions

Short Answer Questions

1. What is the difference between covariance and correlation?
2. What does the slope in a simple linear regression model represent?
3. Define a residual and explain its role in least-squares estimation.
4. Why is one category omitted when dummy variables are used with an intercept?
5. Name two regression assumptions and describe one diagnostic for each.

Long Answer Questions

1. Explain why a strong correlation does not establish causation. Use a business example involving a possible confounding variable.
2. Describe the interpretation of R², adjusted R² and RMSE, and explain why none should be reported without context.
3. Explain the assumptions of multiple linear regression and the consequences of violating each major assumption.
4. Describe how categorical variables and interaction terms can be incorporated into a regression model.
5. Compare the purposes of simple regression, multiple regression and a randomised experiment.

Analytical / Application Questions

1. A retailer finds a correlation of 0.82 between advertising expenditure and sales. Identify at least three reasons why this result may not support an immediate increase in advertising and propose additional analyses.
2. A regression model predicts delivery time using distance, traffic score and weekend status. Explain how you would interpret the weekend coefficient and evaluate whether the model is suitable for deployment.
3. A model has excellent training R² but poor validation RMSE. Diagnose possible causes and recommend a practical modelling workflow to improve reliability.

Multiple Choice Questions with Answers

Question 1

Which statistic measures the strength and direction of a linear relationship on a unit-free scale?

A. Covariance

B. Pearson’s correlation coefficient

C. Residual sum of squares

D. The regression intercept

Correct Answer: B. Pearson’s correlation coefficient

Question 2

In the fitted model ŷ = 12 + 3X, what does 3 represent?

A. The predicted value when X is zero

B. The residual for every observation

C. The predicted increase in Y for a one-unit increase in X

D. The proportion of variation explained

Correct Answer: C. The predicted increase in Y for a one-unit increase in X

Question 3

A residual is calculated as:

A. Fitted value minus the predictor

B. Observed response minus fitted response

C. Observed response divided by fitted response

D. Predictor minus its mean

Correct Answer: B. Observed response minus fitted response

Question 4

If a categorical variable has four categories and the model includes an intercept, how many dummy variables are normally required?

A. 1

B. 2

C. 3

D. 4

Correct Answer: C. 3

Question 5

An R² value of 0.70 means that:

A. The model is 70% accurate for every observation

B. The predictors caused 70% of every outcome

C. 70% of sample variation in the response is explained by the fitted model

D. The correlation must equal 0.70 in every multiple regression

Correct Answer: C. 70% of sample variation in the response is explained by the fitted model

Question 6

A funnel-shaped residual-versus-fitted plot most directly suggests:

A. Heteroscedasticity

B. Perfect correlation

C. Dummy-variable coding

D. An intercept of zero

Correct Answer: A. Heteroscedasticity

Question 7

Why can a correlation close to zero fail to show that two variables are unrelated?

A. Correlation has units

B. Correlation measures only linear association

C. Correlation is always positive

D. Correlation ignores sample size completely

Correct Answer: B. Correlation measures only linear association

Question 8

In multiple regression, a slope coefficient is usually interpreted as the association between its predictor and the response:

A. With all other included predictors held constant

B. After removing the intercept

C. Only when the response is categorical

D. Only when R² equals 1

Correct Answer: A. With all other included predictors held constant

Question 9

Which practice gives the strongest evidence for a causal effect of an intervention?

A. Observing a high correlation

B. Adding more predictors to an observational model

C. Randomly assigning comparable units to treatment conditions

D. Reporting a larger R²

Correct Answer: C. Randomly assigning comparable units to treatment conditions

Question 10

Why is extrapolation risky in regression?

A. The slope is always zero outside the sample

B. Predictions beyond the observed range may rely on an unsupported relationship

C. Residuals cannot be calculated inside the sample

D. Dummy variables become continuous

Correct Answer: B. Predictions beyond the observed range may rely on an unsupported relationship