Association, Correlation and Regression

Introduction

Business analytics frequently begins with a practical question: are two variables related, and if so, can one be used to explain or predict the other? Examples include advertising expenditure and sales, delivery time and customer satisfaction, price and demand, employee training and productivity, and website visits and online orders. Statistical methods for association, correlation and regression provide a structured way to investigate such questions.

Association describes a general relationship between variables. Correlation measures the strength and direction of a linear relationship between quantitative variables. Regression develops an equation that describes how an outcome changes in relation to one or more explanatory variables and can be used for estimation or prediction.

These methods are powerful but must be used carefully. A strong association does not prove causation, a high correlation does not guarantee useful prediction, and a statistically significant model may still be practically unimportant. Sound analysis therefore combines calculation with graphical inspection, subject knowledge, diagnostic testing and ethical judgment.

Learning Objectives

By the end of this chapter, you should be able to:

1. distinguish association, correlation and regression;
2. interpret scatterplots and correlation coefficients;
3. calculate and interpret simple linear regression coefficients;
4. explain multiple regression and interpret partial effects;
5. assess model fit using R², adjusted R², residuals and prediction errors;
6. identify linearity, independence, constant variance and normality issues;
7. detect multicollinearity and understand its consequences;
8. distinguish statistical association from causation; and
9. communicate regression findings and limitations responsibly.

1. Association and the Structure of Relationships

Two variables are associated when their values tend to occur together in a systematic way. The relationship may be positive, negative, linear, curved, weak or strong. Association can involve quantitative, ordinal or categorical variables, although the appropriate measure depends on the measurement scales.

A positive association means that larger values of one variable tend to accompany larger values of the other. A negative association means that larger values of one tend to accompany smaller values of the other. An association may be deterministic, where one value is exactly determined by another, or probabilistic, where substantial variation remains.

Scatterplots

For two quantitative variables, a scatterplot is usually the first analytical tool. The explanatory variable, often denoted X, is placed on the horizontal axis and the response variable, denoted Y, on the vertical axis. A scatterplot can reveal:

• direction: positive, negative or absent;
• form: linear, curved or segmented;
• strength: how closely observations follow a pattern;
• unusual observations: outliers or high-leverage points; and
• changing variability: whether the vertical spread increases or decreases.

A numerical coefficient cannot safely replace a graph. For example, a correlation near zero may occur when a strong U-shaped relationship is present, because positive and negative linear tendencies cancel each other.

Types of Association Measures

The choice of measure depends on the variables and the question.

This chapter focuses on Pearson correlation and linear regression, while recognizing that other methods may be more appropriate in different settings.

2. Pearson Correlation

The Pearson product–moment correlation coefficient, written r, measures the direction and strength of a linear association between paired observations Xᵢ and Yᵢ.

Here x̄ and ȳ are the sample means, and i indexes the n paired observations. The numerator is the sample cross-deviation; the denominator standardizes the result by the variation in each variable. Because the units cancel, r has no units.

The coefficient satisfies −1 ≤ r ≤ 1. Values near +1 indicate a strong positive linear association, values near −1 indicate a strong negative linear association, and values near 0 indicate little linear association. There is no universal boundary that defines “strong” or “weak”; interpretation depends on the context, data quality and consequences of decisions.

Important Conditions for Interpretation

Pearson correlation is most informative when:

• both variables are quantitative;
• observations are paired correctly and are independent of other pairs;
• the relationship is approximately linear;
• extreme outliers are absent or understood; and
• the spread is not so uneven that a few observations dominate the result.

Correlation is symmetric: the correlation of X with Y equals the correlation of Y with X. It does not identify a dependent variable and does not itself provide a prediction equation.

Correlation and Causation

Correlation alone does not establish that changes in X cause changes in Y. A relationship may arise from:

• reverse causality;
• a lurking or confounding variable;
• common time trends;
• selection bias;
• measurement error; or
• coincidence.

For example, ice-cream sales and swimming-pool incidents may rise together because temperature affects both. Temperature is a confounding variable. Establishing causation generally requires a defensible research design, such as randomisation, a natural experiment or a carefully justified causal model.

Worked Example: Correlation

Suppose a manager records weekly online advertising expenditure X and online sales Y for five weeks:

The means are x̄ = 4 and ȳ = 28. The deviations are X: −2, −1, 0, 1, 2 and Y: −8, −4, 0, 3, 9. Therefore:

∑(xᵢ − x̄)(yᵢ − ȳ) = 16 + 4 + 0 + 3 + 18 = 41,

∑(xᵢ − x̄)² = 10, and

∑(yᵢ − ȳ)² = 64 + 16 + 0 + 9 + 81 = 170.

Thus:

The data show a very strong positive linear association in this small sample. This result does not prove that advertising caused sales, especially because other factors may have changed during the five weeks.

3. Simple Linear Regression

Simple linear regression models the mean value of a quantitative response Y as a linear function of one explanatory variable X:

The fitted sample equation is:

where ŷ is the predicted or fitted response, b₀ is the estimated intercept, and b₁ is the estimated slope. The population parameters β₀ and β₁ describe the underlying relationship; b₀ and b₁ are estimates calculated from sample data.

The slope has units of Y per unit of X. It represents the estimated change in the mean response for a one-unit increase in X. The intercept is the predicted mean response when X = 0. It may have little practical meaning if zero is outside the observed or realistic range of X.

Least-Squares Estimation

The ordinary least-squares method chooses b₀ and b₁ to minimize the sum of squared residuals:

The estimated slope and intercept are:

These formulas apply to a model with one quantitative predictor when the denominator is nonzero, meaning that not all X values are identical. Least squares minimizes vertical prediction errors, not horizontal distances or absolute errors.

Worked Example: Advertising and Sales

Using the five-week data above:

The fitted regression equation is therefore:

Interpretation: within the observed setting, an additional thousand currency units of advertising is associated with an estimated increase of 4.1 thousand currency units in mean weekly online sales. The predicted sales at X = 5 are:

The observed value in week 4 is 31, so the residual is:

A negative residual means that the actual value is below the fitted value.

Regression and Correlation

In simple linear regression with an intercept, the slope can also be written as:

where sₓ and sᵧ are the sample standard deviations. This shows why the slope has units, whereas r does not. A positive correlation produces a positive slope, and a negative correlation produces a negative slope. However, the numerical value of the slope cannot be inferred from r alone without knowing the measurement scales.

4. Multiple Linear Regression

Business outcomes commonly depend on several factors. Multiple linear regression extends the model to p predictors:

The fitted equation is:

Each slope bⱼ estimates the expected change in Y associated with a one-unit increase in Xⱼ, holding the other included predictors constant. “Holding constant” is a conditional interpretation; it does not automatically establish a causal effect.

Example of Interpretation

Suppose a delivery company estimates:

Distance is measured in kilometres, peak-period is coded 1 for peak and 0 otherwise, and order-complexity is a score. The coefficient −0.04 would mean that, holding peak status and complexity constant, a one-kilometre increase in distance is associated with a 0.04-minute decrease in predicted delivery time. Although this may seem counterintuitive, it could reflect routing, customer location or selection effects. Regression output must therefore be checked against operational knowledge; an unexpected coefficient is a reason to investigate, not to rewrite the result.

The peak-period coefficient indicates a 2.1-minute difference between peak and non-peak observations with the other predictors held constant. A categorical variable with more than two categories requires appropriate indicator variables, with one category serving as the reference group.

Coding and Transformations

Binary predictors are often coded 0 and 1. A slope then compares the group coded 1 with the reference group coded 0, conditional on other variables. Interaction terms can test whether the effect of one variable changes according to another:

Here β₃ is the change in the slope of X for a one-unit change in Z. Interactions should be included because theory or evidence supports them, not merely because they improve a fit statistic.

Logarithmic or polynomial terms may represent curved relationships. For example, adding X² allows a quadratic pattern, but interpretation and diagnostics become more important because extrapolation can be hazardous.

5. Assessing Model Fit

Decomposing Variation

For a regression with an intercept, total variation in observed Y can be decomposed as:

where:

• SST = ∑(yᵢ − ȳ)² is total sum of squares;
• SSR = ∑(ŷᵢ − ȳ)² is regression or explained sum of squares; and
• SSE = ∑(yᵢ − ŷᵢ)² is residual or unexplained sum of squares.

The coefficient of determination is:

R² is the proportion of sample variation in Y explained by the fitted model relative to a mean-only model. It is dimensionless and usually lies between 0 and 1 when an intercept is included. A high R² does not prove causation, guarantee accurate future predictions or show that the model is correctly specified.

Adding predictors cannot decrease ordinary R², even if the new variables are unhelpful. Adjusted R² penalises unnecessary predictors:

where n is sample size and p is the number of predictors, excluding the intercept. The formula requires n > p + 1. Adjusted R² may decrease when a new predictor contributes little explanatory value.

Prediction Error and Standard Error

The residual for observation i is eᵢ = yᵢ − ŷᵢ. A common overall measure is the root mean squared error:

For estimating the residual standard deviation in a fitted model, analysts commonly use the residual standard error:

RMSE and s have the same units as Y, but their denominators and purposes may differ. A model with a small percentage error may still be unsuitable if the business cost of occasional large errors is high.

Statistical Significance

A slope can be tested using:

where βⱼ,0 is usually 0 under the null hypothesis and SE(bⱼ) is the standard error of the estimated coefficient. The test relies on the regression assumptions and, for exact small-sample inference, normally distributed errors. A small p-value indicates inconsistency with the null hypothesis under the model; it does not measure practical importance or prove causation.

6. Regression Assumptions and Diagnostics

Regression analysis is not completed when software displays a coefficient table. The model should be examined using plots, subject knowledge and diagnostics.

Linearity

The conditional mean of Y should be adequately represented by the linear predictor. A residual-versus-fitted plot should show random scatter around zero. A curve suggests that a transformation, polynomial term, interaction or different model may be needed.

Independence

Observations should be independent, or the dependence should be modelled. Repeated observations from the same customer, employee or store are not automatically independent. Time-series data often contain trend, seasonality and autocorrelation. Ignoring dependence can produce standard errors that are too small and conclusions that are overconfident.

Constant Variance

The variance of errors should be approximately constant across fitted values or relevant predictors. A funnel-shaped residual plot suggests heteroscedasticity. Possible responses include transforming the response, using heteroscedasticity-robust standard errors or applying a model designed for nonconstant variance.

Normality of Errors

Normality is mainly important for small-sample confidence intervals and hypothesis tests, not for calculating fitted values. A histogram or Q–Q plot can reveal severe skewness or heavy tails. Mild departures are often less serious in large samples, but extreme observations still deserve attention.

Outliers and Leverage

An outlier has an unusual response relative to the model. A high-leverage point has an unusual predictor combination. An influential observation substantially changes estimated coefficients or conclusions when removed. Measures such as standardized residuals, leverage values and Cook’s distance can help identify influential cases. An observation should not be deleted simply because it weakens a desired conclusion. Its validity, context and influence should be documented.

Multicollinearity

Multicollinearity occurs when predictors are strongly related to one another. It can make individual coefficients unstable, inflate standard errors and produce unexpected signs, even when the model predicts well overall. It does not necessarily reduce overall R².

A common screening measure is the variance inflation factor:

where Rⱼ² is obtained by regressing predictor Xⱼ on all the other predictors. VIF = 1 indicates no linear relationship with the other predictors. Larger values indicate greater inflation of the variance of bⱼ; thresholds such as 5 or 10 are rules of thumb, not universal laws.

Possible responses include removing redundant variables, combining measures, collecting more informative data, using a theoretically justified variable selection strategy or applying regularisation. Removing a variable solely on an automated threshold may create omitted-variable bias or reduce substantive validity.

7. Prediction, Extrapolation and Model Validation

A fitted model may be used for interpolation, prediction within the observed data range, or extrapolation beyond that range. Extrapolation is particularly risky because the relationship may change outside the observed conditions.

A confidence interval describes uncertainty about a population mean response at a specified predictor value. A prediction interval describes uncertainty for a new individual outcome and is wider because it includes both uncertainty in the mean and individual-level variation. Confusing these intervals can lead to overconfident decisions.

For predictive work, data should be separated into training and test sets when sample size and purpose justify this. Cross-validation can estimate out-of-sample performance. Evaluation metrics should reflect the decision context. Common measures include:

and

MAE is in the units of Y and treats errors linearly. MSE is in squared units and gives greater weight to large errors. Neither metric is automatically best; the selected metric should reflect the costs of overprediction and underprediction.

Data leakage must be avoided. Information that would not be available at the time of prediction must not be used to train or evaluate the model. Random train–test splitting can also be inappropriate for time-ordered data; a temporal validation design is usually more realistic.

8. A Practical Regression Workflow

A professional workflow is iterative rather than mechanical:

1. Define the decision problem. Specify the outcome, prediction horizon, unit of analysis and intended use.
2. Understand the data. Check definitions, missing values, measurement units, coding, duplicates and sampling design.
3. Explore graphically. Use scatterplots, grouped summaries and time plots before fitting a model.
4. Specify a defensible model. Choose predictors based on business knowledge, theory, data availability and the purpose of analysis.
5. Fit the model. Estimate coefficients and record the sample, variables and software settings.
6. Diagnose. Examine residuals, influential observations, multicollinearity and dependence.
7. Validate. Assess performance on data not used to fit the model when prediction is the goal.
8. Interpret in context. Report units, reference groups, uncertainty and practical magnitude.
9. Communicate limitations. State what the model does not establish and when predictions may fail.
10. Monitor after deployment. Relationships, populations and data-generating processes can change.

Applied Case: Forecasting Retail Store Sales

The following is an illustrative case based on a hypothetical retail chain.

Background

A retailer wants to improve weekly sales forecasts for 40 stores. Managers believe that sales depend on promotional spending, store size, local foot traffic and the number of competing stores nearby. The business wants a model for staffing and inventory planning, not a claim that every predictor causes sales.

Problem

A simple correlation between promotion spending and sales is positive, but managers notice that larger stores receive larger promotional budgets. A simple regression may therefore attribute to promotion some effect associated with store size.

Analysis and Approach

The analytics team fits a multiple regression model with weekly sales as the response and promotion spending, store size, foot traffic and competitor count as predictors. Store and week identifiers are examined because repeated observations may violate independence. A residual plot is used to check nonlinearity and unequal variance; VIF values are reviewed because promotion and foot traffic may be correlated.

The team uses earlier weeks for training and later weeks for testing, which reflects the intended forecasting process. It reports MAE in currency units and compares the regression model with a naïve benchmark based on the previous week’s sales. A high in-sample R² is not accepted as evidence of forecasting superiority.

Findings and Decision Use

Assume the regression has lower test-set MAE than the benchmark, but errors increase during holiday weeks. The retailer uses the model for ordinary weeks and adds a holiday indicator after confirming that it is available before the forecast date. Managers receive prediction intervals or error ranges rather than a single apparently precise number.

The model is not used to penalise store managers. A store with lower predicted sales may serve a different market or face temporary disruption. Nor is the promotion coefficient interpreted as a causal return on advertising without an appropriate experimental or quasi-experimental design.

Learning Points

The case demonstrates that multiple regression can reduce confounding in prediction, but it does not automatically solve causal inference. Validation must imitate real use, assumptions must be checked, and operational decisions must account for uncertainty, changing conditions and fairness.

9. Technology, Ethics and Governance

Spreadsheet software, statistical packages and programming languages can calculate regression efficiently, but automation does not replace analytical judgment. Analysts should retain a reproducible record of data preparation, variable definitions, model versions, exclusions, diagnostics and validation results.

AI-assisted modelling creates additional risks. Automated feature selection may exploit chance patterns, especially when many variables are considered. Generated explanations may confuse association with causation. Models trained on historical business decisions may reproduce discrimination or unequal service. Sensitive variables and proxies for protected characteristics require governance, access controls and a clear legitimate purpose.

Good practice includes:

• obtaining data lawfully and respecting privacy principles;
• minimising collection of personal information;
• documenting consent, purpose and retention where applicable;
• checking subgroup performance and error disparities;
• explaining predictions at an appropriate level;
• maintaining human oversight for high-impact decisions; and
• monitoring drift, fairness and performance after deployment.

A model should be judged not only by predictive accuracy but also by validity, transparency, reliability, fairness and fitness for purpose.

Chapter Summary

Association describes a systematic relationship between variables, while correlation quantifies the direction and strength of a linear relationship. Pearson’s r is unit-free and ranges from −1 to 1, but it can be distorted by outliers and cannot detect every form of dependence. Correlation does not establish causation.

Simple linear regression estimates a response from one predictor using an intercept and slope. Least squares chooses coefficients that minimise squared residuals. Multiple regression extends the model to several predictors; each coefficient is interpreted conditionally on the others. Categorical variables, transformations and interactions allow models to represent practical situations more appropriately.

Model quality should be assessed using residuals, R², adjusted R², error measures and, where relevant, statistical tests and confidence or prediction intervals. Diagnostics examine linearity, independence, constant variance, normality and influential observations. Multicollinearity can make individual coefficients unstable even when overall prediction is adequate.

Prediction should be validated using data and procedures that reflect future use. Extrapolation, data leakage, changing populations and omitted variables limit reliability. Ethical practice requires privacy protection, reproducibility, fairness assessment, transparent communication and human oversight. Regression is therefore not merely a calculation: it is a reasoned process connecting data, assumptions, evidence and decisions.

Key Terms

Association: A systematic relationship between variables.

Correlation: A numerical measure of the direction and strength of linear association.

Pearson correlation coefficient: The standardized sample measure of linear association between two quantitative variables.

Scatterplot: A graph displaying paired values of two variables.

Response variable: The outcome that a model describes or predicts.

Explanatory variable: A variable used to explain or predict the response.

Simple linear regression: Regression with one explanatory variable.

Multiple linear regression: Regression with two or more explanatory variables.

Intercept: The fitted mean response when all predictors equal zero.

Slope: The estimated change in mean response for a one-unit predictor change, conditional on other predictors.

Residual: The observed response minus its fitted value.

Least squares: A fitting method that minimises the sum of squared residuals.

R²: The proportion of sample response variation explained relative to a mean-only model.

Adjusted R²: A version of R² that penalises the number of predictors.

Heteroscedasticity: Nonconstant error variance.

Leverage: The unusualness of an observation’s predictor values.

Influential observation: An observation whose removal substantially changes model results.

Multicollinearity: Strong linear dependence among explanatory variables.

Extrapolation: Prediction outside the range of predictor values used to fit the model.

Data leakage: Use of information in model development that would not be available at prediction time.

Review Questions

5 Short Answer Questions

1. What is the difference between positive and negative association?
2. Why does correlation not establish causation?
3. Interpret the slope in a simple linear regression model.
4. What does R² measure?
5. Why can multicollinearity be a problem in multiple regression?

5 Long Answer Questions

1. Explain how a scatterplot, Pearson correlation and regression equation provide different information about two quantitative variables.
2. Describe the least-squares method and explain the meaning of the intercept, slope and residual.
3. Explain the assumptions of linear regression and describe a diagnostic approach for assessing them.
4. Discuss the distinction between in-sample fit and out-of-sample predictive performance.
5. Explain how categorical variables, interactions and transformations can improve a multiple regression model.

3 Analytical / Application Questions

1. A retailer finds a strong positive correlation between discount expenditure and sales. Identify at least three reasons why this result may not represent a causal effect and propose a stronger analytical design.
2. A fitted model has a high R², significant coefficients and a funnel-shaped residual plot. Assess whether the model is ready for business deployment and recommend next steps.
3. A forecasting model performs well on randomly selected test observations but poorly in the most recent month. Analyse possible causes and design a more appropriate validation and monitoring plan.

Multiple Choice Questions with Answers

Question 1

Which statement best describes Pearson’s correlation coefficient?

A. It measures the strength and direction of a linear association between two quantitative variables.

B. It proves that changes in one variable cause changes in another.

C. It gives the predicted value of the response for a specified predictor.

D. It measures the percentage of observations classified correctly.

Correct Answer: A. It measures the strength and direction of a linear association between two quantitative variables.

Question 2

In the regression equation ŷ = 11.6 + 4.1x, what does 4.1 represent?

A. Predicted sales when advertising is zero

B. Estimated change in sales for a one-unit increase in advertising

C. The correlation between advertising and sales

D. The residual for every observation

Correct Answer: B. Estimated change in sales for a one-unit increase in advertising

Question 3

A residual is calculated as:

A. Predicted value minus the predictor

B. Observed response minus fitted response

C. Observed response divided by fitted response

D. Total variation minus correlation

Correct Answer: B. Observed response minus fitted response

Question 4

What is the main purpose of adjusted R²?

A. To convert a slope into a correlation coefficient

B. To test whether residuals are independent

C. To penalise model fit for including additional predictors

D. To identify the largest outlier automatically

Correct Answer: C. To penalise model fit for including additional predictors

Question 5

A funnel shape in a residual-versus-fitted plot most strongly suggests:

A. Constant variance

B. Heteroscedasticity

C. Perfect multicollinearity

D. A zero intercept

Correct Answer: B. Heteroscedasticity

Question 6

If a predictor has VIF = 1, this indicates that:

A. Its coefficient is statistically significant

B. It has no linear relationship with the other predictors

C. The regression has an R² of 1

D. Its units have been standardised

Correct Answer: B. It has no linear relationship with the other predictors

Question 7

Which action best avoids data leakage in forecasting?

A. Using future sales values as predictors of earlier sales

B. Selecting variables using information unavailable at forecast time

C. Ensuring every predictor would be available when the forecast is made

D. Randomly deleting inconvenient observations

Correct Answer: C. Ensuring every predictor would be available when the forecast is made

Question 8

In multiple regression, “holding other predictors constant” means that a coefficient represents:

A. A guaranteed causal effect

B. A conditional association after accounting for included predictors

C. The total variation in the response

D. The sample size used in estimation

Correct Answer: B. A conditional association after accounting for included predictors

Question 9

Using the illustrative equation ŷ = 11.6 + 4.1x, what is the predicted value when x = 5?

A. 20.5

B. 28.0

C. 32.1

D. 57.6

Correct Answer: C. 32.1

Question 10

Which statement about extrapolation is most accurate?

A. It is always safer than interpolation.

B. It predicts outside the observed predictor range and may be unreliable if the relationship changes.

C. It eliminates the effect of outliers.

D. It guarantees a narrower prediction interval.

Correct Answer: B. It predicts outside the observed predictor range and may be unreliable if the relationship changes.