Correlation and Simple Linear Regression

Introduction and Context

Business analysts frequently need to answer questions such as: Do advertising expenditures and sales move together? How strongly is customer satisfaction associated with retention? Can delivery time be used to predict customer complaints? Correlation and simple linear regression provide foundational tools for addressing these questions.

Correlation measures the strength and direction of a linear association between two quantitative variables. Simple linear regression goes further by representing the average relationship mathematically and using one variable to estimate or predict another. These tools are widely used in forecasting, performance analysis, marketing, operations, finance, and quality management.

Neither correlation nor regression, by itself, establishes causation. A strong association may result from a causal relationship, reverse causality, a common third variable, coincidence, or systematic bias. Sound statistical analysis therefore combines numerical results with subject knowledge, research design, and careful interpretation.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain covariance, correlation, and their interpretation.
2. Calculate and interpret Pearson’s correlation coefficient.
3. Formulate and estimate a simple linear regression model.
4. Interpret the slope, intercept, fitted values, and residuals.
5. Test regression coefficients and construct confidence and prediction intervals.
6. Interpret R² and assess model fit.
7. Examine residual assumptions and identify influential observations.
8. Distinguish association from causation and communicate results responsibly.

1. Association Between Two Variables

Let X and Y be quantitative variables observed for n units. A scatterplot is the first and often most informative diagnostic. It displays paired observations (x_i,y_i) and can reveal direction, form, strength, clusters, outliers, and changing variability.

A positive association means that larger values of X tend to occur with larger values of Y. A negative association means that larger values of X tend to occur with smaller values of Y. An association is linear when the points tend to follow a straight-line pattern.

Covariance

Sample covariance measures whether two variables tend to be above or below their means at the same time:

Here, x̄ and ȳ are sample means. Covariance is positive when paired deviations generally have the same sign and negative when they generally have opposite signs. Its units are the product of the units of X and Y, which makes its magnitude difficult to compare across datasets.

Pearson’s Correlation Coefficient

Pearson’s sample correlation coefficient standardizes covariance:

Equivalently,

The quantities s_X and s_Y are sample standard deviations. Correlation has no units and satisfies −1 ≤ r ≤ 1. A value near 1 indicates a strong positive linear association; a value near −1 indicates a strong negative linear association; a value near 0 indicates little linear association.

Correlation is unchanged if either variable is converted between measurement units or if a positive linear rescaling is applied. It is not unchanged when nonlinear transformations are applied. Importantly, r=0 does not prove that variables are unrelated: they may have a strong nonlinear relationship.

2. Worked Example: Computing and Interpreting Correlation

Suppose a retailer records weekly digital advertising expenditure X and weekly online sales Y for five weeks.

The means are x̄=5 and ȳ=30.6. The sums needed for correlation are:

∑(x_i−x̄)(y_i−ȳ)=101, ∑(x_i−x̄)²=34, and ∑(y_i−ȳ)²=324.8. Therefore,

The data show a very strong positive linear association. This result indicates that higher advertising expenditure is observed with higher sales in these five weeks. It does not establish that advertising caused all of the increase. Seasonal demand, promotions, prices, or other factors may also be involved.

Important Cautions About Correlation

• Correlation measures linear association, not general dependence.
• A single outlier can substantially increase or decrease r.
• Restricting the range of one variable can make a real relationship appear weak.
• Aggregated data can produce misleading relationships, a problem often called ecological correlation.
• Statistical significance depends on sample size; a weak correlation can be statistically significant in a very large sample.
• A statistically significant correlation may still be too small to matter commercially.

3. The Simple Linear Regression Model

Simple linear regression models the conditional mean of a response variable using one explanatory variable:

where:

The model implies that the mean response at a particular value of X is E(Y\|X)=β₀+β₁X. The intercept is the expected value of Y when X=0. It may have little practical meaning if zero is outside the observed or meaningful range of X. The slope is the expected change in the mean of Y associated with a one-unit increase in X.

The fitted sample regression equation is:

Lowercase symbols denote estimates calculated from sample data. The least-squares estimates minimize the sum of squared residuals:

The slope and intercept are estimated by:

These formulas apply to ordinary least-squares simple linear regression when the predictor is quantitative and the denominator is nonzero. The fitted line always passes through (x̄,ȳ).

4. Worked Regression Calculation

Using the advertising and sales data above,

and

The fitted equation is therefore:

Because both variables are measured in thousands of dollars, the slope means that an additional $1,000 of advertising is associated with an estimated average increase of approximately $2,971 in weekly sales within the observed range. This is an association-based interpretation, not automatically a causal claim.

At x=7, the predicted sales value is:

Thus, predicted sales are approximately $36,532. This is interpolation because $7,000 lies within the observed advertising range of $2,000 to $9,000. Prediction at, for example, $20,000 would be extrapolation and could be unreliable because the relationship may not continue outside the observed range.

A residual is the observed minus fitted value:

For week 1, ŷ₁=15.735+2.971(2)≈21.677, so the residual is approximately 20−21.677=−1.677 thousand dollars. A negative residual means the observed response is below the fitted line.

5. Decomposing Variation and Measuring Fit

Regression separates total variation in observed responses into explained and unexplained components:

where:

• SST=∑(y_i−ȳ)² is total sum of squares;
• SSR=∑(ŷ_i−ȳ)² is regression or explained sum of squares;
• SSE=∑(y_i−ŷ_i)² is error or residual sum of squares.

The coefficient of determination is:

R² is the proportion of sample variation in the response explained by the fitted linear relationship. In simple linear regression with an intercept, it equals r². In the example, R²≈0.960²≈0.922, so approximately 92.2% of the sample variation in weekly sales is accounted for by the linear predictor advertising expenditure. This does not mean that advertising caused 92.2% of sales, nor does it guarantee accurate future predictions.

A high R² is not always desirable if the model is overfit, and a low R² does not necessarily make a model useless. In fields such as individual-level behavior, moderate explanatory power may still support valuable decisions.

6. Inference for the Slope

To test whether a linear association exists in the population, analysts commonly test:

• Null hypothesis: H₀:β₁=0
• Alternative hypothesis: H_A:β₁≠0

The estimated standard error of the slope is:

where the residual standard error is:

The degrees of freedom are n−2 because two parameters are estimated. The test statistic is:

Usually β_1,0=0. Under the classical assumptions, this statistic follows a t distribution with n−2 degrees of freedom when the null hypothesis is true. A two-sided confidence interval for the population slope is:

Here, α is the significance level and t_α/2,n−2 is the appropriate critical value. If the interval excludes zero, the corresponding two-sided hypothesis test rejects H₀ at level α. Statistical significance should be reported with effect size, uncertainty, data quality, and business relevance.

The intercept can be tested similarly, although inference about it is often unimportant when X=0 is not meaningful or observed.

7. Confidence Intervals and Prediction Intervals

There are two different uncertainties that analysts must distinguish.

A confidence interval for the mean response at x_* estimates the average outcome for all units having that predictor value:

A prediction interval estimates the response for one new individual unit:

The additional 1 inside the prediction-interval expression represents individual random variation. Consequently, a prediction interval is wider than the confidence interval for the mean. Both intervals are most reliable when prediction occurs within the data range and model assumptions are reasonable.

8. Assumptions and Residual Diagnostics

Regression results depend on assumptions. They should be assessed rather than automatically accepted.

Linearity

The conditional mean of Y should be approximately linear in X. A residual-versus-fitted plot that displays curvature suggests that a straight line is inadequate. Possible responses include transforming a variable, adding a nonlinear term, or using a different model.

Independence

Errors should be independent across observations. Time-series data, repeated customer records, and grouped stores often violate this assumption. Autocorrelation can make standard errors too small and significance claims too optimistic.

Constant Variance

The variance of errors should be approximately constant across predictor values. A funnel-shaped residual plot indicates heteroscedasticity. Robust standard errors, transformation, or weighted regression may be appropriate, depending on the objective and data-generating process.

Normality for Small-Sample Inference

For exact small-sample t tests and intervals, errors are commonly assumed to be normally distributed. With larger samples, inference may be more robust, but severe skewness or outliers remains important. Normality of the response itself is not the central requirement; the relevant condition concerns errors around the conditional mean.

Outliers and Influence

An outlier has an unusual response, predictor value, or both. A high-leverage observation has an unusual predictor value. An influential observation substantially changes the fitted model when removed. Analysts should inspect residuals, leverage, and influence measures, investigate data quality, and never delete observations merely because they weaken a desired conclusion.

9. Correlation Does Not Imply Causation

A causal conclusion requires more than a large r or a significant slope. Common threats include:

• Confounding: a third variable affects both predictor and response.
• Reverse causality: the response influences the predictor.
• Selection bias: observed units are not representative of the target population.
• Measurement error: inaccurate variables distort relationships.
• Simultaneous trends: two variables increase over time without a direct relationship.
• Omitted-variable bias: relevant predictors are absent from the model.

Causal evidence is stronger when supported by randomized experiments, credible natural experiments, longitudinal designs, temporal ordering, theoretical mechanisms, and sensitivity analysis. Simple regression is descriptive or predictive unless the research design justifies a causal interpretation.

10. Applied Case: Delivery Time and Customer Complaints

Illustrative case. A parcel company wants to know whether delivery time is associated with complaints. It samples 120 completed deliveries and records delivery time in days and whether the customer submitted a complaint. A manager proposes regressing a complaint indicator on delivery time.

Problem

The proposed response is binary, whereas ordinary simple linear regression assumes a quantitative response with an interpretable conditional mean and can produce fitted values below 0 or above 1. A linear probability model can sometimes be used cautiously, but logistic regression is generally more suitable for modeling a binary probability. This case therefore illustrates the importance of matching the method to the outcome.

Analysis

The analyst first creates a scatterplot or grouped summary of complaint rates by delivery time. If delivery time is continuous and complaints are binary, a logistic model could express the log-odds of a complaint as a linear function of delivery time. The analyst also checks whether complaint rates differ by service type, destination, weather disruption, and shipment value. These variables may confound the relationship.

For an initial descriptive analysis, the company may report the correlation between delivery time and a 0/1 complaint indicator, but it must explain that this is a point-biserial form of Pearson correlation and does not establish causality. A better operational analysis may compare randomized service interventions or use a carefully designed quasi-experiment.

Approach and Possible Outcome

Suppose longer delivery time is strongly associated with complaints. The company should not immediately conclude that reducing time alone will eliminate complaints. Delays may occur disproportionately on difficult routes, during severe weather, or for high-value shipments, each of which may independently increase complaints. The analyst should examine residual or model diagnostics, include relevant controls where justified, and evaluate an intervention such as improved route planning.

Learning Points

1. The response scale determines whether simple linear regression is appropriate.
2. Association is useful for diagnosis but does not by itself identify a causal mechanism.
3. Operational decisions require effect sizes, uncertainty, cost analysis, and validation after implementation.
4. Transparent reporting of limitations is part of professional statistical practice.

11. Professional Practice, Ethics, and Technology

Business analysts should document variable definitions, sampling procedures, missing-data treatment, transformations, excluded observations, model specifications, and software versions. Reproducible code and version-controlled data products improve auditability.

Automated analytics and AI systems can calculate correlations, fit models, and generate visualizations quickly. They cannot reliably determine whether a relationship is causal, whether data are representative, or whether a prediction is ethically appropriate. Analysts should inspect outputs rather than accept automatically generated interpretations.

Fairness is relevant when regression predictions affect credit, employment, insurance, pricing, or access to services. A statistically accurate model may still reproduce historical discrimination. Governance should include privacy protection, access control, bias assessment, explainability appropriate to the decision, monitoring for drift, and human review of consequential decisions.

Best-Practice Workflow

1. Define the business question and target population.
2. Identify the response, predictor, units, and measurement scales.
3. Visualize the data with a scatterplot and summary statistics.
4. Calculate correlation only after checking form, outliers, and range.
5. Fit the regression model and interpret coefficients in context.
6. Quantify fit using R², residual variation, and relevant error measures.
7. Construct confidence or prediction intervals as appropriate.
8. Diagnose residuals, independence, variance, leverage, and influence.
9. Consider confounding and avoid unsupported causal language.
10. Validate predictions on new data and communicate uncertainty clearly.

Chapter Summary

Correlation summarizes the direction and strength of a linear association, while simple linear regression estimates a line for the conditional mean of a response from one predictor. Pearson’s correlation lies between −1 and 1, is unit-free, and must be interpreted with scatterplots and subject knowledge. Least-squares regression estimates the slope and intercept by minimizing squared residuals. The slope describes the expected change in the mean response for a one-unit predictor increase, whereas the intercept describes the fitted response at a predictor value of zero.

The decomposition SST=SSR+SSE supports interpretation of R², the proportion of sample response variation explained by the fitted line. Slope tests, confidence intervals, and prediction intervals quantify uncertainty, with prediction intervals wider because they include individual random variation. Linearity, independence, constant variance, and suitable error behavior are central assumptions. Residual plots, leverage, and influence diagnostics help evaluate them. Finally, neither a strong correlation nor a significant regression coefficient proves causation. Responsible business analytics combines statistical modeling with design, context, ethics, validation, and transparent communication.

Key Terms

Correlation: A standardized measure of linear association between two quantitative variables.

Covariance: A signed measure of joint variation, expressed in product units.

Pearson’s correlation coefficient: The unit-free sample measure r of linear association.

Scatterplot: A graph of paired observations used to inspect relationships and unusual points.

Simple linear regression: A model relating a quantitative response to one predictor through a straight line.

Response variable: The outcome being explained or predicted.

Explanatory variable: The predictor used to explain or estimate the response.

Intercept: The regression model’s expected response when the predictor equals zero.

Slope: The expected change in mean response for a one-unit increase in the predictor.

Residual: The observed response minus its fitted value.

Least squares: The estimation method minimizing the sum of squared residuals.

SSE: The residual or error sum of squares.

SST: The total sum of squares around the response mean.

R²: The proportion of sample response variation explained by the fitted model.

Leverage: The unusualness of an observation’s predictor value.

Influence: The extent to which an observation changes model results.

Confidence interval: An interval estimating a population parameter or mean response.

Prediction interval: An interval estimating the response for one new individual observation.

Confounding: Distortion caused by a third variable related to predictor and response.

Extrapolation: Prediction outside the observed range of the predictor.

Review Questions

Short Answer Questions

1. What does a negative Pearson correlation indicate?
2. Why can r=0 coexist with a strong nonlinear relationship?
3. Interpret the slope in a simple regression model.
4. What is the difference between a residual and a fitted value?
5. Why is a prediction interval wider than a confidence interval for a mean response?

Long Answer Questions

1. Explain how covariance is standardized to obtain Pearson’s correlation coefficient, including the interpretation and limitations of the resulting measure.
2. Derive the role of the least-squares criterion and explain the meanings of the estimated slope and intercept.
3. Explain R² using the decomposition of total, explained, and residual variation.
4. Discuss the assumptions of simple linear regression and describe diagnostic evidence that would suggest violations.
5. Explain why statistical association does not establish causation, using at least three threats to causal interpretation.

Analytical / Application Questions

1. A retailer estimates ŷ=12+4x, where x is advertising expenditure in thousands of dollars and y is sales in thousands of dollars. Interpret both coefficients and calculate the predicted sales when x=6.
2. A regression has R²=0.64. Explain what this value means and identify two conclusions that cannot validly be drawn from it.
3. A residual-versus-fitted plot shows a pronounced funnel shape and one high-leverage observation. Explain the risks to inference and recommend a sequence of diagnostic and modeling actions.

Multiple Choice Questions with Answers

Question 1

Which statement best describes Pearson’s correlation coefficient?

A. It measures the proportion of variation explained by any possible nonlinear model.

B. It measures the direction and strength of linear association and lies between −1 and 1.

C. It proves that changes in one variable cause changes in another.

D. It is expressed in the product units of the two variables.

Correct Answer: B. It measures the direction and strength of linear association and lies between −1 and 1.

Question 2

A fitted regression equation is ŷ=10+3x. What is the predicted value when x=4?

A. 7

B. 12

C. 22

D. 40

Correct Answer: C. 22

Question 3

In simple linear regression, what does the slope represent?

A. The expected change in the mean response for a one-unit increase in the predictor.

B. The response value for every individual when the predictor is zero.

C. The proportion of unexplained variation in the response.

D. The correlation expressed in the response variable’s units.

Correct Answer: A. The expected change in the mean response for a one-unit increase in the predictor.

Question 4

If R²=0.81 in a regression with an intercept, which interpretation is appropriate?

A. The predictor causes exactly 81% of every individual response.

B. The correlation must equal 0.81.

C. The model explains approximately 81% of sample variation in the response.

D. The model will predict future responses with 81% accuracy.

Correct Answer: C. The model explains approximately 81% of sample variation in the response.

Question 5

Which quantity is minimized by ordinary least squares?

A. The sum of absolute predictor values

B. The sum of squared residuals

C. The correlation coefficient

D. The total sample size

Correct Answer: B. The sum of squared residuals

Question 6

Which interval is generally wider for a given predictor value?

A. A prediction interval for one new observation

B. A confidence interval for the mean response

C. Both intervals always have identical width

D. Neither interval contains uncertainty

Correct Answer: A. A prediction interval for one new observation

Question 7

A funnel pattern in a residual-versus-fitted plot most directly suggests:

A. Perfect linearity

B. Constant variance

C. Heteroscedasticity

D. Zero correlation

Correct Answer: C. Heteroscedasticity

Question 8

Why can extrapolation be dangerous in regression?

A. The slope formula cannot be calculated inside a sample.

B. The relationship observed in the data may not continue outside the predictor range.

C. Predictions outside the range always have zero uncertainty.

D. Correlation is automatically negative outside the range.

Correct Answer: B. The relationship observed in the data may not continue outside the predictor range.

Question 9

Which situation is an example of confounding?

A. A third variable affects both the predictor and the response.

B. The response is measured in dollars rather than euros.

C. The scatterplot contains no observations.

D. The regression line passes through the sample means.

Correct Answer: A. A third variable affects both the predictor and the response.

Question 10

For testing whether a population slope differs from zero, the usual simple-regression test statistic uses which reference distribution under classical assumptions?

A. A t distribution with n−2 degrees of freedom

B. A uniform distribution with n degrees of freedom

C. A Poisson distribution with n−1 degrees of freedom

D. A normal distribution with no sampling assumptions

Correct Answer: A. A t distribution with n−2 degrees of freedom