Correlation and Regression Analytics

Introduction and Context

Correlation and regression are central tools for converting data into evidence about relationships. In business administration, they are used to examine questions such as whether advertising expenditure is associated with sales, how price relates to demand, whether employee engagement predicts turnover, and which factors are associated with customer profitability.

These methods are powerful but frequently misused. A correlation does not establish causation; a statistically significant coefficient is not necessarily economically important; and a well-fitting model may still be biased, unstable, or inappropriate for the decision at hand. Graduate-level data literacy therefore requires both computational competence and disciplined interpretation.

This chapter develops Pearson correlation, simple and multiple linear regression, model diagnostics, categorical predictors, interactions, and logistic regression. It also emphasizes assumptions, prediction, uncertainty, confounding, data governance, and managerial communication.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain covariance, correlation, and their appropriate interpretations.
2. Estimate and interpret simple and multiple linear regression models.
3. Distinguish association, prediction, and causal claims.
4. Evaluate linear-regression assumptions using residual diagnostics.
5. Incorporate categorical variables and interaction effects correctly.
6. Explain logistic regression and interpret odds ratios and predicted probabilities.
7. Identify omitted-variable bias, multicollinearity, influential observations, and overfitting.
8. Communicate model results responsibly in managerial and research settings.

1. From Association to Model-Based Explanation

A relationship exists when values of one variable vary systematically with values of another. The variables may be:

• Outcome or response variable (Y): the quantity to be explained or predicted.
• Predictor, explanatory, or independent variable (X): a measured characteristic used to explain or predict the outcome.
• Control variable: an additional predictor included to account for alternative explanations.
• Residual or error (ε): the part of the outcome not explained by the model.

Three claims must be separated:

1. Association: two variables tend to vary together.
2. Prediction: information about one or more variables improves forecasts of an outcome.
3. Causation: changing one variable would change another, under specified conditions.

Regression can support prediction without proving causation. Causal interpretation requires a credible design, such as randomization, a natural experiment, a valid instrumental-variable strategy, difference-in-differences assumptions, or a carefully justified longitudinal design.

A scatterplot should normally precede calculation. It can reveal curvature, clusters, unequal spread, and unusual observations that a single numerical coefficient conceals.

2. Covariance and Pearson Correlation

Covariance measures whether two variables tend to be above or below their means at the same time. For a sample of n observations:

Here, X_i and Y_i are observations, X̄ and Ȳ are sample means, and n − 1 is the degrees-of-freedom adjustment. Covariance has units equal to the product of the units of X and Y, so its magnitude is difficult to compare across datasets.

Pearson’s sample correlation standardizes covariance:

where s_X and s_Y are sample standard deviations. Equivalently:

Correlation is unit-free and lies between −1 and +1. A positive value indicates that larger values of one variable tend to accompany larger values of the other; a negative value indicates the reverse. Values near zero indicate no linear association, not necessarily no relationship.

The coefficient is appropriate for quantitative variables when a linear summary is meaningful. It can be distorted by outliers, nonlinear patterns, restriction of range, and mixtures of distinct subgroups. Pearson correlation also does not distinguish which variable is explanatory and which is the outcome.

Correlation and explained variation

In a simple linear regression with an intercept, the coefficient of determination equals the squared correlation:

Thus, if r = 0.60, then R² = 0.36. In the simple linear setting, 36% of the sample variation in the outcome is represented by the fitted linear relationship. This does not mean that 36% of an individual outcome is caused by the predictor, nor does it guarantee accurate predictions in new settings.

3. Simple Linear Regression

The population model for simple linear regression is:

where β₀ is the intercept, β₁ is the population slope, and ε_i is the unobserved disturbance. The fitted sample equation is:

The ordinary least squares (OLS) estimates choose b₀ and b₁ to minimize the sum of squared residuals:

For one predictor:

The slope has units of outcome per unit of predictor. It represents the expected change in Y associated with a one-unit increase in X, within the observed range and conditional on the model’s assumptions. The intercept is the predicted outcome at X = 0; it may have little substantive meaning when zero is outside the data range.

Worked example: advertising and sales

Suppose a retailer estimates monthly sales, measured in thousands of currency units, from advertising expenditure, also measured in thousands. The fitted model is:

For advertising expenditure of 10 thousand currency units:

The predicted sales are 165 thousand currency units. The slope means that, within the sample and model, an additional thousand currency units of advertising is associated with an average increase of 4.5 thousand currency units in sales. It is not automatically a causal return on advertising, because sales may also reflect seasonality, promotions, prices, distribution, and managerial targeting decisions.

4. Multiple Linear Regression

A multiple regression extends the model to several predictors:

The coefficient β_j is interpreted as the expected difference in Y associated with a one-unit increase in X_j, holding the other included predictors constant. This is a conditional association, not automatically a causal effect.

The OLS objective remains:

In matrix notation, when the design matrix has full column rank:

Here, X is the model matrix including a column of ones for the intercept, Y is the outcome vector, and b is the vector of estimated coefficients. This expression requires that X′X be invertible; exact perfect multicollinearity violates that condition.

Fit statistics

The total variation in the outcome can be partitioned as:

where SST = ∑(Y_i − Ȳ)² is total sum of squares, SSR = ∑(Ŷ_i − Ȳ)² is regression sum of squares, and SSE = ∑(Y_i − Ŷ_i)² is residual sum of squares. The coefficient of determination is:

R² is the proportion of in-sample variation represented by the model. Adding predictors cannot reduce ordinary R², even when they add little useful information. Adjusted R² penalizes unnecessary predictors:

where k is the number of predictors excluding the intercept. Adjusted R² is useful for comparison among models fitted to the same outcome and sample, but it is not a substitute for out-of-sample validation.

Inference for coefficients

Under the classical linear model assumptions, the standard error of coefficient b_j quantifies sampling uncertainty. A commonly used test is:

Usually β_j,0 = 0. A confidence interval is:

where t^* is a critical value from the relevant t distribution. A p-value is the probability, under the null model and specified assumptions, of observing a result at least as incompatible with the null as the observed result. It is not the probability that the null hypothesis is true, nor a measure of practical importance.

5. Assumptions and Diagnostics

OLS estimates can be computed without assumptions, but conventional standard errors, tests, and interpretations depend on assumptions. Important conditions include:

• Linearity: the conditional mean of Y is correctly represented by the model form.
• Independence: observations or errors are not improperly dependent. Time-series and clustered data require special treatment.
• Exogeneity: the error has mean zero conditional on the predictors, often written E(ε_i|X_i) = 0. This is central to unbiased causal interpretation.
• Homoscedasticity: error variance is constant across relevant predictor values. Heteroscedasticity does not necessarily bias OLS coefficients but can invalidate conventional standard errors.
• No perfect multicollinearity: no predictor is an exact linear combination of other predictors.
• Normality of errors: mainly relevant for small-sample exact inference, not a requirement for unbiased OLS coefficients.

Diagnostics should combine substantive knowledge, plots, and numerical measures:

The variance inflation factor for predictor X_j is:

where R_j² is obtained by regressing X_j on the other predictors. High VIF indicates imprecise coefficient estimation due to redundancy; it does not prove that a model is invalid. Thresholds such as 5 or 10 are rules of thumb, not universal laws.

Robust standard errors can provide more reliable inference under heteroscedasticity, but they do not repair nonlinear functional form, omitted variables, measurement error, or dependence. Cluster-robust methods are appropriate when observations are grouped and within-group errors may be correlated, provided the number and structure of clusters support the method.

6. Categorical Predictors

A categorical variable is represented using indicator, or dummy, variables. For a binary variable:

where D_i = 1 for the focal group and D_i = 0 for the reference group. The intercept is the expected outcome for the reference group, and β₁ is the expected difference between the focal and reference groups.

For a categorical variable with g groups, use g − 1 indicators when an intercept is included. Including all g indicators plus an intercept creates perfect multicollinearity, known as the dummy-variable trap. The reference category should be clearly reported and substantively defensible.

A coefficient on a category is a conditional mean difference, not necessarily evidence of discrimination or a causal group effect. Group membership may be associated with other variables, and classification may reflect institutional processes.

7. Interactions and Nonlinearity

An interaction means that the association between one predictor and the outcome depends on another predictor. A linear interaction model is:

The conditional slope of X is:

Therefore, β₁ is the effect or association of X only when Z = 0. If zero is not meaningful, centering Z at its mean or another substantively meaningful value improves interpretation, although centering does not change fitted values.

For a binary group indicator D and a continuous predictor X, the interaction model:

has slope β₁ for the reference group and slope β₁ + β₃ for the focal group. Interaction results are best communicated with predicted values or marginal-effect plots rather than isolated coefficients.

Nonlinearity can be modeled with transformations or polynomial terms, for example:

The marginal association is β₁ + 2β₂X. Polynomial models should be fitted only when supported by theory, design, or diagnostic evidence; high-degree polynomials can behave erratically at the boundaries.

8. Logistic Regression for Binary Outcomes

Linear regression can predict probabilities below 0 or above 1, so a binary outcome requires a suitable link function. Let p_i = P(Y_i = 1|X_i). Logistic regression models the log odds:

The probability is recovered by:

where η_i = β₀ + β₁X_1i + ⋯ + β_kX_ki. The odds ratio for a one-unit increase in X_j, holding other predictors constant, is:

An odds ratio above 1 indicates higher odds of Y = 1, while an odds ratio below 1 indicates lower odds. Odds are not probabilities. For example, odds of 2 correspond to probability 2/(1+2) = 0.667, not 2 or 200%.

Because the probability scale is nonlinear, a fixed coefficient does not represent a constant probability change. Report predicted probabilities, risk differences, or marginal effects for decision-makers. Logistic regression assumes an appropriate specification for the logit, independent observations unless dependence is modeled, and adequate information across outcome groups. Separation—when predictors nearly perfectly distinguish the outcome—can produce unstable or non-finite estimates.

Classification thresholds should reflect decision costs. Accuracy alone can be misleading under class imbalance. Appropriate evaluation may include sensitivity, specificity, precision, recall, the area under the ROC curve, calibration, and expected business loss. A model can discriminate well while producing poorly calibrated probabilities.

9. A Graduate-Level Applied Case

Background

The following is a fictional illustrative case. A subscription software firm wants to understand monthly customer cancellation. Management has transaction records for 8,000 customers, including tenure, monthly fee, number of support contacts, service interruptions, contract type, and cancellation status.

Problem

The firm initially proposes: “Customers with more support contacts are caused to cancel; therefore, reduce support access.” This conclusion risks confusing customer dissatisfaction with the support process itself. Customers experiencing technical problems may both contact support more often and be more likely to cancel.

Analysis

The analytics team first inspects missingness, duplicate records, outcome timing, and whether predictors were measured before cancellation. It plots cancellation rates by tenure and contract type and compares distributions for retained and cancelled customers. It then estimates a logistic model including support contacts, interruptions, tenure, monthly fee, and contract type. It uses a pre-specified validation split and reports confidence intervals and calibration, not only a classification accuracy.

The support-contact coefficient is positive in a bivariate model but becomes smaller after service interruptions are added. This pattern is consistent with confounding or mediation, although the observational data cannot establish which mechanism dominates. The team also tests an interaction between support contacts and interruption status because contacts may be more consequential when service reliability is poor.

Approach and solution

Rather than restricting support, the firm conducts a randomized service-recovery experiment among eligible customers with recent interruptions. One group receives ordinary support, while another receives proactive technical assistance and a temporary service credit. Cancellation is measured over a pre-defined follow-up period. Randomization makes the treatment groups comparable in expectation and allows a stronger causal estimate than the observational regression.

The predictive model is used for triage, not automatic denial of service. A human reviews high-risk cases, and the company audits performance across customer segments. The model is monitored for drift after pricing or product changes.

Outcome and learning points

The observational model identifies useful risk signals, but it does not justify reducing support. The experiment provides evidence about the effect of proactive recovery. The case illustrates five principles: temporal ordering matters; controls do not automatically remove confounding; interactions may reveal heterogeneous relationships; prediction and causal inference require different designs; and deployment requires governance beyond statistical fit.

10. Model Development and Communication Best Practices

A defensible workflow is:

1. Define the decision, outcome, target population, and time horizon.
2. Establish measurement definitions and inspect data quality.
3. Use substantive theory to select candidate predictors.
4. Explore distributions, missingness, and bivariate patterns.
5. Specify the model before repeatedly searching for a favorable result.
6. Estimate the model and inspect diagnostics.
7. Validate predictions on data not used for fitting.
8. Conduct sensitivity analyses for alternative specifications and influential cases.
9. Communicate effect sizes, uncertainty, assumptions, and limitations.
10. Monitor performance and fairness after deployment.

Avoid automatic stepwise selection as a substitute for theory. It can produce unstable variables, biased standard errors, and overoptimistic fit. Separate exploratory analysis from confirmatory claims, and correct or transparently report multiple testing when many hypotheses are examined.

Ethical and governance issues

Regression systems may reproduce historical discrimination, encode protected characteristics through proxies, or impose unequal error costs. Analysts should document data provenance, access controls, variable definitions, missing-data procedures, model versions, and intended use. Sensitive decisions require lawful processing, proportionality, explainability appropriate to the context, human oversight, and appeal mechanisms. Removing a protected variable does not guarantee fairness if correlated proxies remain.

AI-assisted analytics can accelerate coding, feature generation, and reporting, but generated code and interpretations require independent verification. Confidential data should not be placed into unapproved tools. Reproducible scripts, reviewable data transformations, and versioned outputs are preferable to opaque automated workflows.

Challenges and Limitations

Correlation and regression are limited by observational design, measurement error, missing data, selection bias, reverse causality, omitted variables, and extrapolation beyond the observed range. A high R² can arise from common trends or leakage. A low R² can coexist with a valuable predictor when outcomes are inherently noisy or decisions concern individual cases. Statistical significance depends on sample size and does not determine managerial value. Finally, a model calibrated in one market, period, or customer population may fail after structural change.

Chapter Summary

Correlation summarizes the direction and strength of linear association but does not establish causation. Simple regression estimates a fitted line, while multiple regression estimates conditional associations after including several predictors. OLS minimizes squared residuals, and its conventional inference depends on assumptions concerning linearity, independence, exogeneity, variance, and collinearity. Diagnostics are essential because fit statistics alone cannot reveal misspecification or influential cases.

Categorical variables require an explicit reference group, and interactions show that one association varies with another variable. Logistic regression models log odds for binary outcomes; odds ratios must be distinguished from probabilities, and model quality requires both discrimination and calibration. Across all methods, sound analysis depends on clear measurement, appropriate design, validation, transparent uncertainty, ethical governance, and restraint in causal interpretation.

Key Terms

Correlation: A standardized measure of linear association between two quantitative variables.

Covariance: A measure of joint variation whose units are the product of two variables’ units.

Regression: A family of models used to estimate conditional relationships and predictions.

Outcome variable: The response or target quantity modeled by an analysis.

Predictor: A variable used to explain or predict an outcome.

Residual: The observed outcome minus its fitted value.

Ordinary least squares: Estimation that minimizes the sum of squared residuals.

Slope: Expected change in an outcome associated with a one-unit predictor change, conditional on the model.

Intercept: Fitted outcome when all predictors equal zero.

R²: Proportion of in-sample outcome variation represented by a fitted model.

Adjusted R²: A fit measure that penalizes additional predictors.

Heteroscedasticity: Nonconstant conditional variance of model errors.

Multicollinearity: Redundancy or strong linear association among predictors.

Leverage: Unusualness of an observation’s predictor values.

Interaction: A relationship in which one predictor’s association depends on another variable.

Dummy variable: An indicator variable representing membership in a category.

Log odds: The logarithm of the odds of a binary outcome.

Odds ratio: The multiplicative change in odds associated with a predictor change.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Confounding: Distortion of an association by a variable related to both the predictor and outcome.

Review Questions

5 Short Answer Questions

1. Why does correlation not establish causation?
2. Interpret a regression slope while holding other predictors constant.
3. What does R² measure, and what does it not measure?
4. Why is a reference category required when using dummy variables with an intercept?
5. Distinguish odds from probability in logistic regression.

5 Long Answer Questions

1. Explain the OLS objective and discuss the assumptions required for conventional coefficient inference.
2. Compare simple and multiple regression, including the meaning of conditional interpretation.
3. Discuss how residual plots, leverage, Cook’s distance, and VIF contribute to model diagnosis.
4. Explain interaction terms and show how centering can improve interpretation without changing fitted values.
5. Compare prediction-oriented and causal interpretations of regression models.

3 Analytical / Application Questions

1. A firm observes that customers who receive more complaints are more likely to cancel. Design an analysis that distinguishes a predictive relationship from a causal claim.
2. A logistic model reports an odds ratio of 1.40 for a one-unit increase in a risk score. Explain how you would communicate this result and what additional information a manager needs.
3. A multiple regression has a high R², significant coefficients, and a curved residual pattern. Diagnose the problem and propose a defensible next analytical step.

Multiple Choice Questions with Answers

Question 1

Which statement best describes Pearson correlation?

A. It measures any possible relationship between two variables.

B. It measures the strength and direction of a linear association.

C. It proves that changes in one variable cause changes in another.

D. It is expressed in the units of both variables.

Correct Answer: B. It measures the strength and direction of a linear association.

Question 2

In the model Ŷ = 50 + 3X, what is the predicted value when X = 4?

A. 53

B. 62

C. 150

D. 200

Correct Answer: B. 62

Question 3

In multiple regression, the coefficient of a predictor is generally interpreted as:

A. Its unconditional correlation with the outcome.

B. The outcome variance caused by that predictor.

C. Its association with the outcome holding other included predictors constant.

D. The probability that the predictor is statistically significant.

Correct Answer: C. Its association with the outcome holding other included predictors constant.

Question 4

Which condition creates perfect multicollinearity?

A. A predictor has a non-normal distribution.

B. Two predictors are exact linear combinations of each other.

C. Residual variance is constant.

D. The sample contains an influential observation.

Correct Answer: B. Two predictors are exact linear combinations of each other.

Question 5

If r = −0.80, what is R² in a simple linear regression with an intercept?

A. −0.64

B. −0.80

C. 0.16

D. 0.64

Correct Answer: D. 0.64

Question 6

In an interaction model Y = β₀ + β₁X + β₂Z + β₃XZ + ε, the slope of X is:

A. β₁ for every value of Z

B. β₁ + β₃Z

C. β₂ + β₃X

D. β₀ + β₁Z

Correct Answer: B. β₁ + β₃Z

Question 7

If a logistic-regression coefficient is ln(2), the corresponding odds ratio is:

A. 0.5

B. 1

C. 2

D. ln(2)

Correct Answer: C. 2

Question 8

Which statement about a logistic-regression odds ratio is correct?

A. It is always a probability.

B. It describes a multiplicative change in odds, holding other predictors constant.

C. It can only range from 0 to 1.

D. It proves that the predictor causes the outcome.

Correct Answer: B. It describes a multiplicative change in odds, holding other predictors constant.

Question 9

What is a primary purpose of a residual-versus-fitted plot?

A. To select the reference category automatically.

B. To identify possible nonlinearity or unequal residual spread.

C. To calculate an odds ratio.

D. To guarantee causal validity.

Correct Answer: B. To identify possible nonlinearity or unequal residual spread.

Question 10

Why can a high in-sample R² be insufficient evidence of a useful predictive model?

A. Because R² is always negative.

B. Because it may reflect overfitting, leakage, or relationships that do not generalize.

C. Because R² cannot be calculated for regression.

D. Because high fit proves the model is biased.

Correct Answer: B. Because it may reflect overfitting, leakage, or relationships that do not generalize.