7. Association, Regression and Causal Analysis

Introduction and Context

Data analysis frequently begins with a practical question: are two or more variables related? A retailer may ask whether discounts are associated with sales, a hospital may examine whether waiting time is related to patient satisfaction, and a bank may investigate which characteristics are associated with loan default. These questions lead naturally to association measures and regression models.

However, association is not the same as causation. A statistical relationship may arise because one variable influences another, because both are affected by a third variable, because of selection bias, or simply because of random variation. Sound data analysis therefore requires more than calculating a correlation coefficient or fitting a predictive model. It requires attention to measurement, study design, assumptions, uncertainty, confounding, and the purpose for which the analysis will be used.

This chapter develops a connected framework for correlation, linear regression, logistic regression, multivariate regression, causal reasoning, experiments, A/B testing, and introductory quasi-experimental methods. The emphasis is on interpretation and decision-making rather than mechanical computation alone.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish association, prediction, and causation.

2. Select and interpret appropriate measures of correlation.

3. Build and interpret simple and multiple linear regression models.

4. Explain logistic regression and interpret odds ratios and predicted probabilities.

5. Evaluate model fit, uncertainty, assumptions, and influential observations.

6. Recognize confounding, mediation, selection bias, and collider bias.

7. Design and interpret experiments and A/B tests.

8. Explain the logic and limitations of introductory quasi-experimental approaches.

9. Communicate analytical findings responsibly to technical and managerial audiences.

From Association to Explanation

Association describes the degree to which variables vary together. If higher advertising expenditure tends to occur in months with higher sales, the variables are positively associated. If higher product prices tend to accompany lower quantities sold, they may be negatively associated. An association can be useful for prediction even when it does not provide a causal explanation.

Prediction concerns the accuracy with which an outcome can be estimated for new or future observations. A predictive model may use variables that are not causes of the outcome. For example, a model may predict hospital admission using a variable that records a clinical decision made shortly before admission. That variable may be highly predictive but not useful for deciding what intervention would prevent admission.

Causation concerns what would happen to an outcome if a relevant condition or treatment were changed. This is a counterfactual idea: the causal effect for a person compares the outcome under treatment with the outcome that would have occurred for the same person without treatment. Since both states cannot usually be observed simultaneously, causal analysis depends heavily on study design and assumptions.

A useful distinction is:

• Association: variables are statistically related.
• Prediction: information about one or more variables improves estimates of an outcome.
• Causation: changing one variable would produce a change in another under specified conditions.

Confusing these categories is one of the most consequential errors in applied analytics.

Measuring Association

Covariance and Correlation

Covariance indicates whether two variables tend to be above or below their means at the same time. Its magnitude depends on the units of measurement, so it is difficult to compare across datasets. Pearson’s correlation standardizes covariance:

\[ r = \frac{\operatorname{cov}(X,Y)}{s_Xs_Y} \]

The value of r ranges from −1 to +1. A positive value indicates that larger values of one variable tend to accompany larger values of the other; a negative value indicates an inverse linear relationship. A value near zero indicates little linear association, but it does not prove that no relationship exists.

Correlation is sensitive to outliers and can be misleading when the relationship is nonlinear. A curved relationship may have a correlation near zero even though the variables are strongly related. Correlation also does not establish direction or causality. A strong correlation between ice-cream sales and drowning incidents, for example, may reflect a common influence such as hot weather.

Spearman and Rank-Based Association

Spearman’s rank correlation assesses whether the relationship is monotonic: as one variable increases, the other tends consistently to increase or decrease, though not necessarily at a constant rate. It is useful for ordinal variables, non-normal data, and relationships affected by extreme values. Kendall’s tau is another rank-based measure, particularly useful for smaller samples and ordered data with ties.

For categorical variables, association may be examined using contingency tables, proportions, risk ratios, odds ratios, or chi-square tests. Statistical significance should not be confused with practical importance. A very small effect can become statistically significant in a large dataset, while a meaningful effect may be imprecisely estimated in a small sample.

Visual Inspection

A scatterplot should normally precede correlation or regression. It can reveal curvature, clusters, unequal variability, outliers, and data-entry errors. Grouped plots, boxplots, and stratified displays may show that an overall association differs across important subgroups. Visualization is not merely presentation; it is part of model diagnosis.

Linear Regression as a Model of Conditional Association

Simple linear regression represents the expected value of an outcome as a function of one predictor:

\[ Y_i = \beta_0 + \beta_1X_i + \varepsilon_i \]

Here, \(Y_i\) is the outcome, \(X_i\) is the predictor, \(\beta_0\) is the intercept, \(\beta_1\) is the slope, and \(\varepsilon_i\) is the unexplained component. The slope estimates the average change in the outcome associated with a one-unit increase in the predictor.

The intercept is the expected outcome when the predictor equals zero. It may have little practical meaning when zero is outside the observed range. The slope is often more important, but it too must be interpreted within the data range and in the units of measurement.

The fitted model is chosen by ordinary least squares, which minimizes the sum of squared residuals:

\[ \sum_i (Y_i-\hat{Y}_i)^2 \]

A residual is the observed outcome minus its fitted value. The coefficient of determination, \(R^2\), describes the proportion of variation in the observed outcome explained by the fitted linear relationship in the sample. It is not the proportion of causally explained variation, and a high \(R^2\) does not guarantee valid extrapolation or causal interpretation.

Multiple Linear Regression

With several predictors, the model becomes:

\[ Y_i = \beta_0 + \beta_1X_{1i}+\beta_2X_{2i}+\cdots+\beta_pX_{pi}+\varepsilon_i \]

A coefficient in a multiple regression is interpreted as the expected change in the outcome associated with a one-unit increase in that predictor, holding the other included predictors constant. This conditional interpretation is valuable but potentially misleading when the included variables are confounders, mediators, colliders, or consequences of the predictor.

Categorical predictors can be represented using indicator variables. If a variable has k categories, a model generally uses k−1 indicators and selects one category as the reference group. Interactions allow the association between one predictor and the outcome to depend on another predictor. For example, the effect of training hours may differ by employee experience level:

\[ Y = \beta_0+\beta_1X+\beta_2Z+\beta_3(XZ)+\varepsilon \]

The interaction coefficient indicates how the slope for X changes as Z changes. Main effects should not be interpreted in isolation when interactions are present.

Transformations and Nonlinearity

A linear model need not use raw variables. Logarithmic transformations can model diminishing returns and allow percentage interpretations. Polynomial terms, splines, or generalized additive models can represent curvature. The choice should be guided by subject knowledge, diagnostics, and out-of-sample performance rather than by indiscriminate searching for a convenient fit.

Logistic Regression for Binary Outcomes

Linear regression is inappropriate when the outcome is binary because fitted values can fall below zero or above one and because the variance is not constant. Logistic regression models the log-odds of an event:

\[ \log\left(\frac{p}{1-p}\right)=\beta_0+\beta_1X_1+\cdots+\beta_pX_p \]

where p is the probability of the event. Exponentiating a coefficient gives an odds ratio. An odds ratio above one indicates higher odds as the predictor increases; an odds ratio below one indicates lower odds. An odds ratio is not generally the same as a risk ratio, especially when the event is common.

For communication, predicted probabilities are often more understandable than odds ratios alone. The change in predicted probability depends on the starting probability and on the values of other predictors. A coefficient that is constant on the log-odds scale does not imply a constant probability change.

Logistic models can be evaluated using discrimination measures such as the area under the receiver operating characteristic curve, calibration plots, log loss, and Brier score. A model may discriminate well but be poorly calibrated, meaning that predicted probabilities do not correspond accurately to observed frequencies. The decision threshold should reflect the relative costs of false positives and false negatives rather than being selected automatically.

Uncertainty, Fit and Assumptions

An estimated coefficient is not the complete result. A confidence interval describes a range of parameter values compatible with the data under the model and sampling assumptions. A p-value measures incompatibility with a specified null hypothesis; it does not measure the probability that the hypothesis is true, the size of an effect, or its practical importance.

Regression assumptions require careful evaluation. Common considerations include:

• Linearity: the expected outcome is adequately represented by the chosen functional form.
• Independence: observations or errors are not improperly treated as independent when they are clustered, repeated, or time-dependent.
• Constant variance: residual spread is reasonably stable in ordinary linear regression.
• Approximate normality of errors: relevant mainly for small-sample inference, not as a requirement that the outcome itself be normal.
• Limited multicollinearity: predictors are not so strongly related that individual effects become unstable or difficult to separate.
• Appropriate specification: important variables, interactions, and nonlinearities have not been omitted without justification.

Residual plots can identify curvature and heteroscedasticity. Leverage measures identify observations with unusual predictor values, while influence measures such as Cook’s distance assess whether a particular observation substantially changes the fitted model. Removing an observation solely because it is inconvenient is poor practice. Analysts should investigate its provenance, verify data quality, and report sensitivity analyses.

Multicollinearity does not necessarily damage overall prediction, but it can inflate standard errors and make individual coefficients unstable. Remedies include combining related variables, collecting more informative data, using regularization, or changing the inferential question. Standardization can aid comparison of coefficients but does not solve conceptual redundancy.

Model fit should be separated from model validation. Training performance is usually optimistic. Cross-validation or a genuinely held-out test set provides better evidence about performance on new data. Time-dependent data require temporally appropriate validation; randomly mixing future observations into training data can create leakage.

Confounding and Causal Structure

A confounder is a variable associated with both the exposure and the outcome and capable of creating a misleading association if not addressed. Suppose a company observes that employees who attend an optional leadership course receive higher promotions. Prior performance may influence both course attendance and promotion. Adjusting for prior performance may reduce confounding, although the quality of the adjustment depends on measurement and study design.

A directed acyclic graph (DAG) is a useful framework for representing causal assumptions. It distinguishes:

• Confounding: a common cause of exposure and outcome.
• Mediation: a variable on the pathway through which exposure affects outcome.
• Effect modification: the effect differs across levels of another variable.
• Collider bias: conditioning on a common effect of two variables creates a non-causal association.

Adjustment is not automatically beneficial. Controlling for a mediator may remove part of the total effect. Controlling for a collider may introduce bias. A causal DAG helps analysts decide which variables should be included before examining coefficients. Statistical algorithms cannot replace substantive knowledge about how the data were generated.

Other threats include selection bias, measurement error, missing data, reverse causality, interference between units, and treatment changes over time. Missingness mechanisms—missing completely at random, missing at random, or missing not at random—have different implications. Complete-case analysis can be biased when missingness is systematic.

Experiments and A/B Testing

A randomized experiment assigns units to treatment conditions using a random mechanism. Randomization tends, in expectation, to balance both observed and unobserved characteristics between groups. The difference in average outcomes estimates the intention-to-treat effect when implementation and measurement are appropriate.

An A/B test is a common randomized experiment in digital products. Users may be randomly assigned to two interface versions, and outcomes such as conversion, retention, revenue, or error rate are compared. A credible test requires a clearly defined unit of randomization, a prespecified primary outcome, an adequate sample size, consistent exposure, and a plan for handling exclusions.

Analysts should avoid repeatedly checking results and stopping as soon as significance appears, testing many outcomes without adjustment, or changing hypotheses after seeing results. Guardrail metrics may detect harm elsewhere—for example, a higher conversion rate accompanied by increased cancellations. Statistical significance does not establish business value; the estimated effect, uncertainty, implementation cost, and risk must be considered together.

Cluster randomization is necessary when individuals influence one another or when treatment is delivered at the group level. Standard errors must then account for clustering. Ethical experimentation requires informed governance, privacy protection, fairness assessment, and avoidance of unreasonable risk.

Introductory Quasi-Experimental Reasoning

When randomization is impossible, analysts may seek designs that approximate the counterfactual using naturally occurring variation. These approaches require explicit assumptions and should not be presented as automatically causal.

In a difference-in-differences design, changes over time in a treated group are compared with changes in a comparison group. The central assumption is parallel trends: absent treatment, both groups would have followed similar trends. Pre-treatment trends can support but cannot prove this assumption.

A regression discontinuity design exploits a treatment rule based on a threshold. Units just above and below the cutoff may be comparable, allowing estimation of a local treatment effect if individuals cannot precisely manipulate the assignment variable and other relationships are sufficiently smooth at the threshold.

An instrumental variable affects treatment assignment but influences the outcome only through treatment, under strong exclusion and relevance assumptions. Instrumental-variable estimates often apply to a local subgroup rather than to the entire population.

Propensity scores summarize observed covariates related to treatment assignment. Matching, weighting, or stratification can improve comparability on measured variables, but these methods do not eliminate unmeasured confounding. Sensitivity analysis should address how strong an unmeasured factor would need to be to alter the conclusion.

An Illustrative Case: Evaluating a Customer-Service Training Programme

Background

A fictional telecommunications company introduced a voluntary customer-service training programme. Managers observed that trained employees appeared to achieve higher customer-satisfaction scores and proposed making the programme mandatory.

Situation and Problem

The initial comparison showed an average satisfaction score of 86 for trained employees and 79 for untrained employees. Management interpreted the seven-point difference as evidence that training caused improvement. However, participation was voluntary, and employees with stronger prior performance or more flexible schedules may have been more likely to enrol.

Analysis

The analytics team first plotted satisfaction distributions and examined scores over time. It found that trained employees had higher pre-training scores, indicating possible confounding. A linear regression adjusted for pre-training satisfaction, tenure, team, workload, and customer-case complexity. The adjusted association was smaller than the raw difference and had a wider confidence interval.

The team then proposed a randomized rollout. Eligible employees were randomly assigned to immediate training or delayed training. The primary outcome was average satisfaction during the eight weeks following rollout. Secondary outcomes included complaint escalation, handling time, and employee workload. The delayed group preserved access to training while providing a comparison period.

Approach and Solution

The experiment used the employee as the randomization unit, blocked by team to reduce imbalance. Analysis followed intention-to-treat principles. Results were reported with mean differences and confidence intervals, together with effects on guardrail metrics. A subgroup analysis examined whether effects differed by tenure, but this was treated as exploratory unless prespecified.

Outcome

The randomized comparison estimated the causal effect of offering immediate training, rather than merely describing who chose to attend. Management could therefore weigh the estimated improvement against training cost, handling-time changes, and operational capacity. The original observational association was retained as a useful hypothesis but not treated as causal evidence.

Learning Points

The case illustrates why baseline differences matter, why adjustment is not a substitute for randomization when randomization is feasible, and why a decision should consider multiple outcomes. It also demonstrates the distinction between the effect of being offered training and the effect among employees who actually complete it.

Professional and Ethical Practice

Regression and causal findings can influence credit decisions, medical treatment, employment, pricing, and public policy. Analysts should document data provenance, variable definitions, transformations, exclusions, model specifications, and validation procedures. Reproducible code and versioned data improve auditability.

Fairness requires examining whether measurement quality, error rates, and treatment effects differ across protected or vulnerable groups. Removing a sensitive attribute from a model does not guarantee fairness because proxy variables may remain. Privacy protection includes data minimization, access controls, secure storage, and careful reporting of small subgroups.

Communication should distinguish estimates from assumptions. Analysts should state the target population, outcome definition, time horizon, uncertainty, and whether the evidence is predictive or causal. Overstating causality can lead to ineffective or harmful interventions.

Emerging Directions and Best Practices

Modern analytics increasingly combines regression with machine learning, automated feature engineering, causal forests, Bayesian modelling, and probabilistic programming. These tools can improve flexibility or quantify uncertainty, but they do not remove the need for a valid design. Explainable machine learning can describe model behaviour, whereas causal inference requires assumptions about interventions and counterfactuals.

A disciplined workflow is therefore:

1. Define the decision, estimand, population, treatment, and outcome.

2. Understand how the data were collected and whether observations are independent.

3. Visualize distributions, relationships, missingness, and subgroup patterns.

4. Specify a model or design using substantive knowledge before searching results.

5. Check assumptions, uncertainty, sensitivity, and out-of-sample performance.

6. Distinguish association, prediction, and causal claims.

7. Communicate practical significance, risks, limitations, and ethical implications.

Chapter Summary

Association measures describe statistical relationships but do not by themselves establish causation. Correlation is useful for assessing linear or monotonic dependence, while regression models conditional relationships and supports prediction. Linear regression is appropriate for continuous outcomes under suitable assumptions; logistic regression models binary outcomes through log-odds and should be communicated using probabilities as well as odds ratios. Multiple regression requires careful interpretation because adjustment can address confounding but can also create bias when mediators or colliders are controlled.

Reliable analysis requires attention to residuals, nonlinearity, heteroscedasticity, multicollinearity, influence, missingness, leakage, uncertainty, and validation. Randomized experiments and A/B tests provide strong causal evidence when properly designed, while quasi-experimental approaches depend on assumptions such as parallel trends, continuity at a cutoff, or instrumental validity. Professional practice combines statistical reasoning with domain knowledge, ethical governance, transparent reporting, and awareness of the difference between a predictive model and a causal explanation.

Key Terms

Association: A statistical relationship between variables.

Correlation: A standardized measure of linear association, usually ranging from −1 to +1.

Covariance: A measure of joint variation whose magnitude depends on measurement units.

Regression: A modelling framework for relating an outcome to one or more predictors.

Residual: The difference between an observed outcome and its fitted value.

R-squared: The proportion of sample outcome variation represented by a linear regression model.

Multicollinearity: Strong dependence among predictors that can destabilize individual coefficient estimates.

Interaction: A relationship in which the effect of one predictor depends on another variable.

Logistic regression: A regression model for binary outcomes using the logit transformation of probability.

Odds ratio: The multiplicative change in odds associated with a predictor change.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Confounder: A variable related to both exposure and outcome that can distort their relationship.

Mediator: A variable lying on a causal pathway from exposure to outcome.

Collider: A common effect of two variables; conditioning on it can create bias.

Counterfactual: The outcome that would have occurred under an alternative exposure or treatment condition.

Randomization: Assignment by chance to treatment conditions to improve comparability.

Intention-to-treat: Analysis according to assigned treatment, regardless of compliance.

Difference-in-differences: A comparison of changes over time between treated and comparison groups.

Regression discontinuity: A quasi-experimental design using treatment assignment at a threshold.

Instrumental variable: A variable used to identify causal effects under strong relevance and exclusion assumptions.

Review Questions

Short Answer Questions

1. Why does correlation not establish causation?

2. What does a regression slope represent in a multiple regression model?

3. Why are predicted probabilities often preferable to odds ratios for communicating logistic regression results?

4. What is multicollinearity, and how does it affect inference?

5. What is the purpose of a residual plot?

Long Answer Questions

1. Explain the differences among association, prediction, and causation, using an applied example.

2. Describe the principal assumptions of ordinary linear regression and methods for diagnosing violations.

3. Explain confounding, mediation, and collider bias using a causal diagram or suitable examples.

4. Compare randomized experiments with observational regression analysis for estimating causal effects.

5. Discuss how an A/B test should be designed, analyzed, and communicated to support an organizational decision.

Analytical / Application Questions

1. A model shows that customers receiving a discount have higher retention, but discounts are offered mainly to customers already predicted to leave. Identify the likely bias and propose an improved design.

2. A logistic model has excellent discrimination but poor calibration. Explain what this means and recommend actions before deployment.

3. A policy is introduced in one region but not another. The treated region’s outcome improves more after implementation. What assumptions must be examined before using difference-in-differences to claim a causal effect?

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes prediction from causation?

A. Prediction always requires randomization, whereas causation does not.

B. A predictive variable must be a direct cause of the outcome.

C. Prediction concerns accuracy for new cases, while causation concerns the effect of changing an exposure.

D. Causation can be established whenever a model has a high R-squared.

Correct Answer: C. Prediction concerns accuracy for new cases, while causation concerns the effect of changing an exposure.

Question 2

A scatterplot shows a strong U-shaped relationship, but Pearson’s correlation is close to zero. What is the best interpretation?

A. There is no relationship between the variables.

B. The relationship is nonlinear and Pearson correlation fails to capture it adequately.

C. The data must contain a sampling error.

D. The variables are perfectly negatively correlated.

Correct Answer: B. The relationship is nonlinear and Pearson correlation fails to capture it adequately.

Question 3

In a multiple linear regression, the coefficient for training hours is interpreted as:

A. The causal effect of training hours under all circumstances.

B. The expected outcome change for one more training hour, holding included predictors constant.

C. The percentage change in the outcome for one more training hour.

D. The correlation between training hours and the outcome.

Correct Answer: B. The expected outcome change for one more training hour, holding included predictors constant.

Question 4

What does an odds ratio of 0.70 indicate, subject to the model and coding used?

A. The event probability decreases by exactly 30 percentage points.

B. The event odds are multiplied by 0.70 for the specified predictor change.

C. The event is impossible for the comparison group.

D. The model has 70% accuracy.

Correct Answer: B. The event odds are multiplied by 0.70 for the specified predictor change.

Question 5

Which practice is most likely to create data leakage in model validation?

A. Using a held-out test set once after model development.

B. Scaling predictors using parameters estimated from the training data only.

C. Randomly mixing future observations into the training set for a time-dependent forecasting problem.

D. Comparing predictions with outcomes in the test set.

Correct Answer: C. Randomly mixing future observations into the training set for a time-dependent forecasting problem.

Question 6

Why can controlling for a collider bias an analysis?

A. A collider is always measured with error.

B. Conditioning on a common effect can create an artificial association between its causes.

C. A collider is necessarily a mediator.

D. Controlling for a collider always increases sample size.

Correct Answer: B. Conditioning on a common effect can create an artificial association between its causes.

Question 7

What is the central assumption in a basic difference-in-differences analysis?

A. The treatment effect is identical for every individual.

B. The treated group must be twice as large as the comparison group.

C. The groups would have followed parallel trends without treatment.

D. The outcome must be binary.

Correct Answer: C. The groups would have followed parallel trends without treatment.

Question 8

Which feature most directly supports causal interpretation in an A/B test?

A. A high correlation between treatment and outcome.

B. Random assignment to treatment conditions.

C. A large number of predictors in the regression model.

D. A high training-set R-squared.

Correct Answer: B. Random assignment to treatment conditions.

Question 9

A model has unstable coefficients but strong overall predictive performance because predictors are highly correlated. Which issue is most likely present?

A. Heterogeneous treatment effects.

B. Multicollinearity.

C. Perfect calibration.

D. Regression discontinuity.

Correct Answer: B. Multicollinearity.

Question 10

Why should an A/B test include guardrail metrics?

A. To guarantee statistical significance.

B. To replace the primary outcome after analysis.

C. To detect harmful effects on important outcomes not captured by the primary metric.

D. To eliminate the need for randomization.

Correct Answer: C. To detect harmful effects on important outcomes not captured by the primary metric.