Regression, Experimental Design and Causal Analysis

Introduction

Regression analysis and experimental design are central to managerial decision-making because they help explain variation, predict outcomes, evaluate policies and estimate the effects of interventions. A regression model can reveal that advertising expenditure is associated with sales, but association alone does not establish that advertising caused the observed increase. Causal analysis requires a credible counterfactual: what would have happened to the same units, at the same time, had the intervention not occurred?

This chapter develops the logic and practice of simple and multiple regression, categorical variables, interactions and nonlinear effects. It then connects regression to randomized experiments and quasi-experimental designs. The emphasis is not merely on obtaining coefficients, but on diagnosing models, interpreting uncertainty, recognizing threats to validity and communicating results responsibly.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Formulate and interpret simple and multiple linear regression models.
2. Estimate coefficients using ordinary least squares and explain the underlying assumptions.
3. Diagnose nonlinearity, heteroskedasticity, dependence, outliers and influential observations.
4. Incorporate categorical variables, interactions, transformations and nonlinear relationships.
5. Distinguish prediction, association and causation.
6. Design and evaluate randomized experiments and major quasi-experimental approaches.
7. Communicate regression and causal findings with appropriate uncertainty and ethical caution.

1. Regression as a model of conditional relationships

Let Y_i denote an outcome for observation i, and let X_i denote a predictor. The population simple linear regression model is:

Here β₀ is the intercept, β₁ is the population slope and ε_i is an unobserved error. The conditional mean is:

The slope measures the average change in the conditional mean of Y associated with a one-unit increase in X. It is not automatically a causal effect. Causal interpretation requires additional assumptions about confounding, measurement, treatment assignment and model specification.

The fitted sample equation is:

where Ŷ_i is the predicted value and b₀ and b₁ are estimated from data. The residual is e_i = Y_i − Ŷ_i.

Ordinary least squares

Ordinary least squares (OLS) chooses coefficients that minimize the sum of squared residuals:

For simple regression, the slope and intercept are:

The denominator is positive when the predictor varies. The slope has units of outcome per unit of predictor; the intercept has units of the outcome, although it may lack practical meaning when zero is outside the observed range.

Worked example: advertising and sales

Suppose a retailer observes the following data, where advertising expenditure X is measured in ₹ lakh and weekly sales Y in ₹ lakh:

The means are X̄ = 2.5 and Ȳ = 12.75. The cross-product sum is 9.5 and the squared-deviation sum for X is 5. Therefore:

The fitted equation is Ŷ = 8.0 + 1.9X. A one-lakh increase in advertising is associated with an estimated ₹1.9 lakh increase in weekly sales. The prediction at X = 3 is ₹13.7 lakh. This small example describes association; it does not prove that advertising caused sales because stores may differ in location, customer traffic or managerial quality.

2. Multiple regression

With several predictors, the model becomes:

The coefficient β_j represents the expected change in Y associated with a one-unit increase in X_j, holding the other included predictors constant. This is a conditional association. “Holding constant” is a mathematical comparison, not necessarily a realistic intervention.

In matrix notation:

The OLS estimator is:

This expression requires that X′X be invertible, which fails under perfect multicollinearity. X includes a column of ones when an intercept is included.

Goodness of fit

The total variation in the outcome can be decomposed as:

where SST = ∑(Y_i − Ȳ)² is total sum of squares, SSR = ∑(Ŷ_i − Ȳ)² is regression sum of squares and SSE = ∑(Y_i − Ŷ_i)² is error sum of squares. The coefficient of determination is:

R² is the sample proportion of outcome variation explained by the fitted model relative to an intercept-only model. It does not establish causality, and a high value does not guarantee good predictions outside the observed range.

Because R² cannot decrease when predictors are added, adjusted R² is often reported:

Here n is sample size and k is the number of predictors excluding the intercept. Adjusted R² penalizes unnecessary predictors but is not a substitute for theory or out-of-sample validation.

Inference and uncertainty

Under the classical linear model assumptions, the standard error of an estimated coefficient quantifies sampling uncertainty. A commonly used test is:

The null value β_j,0 is often zero. A two-sided confidence interval is:

where t_{α/2, df} is the relevant critical value and df = n−k−1 under the usual model. Statistical significance should not be confused with managerial importance. A tiny effect can be statistically precise yet economically irrelevant; a large effect can be uncertain in a small sample.

3. Assumptions and model diagnosis

OLS coefficients can be computed without all classical assumptions, but interpretation and inference depend on them.

1. Linearity in parameters: the conditional mean is correctly represented by the included functional form.
2. Random sampling or appropriate assignment: observations support the intended population or design-based inference.
3. No perfect multicollinearity: no predictor is an exact linear combination of others.
4. Conditional mean independence: E(ε_i|X_i)=0 for unbiased conditional interpretation. This is especially demanding for causal claims.
5. Constant variance: Var(ε_i|X_i)=σ² for conventional standard errors.
6. Independence: errors are appropriately independent, or dependence is explicitly modeled.
7. Normality: not essential for unbiasedness, but useful for exact small-sample t and F inference under classical assumptions.

Diagnostic practice

A residual-versus-fitted plot can reveal curvature or changing spread. A Q–Q plot helps assess whether residual tails are unusually heavy. A plot of residuals over time can expose serial dependence. Leverage identifies unusual predictor combinations, while influence measures assess how much estimates change when an observation is removed. Cook’s distance is a commonly used influence diagnostic, but no single cutoff should replace substantive investigation.

Heteroskedasticity makes conventional standard errors unreliable. Heteroskedasticity-robust standard errors can improve inference when the mean model is appropriate, but they do not repair omitted variables, reverse causality or functional-form errors. With clustered data—such as employees within firms or customers within branches—cluster-robust standard errors or multilevel models may be needed.

Multicollinearity inflates standard errors and makes individual coefficients unstable, even when the model predicts well. Variance inflation factors are diagnostic indicators, not universal tests of model acceptability. Remedies include better measurement, combining conceptually related predictors, collecting more informative data or reconsidering whether separate effects can be identified.

4. Categorical variables, interactions and nonlinear effects

Indicator variables

A binary variable can be represented by an indicator:

For two groups:

β₀ is the expected outcome for the reference group and β₁ is the difference between the indicator group and the reference group. With g categories, use g−1 indicators when an intercept is present. Including all g indicators plus an intercept creates perfect multicollinearity, known as the dummy-variable trap.

For example, if branch type is urban, suburban or rural, select one reference category and interpret each included coefficient relative to it. The reference choice changes coefficient labels, not fitted values or substantive group comparisons.

Interactions

An interaction permits the effect of one variable to depend on another:

For the reference group D=0, the slope on X is β₁. For D=1, it is β₁+β₃. Thus β₃ is the difference in slopes. Main effects should generally remain in the model when an interaction is included, even if their individual p-values are large. Centering continuous variables can make lower-order coefficients more interpretable, but it does not remove multicollinearity or change fitted values.

Polynomial and transformed models

A quadratic relationship can be represented as:

The marginal association is:

Thus β₁ is not the effect of X at every value; it is the slope at X=0. A turning point, when β₂ ≠ 0, occurs at X* = −β₁/(2β₂), provided that this value lies within a meaningful data range.

Logarithmic transformations can reduce skewness and express proportional relationships. In a log-level model, ln(Y)=β₀+β₁X+ε, a one-unit increase in X is associated approximately with a 100β₁ percent change in Y when β₁ is small. In a log-log model, ln(Y)=β₀+β₁ln(X)+ε, β₁ is an elasticity: a 1% increase in X is associated with approximately a β₁% change in Y. Exact percentage interpretation for a binary regressor in a log outcome model is 100(e^β−1)%, subject to model assumptions.

5. From association to causation

A causal effect compares potential outcomes. Let Y_i(1) be the outcome for unit i if treated and Y_i(0) the outcome if untreated. The individual effect is Y_i(1)−Y_i(0), but only one potential outcome is observed for each unit. The average treatment effect (ATE) is:

The missing potential outcome is the fundamental problem of causal inference. Randomization solves it in expectation by making treatment independent of potential outcomes. In observational data, treatment may be related to prior characteristics that also affect the outcome.

A confounder is a variable associated with both treatment and outcome and not caused by treatment. Omitting it can bias the treatment coefficient. A mediator lies on the pathway from treatment to outcome; controlling for it may remove part of the total effect. A collider is a common consequence of two variables; conditioning on it can create a spurious association. Causal diagrams, or directed acyclic graphs, help distinguish these roles and guide adjustment decisions.

Regression adjustment can estimate a causal effect under assumptions such as consistency, appropriate treatment definition, conditional exchangeability given measured covariates and positivity. Exchangeability means that, conditional on covariates, treated and untreated units are comparable in potential outcomes. Positivity requires a nonzero probability of each treatment level for relevant covariate patterns. These assumptions are partly substantive and cannot be verified solely through a regression output.

6. Experimental design

A randomized controlled experiment assigns units to treatment or control using a known random mechanism. Randomization balances observed and unobserved characteristics on average. A simple post-treatment comparison estimates the intention-to-treat effect:

where Z is randomized assignment. The intention-to-treat effect measures the impact of being assigned to treatment, regardless of compliance. It is often the most policy-relevant effect.

Important design elements include a clearly defined unit of randomization, a prespecified outcome, adequate sample size, blocking or stratification when important prognostic groups are known, blinding where feasible, and protection against interference between units. Power analysis concerns the probability of detecting a substantively important effect under specified assumptions. Multiple outcomes and repeated significance testing increase false-positive risk; preregistration and transparent analysis plans improve credibility.

A factorial design evaluates several interventions simultaneously and can estimate interactions efficiently. A crossover design lets each participant receive multiple treatments, but requires stable conditions and limited carryover. Cluster randomization assigns groups rather than individuals and requires attention to intracluster correlation and effective sample size.

7. Quasi-experimental designs

When randomization is infeasible or unethical, quasi-experimental methods seek a credible comparison without random assignment.

Difference-in-differences

Suppose treated and comparison groups are observed before and after a policy. The estimator is:

The key identifying assumption is parallel trends: absent treatment, the treated group would have experienced the same average change as the comparison group. Pre-treatment trends can provide evidence, but cannot prove future parallel trends. Event-study specifications can display dynamic effects and test for anticipatory changes, while inference should account for group and time dependence.

Regression discontinuity

If treatment is assigned when a running variable crosses a known cutoff, observations just above and below the threshold may be comparable. A local discontinuity estimates the treatment effect at the cutoff. Validity depends on no precise manipulation around the threshold, continuity of potential outcomes and appropriate bandwidth and functional-form choices. The result may not generalize far from the cutoff.

Instrumental variables

An instrument Z affects treatment D but affects outcome Y only through treatment and is not related to unobserved confounders. Under relevance, exclusion and independence assumptions, instrumental-variables methods can estimate a local average treatment effect for compliers. These assumptions are strong and often difficult to defend empirically.

Matching and weighting

Propensity scores summarize observed covariates related to treatment assignment. Matching or inverse-probability weighting can improve balance on measured variables. Neither method solves unmeasured confounding. Balance diagnostics, overlap checks, sensitivity analysis and transparent specification choices are essential.

8. Illustrative case: a digital lending intervention

Background. A bank introduces an automated credit-education message intended to reduce late repayments. Branches are phased into the programme over six months.

Problem. Managers observe that participating branches have lower delinquency and ask whether the message caused the improvement.

Analysis. A raw comparison is inappropriate because early-adopting branches serve more digitally engaged customers. Analysts assemble monthly branch-level data on delinquency, customer mix, loan size, local unemployment and historical repayment. They plot pre-intervention trends and estimate a difference-in-differences model with branch and month fixed effects, clustered standard errors and treatment timing indicators.

Approach and solution. The analysis prespecifies delinquency as the primary outcome, checks whether pre-treatment trends are approximately parallel, tests sensitivity to alternative comparison branches and reports both percentage-point and relative effects. It distinguishes assignment to the messaging programme from actual message receipt. Customer privacy is protected through aggregation and access controls; protected characteristics are not used to deny credit, and fairness checks examine whether the intervention operates differently across groups.

Outcome. The estimated post-intervention reduction is 1.8 percentage points, with a confidence interval that excludes zero under the primary specification. Effects are smaller for customers with limited digital access. The bank therefore expands the programme but adds alternative communication channels and continues monitoring. The evidence supports a programme effect under the design assumptions, not a universal claim that messaging will reduce delinquency in every setting.

Learning points. Design precedes estimation; visual trend assessment is essential; heterogeneous effects matter operationally; statistical evidence must be combined with implementation and equity analysis.

9. Technology, AI and governance

Modern analytics platforms automate model fitting, feature selection and prediction. Machine-learning models may improve predictive accuracy through regularization, trees, ensembles or neural networks, but prediction does not itself identify causal effects. Automated variable selection can amplify data mining, unstable specifications and post-treatment adjustment. A responsible workflow separates training, validation and test data; documents feature construction; monitors drift; reports uncertainty; and preserves human accountability.

In regulated decisions such as lending, hiring and insurance, organisations should assess discrimination, explainability, privacy, data provenance and contestability. Sensitive attributes may be excluded from a model yet still be proxied by location, language or browsing history. Fairness metrics can conflict, particularly when base rates differ, so governance requires explicit normative choices rather than a single technical score.

10. Best practices and limitations

A defensible regression or causal study should begin with a substantive question and a prespecified estimand. Researchers should define the unit, population, treatment, outcome, time horizon and comparison group. They should inspect data quality, missingness, measurement error and influential observations before modelling. Specifications should be justified by theory and design, not selected solely for a preferred p-value. Results should include effect sizes, confidence intervals, sample sizes, diagnostic evidence and sensitivity analyses.

Important limitations remain. Regression is sensitive to omitted variables, measurement error, selection, extrapolation and model misspecification. A statistically significant association can be operationally negligible. A causal estimate can be internally credible but externally narrow. Missing data may be informative, and attrition can undermine experiments. Ethical research requires proportional data collection, confidentiality, informed consent where applicable and honest communication of uncertainty.

Chapter Summary

Regression models describe conditional relationships and can support prediction, explanation and adjustment. OLS estimates coefficients by minimizing squared residuals, but meaningful interpretation requires attention to functional form, dependence, heteroskedasticity, multicollinearity, outliers and influence. Categorical variables require reference categories; interactions allow effects to vary across groups; polynomial and logarithmic terms represent nonlinear relationships.

Causal analysis differs from association because it concerns counterfactual outcomes. Randomization provides a strong basis for causal inference, while quasi-experimental methods rely on design-specific assumptions such as parallel trends, continuity at a cutoff or instrument validity. Credible work combines substantive theory, transparent design, diagnostics, uncertainty, sensitivity analysis and ethical governance. Modern AI can improve prediction and workflow, but it cannot remove the need for causal reasoning or accountability.

Key Terms

Regression: A method for modelling relationships between an outcome and one or more predictors.

Ordinary least squares: Estimation that minimizes the sum of squared residuals.

Residual: The observed outcome minus its fitted value.

Intercept: The model-predicted outcome when all predictors equal zero.

Slope: The expected change in the outcome associated with a one-unit predictor change.

Multicollinearity: Strong linear dependence among predictors.

Heteroskedasticity: Nonconstant conditional variance of errors.

Leverage: The unusualness of an observation’s predictor values.

Influence: The extent to which an observation affects estimated results.

Indicator variable: A variable coded, commonly, as zero or one for category membership.

Interaction: A term allowing one predictor’s association to depend on another.

Confounder: A variable related to both treatment and outcome that can distort their relationship.

Mediator: A variable on the causal pathway between treatment and outcome.

Collider: A common consequence of two variables that can induce bias when conditioned on.

Potential outcome: The outcome a unit would experience under a specified treatment condition.

Average treatment effect: The population average difference between treated and untreated potential outcomes.

Intention-to-treat effect: The effect of assignment to treatment, regardless of compliance.

Difference-in-differences: A comparison of changes over time between treated and comparison groups.

Regression discontinuity: A design exploiting treatment assignment at a threshold.

Instrumental variable: A variable used to identify causal effects under relevance and exclusion assumptions.

Review Questions

Short Answer Questions

1. What does a slope coefficient represent in a multiple regression model?
2. Why are only g−1 indicators normally used for a categorical variable with g categories when an intercept is included?
3. Distinguish heteroskedasticity from multicollinearity.
4. What is the fundamental problem of causal inference?
5. State the parallel-trends assumption in difference-in-differences analysis.

Long Answer Questions

1. Explain the OLS objective, derive the simple-regression slope conceptually and discuss its assumptions.
2. Explain how interactions and quadratic terms change coefficient interpretation.
3. Compare randomized experiments, matching and instrumental-variables approaches.
4. Discuss why a high R² does not establish causation or guarantee useful decisions.
5. Develop a diagnostic and reporting plan for a multiple regression used in a managerial study.

Analytical / Application Questions

1. A firm finds that trained sales staff generate higher revenue. Design an analysis that addresses employee selection, time trends and possible treatment heterogeneity.
2. A policy begins in one state but not neighbouring states. Specify a difference-in-differences model, its identifying assumption and at least three threats to validity.
3. A lending model uses postal code, device type and browsing history. Evaluate prediction, fairness, privacy and causal-interpretation risks, and recommend governance controls.

Multiple Choice Questions with Answers

Question 1

In the simple regression model Y_i = β₀ + β₁X_i + ε_i, what does β₁ represent?

A. The mean of Y for all observations

B. The expected change in Y associated with a one-unit increase in X

C. The unexplained variance in Y

D. The sample size minus two

Correct Answer: B. The expected change in Y associated with a one-unit increase in X

Question 2

If SSE = 40 and SST = 100, what is R²?

A. 0.40

B. 0.60

C. 1.40

D. 2.50

Correct Answer: B. 0.60

Question 3

In a model with an intercept and a three-category variable, how many indicator variables are normally included?

A. One

B. Two

C. Three

D. Four

Correct Answer: B. Two

Question 4

In Y = β₀ + β₁X + β₂D + β₃XD + ε, what is the slope on X when D=1?

A. β₀

B. β₁

C. β₁ + β₃

D. β₂ + β₃

Correct Answer: C. β₁ + β₃

Question 5

Which issue most directly makes conventional OLS standard errors unreliable while leaving the coefficient estimates potentially unbiased under exogeneity?

A. Heteroskedasticity

B. A categorical predictor

C. A nonzero intercept

D. Mean-centering a predictor

Correct Answer: A. Heteroskedasticity

Question 6

The average treatment effect is defined as:

A. The correlation between treatment and outcome

B. The difference in observed means only among treated units

C. The average difference between treated and untreated potential outcomes

D. The regression intercept

Correct Answer: C. The average difference between treated and untreated potential outcomes

Question 7

The key identifying assumption in a standard difference-in-differences design is:

A. Treatment effects must be zero

B. Treated and comparison groups have parallel untreated trends

C. All covariates must be normally distributed

D. The outcome must be binary

Correct Answer: B. Treated and comparison groups have parallel untreated trends

Question 8

Which design exploits a treatment rule based on crossing a known threshold?

A. Regression discontinuity

B. Simple random sampling

C. Factor analysis

D. Moving-average smoothing

Correct Answer: A. Regression discontinuity

Question 9

Why can conditioning on a collider create bias?

A. It necessarily reduces sample size to zero

B. It can induce an association between variables that were otherwise unrelated

C. It guarantees homoskedasticity

D. It removes all measurement error

Correct Answer: B. It can induce an association between variables that were otherwise unrelated

Question 10

In a log-log model, the slope is commonly interpreted as:

A. An elasticity

B. An intercept in percentage points

C. A variance estimate

D. A probability of treatment assignment

Correct Answer: A. An elasticity