Correlation, Regression, and Generalized Linear Models

Introduction and Context

Correlation and regression are central tools for describing relationships, making predictions, and quantifying uncertainty. In artificial intelligence and machine learning, regression provides a foundation for supervised learning, feature analysis, forecasting, experimentation, and model evaluation. Generalized linear models (GLMs) extend ordinary linear regression to outcomes such as binary decisions, counts, and rates.

A relationship between two variables does not, by itself, establish causation. A strong association may arise because one variable causes another, because both are influenced by a third variable, or because of selection bias, measurement error, or chance. Responsible statistical modeling therefore combines computation with subject-matter reasoning, diagnostic analysis, and transparent reporting.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish correlation from regression and causation.

2. Fit and interpret simple and multiple linear regression models.

3. Explain coefficients, residuals, fitted values, confidence intervals, and prediction intervals.

4. Diagnose nonlinearity, heteroscedasticity, dependence, outliers, and influential observations.

5. Interpret interactions, confounding, and transformed predictors.

6. Explain regularization and compare models using suitable criteria.

7. Describe logistic and Poisson regression as generalized linear models.

8. Evaluate models using statistical, predictive, ethical, and domain-specific considerations.

Correlation: Measuring Linear Association

For paired observations \((x_i,y_i)\), the Pearson sample correlation is

\[

r=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}.

\]

Here, \(n\) is the sample size and \(\bar{x}\), \(\bar{y}\) are sample means. The value of \(r\) lies between −1 and 1. Positive values indicate that larger values of one variable tend to accompany larger values of the other; negative values indicate an inverse linear tendency. Values near zero indicate little linear association, although a strong nonlinear relationship may still exist.

Correlation is dimensionless and symmetric: the correlation of \(X\) with \(Y\) equals that of \(Y\) with \(X\). It does not identify a response variable, provide a prediction equation, or prove causation. A scatterplot should accompany correlation because a single number can conceal curvature, clusters, outliers, or changing variability.

Spearman’s rank correlation is useful when the relationship is monotonic but not necessarily linear, or when variables are ordinal or heavily affected by outliers. It is based on ranks rather than original measurements.

Simple Linear Regression

Simple linear regression models the conditional mean of a response \(Y\) using one predictor \(X\):

\[

Y_i=\beta_0+\beta_1X_i+\varepsilon_i.

\]

The parameters are:

• \(\beta_0\): population intercept, the expected response when \(X=0\), if that value is meaningful.
• \(\beta_1\): population slope, the expected change in mean \(Y\) for a one-unit increase in \(X\).
• \(\varepsilon_i\): random error for observation \(i\).

The fitted equation is

\[

\hat{Y}_i=b_0+b_1X_i,

\]

where \(b_0\) and \(b_1\) are estimated from data. Ordinary least squares (OLS) chooses coefficients that minimize the residual sum of squares:

\[

SSE=\sum_{i=1}^{n}(Y_i-\hat{Y}_i)^2.

\]

The residual is \(e_i=Y_i-\hat{Y}_i\). In simple regression, the estimated slope can be written as

\[

b_1=\frac{\sum (X_i-\bar X)(Y_i-\bar Y)}{\sum (X_i-\bar X)^2},

\qquad b_0=\bar Y-b_1\bar X.

\]

These formulas require variation in \(X\); a slope cannot be estimated when all predictor values are identical.

Worked Example: Advertising and Sales

Suppose a retailer records weekly advertising expenditure \(X\), in thousands of pounds, and weekly sales \(Y\), in thousands of units. An estimated model is

\[

\hat{Y}=12.4+3.1X.

\]

The slope means that an additional £1,000 of advertising is associated with an estimated increase of 3.1 thousand units in mean weekly sales. At \(X=5\), predicted sales are

\[

\hat{Y}=12.4+3.1(5)=27.9

\]

thousand units. The intercept, 12.4 thousand units, is the predicted sales level when advertising is zero; it should not be interpreted literally if zero advertising is outside the observed range or if other business conditions change.

Multiple Linear Regression

With \(p\) predictors, the model is

\[

Y_i=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}+\varepsilon_i.

\]

The coefficient \(\beta_j\) represents the expected change in mean response for a one-unit increase in \(X_j\), holding the other included predictors constant. This conditional interpretation is essential. It differs from the marginal association between \(X_j\) and \(Y\).

In matrix notation,

\[

\mathbf y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon,

\]

and, when \(\mathbf X^T\mathbf X\) is invertible,

\[

\hat{\boldsymbol\beta}=(\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y.

\]

The matrix must have full column rank. Perfect multicollinearity makes the inverse unavailable; near multicollinearity produces unstable estimates and large standard errors.

Categorical predictors are represented by indicator variables. For a three-category variable, one category is usually chosen as a reference and two indicators are included. A coefficient then compares its category with the reference category, conditional on other predictors.

Confounding and Omitted Variables

Confounding occurs when a third variable is associated with both a predictor and the response and distorts their apparent relationship. For example, an observed relationship between ice-cream sales and drowning incidents may be confounded by temperature: warm weather increases both.

Adding a confounder can change a coefficient substantially. However, including every available variable is not automatically correct. A variable may be a mediator, collider, post-treatment variable, or source of bias. Causal interpretation requires a defensible design, temporal reasoning, and often a causal diagram—not merely a high regression \(R^2\).

Model Fit and Statistical Inference

The coefficient of determination is

\[

R^2=1-\frac{SSE}{SST},

\qquad SST=\sum_i(Y_i-\bar Y)^2.

\]

It is the proportion of sample variation in \(Y\) explained by the fitted model relative to an intercept-only model. In ordinary least squares with an intercept, \(0\le R^2\le1\). A high value does not prove causation, guarantee good future prediction, or imply that every predictor is useful.

Because \(R^2\) never decreases when predictors are added, adjusted \(R^2\) is sometimes reported:

\[

R^2_{adj}=1-(1-R^2)\frac{n-1}{n-p-1},

\]

where \(p\) is the number of predictors excluding the intercept. It penalizes model complexity, but it is not a substitute for validation.

Under the classical linear model assumptions—correct mean structure, independent errors, constant error variance, and normally distributed errors for exact small-sample inference—the error variance is estimated by

\[

\hat\sigma^2=\frac{SSE}{n-p-1}.

\]

A coefficient estimate is commonly tested with

\[

t=\frac{b_j-\beta_{j,0}}{SE(b_j)},

\]

where \(\beta_{j,0}\) is the null value, often zero. A confidence interval is

\[

b_j\pm t_{1-\alpha/2,n-p-1}SE(b_j).

\]

The normality assumption mainly supports exact tests and intervals; OLS estimation itself does not require normally distributed errors. With large samples, robust or asymptotic methods may be appropriate, but dependence, severe heteroscedasticity, or poor design can still invalidate conclusions.

A confidence interval describes uncertainty about a population parameter under the model and sampling procedure. A prediction interval for a future individual observation is wider because it includes both uncertainty in the mean estimate and individual random error.

Regression Assumptions and Diagnostics

Assumptions should be assessed using design knowledge, plots, and numerical diagnostics rather than accepted automatically.

Linearity

The conditional mean should be adequately represented by the predictors. A residual-versus-fitted plot with systematic curvature indicates that a polynomial term, transformation, spline, or different model may be needed.

Independence

Observations must be independent for standard errors to be valid. Time series, repeated measurements, clustered students, or multiple records per patient violate this assumption unless dependence is modeled using time-series, mixed-effects, clustered-standard-error, or related methods.

Constant Variance

Homoscedasticity means that the variance of errors is approximately constant across fitted values or predictor levels. A funnel-shaped residual plot suggests heteroscedasticity. Remedies include transforming the response, weighted least squares, or heteroscedasticity-robust standard errors.

Outliers and Influence

An outlier has an unusual response or predictor value. A high-leverage observation has an unusual predictor combination. An influential observation substantially changes fitted results when removed. Cook’s distance is one diagnostic for influence, but no single cutoff should replace substantive review. Data errors should be corrected; valid extreme cases should not be deleted merely because they are inconvenient.

Multicollinearity

Correlated predictors can make individual coefficients imprecise and sensitive to small data changes. Variance inflation factors are often used as screening diagnostics, but thresholds are conventions rather than universal laws. Centering can improve interpretation and numerical stability for interactions, but it does not remove correlation between predictors in general.

Interactions and Nonlinearity

An interaction means that the effect of one predictor depends on another. For a continuous predictor \(X\) and binary variable \(G\), an interaction model is

\[

E(Y)=\beta_0+\beta_1X+\beta_2G+\beta_3XG.

\]

When \(G=0\), the slope for \(X\) is \(\beta_1\). When \(G=1\), it is \(\beta_1+\beta_3\). The coefficient \(\beta_2\) is the group difference when \(X=0\), so centering \(X\) can make the main effects more meaningful. Main effects should generally remain in a model when their interaction is included.

Curvature can be modeled with polynomial terms such as \(X^2\), logarithmic transformations, splines, or generalized additive models. A polynomial coefficient should not be interpreted in isolation; the combined fitted curve is the meaningful object.

Regularization and Model Selection

Regularization controls model complexity by adding a penalty to the loss function. Ridge regression minimizes

\[

\sum_i(Y_i-\hat Y_i)^2+\lambda\sum_{j=1}^{p}\beta_j^2,

\]

while lasso minimizes

\[

\sum_i(Y_i-\hat Y_i)^2+\lambda\sum_{j=1}^{p}|\beta_j|.

\]

Here \(\lambda\ge0\) controls penalty strength. The intercept is normally not penalized, and predictors should commonly be standardized when measured on different scales. Ridge shrinks coefficients toward zero and is useful with correlated predictors. Lasso can set some coefficients exactly to zero, supporting sparse representations, although selected variables can be unstable when predictors are highly correlated. Elastic net combines both penalties.

The penalty parameter should be selected using training data only, typically through cross-validation within the training set. A final test set must remain untouched until final evaluation. Feature selection performed before cross-validation can cause data leakage and overly optimistic performance.

Model selection may use adjusted \(R^2\), AIC, BIC, cross-validation error, or domain-specific utility. AIC and BIC rely on likelihood assumptions and compare relative support among candidate models; they do not guarantee predictive superiority. The best model depends on the goal: explanation, causal estimation, prediction, ranking, or decision support.

Generalized Linear Models

Linear regression is inappropriate when the response distribution or mean structure is incompatible with a continuous normal-error model. A GLM has three components:

1. A random component describing the response distribution, such as Bernoulli or Poisson.

2. A systematic component, the linear predictor \(\eta=\beta_0+\beta_1X_1+\cdots+\beta_pX_p\).

3. A link function \(g\) satisfying \(g(\mu)=\eta\), where \(\mu=E(Y\mid X)\).

Logistic Regression

For a binary response, logistic regression models the log odds:

\[

\log\left(\frac{p}{1-p}\right)=\beta_0+\beta_1X_1+\cdots+\beta_pX_p,

\]

where \(p=P(Y=1\mid X)\). Solving for probability gives

\[

p=\frac{1}{1+e^{-\eta}}.

\]

The odds ratio associated with a one-unit increase in \(X_j\), holding other variables constant, is \(e^{\beta_j}\). An odds ratio of 1.5 means the odds—not the probability—are multiplied by 1.5. Probability changes depend on the starting probability.

Classification requires a decision threshold, often 0.5 but not necessarily. Accuracy can be misleading with imbalanced classes. Sensitivity, specificity, precision, recall, the area under the ROC curve, calibration, and decision consequences should be considered together.

Poisson Regression

For count data, Poisson regression commonly uses the log link:

\[

\log(\mu_i)=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}.

\]

Thus \(e^{\beta_j}\) is a multiplicative change in expected count per one-unit increase in \(X_j\), holding other predictors constant. If observations have different exposure times, an offset such as \(\log(t_i)\) may be included with coefficient fixed at one. Poisson models assume conditional mean and variance are equal; overdispersion may require quasi-Poisson or negative binomial methods. Zero inflation and excess zeros require additional investigation rather than automatic model substitution.

Illustrative Case: Predicting Hospital Readmission

Background and Problem

A hospital wants to identify patients at high risk of readmission within 30 days after discharge. The dataset contains readmission status, age, prior admissions, length of stay, diagnosis group, medication count, and a socioeconomic deprivation measure. The purpose is to support follow-up planning, not to deny care.

Analysis

A logistic regression is selected because the outcome is binary. Diagnosis group is encoded using reference categories, and continuous predictors are inspected for nonlinear effects. Prior admissions and medication count are correlated, so standard errors and coefficient stability are examined. Missing data are handled within the training process, and all preprocessing is fitted only on training folds.

The team first fits a main-effects model, then evaluates whether the effect of medication count differs by age using an interaction. Calibration plots and a held-out test set are used in addition to discrimination measures. A coefficient with an odds ratio above one indicates higher conditional odds of readmission, but it does not establish that changing that predictor would reduce risk.

Approach and Outcome

The final model uses penalization to reduce instability and selects a probability threshold based on the capacity of the follow-up service. Performance is reported separately across relevant demographic groups. Where calibration differs, the team investigates data quality, sample size, and clinical differences rather than simply changing thresholds without governance review.

The model is deployed as a decision-support tool. Clinicians can override it, and predictions are recorded for monitoring. Periodic reassessment is scheduled because treatment practices, patient populations, and coding systems change.

Learning Points

The case illustrates that model choice follows outcome type, that prediction is distinct from causation, that validation must reflect deployment, and that fairness and operational consequences are part of statistical quality. A technically accurate model can still be harmful if it uses biased labels, hides uncertainty, or converts risk estimates into automatic decisions without review.

Professional, Ethical, and Governance Considerations

Regression models can influence lending, employment, health care, education, and public services. Practitioners should document data provenance, target definitions, exclusions, transformations, model version, validation design, and known limitations. Sensitive attributes may be needed for fairness auditing even when they are excluded from prediction.

Important risks include proxy discrimination, historical bias, label bias, privacy loss, re-identification, automation bias, and distribution shift. Statistical parity measures can conflict, and no fairness metric resolves underlying policy choices by itself. Access controls, secure storage, consent where required, human oversight, appeal mechanisms, and monitoring for drift are essential. Regulatory requirements vary by jurisdiction and application; compliance should be treated as a design requirement rather than an afterthought.

Best Practices

• Begin with a clearly defined response, prediction horizon, and decision purpose.
• Explore data visually before fitting models.
• Separate exploratory analysis, training, validation, and final testing.
• Report effect sizes and uncertainty, not only p-values.
• Interpret coefficients on their correct scale, especially in GLMs.
• Check residuals, leverage, influence, calibration, and subgroup performance.
• Avoid causal claims from observational regression without an appropriate design.
• Use regularization and cross-validation when prediction is the objective.
• Prefer transparent, reproducible preprocessing and version-controlled code.
• Monitor deployed models for drift, performance degradation, and unintended effects.

Chapter Summary

Correlation measures the strength and direction of linear association but does not establish causation or provide a predictive model. Linear regression estimates conditional mean relationships by minimizing squared residuals; multiple regression permits adjustment for other predictors but requires careful interpretation of confounding, multicollinearity, interactions, and coding choices. Fit statistics such as \(R^2\) must be supplemented with uncertainty estimates, validation, and diagnostics.

Reliable regression analysis examines linearity, independence, variance stability, outliers, leverage, and influence. Regularization trades a small amount of bias for reduced variance and can improve prediction, particularly with many or correlated predictors. GLMs extend regression through a response distribution and link function: logistic regression models binary outcomes through log odds, while Poisson regression models counts through log expected counts. Across all models, appropriate validation, ethical governance, fairness assessment, and monitoring are essential for responsible application.

Key Terms

Correlation: A standardized measure of linear association between two variables.

Regression: A method for modeling a response using one or more predictors.

Slope: Expected change in the conditional mean response for a one-unit predictor change.

Intercept: Model-predicted mean response when all predictors equal zero.

Residual: Observed response minus fitted response.

Ordinary least squares: Estimation by minimizing the sum of squared residuals.

Multiple regression: Regression with two or more predictors.

Confounding: Distortion of an association by another variable related to predictor and response.

Interaction: A relationship in which one predictor’s effect depends on another predictor.

Multicollinearity: Strong linear dependence among predictors.

Leverage: Unusualness of an observation’s predictor values.

Influence: The extent to which an observation affects fitted results.

Heteroscedasticity: Nonconstant conditional error variance.

Regularization: Penalization used to control model complexity.

Ridge regression: Regression using an L2 coefficient penalty.

Lasso: Regression using an L1 coefficient penalty.

Generalized linear model: A model combining a response distribution, linear predictor, and link function.

Odds ratio: Multiplicative change in odds associated with a predictor change.

Offset: A known predictor contribution, commonly used for exposure in count models.

Calibration: Agreement between predicted probabilities and observed frequencies.

Review Questions

Short Answer Questions

1. Why does correlation not establish causation?

2. Interpret a slope of 2.4 in a regression model where the predictor is measured in hours and the response in units produced.

3. What is the purpose of a residual-versus-fitted plot?

4. Distinguish a confidence interval from a prediction interval.

5. What does an odds ratio represent in logistic regression?

Long Answer Questions

1. Explain the assumptions of ordinary least squares regression and describe diagnostic evidence that would indicate violations.

2. Compare ridge regression and lasso regression, including their penalties, strengths, and limitations.

3. Explain confounding, interaction, and multicollinearity, and show why they require different responses.

4. Describe the three components of a generalized linear model and apply them to logistic regression.

5. Discuss why model validation, calibration, fairness, and monitoring are necessary when regression models are deployed.

Analytical / Application Questions

1. A model predicting energy consumption has residuals that increase in spread as fitted consumption increases. What problem is suggested, and what remedies would you investigate?

2. A logistic model has an odds ratio of 2.0 for a binary treatment indicator. Explain why this does not mean that treatment increases probability by 100 percentage points, and give a numerical illustration using a baseline probability of 0.20.

3. A hospital’s count model shows variance much larger than its mean and many zero outcomes. Diagnose the likely modeling concerns and propose a structured evaluation strategy.

Multiple Choice Questions with Answers

Question 1

Which statement best describes Pearson correlation?

A. It measures the causal effect of one variable on another.

B. It measures the strength and direction of linear association.

C. It gives the probability that a response equals one.

D. It is always positive for related variables.

Correct Answer: B. It measures the strength and direction of linear association.

Question 2

In the fitted model Ŷ = 12 + 4X, what is the predicted response when X = 3?

A. 16

B. 20

C. 24

D. 36

Correct Answer: C. 24

Question 3

What does a multiple-regression coefficient usually represent?

A. The unconditional mean of the response

B. The change in response while holding other included predictors constant

C. The probability that the coefficient is causal

D. The proportion of unexplained variation

Correct Answer: B. The change in response while holding other included predictors constant

Question 4

Which pattern most strongly suggests heteroscedasticity?

A. Residuals form a random horizontal band

B. Residuals show a funnel-shaped spread

C. Predictors have different measurement units

D. The response is binary

Correct Answer: B. Residuals show a funnel-shaped spread

Question 5

What is a principal effect of multicollinearity?

A. It guarantees biased response measurements

B. It makes individual coefficient estimates unstable or imprecise

C. It forces all residuals to equal zero

D. It makes correlation impossible to calculate

Correct Answer: B. It makes individual coefficient estimates unstable or imprecise

Question 6

In a logistic regression, exp(βj) is interpreted as the:

A. Change in probability in percentage points

B. Multiplicative change in odds for a one-unit predictor increase

C. Expected count for a one-unit predictor increase

D. Residual standard deviation

Correct Answer: B. Multiplicative change in odds for a one-unit predictor increase

Question 7

Which penalty is associated with lasso regression?

A. The sum of absolute coefficient values

B. The sum of squared residual degrees of freedom

C. The product of all predictor values

D. The inverse of the sample size

Correct Answer: A. The sum of absolute coefficient values

Question 8

Which response is most naturally modeled using Poisson regression?

A. A person’s blood pressure measured continuously

B. Whether a customer churns

C. The number of equipment failures in a month

D. A ranked satisfaction category only

Correct Answer: C. The number of equipment failures in a month

Question 9

Why is a prediction interval generally wider than a confidence interval for the mean response?

A. It includes individual outcome variability as well as mean-estimation uncertainty

B. It always uses fewer observations

C. It ignores the fitted model

D. It is calculated only for categorical predictors

Correct Answer: A. It includes individual outcome variability as well as mean-estimation uncertainty

Question 10

What is a key purpose of cross-validation?

A. To prove that a model is causal

B. To estimate how well a modeling procedure may generalize to new data

C. To eliminate all measurement error

D. To make every coefficient statistically significant

Correct Answer: B. To estimate how well a modeling procedure may generalize to new data