Bias, Confounding, Effect Modification and Evidence Quality

Introduction

Health-care decisions depend on evidence that is not merely statistically precise, but also valid, applicable and trustworthy. An observed association between an exposure and an outcome may reflect a true causal effect, systematic error, random variation, or a combination of these. In epidemiological research, the central task is therefore not simply to calculate a risk ratio or odds ratio, but to determine whether that estimate answers the intended causal question.

This chapter examines four closely related threats to valid inference: bias, confounding, effect modification and random error. It also introduces methods for prevention and control, including restriction, matching, stratification, standardization and multivariable adjustment. The final sections address internal validity, external validity and certainty of evidence, with particular relevance to health-care managers who interpret research for policy, quality improvement and resource allocation.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Distinguish random error from systematic error.
2. Identify selection bias, information bias and common forms of misclassification.
3. Explain confounding and distinguish it from mediation and effect modification.
4. Apply restriction, matching, stratification, standardization and multivariable adjustment appropriately.
5. Interpret interaction on additive and multiplicative scales.
6. Assess internal validity, external validity and the certainty of evidence.
7. Recognize ethical, managerial and governance implications when evidence is biased or uncertain.

1. A framework for interpreting epidemiological associations

An epidemiological estimate can be represented conceptually as:

Observed association = causal effect + bias + confounding + random error.

This is not an arithmetic identity; it is a reasoning framework. A causal effect is the difference in outcome that would result from changing exposure, all else being appropriately controlled. Bias is systematic deviation from the truth caused by study design, conduct or analysis. Confounding is distortion caused by a third variable related to both exposure and outcome. Random error is unpredictable variation arising from sampling and measurement processes.

A useful causal diagram is shown below.

Figure: A conceptual causal diagram distinguishing confounding from mediation. C causes both E and Y; M lies on the pathway from E to Y.

Confounders should generally be controlled when estimating a total causal effect. Mediators should not automatically be controlled, because doing so may remove part of the effect of interest. A variable caused by exposure cannot be a confounder of that exposure–outcome relationship.

2. Random error and systematic error

Random error

Random error produces imprecision rather than a consistent directional distortion. It is reduced by increasing sample size, improving measurement reliability and using efficient study designs. A confidence interval expresses sampling uncertainty. For a risk ratio, a 95% confidence interval that includes 1.0 is compatible, under the model used, with no ratio-scale association; this does not prove absence of an effect.

For a sample proportion p̂ based on n independent observations, the approximate standard error is:

Here p̂ is the observed proportion and n is the sample size. This approximation is most useful when n p̂ and n(1 − p̂) are sufficiently large. It assumes independent observations and is not appropriate without modification for clustered or complex survey data.

A commonly used approximate 95% interval is:

The value 1.96 comes from the standard normal distribution. Exact or alternative intervals may be preferable for small samples or proportions near 0 or 1.

Systematic error

Systematic error, or bias, shifts an estimate away from the target value. A very large study can have a narrow confidence interval around a biased estimate. Precision therefore cannot compensate for invalid design or measurement.

3. Selection bias

Selection bias occurs when inclusion in, retention within, or analysis of a study depends on factors related to exposure and outcome in a way that makes the observed association differ from the association in the target population. It is not simply the fact that a sample is selected; all studies select participants. The problem is differential selection relevant to the research question.

Important forms

Sampling-frame bias arises when the sampling frame excludes relevant groups, such as using only landline telephone numbers to study access to care.

Non-response bias occurs when participants and non-participants differ in exposure, outcome or determinants of both. A high response rate does not guarantee absence of bias, and a low response rate does not prove bias; the relevant issue is the difference between responders and non-responders.

Loss to follow-up bias occurs when withdrawal or incomplete observation is related to prognosis and exposure. If loss is differential, the remaining cohort may no longer represent the original comparison.

Healthy-worker effect is a form of selection in occupational studies: employed people may be healthier than the general population, making occupational hazards appear less harmful when the wrong comparison group is used.

Berkson-type bias can occur in hospital-based studies when admission depends on exposure and disease. The relationship among patients may differ substantially from that in the source population.

Collider bias occurs when analysis conditions on a common consequence of two variables. For example, if both a chronic disease and an unrelated exposure increase the probability of hospital admission, restricting analysis to hospitalized patients can create a spurious association between them.

Prevention and assessment

Prevention includes defining the source population clearly, using appropriate sampling, maximizing follow-up, documenting reasons for non-participation, and analyzing losses transparently. In a managerial setting, patient registries should be checked for systematic exclusion of people without stable addresses, digital access or regular primary-care contact.

4. Information bias and misclassification

Information bias results from inaccurate, inconsistent or differential measurement of exposure, outcome, covariates or timing. Misclassification is the assignment of an observation to the wrong category.

Differential and non-differential misclassification

Misclassification is differential when its probability differs between comparison groups or depends on outcome status. It can bias an estimate in either direction and may create an association or hide a real one.

Misclassification is non-differential when the measurement error is similar across relevant comparison groups. A common oversimplification is that non-differential exposure misclassification always biases toward the null. That result often holds for a simple binary exposure with independent, non-differential error, but it is not universal. Multiple exposure categories, outcome-dependent processes, correlated errors and differential classification can produce other patterns.

Common examples

Recall bias occurs when people with an outcome remember or report previous exposures differently from those without the outcome. It is especially important in retrospective case–control studies.

Interviewer bias occurs when an interviewer’s expectations influence questioning, probing or recording. Blinding interviewers to outcome or exposure status can reduce it.

Observer bias occurs when outcome assessment is influenced by knowledge of treatment or exposure. Objective protocols, independent adjudication and blinded assessment are useful safeguards.

Social desirability bias occurs when respondents under-report stigmatized behavior or over-report desirable behavior.

Diagnostic suspicion bias occurs when knowledge of exposure changes the intensity of testing, thereby changing the probability of detecting disease.

Measurement quality has at least two dimensions. Reliability is consistency of measurement; validity is closeness to the construct or true value. A measure can be reliable but invalid—for example, a blood-pressure device that is consistently calibrated 10 mmHg too high.

5. Confounding

A confounder is a variable that is associated with the exposure, is a cause or proxy for a cause of the outcome, and is not an intermediate step in the causal pathway being estimated. Confounding is a property of a relationship and a causal question, not an intrinsic permanent property of a variable.

For example, suppose coffee consumption is associated with myocardial infarction. Smoking may be associated with coffee consumption and independently increase myocardial infarction risk. The crude association may therefore partly reflect smoking.

A practical diagnostic approach is:

1. Specify the causal question and target population.
2. Use subject-matter knowledge and, where possible, a directed acyclic graph.
3. Identify common causes of exposure and outcome.
4. Avoid adjusting for mediators, colliders and variables affected by exposure when estimating a total effect.
5. Compare crude and adjusted estimates, while remembering that a change in estimate is not itself proof of confounding.

A variable associated with both exposure and outcome is not automatically a confounder. It may be a mediator, collider, consequence of exposure or merely a correlate with no causal relevance.

6. Effect modification and interaction

Effect modification occurs when the effect of an exposure differs across levels of a third variable. The third variable is an effect modifier, and this finding is often scientifically important rather than a nuisance to be removed.

For example, a treatment may reduce mortality more strongly among patients with severe disease than among those with mild disease. Reporting only one average effect may conceal clinically relevant heterogeneity.

Effect modification depends on the effect measure. An exposure can have homogeneous risk ratios across strata but heterogeneous risk differences, or vice versa. Therefore, interaction should be assessed on the scale relevant to the decision.

For two binary exposures A and B, let risks be R₁₁ when both are present, R₁₀ when only A is present, R₀₁ when only B is present, and R₀₀ when neither is present. Additive interaction, often called the relative excess risk due to interaction, is:

RERI = 0 indicates no additive interaction under the specified model; a positive value indicates more joint excess risk than expected from adding individual excess risks. This formula assumes risks are defined over the same time period and categories are correctly specified.

On a multiplicative scale, interaction can be assessed by comparing the joint risk ratio with the product of the two individual risk ratios. The choice between additive and multiplicative scales should follow the causal, clinical or policy question. Additive interaction is often especially relevant to public-health impact because it concerns excess cases or risk differences.

Effect modification should usually be reported through stratum-specific estimates and confidence intervals, not concealed by an overall adjusted estimate.

7. Controlling confounding and improving comparability

No method is universally superior. Design-stage methods reduce opportunities for bias; analysis-stage methods can address measured variables but cannot recover information that was never collected.

Restriction

Restriction limits eligibility to one category of a potential confounder, such as including only non-smokers. It can improve comparability and reduce confounding, but it reduces sample size, limits external validity and prevents estimation of effects in excluded groups. Restriction is most useful when the variable is accurately measured and the target population permits such limitation.

Matching

Matching selects comparison subjects with similar values of selected variables. In a case–control study, each case might be matched to controls of similar age and sex. Matching can improve efficiency, especially for rare outcomes, but matched variables must be handled in the analysis. Ignoring matching can produce invalid standard errors or biased estimates.

Overmatching is matching on variables related to exposure but not confounding, particularly when they are proxies for exposure or lie on the causal pathway. It can reduce efficiency and obscure a genuine association.

Stratification

Stratification estimates associations separately within levels of a confounder. If stratum-specific estimates are similar, they may be combined using a weighted summary, such as the Mantel–Haenszel estimator. For a cohort analysis with strata i:

Here, in stratum i, aᵢ is the number of exposed cases, cᵢ is the number of unexposed cases, N₁ᵢ is the number exposed, N₀ᵢ is the number unexposed and Nᵢ is the total stratum size. The formula is appropriate for a stratified cohort risk-ratio summary under assumptions including correct classification, meaningful strata and a reasonably common underlying ratio across strata. Alternative estimators are needed for odds ratios, incidence rates or complex sampling.

If stratum-specific estimates differ materially, pooling may be misleading because effect modification may be present. Stratification is transparent and useful for teaching and diagnostics, but becomes unwieldy with many covariates.

Standardization

Standardization compares populations as if they had the same distribution of a confounder. In direct standardization, stratum-specific rates from a study population are applied to a standard population:

rᵢ is the observed rate in stratum i and wᵢ is the proportion of the standard population in that stratum, with ∑ᵢ wᵢ = 1. The resulting standardized rate is a weighted average, not necessarily an observed rate in any actual population. It is useful for comparing hospitals or regions with different age structures. Standardized rates should not be interpreted without reporting the standard population and stratum-specific data.

Multivariable adjustment

Regression models estimate associations while simultaneously including several covariates. Logistic regression is commonly used for binary outcomes, Cox regression for time-to-event outcomes, and Poisson or negative-binomial models for rates or counts. The model must reflect the design and estimand; an adjusted odds ratio is not automatically an adjusted risk ratio.

Adjustment is limited by residual confounding, incorrect functional form, missing data, measurement error and model dependence. A statistically significant coefficient does not establish causality. Analysts should pre-specify confounders based on causal reasoning rather than selecting variables only because they are statistically significant.

Propensity-score methods

A propensity score is the probability of receiving the observed exposure conditional on measured pre-exposure covariates. Matching, weighting or stratification by propensity score can improve covariate balance in observational studies. These methods do not control unmeasured confounding and require adequate overlap, correct specification and careful treatment of extreme weights.

8. A worked example

In an illustrative cohort, 40 of 400 exposed patients develop an outcome, compared with 20 of 400 unexposed patients. The risks are 0.10 and 0.05.

The crude risk ratio is 2.0. Suppose age is a confounder and age-specific analyses produce risk ratios of 1.4 in younger patients and 1.5 in older patients. Because the stratum-specific estimates are reasonably similar, there is little evidence of effect modification by age on the ratio scale. A pooled adjusted estimate around 1.45 may be appropriate, subject to the actual stratum sizes and a formal estimator.

If instead the risk ratio were 1.1 among younger patients and 3.0 among older patients, reporting only a pooled ratio would conceal heterogeneity. The correct response would be to report stratum-specific effects and investigate whether age modifies the exposure effect. The difference might reflect biology, treatment practice, measurement, random error or residual confounding; it should not be declared causal without further assessment.

9. Validity and certainty of evidence

Internal validity

Internal validity is the degree to which a study’s estimate is credible for the participants and conditions actually studied. It depends on selection, measurement, confounding control, temporality, missing data, adherence, analysis and outcome ascertainment. Randomization protects against confounding in expectation, but does not prevent non-adherence, attrition, measurement bias or chance imbalance in a particular trial.

External validity

External validity concerns transportability or generalizability to other patients, providers, settings and time periods. A highly internally valid trial may have limited applicability if it excludes older adults, people with multimorbidity or low-resource facilities. Generalizability is not guaranteed by a representative sample alone; it also depends on whether causal mechanisms, baseline risks, treatment delivery and competing risks are similar.

Certainty of evidence

Certainty is a judgment about confidence that the estimated effect is close enough to the truth to support a decision. The GRADE approach commonly begins randomized evidence at high certainty and observational evidence at low certainty, then considers risk of bias, inconsistency, indirectness, imprecision and publication bias. Evidence may be upgraded in selected observational circumstances, such as a very large effect or dose–response relationship, but upgrading is not automatic.

These domains should be considered separately:

Statistical significance is not a synonym for certainty. A small but precise effect may be clinically unimportant, while a clinically important effect may have a wide interval because evidence is sparse.

10. Case study: evaluating a digital hypertension programme

This is an illustrative case. A regional health system introduces a smartphone-supported hypertension programme. After one year, participating clinics report a larger reduction in systolic blood pressure than clinics using usual care.

Background and problem. Participation is voluntary. Patients who enrol are younger, more digitally literate and more motivated. Clinicians in participating clinics also receive additional training. The observed improvement could therefore reflect the programme, patient selection, clinician behaviour or regression to the mean.

Analysis. The programme evaluation team defines the target population and outcome in advance. It compares eligible patients at intervention and comparison clinics, measures baseline blood pressure using a standardized protocol, and records age, baseline pressure, comorbidity, medication adherence and socioeconomic indicators. Clinic-level clustering is addressed in the analysis. Missing follow-up measurements are investigated rather than simply excluded.

A causal diagram suggests that motivation affects both participation and blood-pressure control, making it a potential confounder. Clinic training may be part of the intervention package or a mediator, depending on the decision question. Conditioning only on patients who remain engaged could introduce selection bias if engagement is affected by both the programme and early response.

Approach and solution. The team uses a pre-specified adjusted analysis with overlap checks, conducts sensitivity analyses for missing outcomes, and reports results separately for older adults and people with limited digital access. It also performs a pragmatic cluster-randomized rollout in the next phase, with blinded outcome measurement where feasible.

Outcome. The initial observational estimate is reduced after adjustment, indicating that participant characteristics explained part of the crude difference. The randomized rollout provides more credible evidence of programme effectiveness, while subgroup results show lower reach among people with limited digital access.

Learning points. Voluntary participation can produce selection bias; adjustment cannot guarantee removal of unmeasured motivation; effect modification may reveal unequal benefit; and implementation success requires both effectiveness and equitable reach.

11. Ethical, managerial and governance implications

Biased evidence can allocate resources away from patients who need them most. Health-care managers should ask whether excluded groups, non-responders and people lost to follow-up differ systematically from those analyzed. Data governance should require transparent definitions, audit trails, privacy protection and documentation of missingness and data linkage.

AI and predictive systems may amplify bias when training data under-represent particular populations or encode unequal access to care. A model can have excellent overall accuracy while performing poorly for a minority group. Evaluation should therefore include subgroup calibration, sensitivity, specificity, false-positive and false-negative consequences, and monitoring after deployment. Automated adjustment is not a substitute for causal reasoning or ethical oversight.

12. Best-practice checklist

Before accepting an epidemiological result, ask:

• What is the precise population, exposure, comparator, outcome and time horizon?
• Could selection into the study or analysis depend on exposure and outcome?
• How were exposure, outcome and covariates measured?
• Is misclassification likely to be differential?
• Which variables are common causes, mediators or colliders?
• Were confounders selected using causal knowledge rather than p-values alone?
• Are stratum-specific effects different?
• Does the analysis respect matching, clustering and time at risk?
• Are confidence intervals sufficiently narrow for the decision?
• Is the finding applicable to the intended patients and service setting?
• What are the consequences of acting on a false positive or false negative conclusion?

Chapter Summary

Bias is systematic error; random error is unpredictable imprecision. Selection bias arises from differential inclusion, retention or analysis, while information bias results from inaccurate measurement. Misclassification may be differential or non-differential, and its direction of effect cannot always be assumed. Confounding is distortion by a common cause or proxy for a cause of exposure and outcome; it must be distinguished from mediation and collider bias. Effect modification represents genuine variation in an effect across strata and should be reported, not automatically controlled away.

Restriction, matching, stratification, standardization, regression and propensity-score methods can improve comparability, but each has assumptions and limitations. Internal validity concerns credibility within the study; external validity concerns applicability elsewhere. Certainty of evidence requires consideration of risk of bias, inconsistency, indirectness, imprecision and publication bias. Sound health-care management therefore requires both quantitative analysis and disciplined causal, ethical and contextual judgment.

Key Terms

Bias: Systematic deviation of an estimate from the target truth.

Random error: Unpredictable variation caused by sampling or measurement processes.

Selection bias: Distortion caused by differential selection, retention or analysis.

Information bias: Systematic error in measuring exposure, outcome or covariates.

Misclassification: Assigning an observation to the wrong category.

Differential misclassification: Measurement error differing across comparison groups.

Non-differential misclassification: Measurement error distributed similarly across comparison groups.

Confounding: Distortion by a variable related causally or associatively to exposure and outcome.

Mediator: Variable lying on the causal pathway between exposure and outcome.

Collider: Variable caused by two or more variables being analyzed.

Effect modification: Variation in an exposure effect across levels of another variable.

Interaction: Departure from a specified model of combined effects.

Restriction: Limiting eligibility to a category of a potential confounder.

Matching: Selecting comparison participants with similar characteristics.

Stratification: Estimating effects separately within defined subgroups.

Standardization: Applying stratum-specific rates to a common standard distribution.

Internal validity: Credibility of results for the population studied.

External validity: Applicability of findings to other populations or settings.

Certainty of evidence: Confidence that an estimate is sufficiently close to the truth for decision-making.

Residual confounding: Confounding remaining after attempted adjustment.

Review Questions

Short Answer Questions

1. Distinguish random error from systematic error.
2. What is the difference between differential and non-differential misclassification?
3. State the main conditions that help identify a confounder.
4. Why should effect modification not automatically be adjusted away?
5. Define internal validity and external validity.

Long Answer Questions

1. Explain the major forms of selection bias and describe strategies for prevention.
2. Compare restriction, matching, stratification and multivariable adjustment as methods for controlling confounding.
3. Explain why a mediator and a collider should generally not be treated as ordinary confounders.
4. Discuss additive and multiplicative interaction and their relevance to health-care decisions.
5. Explain the domains used to judge certainty of evidence and distinguish certainty from statistical significance.

Analytical / Application Questions

1. A hospital study finds that a digital follow-up service reduces readmissions. Users are younger and more motivated than non-users. Identify possible sources of bias and propose a stronger evaluation design.
2. An exposure has a risk ratio of 1.2 in one age group and 2.8 in another. Explain how you would determine whether age is an effect modifier, and identify the scale on which your conclusion is made.
3. A registry excludes patients without a permanent address and has 35% loss to follow-up, concentrated among people with severe disease. Assess the likely threats to validity and recommend analytic and governance responses.

Multiple Choice Questions with Answers

Question 1

Which feature most directly distinguishes confounding from effect modification?

A. Confounding is always caused by random error

B. Effect modification represents variation in the effect across strata

C. Effect modification is eliminated by increasing sample size

D. Confounding can occur only in randomized trials

Correct Answer: B. Effect modification represents variation in the effect across strata

Question 2

A study enrolls only people who have been admitted to hospital, although both the exposure and outcome influence admission. What bias is especially possible?

A. Collider bias

B. Recall bias

C. Healthy-worker effect

D. Random error

Correct Answer: A. Collider bias

Question 3

Which method limits study eligibility to one level of a potential confounder?

A. Matching

B. Restriction

C. Standardization

D. Blinding

Correct Answer: B. Restriction

Question 4

In direct standardization, what is multiplied by the standard-population weight for each stratum?

A. The pooled p-value

B. The stratum-specific rate

C. The crude odds ratio only

D. The sample-size confidence level

Correct Answer: B. The stratum-specific rate

Question 5

Why can non-differential misclassification not always be assumed to bias toward the null?

A. It is always caused by confounding

B. Its consequences depend on the measurement structure and variables involved

C. It occurs only in randomized trials

D. It is equivalent to publication bias

Correct Answer: B. Its consequences depend on the measurement structure and variables involved

Question 6

Which variable should generally not be controlled for when estimating the total effect of an exposure?

A. A pre-exposure common cause of exposure and outcome

B. A baseline confounder

C. A mediator caused by the exposure

D. A precisely measured age category

Correct Answer: C. A mediator caused by the exposure

Question 7

If an observed proportion is p̂ = 0.25 in a sample of n = 100, its approximate standard error is closest to:

A. 0.004

B. 0.025

C. 0.043

D. 0.250

Correct Answer: C. 0.043

Question 8

Which statement about internal validity is correct?

A. It refers primarily to applicability to other countries

B. It concerns the credibility of the estimate for the study population

C. It is established solely by a narrow confidence interval

D. It is identical to statistical significance

Correct Answer: B. It concerns the credibility of the estimate for the study population

Question 9

Which is a domain commonly considered when grading certainty of evidence?

A. Brand recognition

B. Publication bias

C. Hospital floor area

D. Number of study authors alone

Correct Answer: B. Publication bias

Question 10

A crude association disappears after appropriate adjustment for a common cause of exposure and outcome. This pattern most directly suggests:

A. Possible confounding

B. Definite effect modification

C. Guaranteed causal protection

D. Absence of random error

Correct Answer: A. Possible confounding