Marketing Research, Statistics, and Causal Inference

Introduction

Marketing decisions increasingly depend on evidence: which customers to target, which message to deliver, which channel to fund, and whether a campaign created additional demand or merely captured demand that would have occurred anyway. Marketing analytics provides the tools for answering these questions, but the tools are not interchangeable. Descriptive statistics summarize observed data; predictive models estimate likely outcomes; causal methods estimate what would have happened under alternative actions.

This distinction is central. A campaign may be associated with higher sales because it worked, because high-intent customers were selected for exposure, or because sales and advertising were both driven by a seasonal event. Sound marketing research therefore combines research design, measurement, sampling, statistical analysis, and managerial judgment.

Learning Objectives

By the end of this chapter, you should be able to:

1. Formulate marketing research questions and select appropriate research designs.
2. Distinguish probability and nonprobability sampling and evaluate survey quality.
3. Apply descriptive statistics, confidence intervals, and hypothesis tests.
4. Interpret correlation and regression without confusing association with causation.
5. Design and evaluate experiments, A/B tests, quasi-experiments, and incrementality studies.
6. Identify threats to validity, ethical risks, and limitations of marketing evidence.
7. Translate analytical findings into defensible managerial decisions.

1. From Business Question to Research Design

A rigorous study begins with a decision, not a dataset. A useful sequence is:

Decision → research question → estimand → design → measurement → analysis → decision rule.

An estimand is the precise quantity the study aims to learn. Examples include the proportion of customers who are aware of a brand, the mean satisfaction score, the difference in conversion rates between two messages, or the incremental revenue caused by advertising.

Research questions should specify the population, treatment or exposure, outcome, time period, and comparison. “Does digital advertising work?” is too broad. “Among eligible customers during the four-week campaign, what is the difference in purchase probability caused by receiving the advertisement rather than not receiving it?” is more analytically useful.

Major Research Designs

Qualitative research is particularly valuable at the problem-definition stage. Interviews can reveal that “price dissatisfaction” actually reflects delivery fees or perceived unfairness. Quantitative studies can then test how widespread that insight is.

A cross-sectional study measures units at one point in time. A longitudinal study follows units or repeated samples over time. A panel repeatedly observes the same units, while a repeated cross-sectional design samples different units from the same target population at successive times.

2. Measurement and Survey Research

Measurement converts concepts into observable variables. A construct such as loyalty, trust, or perceived value cannot be observed directly; it must be represented through indicators. Good measurement requires conceptual clarity, reliability, and validity.

• Reliability concerns consistency. A scale should produce reasonably consistent results under similar conditions.
• Content validity asks whether the measure covers the relevant aspects of the construct.
• Construct validity asks whether the measure behaves as theory predicts.
• Criterion validity asks whether it relates appropriately to an external criterion, such as future retention.

A survey item should measure one idea, use understandable language, avoid leading wording, and specify a meaningful reference period. “How satisfied are you with our affordable and convenient service?” is double-barreled and leading. Better items separately measure affordability and convenience.

Likert-type response scales are common, but the analyst must distinguish the response format from the statistical assumptions. A single five-point item is ordinal: the categories have order, but equal distances cannot be guaranteed. A multi-item scale may be treated as approximately interval when its construction and distribution justify that choice.

Sampling

Let the target population contain N units and let a sample contain n units. In a simple random sample, every unit has a known and equal probability of selection. In stratified sampling, the population is divided into subgroups and sampled within each subgroup; this can improve precision and ensure representation. In cluster sampling, groups such as stores or regions are sampled, and units within them are observed; this can reduce cost but often increases sampling error because units within clusters resemble one another.

Coverage error occurs when the sampling frame excludes relevant members of the population. Nonresponse bias occurs when respondents differ systematically from nonrespondents. A large sample does not correct either problem automatically. Weighting can adjust for known differences, but it cannot fully repair unmeasured differences.

For a proportion estimate p̂ under simple random sampling, an approximate 95% confidence interval is:

Here, n is the sample size and 1.96 is the standard-normal critical value for a two-sided 95% interval. The approximation is most reliable when the expected counts of successes and failures are sufficiently large and observations are independent. For a mean, the corresponding interval is:

where x̄ is the sample mean, s is the sample standard deviation, n is sample size, and t_0.975,n−1 is the appropriate t critical value. A confidence interval describes uncertainty from sampling under the model; it does not guarantee that the population estimate lies in the interval with 95% probability after the data have been observed.

3. Descriptive Statistics and Hypothesis Testing

Descriptive analysis should precede modeling. Examine counts, missingness, distributions, outliers, cross-tabulations, and time patterns. Means can be misleading when distributions are skewed; medians and percentiles may better represent order values or customer spend.

The sample mean is:

The sample variance is:

The denominator n−1 provides the usual unbiased estimator of population variance under independent random sampling. Standard deviation has the same units as the measured variable; variance has squared units.

A hypothesis test compares data with a null model. Let H₀ represent no effect and H₁ represent an effect or specified direction. The p-value is the probability, under H₀ and the stated analysis assumptions, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true.

A test statistic for a one-sample mean is:

where μ₀ is the null hypothesized mean. A two-sided test at significance level α = 0.05 rejects the null when the p-value is below 0.05, but statistical significance does not establish practical importance. A small effect can become statistically significant in a very large sample, while a meaningful effect can fail to reach significance in a small sample.

Type I error is rejecting a true null hypothesis; Type II error is failing to reject a false null hypothesis. Statistical power is the probability of detecting an effect of a specified size under the alternative. Power increases with effect size, sample size, and measurement precision, and generally decreases with greater outcome variability.

Multiple testing creates a further risk: testing many outcomes, segments, or time windows increases the chance of at least one apparently significant result by chance. Pre-specification, correction procedures, and transparent reporting are therefore important.

4. Correlation, Prediction, and Regression

The Pearson correlation between variables X and Y is:

The value lies between −1 and 1 and measures linear association. It is sensitive to outliers and does not imply causation. A third variable may create a spurious association, reverse the apparent relationship, or mask a real relationship.

Ordinary least squares regression models the conditional mean of an outcome:

Here, Y_i is the outcome, X_i is a predictor, β₀ is the intercept, β₁ is the expected change in Y associated with a one-unit increase in X under the model, and ε_i is the residual. Multiple regression extends the model:

Regression coefficients are not automatically causal effects. Causal interpretation requires credible identification: random assignment or assumptions that adequately address confounding, selection, measurement error, and model misspecification. Including a control variable can reduce confounding, but controlling for a mediator or a collider can create bias.

Important diagnostic concerns include nonlinearity, heteroskedasticity, influential observations, multicollinearity, dependence over time, and missing-data mechanisms. Robust standard errors can address some forms of unequal variance, but they do not solve selection bias or omitted-variable bias.

Prediction and explanation have different objectives. A high-performing predictive model may exploit stable associations without identifying what would happen if a manager changed a variable. Causal analysis instead asks about interventions and counterfactual outcomes.

5. Experiments and A/B Testing

The causal effect for unit i can be expressed through potential outcomes: Y_i(1) is the outcome if treated and Y_i(0) is the outcome if untreated. The individual effect is Y_i(1) − Y_i(0), but both potential outcomes cannot normally be observed for the same unit at the same time. This is the fundamental problem of causal inference.

The average treatment effect is:

Random assignment makes treatment status independent of potential outcomes in expectation. Therefore, the difference in mean outcomes between treatment and control estimates the ATE when implementation is valid and interference is limited.

In an A/B test, eligible users are randomly assigned to version A or version B. Primary design decisions include the unit of randomization, treatment exposure, outcome definition, test duration, sample size, and stopping rule. Randomization at the user level may be inappropriate if users interact across accounts or if treatment spills over through social networks. In such cases, geographic, store-level, or account-level randomization may be preferable.

For binary conversion, the estimated difference is:

where p̂_T and p̂_C are treatment and control conversion rates. Relative lift is:

Absolute lift and relative lift answer different managerial questions. If treatment conversion is 5.5% and control conversion is 5.0%, the absolute lift is 0.5 percentage points and relative lift is 10%.

A credible test requires balance checks, treatment-fidelity monitoring, intent-to-treat analysis, and a pre-specified primary outcome. The intent-to-treat estimate preserves the benefit of randomization and measures the effect of assignment, even when some users do not comply. Per-protocol analyses can be informative but may reintroduce selection bias.

Do not repeatedly inspect results and stop when significance appears unless the testing method accounts for sequential monitoring. Also avoid declaring a winner based on many secondary metrics without adjustment. A statistically significant result may still be commercially unimportant if the effect is small, short-lived, or expensive to implement.

6. Quasi-Experiments and Incrementality

Randomized experiments are not always feasible. Quasi-experimental methods use design features or assumptions to approximate the missing counterfactual.

Difference-in-Differences

Suppose one market receives a campaign and another comparable market does not. The difference-in-differences estimator is:

The key assumption is parallel trends: absent treatment, the treated and comparison groups would have changed similarly. This assumption cannot be proved from post-treatment data, but pre-treatment trends, placebo tests, and event-study graphs can provide evidence. Anticipation, simultaneous interventions, migration, and differential seasonality threaten validity.

Regression Discontinuity

When treatment is assigned at a cutoff, units just above and below the threshold can be compared. The effect is local to the cutoff and requires no precise functional form if estimation is appropriately localized, but manipulation around the threshold and discontinuities in other variables must be examined.

Instrumental Variables

An instrument affects treatment assignment but affects the outcome only through treatment and is not related to unobserved confounders. Under strong assumptions, instrumental-variables estimates often identify a local average treatment effect for compliers, not necessarily the effect for every customer.

Incrementality Analysis

Incrementality asks: how much outcome would not have occurred without the marketing intervention? Common approaches include randomized holdouts, geo experiments, matched markets, and modeled counterfactuals. The incremental return on ad spend can be expressed as:

The units in the numerator and denominator must be compatible; revenue-based iROAS is not the same as profit-based return. If incremental revenue is $120,000 and incremental media spend is $30,000, iROAS is 4.0. This does not mean total revenue divided by total spend is 4.0.

Attribution models allocate observed conversions among touchpoints. Incrementality studies estimate the change caused by an intervention. Attribution can support optimization, but it should not be treated as causal evidence unless its assumptions are justified.

7. Applied Case: The Incremental Effect of Retargeting

The following is an illustrative case.

Background and Problem

An online retailer retargets visitors who viewed a product but did not purchase. Its platform reports that retargeted users purchase at a rate of 8%, compared with 3% among users who were not shown the advertisement. Management is considering doubling the budget.

Analysis

The observed difference is 5 percentage points, but retargeted users may have been more motivated before advertising. The platform’s comparison is therefore descriptive, not necessarily causal. The retailer defines the estimand as the difference in 14-day purchase probability caused by ad eligibility and delivery among qualified visitors.

The company randomly assigns eligible users to a treatment group, which can receive retargeting advertisements, and a holdout group, which is suppressed from those advertisements. Assignment is at the user level, the primary outcome is purchase within 14 days, and the analysis follows intent to treat.

Suppose 50,000 treatment users produce 3,250 purchases and 50,000 holdout users produce 3,000 purchases. Then:

p̂_T = 3,250/50,000 = 0.065

and

p̂_C = 3,000/50,000 = 0.060.

The incremental conversion estimate is 0.005, or 0.5 percentage points. Estimated incremental purchases are approximately 50,000 × 0.005 = 250. The observed treatment rate is 8% only in the original platform comparison; the randomized estimate is 6.5% versus 6.0%.

If contribution margin per incremental purchase is $24 and incremental advertising cost is $4,000, estimated incremental contribution is 250 × $24 = $6,000, producing incremental contribution return of $6,000 ÷ $4,000 = 1.5. The campaign may be worth continuing, but the decision should also consider long-term effects, frequency, brand outcomes, and uncertainty around the estimate.

Outcome and Learning Points

The experiment shows that most of the platform-observed difference was likely selection rather than incremental impact. The study should be repeated across seasons and customer segments, with safeguards against contamination and excessive frequency. The key lesson is that marketing performance should be evaluated against a credible counterfactual, not merely against an unexposed group that differs systematically from exposed customers.

8. Ethics, Governance, and Contemporary Practice

Marketing research must respect informed consent, privacy, security, and applicable data-protection law. Analysts should collect only necessary data, define retention periods, restrict access, and document legitimate uses. Sensitive attributes require heightened governance, especially when models influence prices, credit, eligibility, or service access.

Consent is not merely a checkbox. Participants should understand material risks, data uses, and withdrawal options. Deception in experiments requires strong justification, minimal risk, and appropriate review. A/B testing should not knowingly expose customers to unsafe or materially harmful experiences.

Fairness analysis should examine whether sampling, measurement, treatment delivery, or model performance differs across relevant groups. Removing a protected attribute does not remove bias if correlated variables remain. Privacy-preserving methods such as aggregation, access controls, and carefully designed experimentation can reduce risk, but no method eliminates all governance obligations.

AI can accelerate questionnaire coding, segmentation, text analysis, and experimentation, but generated interpretations require validation. Automated systems may hallucinate findings, reproduce historical bias, or optimize a proxy that conflicts with customer welfare. Reproducible workflows should preserve data definitions, code, versions, exclusions, pre-analysis plans, and decision logs.

9. Best Practices for Professional Analysts

1. Define the decision and estimand before selecting a method.
2. Separate exploratory analysis from confirmatory testing.
3. Use sampling designs that match the target population and disclose coverage limitations.
4. Predefine primary outcomes, treatment rules, and stopping criteria.
5. Prefer randomized holdouts when feasible for causal marketing questions.
6. Check balance, compliance, contamination, missingness, and attrition.
7. Report effect sizes and confidence intervals, not p-values alone.
8. Distinguish absolute lift, relative lift, revenue, margin, and lifetime value.
9. Test assumptions with diagnostics and sensitivity analyses.
10. Document limitations and avoid claims stronger than the design supports.
11. Protect customers through privacy, fairness, security, and responsible experimentation.
12. Replicate important findings across time, markets, and customer contexts.

Chapter Summary

Marketing research begins with a managerial decision and a precise estimand. Exploratory, descriptive, correlational, experimental, and quasi-experimental designs answer different questions. Sampling quality depends not only on sample size but also on coverage, nonresponse, and selection processes. Surveys require reliable and valid measures, while descriptive statistics and confidence intervals quantify patterns and uncertainty.

Correlation and regression describe relationships or support prediction; they do not automatically establish causation. Causal inference requires a credible counterfactual, most directly supplied by random assignment. A/B tests should use appropriate randomization units, pre-specified outcomes, intent-to-treat analysis, and disciplined monitoring. When randomization is unavailable, difference-in-differences, regression discontinuity, instrumental variables, and geo experiments can help, but each depends on assumptions.

Incrementality analysis estimates the outcomes caused by marketing rather than merely attributed to it. Effective analysts combine statistical evidence with business economics, ethical governance, reproducibility, and critical evaluation of assumptions.

Key Terms

Estimand: The precise quantity a study aims to learn.

Target population: The full group to which a study intends to generalize.

Sampling frame: The operational list or source from which a sample is selected.

Reliability: The consistency of a measurement.

Validity: The extent to which a measure or design supports the intended interpretation.

Confidence interval: A range produced by a statistical procedure to express sampling uncertainty.

P-value: The probability, under a null model, of data at least as extreme as those observed.

Type I error: Rejecting a true null hypothesis.

Statistical power: The probability of detecting a specified effect under the alternative.

Correlation: A measure of linear association between two variables.

Regression: A model relating an outcome to one or more predictors.

Confounding: Distortion of an exposure-outcome relationship by a common cause.

Potential outcome: The outcome a unit would have under a specified treatment condition.

Average treatment effect: The population average difference between treated and untreated potential outcomes.

A/B test: A randomized comparison of two versions or conditions.

Intent-to-treat: Analysis according to assigned treatment, regardless of compliance.

Difference-in-differences: A quasi-experimental estimator comparing changes across treated and comparison groups.

Incrementality: The outcome caused by an intervention beyond what would otherwise have occurred.

Attribution: Allocation of observed outcomes among marketing touchpoints.

External validity: The extent to which findings generalize to other settings or populations.

Review Questions

Short Answer Questions

1. Why is an estimand important when formulating a marketing research question?
2. Distinguish reliability from validity in survey measurement.
3. What does a p-value measure, and what does it not measure?
4. Why does correlation not establish causation?
5. State the key parallel-trends assumption in difference-in-differences analysis.

Long Answer Questions

1. Compare exploratory, descriptive, correlational, experimental, and quasi-experimental research designs, including an appropriate marketing use for each.
2. Explain how coverage error, nonresponse bias, and sample size affect survey conclusions.
3. Describe the potential-outcomes framework and explain why random assignment supports causal inference.
4. Discuss the design requirements and common threats to validity in an A/B test.
5. Explain why incrementality may differ substantially from platform-reported attribution and how a holdout study can estimate it.

Analytical / Application Questions

1. An email campaign produces a 7% conversion rate among recipients and 5% among nonrecipients. Identify at least two reasons this difference may not be causal and propose a stronger design.
2. A treated region’s sales rise from $800,000 to $920,000, while a comparison region’s sales rise from $600,000 to $660,000. Calculate the difference-in-differences estimate and explain the assumption required for interpretation.
3. A randomized test assigns 20,000 customers to treatment and 20,000 to control. Treatment conversion is 4.8% and control conversion is 4.2%. Explain the difference between absolute lift and relative lift, and identify additional information needed before recommending rollout.

Multiple Choice Questions with Answers

Question 1

Which quantity is the primary target of an incrementality study?

A. Total observed sales among exposed customers

B. The outcome caused by the intervention relative to a credible counterfactual

C. The number of advertisements delivered

D. The percentage of customers who opened an advertisement

Correct Answer: B. The outcome caused by the intervention relative to a credible counterfactual

Question 2

A five-point single-item satisfaction response is most directly treated as what type of measurement?

A. Nominal

B. Ordinal

C. Ratio

D. Binary

Correct Answer: B. Ordinal

Question 3

What does a p-value represent?

A. The probability that the null hypothesis is true

B. The probability that the alternative hypothesis is true

C. The probability, under the null model, of data at least as extreme as observed

D. The probability that the sample is representative

Correct Answer: C. The probability, under the null model, of data at least as extreme as observed

Question 4

If treatment conversion is 5.5% and control conversion is 5.0%, what is the relative lift?

A. 0.5%

B. 5%

C. 10%

D. 50%

Correct Answer: C. 10%

Question 5

Which condition is essential for interpreting a standard difference-in-differences estimate causally?

A. Treatment and comparison groups have identical sample sizes

B. The groups would have followed parallel trends without treatment

C. The outcome must be binary

D. All customers must receive treatment

Correct Answer: B. The groups would have followed parallel trends without treatment

Question 6

Why does random assignment support causal inference in an A/B test?

A. It guarantees a large effect

B. It makes treatment status independent of potential outcomes in expectation

C. It eliminates all measurement error

D. It ensures every participant complies

Correct Answer: B. It makes treatment status independent of potential outcomes in expectation

Question 7

Which statement best distinguishes prediction from causal inference?

A. Prediction never uses historical data

B. Causal inference asks what would happen under an intervention

C. Prediction always requires randomization

D. Causal inference ignores uncertainty

Correct Answer: B. Causal inference asks what would happen under an intervention

Question 8

A sample excludes customers without email addresses even though the target population includes them. What problem is most directly present?

A. Coverage error

B. Type I error

C. Multicollinearity

D. Regression to the mean

Correct Answer: A. Coverage error

Question 9

If incremental revenue is $120,000 and incremental media spend is $30,000, what is revenue-based iROAS?

A. 0.25

B. 2.0

C. 4.0

D. 90,000

Correct Answer: C. 4.0

Question 10

Which analysis preserves the causal benefit of original random assignment when some assigned customers do not comply?

A. Intent-to-treat analysis

B. Per-protocol analysis only

C. Analysis of only exposed customers

D. Analysis of only purchasers

Correct Answer: A. Intent-to-treat analysis