Research Design and Methodological Planning

Introduction

Research design is the logical architecture of a study. It connects a research problem to evidence, data-collection procedures, analysis, and defensible conclusions. In business analytics, design decisions determine whether an observed pattern reflects a genuine business relationship, a measurement problem, a selection effect, or simple chance.

A well-designed study answers four linked questions:

1. What is the research question?
2. What evidence would answer it?
3. How will the evidence be collected and analysed?
4. What claims can legitimately be made from the evidence?

Research design is broader than selecting a statistical technique. A regression model cannot repair a biased sample, an invalid measure, or a design that cannot establish temporal order. Conversely, an experiment is not automatically strong if treatment assignment is poorly implemented or outcomes are measured unreliably. Methodological planning therefore requires alignment among the research question, concepts, design, sample, measures, procedures, analysis, and ethics.

Learning Objectives

By the end of this chapter, students should be able to:

• distinguish major research designs used in business analytics;
• select a design appropriate to descriptive, relational, predictive, and causal questions;
• align research questions, variables, sampling, measurement, and analysis;
• explain internal, external, construct, and statistical conclusion validity;
• identify common sources of bias and threats to credible inference;
• design basic cross-sectional, longitudinal, case-study, survey, observational, experimental, and quasi-experimental studies;
• evaluate ethical, operational, and governance issues in data-driven research.

1. From Research Question to Design

A research question should determine the design, rather than the availability of a convenient dataset. Four broad question types are particularly useful.

A descriptive question may require a representative survey. A causal question requires more demanding evidence: the presumed cause must precede the outcome, alternative explanations must be addressed, and treatment and comparison conditions must be made comparable.

A useful planning chain is:

Research problem → research question → conceptual framework → operational definitions → design → sampling → data collection → analysis → inference.

Concepts, Variables, and Operationalisation

A concept is an abstract idea, such as customer loyalty, employee engagement, or service quality. A variable is an observable representation that can take different values. Operationalisation specifies exactly how a concept will be measured.

For example, “customer loyalty” might be operationalised as repeat purchase within 90 days, annual purchase frequency, or a validated intention-to-repurchase scale. These are not interchangeable. Each captures a different aspect of the concept and creates different validity implications.

Variables may be classified as:

• Independent variable: a presumed predictor, exposure, or treatment.
• Dependent variable: the outcome to be explained or estimated.
• Confounder: a variable associated with both the exposure and outcome that can create a misleading association.
• Mediator: a mechanism through which an exposure may influence an outcome.
• Moderator: a variable that changes the strength or direction of a relationship.
• Control variable: a variable included to improve comparability or precision, provided its inclusion is theoretically justified.

A conceptual framework should distinguish these roles before data analysis begins. Including every available variable is not a substitute for reasoning; controlling for a mediator or a consequence of treatment can produce misleading estimates.

2. Major Research Designs

2.1 Cross-Sectional Design

A cross-sectional study measures variables at one point in time or during a short field period. It is efficient for estimating prevalence and examining associations.

For example, a retailer might survey 1,000 customers in June to estimate satisfaction and examine whether satisfaction is associated with mobile-app usage. The design is economical, but it usually cannot establish whether app usage increased satisfaction or satisfied customers were more likely to use the app. Temporal order is unclear, and common-method bias may arise when the same respondent reports both variables at the same time.

Cross-sectional designs are appropriate when the objective is to describe a current population or generate hypotheses. They are weaker for claims about change and causation.

2.2 Longitudinal Design

A longitudinal study collects data from the same units or a defined population over multiple time points. It can examine change, persistence, and temporal order.

Common forms include:

• Panel study: repeatedly measures the same individuals, firms, or accounts.
• Cohort study: follows a group sharing a defining characteristic, such as employees hired in the same year.
• Repeated cross-sectional study: surveys different samples from the same population at several times.
• Time-series study: records observations on one unit at regular intervals, such as daily sales.

Longitudinal designs help distinguish a temporary fluctuation from a trend. They also introduce attrition, panel conditioning, changing measurement systems, and history effects. If participants who leave differ systematically from those who remain, estimates may become biased.

A simple before-and-after comparison is not automatically causal. Sales may change after a campaign because of seasonality, competitor actions, economic conditions, or regression to the mean. A comparison group or stronger quasi-experimental strategy may be needed.

2.3 Case-Study Design

A case study is an intensive investigation of a bounded case—such as a firm, project, market, team, implementation, or event—using multiple sources of evidence. It is especially useful for questions involving “how,” “why,” processes, context, and organizational complexity.

Case studies may be:

• Single-case or multiple-case;
• Holistic or embedded, depending on whether the case is analysed as one unit or through subunits;
• Exploratory, descriptive, or explanatory.

Rigour requires a clear case boundary, a defined protocol, systematic evidence collection, a chain of evidence, and attention to rival explanations. Interviews, documents, system logs, observations, and performance data can be triangulated. A case study is not merely an anecdote, and one case should not be presented as statistically representative of all firms. Its value may instead lie in theoretical insight, process explanation, or transferability to sufficiently similar contexts.

2.4 Survey Design

A survey collects standardised information from respondents through questionnaires, interviews, or digital instruments. It is useful for attitudes, perceptions, self-reported behaviour, and characteristics not available in administrative data.

Important survey planning decisions include:

• defining the target population and sampling frame;
• choosing probability or non-probability sampling;
• writing clear, neutral, single-focus questions;
• selecting response scales;
• piloting the instrument;
• planning invitations, reminders, and non-response assessment;
• protecting confidentiality and data quality.

Probability sampling gives each population unit a known, non-zero selection probability. Simple random sampling selects units directly; stratified sampling samples within important subgroups; cluster sampling selects groups and then units within groups. Convenience and voluntary-response samples may be practical but often have unknown selection probabilities and limited generalisability.

Survey errors include coverage error, sampling error, non-response error, recall error, social desirability bias, acquiescence, ambiguous wording, and mode effects. A large sample does not eliminate systematic error.

For a proportion estimate under simple random sampling, the approximate standard error is:

Here, p̂ is the sample proportion and n is the sample size. This approximation assumes a probability sample, independent observations, and a sufficiently large sample. It describes sampling variability, not non-response or measurement bias. For a finite population of size N, a finite-population correction may be relevant when sampling without replacement and the sampling fraction is substantial:

2.5 Observational Design

In observational research, the researcher records exposures and outcomes without assigning the exposure. Examples include analysing customer churn from transaction records or examining whether flexible work is associated with employee retention.

Observational data can be rich, timely, and ethically preferable when manipulation is impossible or undesirable. Its central limitation is confounding. Customers who adopt a premium service may differ from non-adopters in income, loyalty, or prior usage. An observed outcome difference may reflect these pre-existing differences rather than the service itself.

Directed acyclic graphs, subject-matter knowledge, matching, stratification, regression adjustment, and sensitivity analysis can help address confounding, but no statistical adjustment guarantees causal identification. The quality of adjustment depends on measurement and assumptions.

2.6 Experimental Design

An experiment assigns units to treatment conditions and observes outcomes under controlled procedures. Random assignment makes treatment groups comparable in expectation, so differences in average outcomes can be interpreted causally when implementation and measurement are sound.

A basic parallel-group experiment contains:

• an experimental unit, such as a customer or store;
• a treatment condition;
• a control or comparison condition;
• random assignment;
• a specified outcome and follow-up period.

If Y_i(1) denotes unit i’s outcome under treatment and Y_i(0) its outcome under control, the individual treatment effect is:

Only one potential outcome is observed for each unit. The average treatment effect is:

The notation E denotes a population average. In a randomised study, the difference between sample mean outcomes in treatment and control groups estimates the ATE under assumptions such as valid randomisation, adequate implementation, and a well-defined treatment. The units of the ATE are the units of the outcome—for example, percentage points, dollars, or minutes.

Experiments may be between-subjects, within-subjects, factorial, field-based, or laboratory-based. Blocking or stratification before randomisation can improve balance on important variables. Blinding, standardised procedures, preregistered outcomes, and intention-to-treat analysis reduce bias.

Ethical constraints matter. Researchers should not expose participants to unreasonable risks, deceive them without justification, or withhold essential services merely to create a control group.

2.7 Quasi-Experimental Design

A quasi-experiment estimates causal effects without full random assignment. It is valuable when randomisation is impractical, politically unacceptable, or unethical.

Important designs include:

Difference-in-differences. A treated group and comparison group are observed before and after an intervention. The estimator is:

Here, Ȳ represents a group mean; subscripts T and C indicate treated and comparison groups, while pre and post indicate time. The key identifying assumption is parallel trends: without treatment, the treated and comparison groups would have followed similar average trends. This assumption cannot be proven from post-treatment data, but pre-treatment trends and substantive reasoning can be examined.

Regression discontinuity. Treatment is assigned according to whether a continuous running variable crosses a known cutoff, such as a credit score threshold. Units just above and below the cutoff may be comparable, allowing a local causal estimate. The design requires no precise manipulation around the threshold and careful modelling near the cutoff.

Interrupted time series. A long sequence of outcomes is analysed before and after an intervention. A change in level or slope may indicate an effect, but concurrent events, seasonal patterns, and changes in measurement must be considered.

Matching and weighting. Treated units are paired or weighted to resemble comparison units on measured pre-treatment characteristics. These approaches address measured differences but not unmeasured confounding.

Quasi-experiments require explicit assumptions. Analysts should report diagnostics, alternative specifications, placebo tests where appropriate, and sensitivity to the chosen comparison period or bandwidth.

3. Aligning Design with the Question

Methodological alignment can be assessed through a design matrix.

The unit of analysis must match the research question. Analysing store-level averages and then claiming individual-level conclusions creates an ecological fallacy. Conversely, treating thousands of transactions from a few stores as independent observations can produce pseudoreplication and underestimated uncertainty.

A practical planning sequence is:

1. State the estimand or claim: prevalence, association, prediction accuracy, or causal effect.
2. Define the population, unit, setting, and time period.
3. Specify variables and operational measures before observing results.
4. Select the design that supplies the required comparison and temporal structure.
5. Develop the sampling and recruitment plan.
6. Pilot data collection and test procedures.
7. Predefine primary outcomes, exclusions, and analysis decisions where feasible.
8. Conduct a power or precision assessment for quantitative studies.
9. Document data governance, ethics, and quality checks.
10. Report limitations and the scope of inference.

4. Validity and Threats to Credible Inference

Validity concerns whether a study supports the interpretation claimed. Four forms are particularly important.

Internal Validity

Internal validity concerns whether the observed difference or relationship is attributable to the proposed explanation within the study.

Threats include:

• History: an external event affects outcomes during the study;
• Maturation: units change naturally over time;
• Selection: groups differ before treatment;
• Regression to the mean: extreme observations tend to be followed by less extreme ones;
• Attrition: dropout differs across conditions;
• Contamination: control units receive part of the treatment;
• Instrumentation: measurement procedures change;
• Confounding: a third variable explains the association.

Randomisation, allocation concealment, comparison groups, repeated measures, consistent protocols, retention efforts, and sensitivity analyses can reduce these threats.

External Validity

External validity concerns generalisation to other people, settings, times, measures, and treatments. A controlled experiment may have strong internal validity but limited realism. A probability survey may generalise to its target population but not to other countries or future periods.

Generalisation requires attention to sampling, response rates, context, treatment variation, and replication. Statistical generalisation is strongest when the sample is probabilistically drawn from a clearly defined population. Analytic or theoretical generalisation relies on reasoning about whether mechanisms or conditions transfer to another setting.

Construct Validity

Construct validity asks whether the operational measure represents the intended concept. A sales count may measure purchase behaviour but not customer loyalty. A single satisfaction item may omit reliability, responsiveness, and perceived value.

Threats include poorly worded items, mono-method measurement, demand characteristics, proxy mismatch, and changes in definitions. Multiple indicators, validated scales, cognitive interviewing, pilot testing, and convergent or discriminant evidence can strengthen construct validity.

Statistical Conclusion Validity

Statistical conclusion validity concerns whether the analysis supports claims about relationships or differences. Threats include low statistical power, unreliable measures, non-independence, model misspecification, outliers, selective reporting, multiple testing, and violations of assumptions.

Statistical significance does not establish practical importance or causality. A small effect can be precisely estimated in a large sample, while a meaningful effect can be uncertain in a small sample. Analysts should report effect sizes, uncertainty intervals, assumptions, and sensitivity analyses rather than relying only on a threshold such as p < 0.05.

Bias, Precision, and Error

Bias is systematic deviation from the target quantity; random error varies unpredictably across samples. Increasing sample size generally reduces random sampling error but does not remove systematic bias. A precisely estimated biased quantity remains wrong.

A useful diagnostic asks:

• Who was included and excluded?
• Who provided data, and who did not?
• Were treatment and comparison conditions comparable?
• Were measures valid and consistent?
• Could the outcome influence later exposure or participation?
• Were analysis decisions made after inspecting results?

5. A Detailed Applied Case

Illustrative case: Evaluating a personalised recommendation system

Background. An online retailer introduces a machine-learning recommendation panel intended to increase completed purchases. Managers want to know whether the panel causes higher conversion, not merely whether users who see it purchase more.

Problem. The system initially displays recommendations to users predicted to be highly engaged. A simple comparison between exposed and unexposed customers would be confounded because high-engagement customers are more likely to receive the panel.

Approach. The analytics team designs a two-week randomised field experiment. Eligible sessions are randomly assigned to a recommendation condition or a visually identical control condition without recommendations. The primary outcome is purchase within the session. Secondary outcomes are order value and return rate within 30 days. Assignment is recorded before the outcome occurs, exposure logs are checked, and the analysis follows intention to treat.

Suppose the treatment group contains 20,000 sessions, of which 2,000 result in purchase. The control group contains 20,000 sessions, of which 1,800 result in purchase. Conversion rates are:

The estimated absolute effect is:

Thus, the estimated increase is 0.01, or 1 percentage point. The relative conversion ratio is:

So conversion is approximately 11% higher relative to control. The absolute effect is often more useful for forecasting incremental purchases. With 100,000 eligible sessions under similar conditions, a 1-percentage-point effect would correspond to approximately 1,000 additional purchases, subject to uncertainty and stable implementation.

Validity assessment. Random assignment supports internal validity, but several issues remain. Users may encounter recommendations on another device, creating contamination. Tracking failures may make treatment assignment differ from actual exposure. A short study may not reveal effects on returns or long-term loyalty. If the experiment occurs during a holiday promotion, external validity may be limited. The team should report confidence intervals, check balance and logging integrity, analyse predefined subgroups cautiously, and replicate across periods and customer segments.

Outcome. Management decides to deploy the panel only after confirming that the positive short-term effect does not increase returns or reduce margin. The result supports a causal claim for eligible sessions under the tested conditions—not a universal claim that recommendations improve every customer outcome.

Learning points. The case demonstrates the difference between association and causation, the importance of a counterfactual, the value of a primary outcome, and the need to examine operational, ethical, and commercial consequences beyond a single metric.

6. Ethics, Governance, and Contemporary Practice

Research involving people requires respect for autonomy, informed consent where applicable, minimisation of harm, fair selection, and appropriate confidentiality. Business analytics also raises issues involving profiling, employee monitoring, automated decisions, and secondary use of customer data.

Researchers should collect only data needed for the stated purpose, restrict access, protect identifiers, document retention periods, and assess whether data may be re-identified. Consent is not always the only lawful basis for organisational data use, but legal compliance does not by itself establish ethical acceptability. Applicable privacy and data-protection requirements must be reviewed with institutional or organisational governance teams.

AI and automated analytics create additional methodological risks. Training data may encode historical discrimination; labels may be inconsistent; model performance may differ across groups; and data drift may weaken external validity. Researchers should document data provenance, version datasets and code, monitor subgroup performance, distinguish prediction from causal explanation, and avoid presenting a high-performing model as evidence that its inputs cause the outcome.

Open research practices strengthen credibility. Preregistration, analysis plans, reproducible code, audit trails, transparent exclusions, and reporting of null or adverse findings reduce questionable research practices. These practices do not eliminate judgement, but they make judgement visible and reviewable.

7. Best-Practice Checklist

Before collecting data, confirm that:

• the research question specifies the intended type of inference;
• the population and unit of analysis are explicit;
• concepts have defensible operational definitions;
• the design provides a credible comparison or temporal structure;
• sampling and recruitment procedures are documented;
• primary outcomes and decision rules are specified;
• measurement quality and pilot results are reviewed;
• likely validity threats have mitigation plans;
• the sample size supports the desired precision or power;
• privacy, consent, security, and retention requirements are addressed;
• the analysis matches the design and dependence structure;
• conclusions will be limited to the population and conditions justified by the evidence.

Chapter Summary

Research design is the architecture linking questions to evidence and inference. Cross-sectional studies describe current conditions efficiently, whereas longitudinal designs reveal change and temporal ordering. Case studies explain context and process; surveys measure standardised self-reports; observational studies examine naturally occurring exposures; experiments use random assignment to support causal inference; and quasi-experiments use structured comparisons when randomisation is unavailable.

Good design requires alignment among the research question, population, unit of analysis, measures, sampling, comparison, time, and analysis. Internal validity concerns alternative explanations within the study; external validity concerns generalisation; construct validity concerns the adequacy of measures; and statistical conclusion validity concerns whether the analysis supports the stated claim. Bias cannot generally be eliminated through a larger sample alone.

Business analytics researchers must also address ethics, privacy, data governance, algorithmic fairness, reproducibility, and the distinction between prediction and causation. The strongest conclusions are appropriately ambitious: they state what the design supports and clearly identify what it cannot establish.

Key Terms

Research design: The overall plan connecting a research question to data collection, analysis, and inference.

Cross-sectional design: A design measuring variables at one point or short period.

Longitudinal design: A design collecting observations across multiple time points.

Panel study: A longitudinal study repeatedly measuring the same units.

Case study: An intensive investigation of a bounded case using multiple evidence sources.

Survey: A standardised method for collecting information from respondents.

Observational study: A study in which exposures are recorded rather than assigned.

Experiment: A study in which researchers assign units to treatment conditions.

Quasi-experiment: A causal design without complete random assignment.

Confounder: A variable related to both an exposure and an outcome that can distort their association.

Mediator: A variable representing a mechanism through which an exposure affects an outcome.

Moderator: A variable that changes the strength or direction of a relationship.

Internal validity: The credibility of a study’s explanation within its observed setting.

External validity: The extent to which findings generalise to other settings or populations.

Construct validity: The extent to which measures represent the intended concepts.

Statistical conclusion validity: The credibility of inferences drawn from statistical analysis.

Counterfactual: The outcome that would have occurred for a unit under an alternative condition.

Average treatment effect: The average difference between potential outcomes under treatment and control.

Difference-in-differences: A comparison of changes over time between treated and comparison groups.

Operationalisation: The process of defining how an abstract concept will be measured.

Review Questions

Short Answer Questions

1. What is the principal difference between a cross-sectional and a longitudinal design?
2. Why does random assignment strengthen causal inference?
3. Define confounding and give one business example.
4. What is construct validity?
5. State the key assumption underlying a basic difference-in-differences analysis.

Long Answer Questions

1. Compare survey, observational, and experimental designs in terms of purpose, strengths, limitations, and suitable research questions.
2. Explain the four forms of validity discussed in this chapter and describe one threat to each.
3. Discuss how a researcher could plan a longitudinal study of employee retention, including sampling, measurement, attrition, and analysis.
4. Explain why a large sample cannot by itself eliminate research bias.
5. Describe the methodological and ethical issues involved in evaluating an AI-based recommendation system.

Analytical / Application Questions

1. A company observes that customers receiving a loyalty discount spend more than customers who do not receive one. Design an observational or experimental study that would more credibly estimate the discount’s causal effect.
2. A city introduces a public transport app in one district but not another. Explain how difference-in-differences could be used, and identify the evidence needed to assess the parallel-trends assumption.
3. A survey reports high employee engagement using one voluntary online question. Evaluate the likely threats to sampling, construct, internal, and statistical conclusion validity, and propose improvements.

Multiple Choice Questions with Answers

Question 1

Which research design is most appropriate for estimating current customer satisfaction across a defined population?

A. Cross-sectional survey

B. Laboratory experiment

C. Interrupted time series

D. Single-case ethnography

Correct Answer: A. Cross-sectional survey

Question 2

What is the main purpose of random assignment in an experiment?

A. To guarantee a large sample

B. To make treatment groups comparable in expectation

C. To eliminate measurement error

D. To ensure findings generalise to every population

Correct Answer: B. To make treatment groups comparable in expectation

Question 3

Which threat occurs when participants who leave a longitudinal study differ systematically from those who remain?

A. History

B. Contamination

C. Attrition

D. Instrumentation

Correct Answer: C. Attrition

Question 4

A researcher studies whether app use and satisfaction are associated in a one-time customer survey. Which conclusion is least justified?

A. The variables are associated in the sample

B. The association can generate a hypothesis

C. The results describe the surveyed population, subject to sampling quality

D. App use caused satisfaction to increase

Correct Answer: D. App use caused satisfaction to increase

Question 5

In the expression τ_i = Y_i(1) − Y_i(0), what does τ_i represent?

A. The individual treatment effect for unit i

B. The sample size

C. The average response rate

D. The measurement error for unit i

Correct Answer: A. The individual treatment effect for unit i

Question 6

A treatment group’s conversion rate is 10% and a control group’s rate is 9%. What is the absolute difference in conversion rates?

A. 0.1 percentage point

B. 1 percentage point

C. 9 percentage points

D. 10 percentage points

Correct Answer: B. 1 percentage point

Question 7

What is the key identifying assumption in a basic difference-in-differences design?

A. The treatment effect is zero before treatment

B. The groups have identical sample sizes

C. The groups would have followed parallel trends without treatment

D. All outcomes are normally distributed

Correct Answer: C. The groups would have followed parallel trends without treatment

Question 8

Which validity type asks whether a measure actually represents the intended concept?

A. External validity

B. Internal validity

C. Statistical conclusion validity

D. Construct validity

Correct Answer: D. Construct validity

Question 9

Why can a very large sample still produce a misleading estimate?

A. Large samples always create regression to the mean

B. Systematic bias is not necessarily removed by increasing sample size

C. Large samples cannot contain variables

D. Sampling error increases automatically with sample size

Correct Answer: B. Systematic bias is not necessarily removed by increasing sample size

Question 10

Which design is especially suited to examining whether an intervention changes an outcome at a known cutoff on a running variable?

A. Regression discontinuity

B. Convenience survey

C. Cross-sectional case study

D. Simple randomised block design

Correct Answer: A. Regression discontinuity