Research Design and Quality

Introduction

Research design is the logical architecture of an investigation. It connects a research question to the evidence required to answer it and specifies how that evidence will be collected, analysed and interpreted. In business research, design choices influence whether managers can reasonably conclude that a marketing campaign caused higher sales, that a leadership programme improved employee retention, or that a governance reform reduced compliance failures.

A design is therefore more than a timetable or data-collection technique. A questionnaire, interview or regression model is a method; research design explains how that method fits the research question, comparison structure, timing, sampling strategy and causal logic. High-quality design seeks credible inference: conclusions that are appropriately supported by the evidence and not merely plausible narratives.

For MBA researchers, design quality has two dimensions. First, the design must be technically credible. Second, it must be feasible, ethical and useful for managerial decision-making. This chapter examines major research designs, criteria for selecting among them, and threats to internal validity, external validity, construct validity and unbiased inference.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define research design and distinguish major quantitative, qualitative and mixed-method designs.
2. Select a design using criteria such as purpose, causal ambition, time, feasibility and ethics.
3. Evaluate internal, external and construct validity.
4. Identify selection bias, measurement bias, researcher bias, confounding and other threats to credible inference.
5. Apply design-quality principles to a managerial research problem.

1. The Logic of Research Design

A research design answers five connected questions:

• What is the research question?
• What phenomenon, population or process is being studied?
• What evidence would count as a convincing answer?
• How will alternative explanations be controlled or assessed?
• To whom, when and under what conditions may the findings apply?

A useful distinction is between description, explanation, prediction and evaluation. Descriptive research estimates what exists, such as employee engagement levels. Explanatory research investigates why an outcome occurs, such as whether autonomy improves engagement. Predictive research focuses on forecasting future outcomes, while evaluation research assesses the effects or value of a programme or policy.

Research design also reflects the researcher’s assumptions about knowledge. Quantitative designs commonly emphasise measurement, comparison and statistical estimation. Qualitative designs emphasise meaning, context, interpretation and process. Mixed-method designs combine these approaches when numerical patterns and contextual explanations are both necessary.

Variables, units and comparisons

The unit of analysis is the entity about which conclusions are drawn: an employee, customer, firm, business unit, transaction or country. A common error is the ecological fallacy—drawing conclusions about individuals from relationships observed among firms or countries. The opposite error, sometimes called the atomistic fallacy, occurs when individual-level evidence is used to make claims about organisations without justification.

An independent variable is a presumed cause or explanatory factor; a dependent variable is the outcome. A mediator explains how an effect occurs, whereas a moderator changes the strength or direction of a relationship. For example, training may improve performance through skill development, while the effect may be stronger for inexperienced employees than for experienced employees.

Causal reasoning requires more than observing that two variables are associated. A causal claim generally requires temporal precedence, covariation and credible exclusion of alternative explanations. If firms adopting flexible work also report higher productivity, the association may reflect flexibility, but it may alternatively be caused by technology investment, managerial quality or the prior productivity of adopting firms.

2. Major Types of Research Design

Exploratory design

Exploratory research is appropriate when a problem is poorly understood, concepts are unclear or relevant variables have not yet been identified. Interviews, focus groups, observation, document analysis and exploratory case studies are common methods. Its purpose is usually conceptual development rather than population estimation or definitive causal testing.

For example, before designing a customer-satisfaction survey, a firm may interview customers who recently cancelled subscriptions. The interviews may reveal that “ease of use” includes onboarding, search speed and integration with other software. Exploration improves later measurement but should not be presented as representative evidence unless an appropriate sampling design supports that claim.

Descriptive design

Descriptive research systematically estimates characteristics, frequencies or distributions. Cross-sectional surveys, organisational dashboards and structured observation are typical examples. A descriptive design can answer questions such as: What proportion of employees intend to leave? Which customer segments report the highest service complaints?

Descriptive accuracy depends on sampling, question wording, nonresponse management and measurement quality. A large sample does not compensate for systematic undercoverage—for example, surveying only customers who use a mobile application when the research concerns all customers.

Explanatory and correlational design

Correlational designs examine relationships among naturally occurring variables. They are useful when manipulation is impossible or inappropriate, such as studying the relationship between executive experience and governance performance. However, correlation alone does not establish causation because of confounding, reverse causality and selection effects.

Longitudinal correlational designs improve temporal analysis by observing units repeatedly. Panel data can reveal whether changes in one variable precede changes in another, although temporal order by itself does not eliminate confounding.

Experimental design

An experiment deliberately varies a treatment and compares outcomes with a control condition. In a randomised controlled trial, eligible units are randomly assigned to treatment or control groups. Randomisation tends, in expectation, to balance both observed and unobserved characteristics across groups, strengthening causal inference.

For a simple two-group design, the estimated average treatment effect is:

Here, Y(1) is the potential outcome if a unit receives treatment, Y(0) is the potential outcome if it does not, and E denotes the population mean. The fundamental problem of causal inference is that each unit is observed in only one treatment state. Randomisation allows the difference between group means to estimate the average effect, subject to assumptions such as consistent treatment definitions, valid assignment and adequate outcome measurement.

A true experiment includes manipulation, comparison and random assignment. A quasi-experiment includes a treatment or policy change but lacks full random assignment. Examples include difference-in-differences designs, regression discontinuity and interrupted time-series designs. These can be powerful, but their assumptions must be examined rather than treated as automatically causal.

In a difference-in-differences design, the estimated effect is commonly expressed as:

The symbols represent the treated group T and comparison group C before and after an intervention. The central parallel-trends assumption is that, without treatment, both groups would have experienced similar average changes. If the groups were already following different trends, the estimate may be biased.

Cross-sectional and longitudinal designs

A cross-sectional design collects evidence at one point or during one short period. It is efficient for estimating prevalence and comparing groups, but it is weak for establishing temporal order. A longitudinal design collects data at multiple times. It supports analysis of change, stability, event sequences and lagged relationships, but costs more and faces attrition—the loss of participants over time.

Longitudinal designs may be panel studies, cohort studies or repeated cross-sectional studies. A panel follows the same units; a cohort follows units sharing a defining experience; repeated cross-sectional studies sample different units from the same population at different times.

Case study design

A case study is an intensive investigation of a bounded case, such as a firm, project, merger or supply chain. It is particularly valuable when context and process are central. Single-case studies can provide depth and theory development; multiple-case studies permit systematic comparison and replication of patterns.

Case evidence is not automatically weak, but its claims must be proportionate to the design. A case may support analytical generalisation—refining or illustrating theory—without supporting statistical generalisation to every organisation.

Qualitative designs

Qualitative designs include phenomenology, grounded theory, ethnography, narrative inquiry and qualitative case study. Their purposes differ. Phenomenology investigates lived experience; grounded theory develops concepts from systematically analysed data; ethnography examines culture and practice through extended engagement; narrative inquiry studies how people construct meaning through stories.

Quality in qualitative research involves credibility, dependability, confirmability and transferability. Useful practices include purposive sampling, triangulation, reflexive analysis, negative-case analysis, transparent coding and member reflection. These practices do not make qualitative work “quantitative”; they make interpretive claims more accountable.

Mixed-method design

Mixed-method research integrates quantitative and qualitative evidence in one coherent study. In a sequential exploratory design, qualitative findings inform a later quantitative instrument. In a sequential explanatory design, quantitative results are followed by qualitative work to explain unexpected patterns. In a convergent design, both forms of evidence are collected and compared during interpretation.

Integration must occur at the level of research questions, sampling, data collection or interpretation. Simply placing interviews beside survey results is not sufficient. Mixed methods are especially useful when decision-makers need both an estimate of scale and an explanation of mechanisms.

3. Criteria for Selecting a Research Design

Design selection should begin with the question, not with a preferred method. The following criteria are particularly important.

A strong design is not necessarily the most elaborate design. Researchers should match the strength of the claim to the strength of the evidence. If random assignment is impossible, a carefully justified quasi-experiment may be more credible than a poorly implemented experiment. If the purpose is to understand employee identity after a merger, a rich qualitative design may be superior to a superficial questionnaire.

4. Validity and Quality of Inference

Validity concerns whether a study supports the interpretation the researcher gives to its findings. Reliability—consistency of measurement—is important but not sufficient. A measure can be highly reliable and systematically wrong. For example, a scale that consistently omits workload from “job quality” may produce stable but incomplete results.

Internal validity

Internal validity is the credibility of a causal conclusion within the study. It asks whether the observed outcome difference was caused by the intervention or by another factor.

Important threats include:

• History: an external event occurs between pretest and post-test, such as a recession during a training evaluation.
• Maturation: participants change naturally over time, through learning, fatigue or organisational tenure.
• Testing effects: completing a pretest changes later responses.
• Instrumentation: the measurement instrument or procedure changes over time.
• Regression to the mean: unusually high or low observations tend to be followed by less extreme observations.
• Attrition: dropout differs between groups or is related to outcomes.
• Contamination: control participants receive part of the treatment.
• Selection bias: treatment and comparison groups differ before treatment.
• Confounding: a third variable influences both the presumed cause and outcome.
• Spillover: treatment affects people outside the treatment group.

Randomisation, allocation concealment, baseline comparison, blinding where possible, consistent measurement, preregistered outcomes, retention strategies and intention-to-treat analysis can strengthen internal validity. Statistical adjustment can help with measured differences, but it cannot reliably control for important unmeasured confounders.

External validity

External validity concerns the extent to which findings apply beyond the study’s participants, setting, treatment and time. It includes generalisability to other people, organisations, contexts and periods.

A study can have high internal validity but limited external validity. For example, a tightly controlled experiment with volunteer MBA students may identify a causal psychological mechanism but provide uncertain evidence about factory workers or senior executives. Conversely, a broad survey may generalise well descriptively but offer weak causal evidence.

External validity is supported by representative sampling, replication across settings, realistic treatment conditions, analysis of effect moderators and transparent description of context. Generalisation is not a property of sample size alone. Researchers should specify the target population, accessible population and study sample, then justify the bridge from sample to target.

Construct validity

Construct validity concerns whether operational measures accurately represent the theoretical concepts they are intended to measure. Constructs such as trust, innovation capability, organisational justice and customer loyalty are not directly observable in a simple way.

Threats include:

• Construct underrepresentation: important dimensions are omitted.
• Construct-irrelevant variance: scores reflect influences unrelated to the intended construct, such as reading ability in a complex questionnaire.
• Common-method bias: predictor and outcome are collected from the same source, at the same time and with the same response format, creating artificial covariance.
• Demand characteristics: participants infer the study purpose and alter responses.
• Social desirability: respondents give acceptable rather than accurate answers.

Construct validation uses several forms of evidence. Content validity asks whether items cover the conceptual domain. Convergent validity is supported when related measures correlate as theory predicts. Discriminant validity is supported when distinct constructs are not excessively correlated. Criterion-related validity examines association with relevant outcomes. Clear definitions, multiple data sources, pilot testing and theoretically justified measures improve construct validity.

Statistical conclusion validity

Statistical conclusion validity concerns whether the evidence supports the claimed relationship or difference. Low statistical power, unreliable measurement, non-independence of observations, inappropriate model assumptions, selective reporting and multiple unplanned tests can produce misleading conclusions. Researchers should specify analyses in advance where possible, report uncertainty, inspect missing data and distinguish statistical significance from practical importance.

A small p-value does not show that an effect is large or managerially valuable. Similarly, a non-significant result does not prove that no effect exists; it may reflect an imprecise estimate or inadequate sample size.

5. Bias and Credible Inference

Bias is systematic deviation from the quantity or interpretation the study seeks to estimate. Unlike random error, bias does not necessarily disappear with a larger sample.

Sampling bias occurs when selection into the sample systematically differs from the target population. Nonresponse bias occurs when respondents differ from nonrespondents in relevant ways. Survivorship bias arises when analysis includes only organisations or products that remain successful. Recall bias occurs when participants remember some events more accurately than others. Observer or researcher bias occurs when expectations influence data collection, coding or interpretation.

Causal studies also face endogeneity, meaning that an explanatory variable is related to unobserved influences in the outcome equation. Common sources are omitted variables, reverse causality and measurement error. The appropriate response is not to claim that statistical control has solved endogeneity. Researchers should use design solutions—randomisation, natural experiments, temporal ordering, instruments or fixed-effects strategies where defensible—and conduct sensitivity analyses.

A credible inference chain can be represented as:

A weakness at any stage can undermine the final claim. For example, a sophisticated regression cannot rescue a measure that does not represent the construct or a sample that excludes the relevant population.

6. An Applied Case: Evaluating a Flexible-Work Policy

Background

Illustrative case: Meridian Services, a 1,200-employee business-process firm, introduced a two-day-per-week remote-work policy in selected departments. Senior management wanted to know whether the policy improved productivity without harming service quality or employee retention.

Problem

The first proposal was a before-and-after survey of participating employees. This design could describe perceptions but could not confidently attribute changes to the policy. Productivity might improve because demand declined, new software was introduced or experienced employees self-selected into participating departments.

Design analysis

A stronger design used departments scheduled to adopt the policy in September and comparable departments scheduled to adopt it in January. Productivity, service-quality indicators and turnover intentions were measured for six months before and after implementation. Administrative performance data were combined with employee surveys and interviews.

The design was a quasi-experimental, longitudinal, mixed-method study. The comparison depended on the parallel-trends assumption: before implementation, early and later departments should show similar outcome trends. Researchers inspected pre-policy trends and controlled only for clearly justified baseline differences. They also monitored contamination, since employees in the comparison group might work remotely informally.

Validity assessment

Internal validity was threatened by department-level selection, seasonal demand and differential attrition. External validity was limited because the firm operated in one industry and the policy applied only to selected roles. Construct validity required distinguishing productivity from hours logged; therefore, productivity was measured through completed cases adjusted for complexity, while service quality used error and customer-escalation rates.

Common-method bias was reduced by combining administrative records with survey responses. Interviews investigated mechanisms, including commuting time, coordination difficulties and managerial trust. The researchers reported effect estimates with uncertainty rather than claiming universal benefits.

Outcome and learning points

The evaluation found a moderate productivity improvement in eligible roles, no meaningful deterioration in service quality and improved reported work-life balance. Effects were weaker where tasks required frequent real-time coordination. The conclusion was conditional: flexible work appeared beneficial for suitable roles under adequate coordination systems, not as a universal policy.

The case illustrates five principles: begin with the causal question; use a comparison that approximates the counterfactual; measure constructs carefully; combine evidence when mechanisms matter; and state boundary conditions rather than overstating generalisation.

7. Ethics, Governance and Contemporary Practice

Research design is also an ethical decision. Participants should receive understandable information, provide voluntary consent where required, and face proportionate risks. Confidentiality is especially important in employee research because identifiable responses may affect careers. Researchers should minimise coercion, protect vulnerable participants and explain data retention and access.

Organisational analytics and artificial intelligence create additional risks. Automated systems may reproduce historical discrimination, infer sensitive attributes or make participation effectively compulsory. Researchers should document data provenance, assess subgroup performance, protect personally identifiable information and retain human oversight over consequential decisions. AI-generated coding or text analysis should be validated against human-reviewed material, with uncertainty and model limitations disclosed.

Preregistration, version-controlled protocols, audit trails, transparent inclusion criteria, open materials where confidentiality permits, and reporting of null or adverse findings improve research integrity. These practices do not eliminate bias, but they make design decisions visible and contestable.

Best-Practice Checklist

Before collecting data, ask:

1. Does the design answer the stated question rather than a more convenient question?
2. Is the unit of analysis consistent with the intended conclusion?
3. Is the comparison or counterfactual credible?
4. Have major threats to internal, external and construct validity been anticipated?
5. Are measures theoretically justified and tested for reliability and validity?
6. Could sampling, nonresponse, attrition or common-method bias distort findings?
7. Are ethical risks, privacy and data governance addressed?
8. Are claims limited to the population, context and time supported by the evidence?
9. Is the study feasible without compromising quality?
10. Have alternative explanations and negative cases been considered?

Chapter Summary

Research design is the integrated plan linking a research question to evidence and inference. Exploratory, descriptive, correlational, experimental, quasi-experimental, qualitative, case-study, longitudinal and mixed-method designs serve different purposes. Selection should reflect the research objective, causal ambition, unit of analysis, timing, feasibility, ethics, measurement needs and desired generalisability.

Internal validity concerns whether a causal conclusion is credible within the study; external validity concerns transfer beyond the study; construct validity concerns whether operational measures represent theoretical concepts; and statistical conclusion validity concerns whether the analysis supports the claimed relationship. Selection bias, confounding, attrition, measurement error, common-method bias, researcher influence and endogeneity threaten credible inference. Strong research anticipates these threats, uses appropriate design safeguards, integrates evidence where useful and states conclusions with disciplined qualifications.

Key Terms

Research design: The overall plan connecting questions, evidence, analysis and inference.

Unit of analysis: The entity about which conclusions are drawn.

Independent variable: A presumed explanatory or causal factor.

Dependent variable: The outcome being explained or evaluated.

Mediator: A variable explaining how an effect occurs.

Moderator: A variable changing the strength or direction of a relationship.

Internal validity: Credibility of a causal conclusion within a study.

External validity: Applicability of findings beyond the study context.

Construct validity: Accuracy with which measures represent theoretical constructs.

Statistical conclusion validity: Adequacy of evidence for a claimed statistical relationship.

Confounding: Distortion caused by a factor related to both presumed cause and outcome.

Selection bias: Systematic difference created by how units enter groups or samples.

Common-method bias: Artificial covariance caused by using the same measurement source or procedure.

Quasi-experiment: An intervention study without full random assignment.

Randomisation: Assignment by chance to reduce systematic baseline differences.

Attrition: Loss of participants or observations during a study.

Triangulation: Use of multiple sources, methods or perspectives to strengthen interpretation.

Generalisation: Extension of findings to a wider population or context.

Endogeneity: Association between an explanatory variable and unobserved outcome influences.

Credible inference: A conclusion proportionate to the quality and limits of the evidence.

Review Questions

Short Answer Questions

1. What is the difference between a research method and a research design?
2. Define internal validity and give two threats to it.
3. Why does a large sample not guarantee external validity?
4. Distinguish a mediator from a moderator.
5. What is common-method bias?

Long Answer Questions

1. Compare experimental, quasi-experimental and correlational designs for evaluating a management intervention.
2. Explain how construct validity can be assessed in research on organisational trust.
3. Discuss the criteria an MBA researcher should use when selecting between cross-sectional and longitudinal designs.
4. Explain the relationship between sampling bias, nonresponse bias and generalisability.
5. Evaluate the role of mixed methods in business research.

Analytical / Application Questions

1. A company compares productivity in departments that voluntarily adopted remote work with productivity in departments that did not. Identify the principal threats to causal inference and propose a stronger design.
2. A customer-loyalty survey uses one five-point questionnaire completed in a single sitting. Analyse possible construct and common-method validity problems and recommend remedies.
3. A policy evaluation reports a statistically significant improvement but uses a small, nonrepresentative volunteer sample. Assess what the organisation may and may not conclude.

Multiple Choice Questions with Answers

Question 1

Which feature most directly distinguishes a true experiment from a quasi-experiment?

A. Use of numerical data

B. Random assignment to conditions

C. Collection of data over time

D. Use of a control variable

Correct Answer: B. Random assignment to conditions

Question 2

Internal validity primarily concerns whether:

A. Findings apply to other populations

B. Measures represent theoretical constructs

C. An observed effect was caused by the intervention

D. The sample is sufficiently large

Correct Answer: C. An observed effect was caused by the intervention

Question 3

In a difference-in-differences evaluation, the key identifying assumption is usually:

A. Perfect measurement reliability

B. Parallel trends absent the intervention

C. Equal sample sizes at every time point

D. A statistically significant pretest difference

Correct Answer: B. Parallel trends absent the intervention

Question 4

Which example best illustrates construct underrepresentation?

A. Respondents exaggerate socially desirable behaviour

B. A job-quality scale measures pay but omits safety and autonomy

C. Treatment participants leave the study more often

D. A researcher misrecords the survey date

Correct Answer: B. A job-quality scale measures pay but omits safety and autonomy

Question 5

A mediator is a variable that:

A. Changes the strength of a relationship

B. Explains the process through which an effect occurs

C. Determines who enters the sample

D. Creates random assignment

Correct Answer: B. Explains the process through which an effect occurs

Question 6

Which statement about external validity is most accurate?

A. It is guaranteed by a large sample

B. It concerns whether findings apply beyond the study context

C. It is identical to measurement reliability

D. It can be established without describing the target population

Correct Answer: B. It concerns whether findings apply beyond the study context

Question 7

A study surveys only current customers to estimate why customers cancel subscriptions. The most immediate concern is:

A. Survivorship or coverage bias

B. Random assignment

C. Regression discontinuity

D. Parallel trends

Correct Answer: A. Survivorship or coverage bias

Question 8

If treatment and control groups differ systematically before an intervention, the design faces:

A. Selection bias

B. Convergent validity

C. Maturation control

D. Analytical generalisation

Correct Answer: A. Selection bias

Question 9

Which practice most directly reduces common-method bias?

A. Collecting predictor and outcome from different sources where feasible

B. Increasing the font size of survey items

C. Removing all comparison groups

D. Using only one response format

Correct Answer: A. Collecting predictor and outcome from different sources where feasible

Question 10

A study finds a statistically significant effect in a volunteer sample from one firm. The most defensible conclusion is that:

A. The effect applies to all firms

B. The intervention caused the effect in every industry

C. The result is evidence of an effect in the studied context, subject to design limitations

D. The effect must be practically important

Correct Answer: C. The result is evidence of an effect in the studied context, subject to design limitations