Chapter 2: 2. Research Design, Sampling and Analytical Planning

1. Introduction

Research findings are only as defensible as the design decisions that produce them. Sophisticated statistical software cannot compensate for a poorly defined research question, an inappropriate sample, uncontrolled sources of bias, or an analysis plan developed after examining the results. In information technology research, these issues are especially important because data are generated by complex sociotechnical systems involving people, organizations, algorithms, platforms, and changing environments.

Research design provides the logical architecture of a study. It connects research questions with evidence, data-collection procedures, sampling decisions, analytic methods, and conclusions. Quantitative designs emphasize measurement, comparison, estimation, prediction, and causal inference. Qualitative designs emphasize meaning, context, process, interpretation, and the perspectives of participants. Mixed-methods designs integrate both forms of evidence.

This chapter examines experimental, quasi-experimental, observational, case-study, and field designs. It explains probability and purposive sampling, power analysis, qualitative saturation, preregistration, and analysis planning. The central argument is that methodological defensibility depends on alignment: the design, sample, measures, analytic procedures, and claims must be appropriate to the question being asked.

2. Learning Objectives

By the end of this chapter, readers should be able to:

1. Distinguish major quantitative, qualitative, and mixed-methods research designs.

2. Match research questions and causal claims to suitable designs.

3. Select probability or purposive sampling strategies appropriately.

4. Explain statistical power, minimum detectable effects, and qualitative saturation.

5. Develop a transparent analysis plan and preregistration.

6. Identify threats to validity, bias, confounding, and transferability.

7. Integrate ethical, governance, and data-management requirements into study planning.

8. Evaluate research designs in organizational and digital-technology settings.

3. Meaning and Concept

Research design is the coherent plan specifying what will be studied, from whom or what data will be collected, how variables or experiences will be measured, when observations will occur, and how evidence will be analyzed. It is not simply a choice between a survey and an interview. It is a system of decisions that determines the credibility of inferences.

A design should begin with the inferential purpose of the study. Descriptive questions ask what exists or how often something occurs. Relational questions ask whether variables are associated. Explanatory questions ask why an outcome occurs. Causal questions ask whether an intervention produces a change. Interpretive questions ask how people understand an experience, while process questions examine how events unfold over time.

A useful distinction is between the target population, the accessible population, the sample, and the unit of analysis. The target population is the broader group to which conclusions are intended to apply. The accessible population is the group from which cases can realistically be reached. The sample is the selected subset. The unit of analysis may be an individual, team, organization, transaction, software artifact, online community, or event. Confusing these levels can produce ecological fallacies, individual-level conclusions drawn from group data, or atomistic fallacies, group conclusions drawn from individual data.

4. Fundamental Principles

Alignment

Research questions, design, sampling, measures, and analysis must support one another. A randomized experiment is appropriate for estimating an intervention effect, but not necessarily for understanding participants’ lived experiences. An interview study may illuminate adoption barriers but cannot by itself estimate population prevalence.

Validity

Internal validity concerns whether observed effects are attributable to the proposed explanation rather than to confounding, selection, maturation, measurement changes, or history. External validity concerns the extent to which findings apply in other populations, settings, or times. Construct validity concerns whether operational measures represent the intended concepts. Statistical conclusion validity concerns whether the analysis supports accurate inferences.

Transparency and reproducibility

Researchers should document hypotheses, exclusions, measures, transformations, missing-data rules, analytic code, and deviations from the original plan. Transparency does not require that every study be fully reproducible in practice, especially where confidential data are involved, but it requires that decisions be traceable and justified.

Uncertainty

Estimates are not facts without error. Confidence intervals, sensitivity analyses, posterior distributions, qualitative disconfirming evidence, and explicit limitations communicate uncertainty. Precision should not be confused with truth.

Reflexivity

In qualitative and field research, the researcher is part of the knowledge-production process. Assumptions, institutional position, relationships with participants, and interpretive choices should be examined rather than treated as irrelevant.

Ethical proportionality

The burden and risk imposed on participants should be proportionate to the value of the knowledge sought. Data minimization, informed consent, privacy protection, and fair participant selection are methodological as well as ethical requirements.

5. Major Components / Elements

Research questions and hypotheses

A strong question identifies the population, exposure or intervention, outcome, comparison, and relevant time frame when appropriate. Quantitative hypotheses should specify expected direction and relationships. Qualitative questions should be sufficiently focused to guide inquiry while remaining open to unexpected meanings.

Conceptual framework

A conceptual framework describes expected relationships among constructs. For example, a technology-acceptance study might propose that perceived usefulness affects intention to use, with organizational support moderating that relationship. The framework guides operationalization and prevents the analysis from becoming a disconnected search for significant variables.

Variables and operationalization

Constructs such as trust, usability, security awareness, or innovation capability must be translated into observable indicators. Researchers should define measurement level, scoring, reliability, validity, and acceptable ranges. Composite scales require attention to dimensionality rather than automatic averaging.

Design type and timing

Cross-sectional studies measure variables at one point; longitudinal studies examine change. Panel designs repeatedly measure the same units, whereas repeated cross-sectional designs sample different units from the same population over time. A cohort follows units sharing a defining characteristic. Time-series and interrupted-time-series designs use repeated observations to assess changes associated with an intervention or event.

Sampling and recruitment

Sampling determines whose evidence enters the study. Recruitment procedures also influence participation, nonresponse, and the social meaning of the data. A sample should be justified in relation to the intended inference rather than selected only because it is convenient.

Measurement and data quality

A design should specify instrument piloting, data validation, duplicate detection, coding rules, missing-data treatment, and quality checks. In digital research, automated logs may appear objective but can contain systematic errors, changing definitions, bot activity, and unequal opportunities for users to generate data.

Analysis plan

The analysis plan should identify outcomes, predictors, estimands, models, assumptions, contrasts, subgroup analyses, missing-data procedures, and sensitivity analyses before outcome examination whenever feasible. Qualitative plans should specify the analytic approach, coding strategy, unit of meaning, reflexive procedures, and standards for establishing credibility.

6. Process / Framework / Methodology

A defensible planning process can be organized into the following stages.

Stage 1: Define the inferential aim

State whether the study is descriptive, predictive, explanatory, causal, interpretive, evaluative, or exploratory. Specify the claim that the evidence is expected to support. A causal claim requires stronger design assumptions than a descriptive claim.

Stage 2: Select the design

An experiment randomly assigns units to conditions. A quasi-experiment estimates effects without full randomization, using designs such as difference-in-differences, regression discontinuity, matched comparison groups, or interrupted time series. An observational study measures naturally occurring exposures and outcomes without researcher assignment. A case study investigates a bounded system in depth using multiple sources. Field research examines phenomena in natural organizational or community settings.

Stage 3: Specify the population and sampling frame

Define inclusion and exclusion criteria, the sampling frame, recruitment channels, expected response rate, and the unit of sampling. Researchers should identify groups that may be systematically omitted, such as nonusers, temporary workers, people with limited connectivity, or employees reluctant to criticize management.

Stage 4: Plan sample size or information sufficiency

For quantitative work, sample size depends on the outcome, design, effect size of interest, variability, significance threshold, desired power, attrition, clustering, and multiplicity. For a simple two-group comparison, a conventional planning expression is

n per group approximately equal to 2(Z alpha/2 + Z beta)^2 sigma squared divided by delta squared,

where delta is the minimum effect of practical interest, sigma is the standard deviation, alpha is the Type I error rate, and 1 minus beta is power. This formula is only a starting point; regression, repeated measures, clustered designs, and unequal allocation require specialized calculations. Researchers should distinguish an effect that is statistically detectable from one that is practically important.

Qualitative sample adequacy depends on the study purpose, heterogeneity of the population, depth of interviews, complexity of the phenomenon, and quality of the data. Saturation is not a mechanical number. It may refer to code saturation, where few new codes appear, or meaning saturation, where deeper dimensions of already identified themes have been sufficiently explored. Researchers should use stopping criteria and document the reasoning.

Stage 5: Develop measures and protocols

Pilot instruments with participants resembling the target population. Test comprehension, response burden, technical reliability, interview sequencing, and data-security procedures. A pilot should be used to improve the design, not merely to generate a small version of the final study.

Stage 6: Preregister and document

Preregistration records questions, hypotheses, design, primary outcomes, sample-size rationale, stopping rules, and planned analyses before examining confirmatory outcomes. It does not prevent exploratory work; it distinguishes confirmatory analyses from analyses generated after seeing the data. Registrations may be time-stamped and stored in a public or restricted repository, depending on confidentiality and intellectual-property requirements.

Stage 7: Collect and monitor data

Monitoring should detect recruitment imbalance, attrition, protocol deviations, adverse events, instrument failure, and changes in the environment. Monitoring must not become an unplanned opportunity to stop when results become favorable unless stopping rules were specified or a justified safety decision is recorded.

Stage 8: Analyze and conduct sensitivity checks

Apply the prespecified primary analysis, then assess robustness to alternative plausible choices. Examples include different missing-data assumptions, covariate specifications, outlier rules, operational definitions, and analytic populations. Qualitative researchers may compare interpretations across coders, cases, time points, or stakeholder groups.

Stage 9: Interpret and report

Report estimates with uncertainty, disclose deviations, distinguish association from causation, and explain the boundaries of transferability. Negative or inconclusive findings are informative when the study had adequate power and sound measurement.

7. Important Theories and Models

Campbell and Stanley validity framework

The distinction between internal and external validity remains central to design evaluation. A study may provide strong control over confounding but have limited generalizability, or it may closely represent field conditions while permitting weaker causal inference.

Rubin causal model and potential outcomes

For each unit, the potential-outcomes framework imagines an outcome under treatment and another under control. The individual treatment effect is unobservable because a unit cannot simultaneously occupy both conditions. Randomization makes treatment assignment independent of potential outcomes in expectation, allowing average treatment effects to be estimated. Observational causal inference requires additional assumptions, such as no unmeasured confounding, consistency, and adequate overlap.

Logic models and program theory

A logic model links inputs, activities, outputs, short-term outcomes, and long-term impacts. It is valuable in evaluating digital transformation, cybersecurity training, or analytics adoption because it clarifies the mechanisms through which an intervention is expected to work and the indicators appropriate at each stage.

Maximum variation and information-power models

In qualitative inquiry, maximum-variation sampling deliberately includes diverse cases to examine common patterns and meaningful differences. Information power suggests that a narrower study with a strong theoretical basis, specific participants, high-quality dialogue, and focused analysis may require fewer participants than a broad exploratory study.

Mixed-methods integration

Convergent designs collect quantitative and qualitative data during a similar period and integrate findings. Explanatory sequential designs begin quantitatively and use qualitative inquiry to explain unexpected patterns. Exploratory sequential designs develop measures or interventions from qualitative findings and then test them quantitatively.

8. Practical Applications

A university evaluating a new learning-management system might use a randomized experiment if classes can be assigned to the new interface without contamination. If departments adopt the system at different times, a quasi-experimental staggered-adoption design may be more realistic. If the objective is to understand why instructors resist the system, purposive interviews and observation may be appropriate. A mixed-methods design could estimate changes in completion rates and then explore student and instructor experiences.

A cybersecurity researcher studying phishing awareness might randomly assign training formats, measure simulated-phishing responses, and use a delayed post-test. An observational alternative could analyze naturally occurring security incidents, but conclusions would be vulnerable to differences between trained and untrained employees. A qualitative field study could examine how organizational norms influence reporting behavior.

In software engineering, repository data can support observational analyses of code-review latency and defect rates. However, teams differ in project complexity, staffing, and release pressure. Multilevel models, carefully defined exposure periods, and sensitivity analyses are needed. Interviews can explain why a metric changes, but participants may describe formal policy rather than actual practice; triangulation with logs and observation is therefore valuable.

9. Examples

Consider the question: “Does an automated feedback tool improve doctoral students’ research-proposal quality?” A strong experimental design would randomly assign students to automated feedback or usual support, use a blinded rubric, define the primary outcome before data collection, and account for baseline writing ability. If students choose whether to use the tool, the study becomes observational or quasi-experimental and requires methods such as matching or inverse-probability weighting, with weaker causal assumptions.

For the question “How do remote teams develop trust when using collaboration platforms?”, a qualitative multiple-case design could purposively select teams differing in geographic distribution, tenure, and platform intensity. Data might include interviews, meeting observations, and communication artifacts. Analysis would proceed through within-case coding followed by cross-case comparison. Claims would concern processes and meanings, not population prevalence.

10. Technology and Digital Applications

Research teams may use electronic data-capture platforms for consent, surveys, randomization, audit trails, and secure data transfer. Statistical environments such as R, Python, Stata, or specialized power-analysis software support reproducible calculations and scripted workflows. Qualitative analysis platforms can organize transcripts, codes, memos, and retrieval, but software does not replace interpretation.

Version-control systems allow researchers to preserve code, preregistration amendments, analysis scripts, and documentation. Reproducible computational notebooks connect data preparation, analysis, and visualizations. Digital trace data, application logs, sensors, and platform records can provide high-frequency behavioral evidence, but researchers must verify event definitions, timestamps, missingness, access rights, and algorithmic changes. Automated machine-learning classification should be validated against human-coded data and assessed for differential error across groups.

11. Strategic / Managerial Importance

Research design affects whether managers can trust evidence used for investment, workforce, cybersecurity, and digital-transformation decisions. A poorly designed evaluation may attribute benefits to a new system when the real cause is a simultaneous policy change. An underpowered study may lead leaders to abandon an effective intervention. An unrepresentative employee survey may conceal risks experienced by contractors or night-shift workers.

Managers should therefore ask what claim the study supports, how participants were selected, what alternative explanations remain, whether outcomes are practically meaningful, and whether implementation conditions resemble the intended operating environment. Strong design improves resource allocation and reduces the risk of scaling ineffective or inequitable technologies.

12. Detailed Case Study

Background

The following is a fictional illustrative case. A regional bank introduces an AI-supported service-desk system intended to reduce the time required for employees to resolve internal IT requests.

Problem

After three months, managers observe that average ticket-resolution time has declined. They want to conclude that the AI system caused the improvement. However, the same period included a staffing increase, a simplified ticket form, and seasonal variation in demand.

Analysis

The initial before-and-after comparison has weak internal validity. Resolution time may have changed because of staffing, ticket composition, learning, or regression to the mean. Moreover, average time conceals whether complex cases improved and whether some employee groups experienced poorer service.

The evaluation team defines the primary outcome as median resolution time for eligible tickets, with secondary outcomes including first-contact resolution, reassignment rate, user satisfaction, and escalation. Tickets are nested within teams, so observations are not independent. The team examines pre-intervention trends and identifies comparable departments that have not yet adopted the system.

Strategy / Solution

Because organization-wide randomization is impractical, the team uses a difference-in-differences design comparing changes in adopting and nonadopting departments. Adoption timing is recorded, and the team tests whether pre-intervention trends are reasonably parallel. A multilevel model accounts for team clustering and ticket complexity. The analysis plan specifies exclusions, outcome definitions, missing-data treatment, and subgroup checks by user role and ticket type.

Interviews with service-desk staff and employees are purposively sampled to include heavy users, infrequent users, supervisors, and staff whose tickets were escalated. Interview data are analyzed thematically and integrated with the quantitative results. Data access is restricted, personally identifying content is removed, and the AI recommendation is not used as the sole basis for employee performance evaluation.

Outcome

The adjusted analysis finds a moderate reduction in median resolution time, but the benefit is concentrated in routine tickets. Complex tickets show no reliable improvement, and new employees report difficulty understanding automated recommendations. The qualitative evidence explains that experienced staff use the tool selectively, whereas novices tend to accept suggestions without checking them.

Learning Points

The case demonstrates why a simple pre-post comparison is insufficient, why unit-of-analysis and clustering matter, and why mixed methods can explain heterogeneous effects. It also shows that implementation quality and human judgment are part of the causal pathway, not merely background conditions.

13. Advantages / Benefits

Careful design improves causal reasoning, measurement quality, efficiency, transparency, and ethical protection. Probability sampling supports defensible population estimates, while purposive sampling provides access to information-rich cases. Power analysis reduces the risk of inconclusive quantitative studies, and saturation planning limits unnecessary qualitative burden. Preregistration reduces undisclosed analytic flexibility and strengthens the distinction between confirmation and exploration. Integration of multiple methods can reveal both the magnitude of an effect and the mechanisms or experiences behind it.

14. Limitations / Challenges

No design eliminates uncertainty. Randomized experiments may be expensive, artificial, or ethically unavailable. Quasi-experiments depend on assumptions that are difficult to verify. Observational studies may suffer from confounding, selection bias, reverse causality, and measurement error. Case studies provide contextual depth but usually support analytic rather than statistical generalization. Surveys are vulnerable to nonresponse and social desirability. Interviews can be affected by recall, power relationships, and researcher interpretation.

Power calculations depend on assumptions that may be wrong, and large samples can make trivial effects statistically significant. Saturation judgments can be vague or used to justify premature closure. Preregistration may constrain legitimate adaptation if treated rigidly. Digital data can create an illusion of completeness while excluding people who leave fewer traces or lack access to the system.

15. Ethical and Governance Considerations

Researchers must obtain appropriate consent or justify a lawful alternative, explain foreseeable risks, and protect participants from coercion. In organizational studies, employees may feel that participation is not voluntary when supervisors recruit them. Confidentiality requires careful handling of small cells, quotations, metadata, and potentially re-identifiable logs.

Governance should address data ownership, retention, access controls, encryption, cross-border transfer, secondary use, and secure destruction. Sensitive data require data-protection impact assessment where applicable. Researchers should consider fairness in recruitment and whether an intervention could disadvantage a group. Algorithmic studies should document model versions, training data, performance differences, and human-oversight procedures. Research integrity requires accurate reporting, disclosure of conflicts, correction of errors, and refusal to manipulate outcomes or omit inconvenient evidence.

16. Current and Emerging Trends

Open science is expanding through preregistration, registered reports, open code, data dictionaries, and reproducible workflows. Causal machine learning is being used to estimate heterogeneous treatment effects, although interpretability and transportability remain challenges. Digital phenotyping and passive sensing enable intensive longitudinal research but raise serious privacy concerns.

Synthetic data and privacy-preserving computation may allow collaboration without releasing identifiable records. Federated analysis can keep data within institutional environments while sharing model results. Adaptive and platform trials permit planned modification of interventions as evidence accumulates. Bayesian and sequential designs provide flexible updating, but their stopping and decision rules must be specified transparently. Participatory and community-based research increasingly involves stakeholders in defining questions, interpreting findings, and determining acceptable data use.

17. Best Practices

1. Begin with a precise question and define the intended inference.

2. Select the design according to the claim, not according to the easiest dataset.

3. Define the unit of analysis, target population, sampling frame, and eligibility rules.

4. Use probability sampling for population estimation when a suitable frame exists; use purposive logic when depth and information richness are the priority.

5. Specify practically meaningful effect sizes and conduct power or precision planning.

6. Plan for attrition, clustering, missingness, multiplicity, and protocol deviations.

7. Pilot instruments and procedures with comparable participants.

8. Preregister confirmatory hypotheses, outcomes, exclusions, and analytic rules.

9. Preserve an audit trail for data cleaning, coding, and deviations.

10. Use triangulation, sensitivity analysis, and negative-case analysis to test robustness.

11. Separate exploratory findings from confirmatory claims.

12. Report uncertainty, limitations, null results, and subgroup differences responsibly.

13. Protect privacy throughout collection, analysis, publication, and archiving.

14. Involve relevant stakeholders without allowing organizational interests to suppress inconvenient findings.

18. Future Scope

Future research design will increasingly address dynamic interventions, continuously changing algorithms, network effects, and feedback between measurement and behavior. Researchers will need methods that combine causal inference with qualitative explanation and real-time operational data. Designs will also need to account for fairness, sustainability, accessibility, and the environmental cost of large-scale computation.

Doctoral researchers can contribute by developing transparent methods for evaluating AI systems in realistic settings, designing privacy-preserving longitudinal studies, improving standards for qualitative saturation, and creating models that integrate individual, team, organizational, and platform-level data. The most valuable advances will combine technical rigor with social and ethical responsibility.

19. Chapter Summary

Research design is the structure connecting questions, evidence, sampling, measurement, analysis, and claims. Experiments provide the strongest basis for causal inference when randomization is feasible, while quasi-experimental, observational, case-study, and field designs answer important questions under less controlled conditions. Probability sampling supports generalization to defined populations; purposive sampling supports depth, variation, and theoretical development. Quantitative sample planning should consider power, precision, practical importance, clustering, and attrition. Qualitative adequacy depends on information richness and justified saturation rather than a universal number.

Preregistration and analysis plans make decisions visible and reduce undisclosed flexibility. Validity, uncertainty, reflexivity, ethics, and governance must be addressed throughout the research process. In information technology research, digital traces and computational tools expand analytical possibilities but also introduce privacy, measurement, algorithmic, and access-related risks. Defensible research is therefore not merely technically sophisticated; it is aligned, transparent, context-sensitive, and ethically governed.

20. Key Terms

1. Research design: The integrated plan linking research questions, data collection, sampling, measurement, and analysis.

2. Target population: The broader population to which findings are intended to apply.

3. Sampling frame: The operational list or source from which a sample is selected.

4. Unit of analysis: The entity about which conclusions are drawn.

5. Probability sampling: Sampling in which units have known, nonzero selection probabilities.

6. Purposive sampling: Deliberate selection of information-rich cases according to stated criteria.

7. Randomized experiment: A design assigning units to conditions by chance.

8. Quasi-experiment: An intervention evaluation without complete random assignment.

9. Observational study: A study in which exposures are observed rather than assigned.

10. Internal validity: Credibility that an observed effect is caused by the proposed factor.

11. External validity: Extent to which findings apply across populations, settings, or times.

12. Construct validity: Degree to which measures represent the intended construct.

13. Statistical power: Probability of detecting an effect of a specified size when it exists.

14. Minimum detectable effect: Smallest effect a design is adequately powered to identify.

15. Saturation: Point at which additional qualitative data yield insufficiently new insight for the study purpose.

16. Preregistration: Time-stamped documentation of hypotheses and planned methods before confirmatory analysis.

17. Confounding: A distortion caused by a factor related to both exposure and outcome.

18. Triangulation: Use of multiple data sources, methods, investigators, or theories to examine a finding.

19. Estimand: Precisely defined quantity a statistical analysis seeks to estimate.

20. Transferability: The reasoned applicability of qualitative findings to other contexts.

21. Review Questions

Short Answer Questions

1. What is the difference between a target population and a sampling frame?

2. Why does randomization strengthen causal inference?

3. Distinguish code saturation from meaning saturation.

4. What information should a preregistration contain?

5. Why must the unit of analysis be identified before analysis begins?

Long Answer Questions

1. Compare experimental, quasi-experimental, and observational designs for evaluating a digital intervention.

2. Explain how probability and purposive sampling serve different inferential purposes.

3. Discuss the relationship among power, effect size, practical significance, and sample size.

4. Describe how a mixed-methods design can improve evaluation of an organizational information system.

5. Explain major threats to internal and external validity in field research.

Analytical / Application Questions

1. A company introduces an AI recruitment tool in half of its branches, while the other branches continue using conventional procedures. Design an evaluation, identify plausible confounders, and propose at least two sensitivity analyses.

2. A researcher wants to study trust in remote software teams across three industries. Propose a purposive sampling strategy, criteria for saturation, and a plan for integrating interviews with digital communication data.

3. A survey reports that employees who use a cybersecurity platform more frequently experience fewer incidents. Evaluate whether this supports a causal claim and recommend a stronger design.

22. Multiple Choice Questions with Answers

Question 1

Which design provides the strongest basis for estimating the causal effect of an intervention when ethically and practically feasible?

A. Cross-sectional survey

B. Randomized experiment

C. Convenience case study

D. Unstructured observation

Correct Answer: B. Randomized experiment

Question 2

A researcher deliberately selects teams that vary in size, location, and organizational maturity to study common and differing experiences. This is an example of:

A. Simple random sampling

B. Systematic sampling

C. Maximum-variation purposive sampling

D. Cluster randomization

Correct Answer: C. Maximum-variation purposive sampling

Question 3

What is the primary purpose of preregistration?

A. To guarantee statistically significant findings

B. To eliminate the need for exploratory research

C. To document planned confirmatory decisions before outcome examination

D. To increase the number of participants automatically

Correct Answer: C. To document planned confirmatory decisions before outcome examination

Question 4

Which factor generally increases the sample size required for a quantitative study?

A. A larger expected effect size

B. Lower desired statistical power

C. Greater expected attrition

D. Lower outcome variability

Correct Answer: C. Greater expected attrition

Question 5

In a qualitative study, meaning saturation is reached when:

A. Every participant gives identical answers

B. Deeper dimensions of existing themes are sufficiently understood

C. The sample has reached a predetermined universal number

D. All collected interviews have been transcribed

Correct Answer: B. Deeper dimensions of existing themes are sufficiently understood

Question 6

A before-and-after comparison of an IT system is threatened most directly by which problem?

A. Random assignment

B. History and other changes occurring at the same time

C. Excessive blinding

D. Known selection probabilities

Correct Answer: B. History and other changes occurring at the same time

Question 7

What is the main reason for accounting for clustering in an analysis of tickets nested within service teams?

A. Tickets within a team may not be statistically independent

B. Clustering always increases the treatment effect

C. It removes the need to define outcomes

D. It guarantees external validity

Correct Answer: A. Tickets within a team may not be statistically independent

Question 8

Which statement best describes an estimand?

A. A list of participants who withdrew

B. A precisely defined quantity that an analysis seeks to estimate

C. A software package for qualitative coding

D. A method for encrypting research data

Correct Answer: B. A precisely defined quantity that an analysis seeks to estimate

Question 9

A study finds a statistically significant improvement that is too small to affect organizational decisions. This illustrates the difference between:

A. Power and sampling frame

B. Statistical significance and practical significance

C. Reliability and confidentiality

D. Saturation and transferability

Correct Answer: B. Statistical significance and practical significance

Question 10

Which practice best supports reproducible computational analysis?

A. Changing exclusion rules without recording them

B. Keeping only the final charts

C. Using version-controlled code and documenting data transformations

D. Removing all exploratory analyses from the research record

Correct Answer: C. Using version-controlled code and documenting data transformations