Chapter 6: Sampling, Measurement, and Data Quality

1. Introduction

Research conclusions are only as credible as the evidence on which they rest. In information technology (IT), evidence may be obtained from employees, customers, software repositories, transaction logs, devices, online communities, or organizational records. Each source presents distinctive challenges. A researcher may study a population that is too large to observe completely, a construct that cannot be directly seen, or a data set containing missing, duplicated, biased, or poorly documented observations. Sampling, measurement, and data-quality management provide the methodological foundation for addressing these challenges.

Sampling determines who or what is included in a study. Measurement translates theoretical constructs into observable indicators. Data quality concerns whether observations are sufficiently accurate, complete, consistent, timely, credible, and fit for the intended purpose. These activities are interdependent. A perfectly designed instrument cannot correct a biased sample, and a representative sample cannot compensate for invalid measures. Doctoral-level IT research therefore treats sampling, measurement, and data quality as an integrated design problem rather than as isolated technical tasks.

2. Learning Objectives

By the end of this chapter, readers should be able to:

1. Distinguish probability from non-probability sampling and select an appropriate design.

2. Define populations, sampling frames, units of analysis, and sample sizes.

3. Operationalize abstract IT constructs using defensible indicators.

4. Develop, adapt, and validate questionnaires, interview protocols, tests, and digital measures.

5. Assess reliability, validity, and measurement invariance.

6. Diagnose missing data, sampling error, nonresponse, and common-method bias.

7. Evaluate data provenance, lineage, and fitness for purpose.

8. Apply ethical, governance, and quality-control principles throughout the research lifecycle.

3. Meaning and Concept

A population is the complete set of units to which a researcher intends to generalize. A population might consist of software developers in a country, users of a government portal, or all cybersecurity incidents recorded by an organization during a year. A target population is the conceptual population of interest, whereas an accessible population is the portion that can practically be reached.

A sample is a subset of the population. A sampling frame is the operational list or mechanism from which the sample is selected. The frame may be an employee directory, customer database, registry, API-accessible repository, or list of eligible systems. Differences between the target population and the frame create coverage error. The unit of analysis is the entity about which conclusions are made, such as an individual, team, project, organization, system, or event. The unit of observation is the entity from which data are collected. These may differ: employees may report on teams, while system logs record events.

Measurement is the systematic assignment of values to attributes according to rules. Many important IT concepts are latent constructs: trust in automation, technological readiness, perceived security, innovation capability, or developer well-being. Such constructs cannot be observed directly. Researchers represent them through indicators, such as questionnaire items, behavioral events, performance scores, or coded interview statements.

Data quality is multidimensional. Accuracy concerns closeness to the true value; completeness concerns the presence of required values; consistency concerns agreement across records and systems; timeliness concerns availability at the appropriate time; validity concerns whether data represent the intended phenomenon; uniqueness concerns absence of unwanted duplication; and provenance concerns knowledge of where data came from and how they were transformed.

4. Fundamental Principles

The first principle is alignment. The population, research question, design, measure, and analysis must correspond. A study claiming to explain organizational adoption should not rely only on enthusiastic users, and a study of individual attitudes should not treat departments as interchangeable observations.

The second principle is transparency. Researchers should document inclusion criteria, frame construction, recruitment, refusals, exclusions, instrument revisions, coding decisions, transformations, and quality checks. Transparent procedures permit replication and allow readers to judge the credibility of generalizations.

The third principle is fitness for purpose. Data are not inherently good or bad. A data set suitable for predicting service demand may be unsuitable for estimating causal effects. A short single-item measure may be acceptable for operational monitoring but inadequate for a doctoral study of a complex psychological construct.

The fourth principle is error awareness. Total survey error includes coverage error, sampling error, nonresponse error, measurement error, and processing error. Digital studies add sensor error, logging gaps, algorithmic bias, and changes in platform architecture. Quality management requires identifying and reducing these sources rather than assuming that large samples eliminate them.

The fifth principle is proportionality. The rigor, burden, cost, and sensitivity of data collection should be proportionate to the research purpose. Collecting extensive personal data without a defensible analytic need violates both ethical and methodological standards.

5. Major Components / Elements

Sampling designs

Probability sampling gives every eligible population element a known, non-zero probability of selection. Simple random sampling selects units by chance from a complete frame. Systematic sampling selects every kth unit after a random start. Stratified sampling divides a population into relevant subgroups and samples within each; proportional allocation reflects subgroup sizes, while disproportionate allocation can ensure adequate representation of small but important groups. Cluster sampling selects naturally occurring groups, such as firms or classrooms, and then studies units within them. Multistage sampling combines these approaches.

Non-probability sampling does not provide known selection probabilities. Convenience sampling uses accessible participants. Purposive sampling selects information-rich cases according to explicit criteria. Quota sampling fills subgroup targets without random selection. Snowball sampling uses participant referrals and may be useful for hidden populations. Theoretical sampling, associated with grounded theory, selects cases as concepts develop. Non-probability designs can be appropriate for exploratory, qualitative, rare, or difficult-to-access populations, but statistical generalization requires caution.

Sample size and power

Sample size depends on the research design, population variability, desired precision, confidence level, effect size, statistical power, number of predictors, clustering, expected attrition, and missingness. Power analysis is preferable to rules of thumb. In a quantitative study, the researcher should specify a substantively meaningful effect, alpha level, desired power, and model. Clustered or longitudinal data require inflation because observations may not be independent. Qualitative sample adequacy is assessed through information power, depth, diversity, and theoretical saturation rather than a universal number.

Operationalization

Operationalization proceeds from construct definition to dimensions, indicators, response format, scoring, and interpretation. A construct should be defined conceptually before items are written. For example, “cloud migration readiness” might include strategic commitment, technical capability, governance preparedness, and workforce capability. Researchers must distinguish formative measures, in which indicators create or define a construct, from reflective measures, in which indicators are effects of an underlying construct. Removing a formative indicator may change the construct’s meaning; removing a redundant reflective item may not.

Instrument development

Instruments may be adopted, adapted, or newly developed. Existing validated scales can improve comparability, but their meaning may change across technologies, languages, cultures, or populations. Development generally includes literature review, construct mapping, item generation, expert review, cognitive interviewing, pilot testing, field testing, and revision. Digital behavioral measures require additional documentation of event definitions, logging rules, time windows, and data-processing scripts.

Reliability and validity

Reliability concerns consistency, not truth. Internal consistency is often assessed with Cronbach’s alpha or, preferably in many latent-variable applications, omega. Test–retest reliability evaluates stability over time; inter-rater reliability evaluates agreement among coders. Validity concerns the defensibility of interpretations. Content validity asks whether indicators adequately cover the construct. Construct validity includes convergent and discriminant evidence. Criterion-related validity examines association with an external criterion. Nomological validity evaluates whether relationships conform to theory. No single statistic proves validity; evidence must accumulate across sources and contexts.

6. Process / Framework / Methodology

A rigorous workflow begins by translating the research problem into a clearly bounded population and unit of analysis. The researcher specifies inclusion and exclusion criteria, geographic and temporal boundaries, and the intended level of generalization. Next, the sampling frame is assessed for coverage, duplication, outdated entries, and eligibility.

The design is then selected. Probability sampling is preferred when estimating population parameters or testing claims requiring statistical generalization. Purposeful or theoretical sampling may be stronger when the aim is explanation, discovery, or understanding of a rare phenomenon. Mixed-methods studies may use sequential designs, such as a broad survey followed by purposive interviews with contrasting cases.

Sample size or adequacy is justified before collection. Quantitative researchers conduct power or precision calculations and allow for nonresponse. Qualitative researchers specify anticipated diversity and criteria for judging saturation. Recruitment procedures should be standardized while allowing reasonable accessibility accommodations.

The measurement model is developed next. Each construct is defined, its dimensions identified, and indicators mapped to research questions. Items should be clear, singular, relevant, and appropriately scaled. Double-barreled, leading, ambiguous, and unnecessarily technical wording should be avoided. Response options must be mutually exclusive and collectively appropriate. Reverse-coded items should be used sparingly because they can introduce confusion.

Expert review evaluates relevance, clarity, and coverage. Cognitive interviews investigate how participants understand questions and retrieve answers. A pilot study tests administration time, technical operation, response distributions, routing, reliability, and unexpected interpretations. The final protocol should be version-controlled.

During collection, researchers monitor response rates, subgroup representation, duplicate submissions, straight-lining, implausibly fast completion, device failures, and changes in the data-generating process. Monitoring must not become arbitrary post hoc exclusion. Every exclusion rule should be documented and, where possible, established before analysis.

Data preparation includes validation checks, range and logic checks, deduplication, coding, de-identification, and linkage verification. A data dictionary defines variables, formats, permissible values, missing-value codes, and derivations. A provenance record identifies source systems, collection dates, transformations, responsible persons, and software versions.

Finally, researchers assess reliability, validity, missingness, and invariance before substantive analysis. Findings should report limitations, sensitivity analyses, and the boundaries of generalization.

7. Important Theories and Models

Classical test theory represents an observed score as true score plus error. It encourages researchers to reduce random error, but reliability estimates are population- and context-dependent. Generalizability theory extends this perspective by examining multiple sources of error, such as persons, items, occasions, and raters, and estimating how well scores generalize across conditions.

Item response theory models the probability of a response as a function of a person’s latent trait and item characteristics, such as difficulty and discrimination. It is useful for adaptive testing, scale refinement, and comparing item performance, although it requires suitable assumptions and sample sizes.

Confirmatory factor analysis tests whether observed indicators fit a hypothesized factor structure. Measurement invariance is typically examined in stages: configural invariance tests whether the pattern of factors is similar; metric invariance tests equality of factor loadings; scalar invariance tests equality of intercepts or thresholds; and strict invariance tests equality of residual variances. Comparing latent means across groups generally requires at least scalar invariance. If full invariance fails, partial invariance or alignment approaches may be considered, provided their assumptions are explained.

The Total Survey Error framework organizes errors across coverage, sampling, nonresponse, measurement, and processing. It is particularly valuable because it shifts attention from sampling alone to the entire evidence-production process.

For missing data, missing completely at random (MCAR) means missingness is unrelated to observed and unobserved values; missing at random (MAR) means missingness can be explained by observed information; missing not at random (MNAR) means missingness depends on unobserved or incompletely observed values. These assumptions cannot usually be proven from the data alone and should be assessed using substantive knowledge and sensitivity analysis.

8. Practical Applications

In enterprise IT, stratified sampling can ensure representation of technical, managerial, and operational employees when studying cybersecurity behavior. In software engineering, purposive sampling may select projects differing in size, development method, and regulatory context for comparative case research. In human–computer interaction, laboratory participants can be supplemented by real-world telemetry to compare stated usability with actual behavior.

For algorithm evaluation, the test set should reflect deployment conditions, including minority and edge cases. Randomly splitting data is insufficient when observations are temporally or organizationally dependent. Time-based validation may be necessary to avoid leakage from future information. For digital health or smart-city research, sensor data should be checked for device calibration, intermittent connectivity, changes in firmware, and unequal coverage across communities.

9. Examples

Suppose a researcher studies trust in an AI decision-support system. Trust is defined as willingness to rely on the system under conditions of vulnerability. Indicators may include perceived competence, predictability, integrity, and willingness to follow recommendations. A survey alone may overestimate trust because participants report intentions rather than behavior. A stronger design combines validated items, recommendation-acceptance logs, task performance, and interviews.

Consider a survey of software developers drawn from an online professional forum. The accessible sample may overrepresent active, English-speaking, highly engaged developers. The researcher should describe the frame, compare respondents with available population benchmarks, use recruitment across multiple channels, and avoid claiming that results represent all developers.

For a multinational study of remote-work technology acceptance, translation and cultural adaptation are essential. Back-translation is useful but insufficient. Cognitive interviews in each language and measurement-invariance testing are needed before comparing country-level means.

10. Technology and Digital Applications

Survey platforms support branching, randomized item presentation, response-time monitoring, and secure export, but platform defaults can affect measurement and must be documented. Statistical software can perform power analysis, reliability estimation, factor analysis, multiple imputation, and invariance testing. Reproducible scripts reduce processing error.

Research data-management platforms and workflow tools can record metadata, versions, transformations, and approvals. Data catalogs and lineage systems help connect derived variables to source records. Electronic laboratory notebooks and repository platforms support audit trails. In machine-learning research, dataset cards, model cards, and benchmark documentation provide structured information about provenance, intended use, limitations, and subgroup performance.

Automation can detect duplicates, impossible values, anomalous timestamps, and schema changes. However, automated cleaning may remove legitimate rare cases or encode hidden assumptions. Human review and a preserved raw-data layer remain necessary.

11. Strategic / Managerial Importance

Organizations make decisions about cybersecurity investment, technology adoption, digital inclusion, workforce development, and system performance using empirical evidence. Poor sampling can conceal vulnerable groups; poor measurement can misdirect resources; and poor provenance can make findings impossible to audit. Managers should therefore treat research data as governed organizational assets. Clear quality standards improve decision confidence, regulatory defensibility, vendor evaluation, and the ability to compare performance across time.

12. Detailed Case Study

Background

The following case is fictional and illustrative. “Northbridge University” introduced an AI advising system intended to help students select courses and identify academic risks. The university commissioned a doctoral study of student trust, usability, and actual use.

Problem

An initial online survey received 1,200 responses, but 82 percent came from students who had voluntarily used the system. The instrument was adapted from a general technology-acceptance scale without testing whether its items applied to academic advising. Usage logs contained duplicate sessions, inconsistent user identifiers, and missing demographic fields.

Analysis

The target population was all enrolled students, while the sampling frame was the registrar’s current enrollment list. The original survey used self-selection and therefore suffered from coverage and nonresponse bias. The researchers also found that “trust” items combined competence, benevolence, and privacy concerns. Logins were counted as use even when students abandoned the session after the first screen. Missingness was concentrated among students using mobile devices and among those who declined optional demographic questions.

Strategy / Solution

The team drew a stratified random sample by academic level and faculty, oversampling international and part-time students. Weights were prepared for population estimation. A revised instrument defined trust as a multidimensional construct and included separate measures of competence, transparency, privacy, and reliance intention. Expert review and cognitive interviews preceded a pilot. Usage was redefined as completion of a meaningful advising task, and a stable pseudonymous identifier was created through a documented linkage process.

The team compared early and late respondents, reported response patterns by stratum, conducted reliability and factor analyses, and tested measurement invariance across domestic and international students. Missing demographic values were not silently coded as zero. Multiple imputation was used for appropriate survey variables, while sensitivity analyses examined plausible departures from MAR. Raw logs were retained, transformed data were versioned, and access was restricted.

Outcome

The revised study found that students generally considered the system competent but were reluctant to rely on it when explanations were unclear. Trust scores differed across groups before invariance testing but showed smaller differences after noninvariance in privacy items was addressed. The university prioritized explanation design, privacy communication, and outreach to underrepresented students rather than simply increasing system promotion.

Learning Points

The case demonstrates that a large response count does not guarantee representativeness. Construct definition must precede item selection, and behavioral data require explicit event definitions. Missingness can reveal unequal participation rather than merely a technical inconvenience. Finally, provenance and version control are essential when research combines surveys, administrative records, and logs.

13. Advantages / Benefits

Careful sampling improves the credibility and efficiency of generalization. Strong operationalization makes abstract concepts observable and analyzable. Validated instruments support cumulative research and comparison. Reliability assessment reduces random error, while invariance testing prevents misleading group comparisons. Comprehensive quality procedures improve reproducibility, auditability, and the practical value of findings.

14. Limitations / Challenges

Probability sampling can be expensive and depends on a usable frame. Non-probability sampling may be unavoidable for hidden or emerging populations, but selection probabilities are unknown. Reliability coefficients can be inflated by redundant items and do not establish validity. Factor structures may vary across settings. Missing-data assumptions are uncertain, and imputation cannot recover information that was never collected. Digital traces may appear objective while reflecting platform design, unequal access, or undocumented changes. Quality management also requires time, expertise, and organizational cooperation.

15. Ethical and Governance Considerations

Researchers must obtain informed consent where required, minimize data collection, protect confidentiality, and explain foreseeable risks. Identifiable survey responses should be separated from contact information, and pseudonymization should not be mistaken for complete anonymity. Sensitive IT data may reveal health, employment, political, behavioral, or security information.

Governance includes lawful collection, purpose limitation, retention schedules, access controls, encryption, secure transfer, incident response, and controlled data sharing. Researchers should assess whether participants can genuinely refuse, especially in workplace or educational settings. Sampling should avoid systematically excluding vulnerable groups, while oversampling should be justified and privacy-protective. Algorithmic measures should be examined for disparate error rates and historical bias. Data provenance must include transformations, not merely the original source.

16. Current and Emerging Trends

Increasingly, researchers combine surveys with passive sensing, platform telemetry, administrative records, and open data. This creates opportunities for triangulation but also increases linkage and consent risks. Privacy-enhancing technologies, including differential privacy, secure enclaves, federated analysis, and synthetic data, are developing as alternatives to unrestricted data sharing.

Automated and adaptive instruments can personalize item difficulty, but their algorithms require validation and fairness assessment. FAIR principles—findable, accessible, interoperable, and reusable—are influencing research-data management, subject to ethical and legal restrictions. Reproducible computational pipelines, preregistration, registered reports, and machine-readable metadata are becoming more common. Researchers are also giving greater attention to dataset shift, concept drift, and the continuing validity of measures after technologies and populations change.

17. Best Practices

1. Define the target population, unit of analysis, and intended generalization explicitly.

2. Evaluate the sampling frame before selecting cases.

3. Use probability designs for population estimation when feasible.

4. Justify non-probability designs and avoid overstated generalization.

5. Conduct power or precision analysis rather than relying on simplistic sample-size rules.

6. Define constructs conceptually before writing or selecting items.

7. Combine expert review, cognitive interviewing, piloting, and statistical validation.

8. Distinguish formative from reflective measurement models.

9. Assess reliability alongside multiple forms of validity.

10. Test measurement invariance before comparing groups.

11. Establish missing-data codes, assumptions, and sensitivity analyses in advance.

12. Preserve raw data and document every transformation.

13. Monitor coverage, nonresponse, duplicates, and data-generating-process changes.

14. Predefine quality exclusions and retain an audit trail.

15. Report limitations and boundaries of inference honestly.

18. Future Scope

Future IT research will increasingly require methods capable of integrating heterogeneous, streaming, and adaptive data. Sampling may become dynamic, with designs responding to observed representation gaps while preserving inferential validity. Measurement systems will need to address algorithmic constructs whose definitions evolve as models are retrained. Advances in causal inference, privacy-preserving computation, multimodal measurement, and automated provenance may improve evidence quality, but they will also demand stronger theoretical and governance expertise. Doctoral researchers should therefore develop both statistical competence and critical understanding of how technological infrastructures shape the data they produce.

19. Chapter Summary

Sampling determines the cases to which a study can speak, measurement determines how constructs are represented, and data-quality practice determines whether observations are trustworthy and usable. Probability sampling supports defensible population inference, while non-probability strategies can be valuable for exploratory and hard-to-reach research. Operationalization requires clear construct definitions and appropriate indicators. Reliability is necessary but insufficient; validity requires converging theoretical and empirical evidence. Measurement invariance is essential for fair group comparisons. Missingness, provenance, and processing decisions must be treated as methodological issues. Ethical governance, transparency, and reproducibility are integral to high-quality IT research.

20. Key Terms

1. Accessible population: The portion of the target population that can practically be reached.

2. Cluster sampling: Selection of naturally occurring groups rather than isolated units.

3. Construct: An abstract concept represented through observable indicators.

4. Data provenance: Documentation of data origin, ownership, processing, and transformations.

5. Formative measure: A measure in which indicators collectively define a construct.

6. Measurement invariance: Evidence that a measure functions comparably across groups or occasions.

7. Missing at random: Missingness explainable by observed variables, conditional on the model.

8. Nonresponse error: Bias arising when participation differs systematically from nonparticipation.

9. Operationalization: Translation of a theoretical construct into measurable indicators and rules.

10. Probability sampling: Sampling with known, non-zero selection probabilities.

11. Purposive sampling: Deliberate selection of cases according to relevant criteria.

12. Reliability: Consistency or stability of measurement.

13. Sampling frame: The list or mechanism from which a sample is selected.

14. Sampling error: Difference between a sample estimate and the population value due to sampling.

15. Stratified sampling: Sampling separately within defined population subgroups.

16. Unit of analysis: Entity about which conclusions are drawn.

17. Validity: Credibility of interpretations and uses made from measurements.

18. Weighting: Adjustment of observations to reflect unequal selection or response probabilities.

21. Review Questions

Short Answer Questions

1. What is the difference between a target population and a sampling frame?

2. Why does reliability not establish validity?

3. Define measurement invariance and explain its importance.

4. What are the principal dimensions of data quality?

5. Distinguish formative and reflective measurement.

Long Answer Questions

1. Compare probability and non-probability sampling in relation to generalizability, cost, access, and research purpose.

2. Explain a complete process for developing and validating an IT research instrument.

3. Discuss the Total Survey Error framework and its relevance to digital research.

4. Explain MCAR, MAR, and MNAR and evaluate appropriate responses to each.

5. Discuss how provenance, reproducibility, and governance affect the credibility of IT research.

Analytical / Application Questions

1. Design a sampling and measurement strategy for studying cybersecurity behavior across a multinational organization. Justify the strata, sample-size approach, instrument, and validity checks.

2. A platform study reports large differences in technology trust between two demographic groups. Propose an analysis to determine whether the difference reflects true attitudes, measurement noninvariance, sampling bias, or missingness.

3. Develop a data-quality and provenance protocol for a study combining survey responses, application logs, and administrative records. Identify likely errors and controls for each stage.

22. Multiple Choice Questions with Answers

Question 1

Which sampling approach is most appropriate when a researcher wants defensible estimates for defined population subgroups?

A. Convenience sampling

B. Stratified probability sampling

C. Snowball sampling

D. Self-selection sampling

Correct Answer: B. Stratified probability sampling

Question 2

What does measurement invariance primarily assess?

A. Whether a sample is sufficiently large

B. Whether missing values are random

C. Whether an instrument functions comparably across groups

D. Whether data are stored securely

Correct Answer: C. Whether an instrument functions comparably across groups

Question 3

A measure in which indicators collectively form the meaning of a construct is called:

A. Reflective

B. Formative

C. Nomological

D. Criterion-based

Correct Answer: B. Formative

Question 4

Which practice best reduces the risk of data leakage in a time-dependent machine-learning evaluation?

A. Randomly mixing all historical and future observations

B. Removing all rare cases

C. Using time-based validation

D. Increasing questionnaire length

Correct Answer: C. Using time-based validation

Question 5

Which statement about reliability is correct?

A. Reliability alone proves construct validity

B. Reliability concerns consistency of measurement

C. Reliability eliminates sampling bias

D. Reliability guarantees measurement invariance

Correct Answer: B. Reliability concerns consistency of measurement

Question 6

When missingness depends on an unobserved value itself, the situation is most consistent with:

A. MCAR

B. MAR

C. MNAR

D. complete randomization

Correct Answer: C. MNAR

Question 7

Which document most directly records variable definitions, formats, permissible values, and derivations?

A. Data dictionary

B. Consent form

C. Power analysis

D. Sampling quota

Correct Answer: A. Data dictionary

Question 8

Why are cognitive interviews useful during instrument development?

A. They calculate statistical power

B. They reveal how participants interpret questions

C. They replace all validity testing

D. They guarantee representative sampling

Correct Answer: B. They reveal how participants interpret questions

Question 9

Which condition is generally needed before comparing latent means across groups?

A. Only convenience sampling

B. Scalar measurement invariance

C. A zero response rate

D. Strictly identical sample sizes

Correct Answer: B. Scalar measurement invariance

Question 10

What is a central purpose of data provenance documentation?

A. To conceal transformations from reviewers

B. To increase the number of observations automatically

C. To show where data came from and how they were changed

D. To replace informed consent

Correct Answer: C. To show where data came from and how they were changed