Sampling, Measurement and Instrument Design

Introduction

A dissertation is only as credible as the evidence on which its conclusions rest. In human resource management (HRM), researchers rarely study every employee, applicant, manager or organisation in a population. They select a sample, measure abstract constructs such as engagement or perceived organisational support, and collect evidence through questionnaires, interviews, observations or records. Each decision can improve or weaken the quality of the study.

Sampling concerns who or what is included in a study. Measurement concerns how concepts are translated into observable indicators. Instrument design concerns the practical tools used to collect data. These activities are interdependent: a carefully designed questionnaire cannot compensate for a biased sample, and a representative sample cannot rescue an instrument that does not measure the intended construct.

This chapter develops a systematic approach to sampling, sample-size planning, operationalisation, questionnaire design, interview protocols and observation schedules. It also explains reliability, validity, bias and pilot testing, with particular attention to ethical HR research.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish populations, sampling frames, samples and units of analysis.
2. Select appropriate probability and non-probability sampling strategies.
3. Estimate a defensible sample size and explain its assumptions.
4. Operationalise HR constructs using dimensions and indicators.
5. Design questionnaires, interview protocols and observation tools.
6. Assess reliability, validity, bias and pilot-test findings.
7. Apply ethical and data-governance principles to recruitment and measurement.

1. The Logic of Sampling

The target population is the complete group to which the researcher wishes to generalise. For example, a study may target all full-time employees in medium-sized private hospitals in a country. The accessible population is the part of that population to which the researcher can realistically gain access. A sampling frame is the operational list from which the sample is drawn, such as an employee register.

The sampling unit is the element selected at a particular stage: an employee, department, branch or organisation. The unit of analysis is the entity about which conclusions are made. These may differ. A researcher may sample employees but analyse departments using aggregated responses. Confusing these levels can produce incorrect conclusions, especially when employees are nested within teams.

A sample should be judged by more than its size. Its quality depends on:

• coverage of the target population;
• the selection mechanism;
• the response rate and patterns of non-response;
• the suitability of the measurement instrument; and
• the match between the sample and the intended inference.

A large convenience sample may be less useful than a smaller, well-designed probability sample.

2. Probability Sampling Strategies

In probability sampling, every eligible population element has a known, non-zero probability of selection. This supports statistical generalisation and permits estimation of sampling error, although it does not automatically eliminate non-response or coverage problems.

Simple random sampling

Each member of the sampling frame has an equal chance of selection. A random-number generator can select employee identification numbers. The method is conceptually straightforward but requires a complete and accurate frame.

Systematic sampling

After selecting a random starting point, the researcher selects every kth element. If the population contains N elements and the desired sample is n, the sampling interval is:

Systematic sampling is efficient, but the list must not contain periodic patterns that coincide with k. For example, a payroll list arranged in repeating day and night shifts could create an unrepresentative sample.

Stratified random sampling

The population is divided into relevant subgroups, or strata, such as job level, location or employment status. A random sample is then taken within each stratum. Proportionate allocation is:

where nₕ is the sample from stratum h, n is the total sample, Nₕ is the size of stratum h, and N is the population size. Stratification improves representation when subgroups differ meaningfully. Disproportionate allocation may be justified when a small subgroup requires adequate analytical representation, but weights may then be needed when estimating population-level results.

Cluster sampling

The researcher samples naturally occurring groups, such as branches, teams or schools, and surveys members within selected clusters. It reduces travel and administrative cost but usually increases sampling error because members of the same cluster tend to resemble one another.

Multistage sampling

Selection occurs in stages, for example: regions, organisations, departments and employees. It is practical for dispersed populations, but each stage can introduce coverage and selection error. The sampling procedure must be documented clearly.

3. Non-Probability Sampling Strategies

Non-probability sampling does not give every population member a known selection probability. It is often appropriate for exploratory, qualitative or hard-to-reach research, but claims of statistical generalisation must be cautious.

• Convenience sampling selects accessible participants. It is inexpensive but vulnerable to systematic bias.
• Purposive sampling selects participants because they possess relevant experience, such as HR directors who implemented a new appraisal system.
• Quota sampling fills predetermined subgroup quotas without random selection within groups.
• Snowball sampling asks participants to refer others. It can reach hidden populations but may produce socially connected and homogeneous participants.
• Theoretical sampling is used in some qualitative designs, especially grounded theory, where subsequent cases are selected to develop emerging concepts.
• Volunteer sampling relies on self-selection. Participants may be especially interested, dissatisfied or motivated.

Qualitative sample adequacy is not normally determined by a statistical margin of error. Researchers consider information richness, diversity of perspectives, analytic purpose and thematic sufficiency—the point at which additional interviews provide little substantially new insight. This should be demonstrated rather than claimed mechanically.

4. Determining Sample Size

Sample-size planning should begin with the research question and proposed analysis. A descriptive survey, a comparison of two groups, a regression model and an interview study require different reasoning. Researchers should consider population size, expected variability, desired precision, confidence level, statistical power, design effect, subgroup analysis and anticipated non-response.

Cochran’s initial formula for a proportion

For a large population, an initial sample size for estimating a population proportion is:

where:

• n₀ = initial required sample size;
• Z = standard-normal value associated with the confidence level;
• p = anticipated population proportion;
• e = desired margin of error, expressed as a proportion.

For a 95% confidence level, Z is approximately 1.96. If no prior estimate of p is available, p = 0.50 is conservative because it produces the largest required sample for a given confidence level and precision. The formula assumes a probability sample, a sufficiently large population, independent observations and a valid estimate of a proportion.

Worked example. Suppose a researcher wants to estimate the proportion of employees reporting high engagement with 95% confidence and a ±5 percentage-point margin of error, with no prior estimate.

The result is rounded upward to 385 respondents. This is a precision-based estimate, not a universal rule for every HR study.

Finite population correction

If the population is not large relative to the initial sample, the finite population correction is appropriate:

where N is the population size. If the population is 1,000 and n₀ = 384.16:

The required sample is therefore rounded upward to 278 completed responses, subject to design and non-response adjustments.

Non-response adjustment

If the anticipated response rate is r, the number invited can be estimated as:

For 278 completed responses and an anticipated response rate of 70%:

The researcher should invite at least 398 eligible people, allowing for screening failures where relevant. This formula assumes that the response-rate estimate is reasonably realistic; increasing invitations does not correct non-response bias if responders differ systematically from non-responders.

Power-based planning

For hypothesis tests, sample size should also reflect statistical power, commonly targeted at 0.80 or higher, the significance level, expected effect size and number of predictors or groups. Power calculations should be conducted using an appropriate method or software before data collection. Rules such as “ten respondents per variable” can provide rough orientation but are not substitutes for a design-specific power analysis.

Cluster samples require additional respondents because observations within clusters are correlated. A common design-effect expression is:

where m is the average number of respondents per cluster and ρ is the intraclass correlation coefficient. The effective sample size is approximately the achieved sample divided by DEFF. This expression assumes roughly equal cluster sizes and a correlation structure suited to the design.

5. From HR Constructs to Measurable Variables

Many HRM concepts are latent constructs: they cannot be observed directly but are inferred from indicators. Examples include job satisfaction, psychological safety, organisational commitment, perceived fairness and leadership support. Operationalisation is the process of specifying what a construct means in the study and how it will be measured.

A strong operational definition includes:

1. the conceptual meaning of the construct;
2. its dimensions, if multidimensional;
3. observable indicators;
4. the response scale or coding system;
5. the source of items or observations; and
6. the method of scoring and interpretation.

Consider “training transfer climate.” Conceptually, it may refer to employees’ perception that supervisors, peers and organisational systems support the application of training. Dimensions could include supervisor support, peer support and opportunity to use new skills. Indicators might ask whether supervisors allow employees to practise newly learned behaviours and whether performance systems recognise their use.

Researchers must distinguish formative and reflective measurement. In a reflective model, the construct is presumed to cause responses to items; highly similar items are expected to correlate. In a formative model, indicators combine to form the construct; removing an indicator may change its meaning, and high inter-item correlation is not necessarily expected. This distinction affects item selection and reliability analysis.

A conceptual framework should identify independent, dependent, mediating, moderating and control variables. For example, supervisor support may influence engagement indirectly through psychological safety, while job level may moderate the relationship. The framework should be theoretically justified rather than created solely from available questionnaire items.

6. Designing Questionnaires

A questionnaire should be purposeful, readable and economical. Begin with the research questions and a measurement blueprint. Map every item to a construct, dimension and planned analysis. Remove items that do not contribute to the research objectives.

Item-writing principles

Good items:

• address one idea at a time;
• use language familiar to respondents;
• specify a clear time frame where needed;
• avoid unnecessary technical terminology;
• avoid assumptions about experience;
• do not imply a desirable answer;
• provide balanced response options; and
• distinguish “not applicable” from a substantive response when appropriate.

Avoid double-barrelled items such as “My manager is supportive and fair,” because respondents may agree with one part but not the other. Avoid double negatives, vague words such as “often” without a defined reference period, and emotionally loaded wording.

A typical attitudinal item may use a five- or seven-point agreement scale. A frequency scale should use frequency anchors, such as “never” to “very often,” rather than agreement anchors. A knowledge test requires objectively scored answers and should not be disguised as an attitude scale.

Question order should usually move from simple, non-threatening questions to substantive items, followed by sensitive demographics. Screening questions must appear early. Instructions should explain the purpose, estimated completion time, voluntary nature of participation and how to request assistance.

Reverse-worded items are sometimes used to reduce acquiescence, but excessive or confusing reversal can create method effects and respondent error. If used, they must be clear and correctly recoded before analysis.

Questionnaire structure

A practical structure is:

1. participant information and consent;
2. eligibility and screening;
3. core construct measures;
4. behavioural or factual questions;
5. relevant demographic and employment variables; and
6. optional open-ended comments.

Online questionnaires should be mobile-compatible, accessible, secure and configured to prevent accidental collection of unnecessary identifiers.

7. Designing Interview Protocols

An interview protocol provides consistency while preserving the depth of qualitative inquiry. It should include the study purpose, eligibility criteria, opening script, consent procedure, principal questions, probes, closing script and data-security instructions.

Open questions invite participants to describe experience: “Can you describe how performance feedback is provided in your team?” Probes clarify meaning: “What happened next?” “How did that affect your work?” “Can you give a recent example?” Questions should not presume that a policy is effective or that a participant agrees with management.

A semi-structured protocol normally contains fewer, broader questions than a questionnaire. The interviewer should listen actively, avoid leading participants, manage their own reactions and distinguish participant language from interpretation. Sensitive HR topics require careful handling. Participants should know that declining a question will not affect employment, and researchers should avoid promising absolute confidentiality where legal or organisational limits exist.

Interviews may be recorded only with explicit consent. Transcripts should remove direct identifiers and indirect identifiers where re-identification is plausible. A reflexive log can document the interviewer’s assumptions, relationship with the organisation and changes made during data collection.

8. Designing Observation Tools

Observation is useful when behaviour, workflow or workplace interaction is central to the research question. An observation schedule should specify:

• setting, date and duration;
• behaviours or events to observe;
• operational definitions and inclusion rules;
• frequency, duration or sequence coding;
• contextual notes;
• observer identity and training; and
• procedures for unusual or sensitive events.

For example, “manager provides feedback” should be defined in observable terms, such as a manager giving information about performance linked to a specific task. “Manager is supportive” is an interpretation and requires indicators or a separate rating rubric.

Structured observation improves consistency but may miss unexpected events. Unstructured field notes provide richness but require transparent analytic procedures. In either form, observers should minimise disruption and comply with workplace privacy requirements.

9. Reliability

Reliability concerns consistency and dependability of measurement. It is necessary but not sufficient for validity.

Internal consistency

Cronbach’s alpha is commonly reported for multi-item scales. For k items, it can be represented as:

where σᵢ² is the variance of item i and σₜ² is the variance of the total score formed from the items. Alpha assumes, approximately, that items measure one construct with suitable covariance and that scoring is coherent. A high alpha may result from many repetitive items and does not prove unidimensionality. Values around 0.70 are often treated as a rough starting point in applied research, but acceptable levels depend on the construct, purpose, number of items and stage of development. Alpha should be interpreted with item-total correlations and dimensionality evidence, not used mechanically.

Other forms include test–retest reliability, inter-rater reliability and agreement statistics. For categorical ratings, Cohen’s kappa may be relevant; for continuous ratings, intraclass correlation may be appropriate. The statistic must match the data and design.

10. Validity and Bias

Validity is the degree to which evidence supports the intended interpretation and use of scores. It is not a permanent property of an instrument independent of context.

• Content validity asks whether items adequately cover the construct domain. Expert review and a construct blueprint are useful.
• Face validity concerns whether the instrument appears relevant to respondents. It may support acceptability but is weak evidence by itself.
• Construct validity examines whether scores behave as theory predicts. Convergent evidence involves association with related constructs; discriminant evidence involves distinction from unrelated constructs.
• Criterion-related validity examines association with a relevant outcome or criterion, such as linking a selection test with later performance, while recognising that the criterion itself must be credible.
• Internal validity concerns whether the study design supports causal claims. Confounding, history, maturation and selection can threaten it.
• External validity concerns transfer or generalisation beyond the study sample and setting.

Common sources of bias include:

• coverage bias: some population members are absent from the frame;
• selection bias: inclusion probabilities differ in a systematic way;
• non-response bias: responders differ from eligible non-responders;
• social desirability bias: participants report socially approved views;
• recall bias: memory is incomplete or reconstructed;
• acquiescence bias: respondents tend to agree;
• common-method bias: predictor and outcome measures collected from the same source and method create artificial association;
• interviewer or observer bias: researcher expectations influence interaction or coding; and
• survivorship bias: only those remaining in the organisation are observed.

Mitigation may include probability sampling, anonymous collection, neutral wording, multiple data sources, temporal separation, objective records, interviewer training, standardised coding and transparent reporting. No single remedy eliminates all bias.

11. Pilot Testing

A pilot is a small-scale trial of the recruitment, instrument and data procedures. It is not merely a way to obtain a preliminary result. A pilot should test comprehension, completion time, item variability, technical operation, consent procedures, skip patterns, interview flow, recording and data export.

Cognitive interviewing can reveal how participants interpret questions. Participants are asked to explain what they thought an item meant and how they selected an answer. Expert review can identify missing content, inappropriate assumptions and cultural or linguistic problems.

Pilot data may be examined for:

• missing responses and problematic skip patterns;
• items with nearly identical answers from everyone;
• confusing or frequently queried wording;
• excessive completion time;
• unexpected floor or ceiling effects;
• preliminary internal consistency; and
• evidence that respondents interpret items differently from the intended construct.

A low alpha does not automatically justify deleting an item. The researcher should inspect wording, dimensionality, response distributions and theoretical coverage. Deleting items solely to raise alpha may narrow the construct and weaken content validity. Changes after piloting should be documented, and if substantial, the revised instrument may require another pilot.

12. Applied Case: Employee Voice in a Growing Technology Firm

Background

This illustrative case concerns “NovaApps,” a technology firm with 1,200 employees across three offices. Senior management believes that employees hesitate to raise concerns, but no systematic evidence exists. A dissertation student proposes to study psychological safety and employee voice.

Problem

A convenience sample of volunteers would be easy to obtain through an internal email, but employees who feel strongly may be more likely to respond. The topic is sensitive because employees may fear that criticism will affect promotion.

Analysis and approach

The student defines the target population as all employees with at least three months’ service. The HR register is used as a sampling frame after removing contractors who are outside the study scope. To ensure representation, employees are stratified by office and job level, with proportionate random selection. Because the population is 1,200, the finite-population calculation indicates that approximately 300 completed responses are needed for the planned precision assumptions. Anticipating a 60% response rate, approximately 500 eligible employees are invited.

Psychological safety is operationalised through items concerning whether employees can raise mistakes, ask for help and disagree respectfully. Employee voice is measured through items concerning constructive suggestions and speaking up about risks. The blueprint distinguishes the two constructs and includes an open-ended question for examples.

Before launch, HR experts and two academic reviewers assess content coverage. Six employees complete cognitive interviews. They identify that “I can challenge decisions” is interpreted differently by managers and junior staff; the item is revised to specify respectful discussion of work-related decisions. The pilot with 35 employees shows that one reverse-worded item causes confusion, so it is removed rather than retained merely for balance.

To reduce social desirability, the survey is administered by the university researcher, not line managers. The invitation states that managers will receive only aggregated results and that small subgroup results will not be reported where identification is possible. A separate interview phase purposively selects employees from different offices and job levels, including those who reported both high and low psychological safety. Interviews explore examples rather than asking participants to repeat survey scores.

Outcome

The final study reports the sampling frame, selection procedure, response rate, missing data, scale construction, reliability evidence and limitations. Survey results are interpreted as evidence of association, not proof that psychological safety causes voice. Interview accounts help explain mechanisms and organisational context but are not presented as statistically representative.

Learning points

The case demonstrates that sampling, operationalisation, instrument design and ethics must be planned together. Independence from management can improve candour, but it must be accompanied by secure data handling. Mixed methods can be valuable when quantitative patterns require contextual explanation, provided the researcher does not treat the two forms of evidence as interchangeable.

13. Ethics, Governance and Contemporary Practice

HR research frequently involves employment records and power relationships. Ethical practice requires informed consent, voluntary participation, minimisation of harm, confidentiality, secure storage and a clear retention and deletion plan. Researchers should obtain institutional ethics approval where required and comply with applicable data-protection law and organisational policy.

Do not collect identifiers “just in case.” If linking survey responses to performance records is necessary, explain the linkage, limit access and separate identifying information from research data using a coded key. Small-cell reporting can reveal individuals even when names are removed. Researchers should assess re-identification risk, particularly in small departments or specialised roles.

Digital survey platforms, automated transcription and AI-assisted coding can improve efficiency, but they introduce risks involving data transfer, vendor retention, inaccurate transcription, fabricated or overconfident coding and confidentiality. Sensitive employee data should not be entered into an AI tool unless governance, contractual protection and informed consent permit it. AI-generated coding must be checked against source data, and the researcher remains responsible for methodological decisions.

Contemporary instruments increasingly use mobile delivery, multilingual versions, accessible design and mixed-mode collection. Translation should use a documented process, such as independent translation, review and reconciliation, rather than literal word substitution. Measurement equivalence should be considered when comparing language or cultural groups; a scale that functions differently across groups may produce misleading comparisons.

Best-Practice Checklist

Before data collection, the researcher should be able to answer yes to the following questions:

• Is the target population and unit of analysis clearly defined?
• Is the sampling frame suitable and sufficiently complete?
• Does the sampling strategy match the inference being claimed?
• Is the sample-size rationale linked to the analysis and response rate?
• Is each construct operationally defined and theoretically justified?
• Are questionnaire, interview and observation items clear and non-leading?
• Have content, construct and criterion considerations been addressed?
• Are reliability statistics appropriate to the scale and design?
• Have likely sources of bias been anticipated and mitigated?
• Has the instrument been piloted and revised systematically?
• Are consent, confidentiality, data security and reporting safeguards in place?

Chapter Summary

Sampling determines which cases contribute evidence and whether conclusions can reasonably extend beyond them. Probability strategies support statistical inference when coverage and response are adequate; non-probability strategies are often valuable for exploratory, qualitative and hard-to-reach research but require restrained claims. Sample size should reflect precision, power, population size, design effects and anticipated non-response rather than a universal rule.

Operationalisation translates abstract HR constructs into dimensions, indicators and scoring procedures. Questionnaires require clear, single-focus items and appropriate response scales. Interview protocols should combine consistency with probing depth, while observation tools require observable definitions and coding rules. Reliability addresses consistency; validity concerns the defensibility of score interpretations and research conclusions. Both require evidence and contextual judgement. Bias can arise at every stage, and pilot testing provides an opportunity to identify comprehension, technical, ethical and measurement problems before the main study. Sound HR research integrates methodological quality with respect for employee autonomy, privacy and dignity.

Key Terms

Target population: The complete group to which the researcher intends to generalise.

Accessible population: The portion of the target population realistically available to the researcher.

Sampling frame: The list or system from which a sample is selected.

Sampling unit: The entity selected at a particular stage of sampling.

Unit of analysis: The entity about which conclusions are drawn.

Probability sampling: Sampling in which selection probabilities are known and non-zero.

Non-probability sampling: Sampling without known selection probabilities for all eligible units.

Stratified sampling: Sampling separately within defined population subgroups.

Cluster sampling: Sampling naturally occurring groups rather than, or before, individuals.

Operationalisation: Translation of an abstract concept into measurable indicators and procedures.

Latent construct: A concept not directly observed but inferred from indicators.

Reflective measure: A measure in which the construct is theorised to cause item responses.

Formative measure: A measure in which indicators combine to define or form the construct.

Reliability: Consistency or dependability of measurement.

Validity: Evidence supporting the intended interpretation and use of measurements.

Sampling bias: Systematic distortion caused by how cases enter or remain in a sample.

Common-method bias: Artificial association caused by measuring variables with the same method or source.

Pilot test: A preliminary trial of an instrument and data-collection procedure.

Thematic sufficiency: A point at which additional qualitative data yield little substantially new insight.

Review Questions

Short Answer Questions

1. Distinguish a target population from a sampling frame.
2. What is the main advantage of stratified random sampling?
3. Define operationalisation and give one HRM example.
4. Why does reliability not by itself establish validity?
5. State two purposes of pilot testing.

Long Answer Questions

1. Compare probability and non-probability sampling, including appropriate uses and limitations of each.
2. Explain how a researcher could determine and justify the sample size for an employee survey.
3. Describe the process of operationalising a latent HR construct.
4. Discuss principles for designing a high-quality questionnaire and semi-structured interview protocol.
5. Explain major forms of validity and how bias can threaten HR research conclusions.

Analytical / Application Questions

1. An organisation has 2,000 employees distributed across four branches and three job levels. Design a sampling strategy for a survey comparing job levels, and explain how you would protect subgroup representation.
2. A pilot questionnaire has high internal consistency, but employees report that several items are confusing and repetitive. Analyse what the researcher should investigate before revising the instrument.
3. A study finds that employees who report high engagement also report high supervisor support, using one anonymous questionnaire completed at one time. Evaluate the conclusion that supervisor support causes engagement and propose a stronger design.

Multiple Choice Questions with Answers

Question 1

Which sampling method divides a population into relevant subgroups and then randomly samples within each subgroup?

A. Cluster sampling

B. Stratified random sampling

C. Snowball sampling

D. Convenience sampling

Correct Answer: B. Stratified random sampling

Question 2

In the formula k = N ÷ n, what does k represent?

A. The confidence coefficient

B. The response rate

C. The sampling interval

D. The population proportion

Correct Answer: C. The sampling interval

Question 3

Using Cochran’s initial formula with 95% confidence, p = 0.50 and e = 0.05 produces an initial sample of approximately:

A. 100

B. 196

C. 278

D. 385

Correct Answer: D. 385

Question 4

Which statement best describes operationalisation?

A. Selecting only participants who volunteer

B. Translating a construct into dimensions, indicators and measurement procedures

C. Removing all items with missing responses

D. Increasing the sample until results are significant

Correct Answer: B. Translating a construct into dimensions, indicators and measurement procedures

Question 5

Which questionnaire item is double-barrelled?

A. My supervisor gives feedback about my work.

B. I receive training at least once a year.

C. My manager is supportive and fair.

D. I can ask for help when needed.

Correct Answer: C. My manager is supportive and fair.

Question 6

What does Cronbach’s alpha primarily assess?

A. Population coverage

B. Internal consistency among items

C. Causal validity

D. Interview saturation

Correct Answer: B. Internal consistency among items

Question 7

Which is an example of non-response bias?

A. A question uses unfamiliar technical language

B. Employees who do not respond differ systematically from those who respond

C. Two interviewers code the same behaviour differently

D. A sample includes every fifth name on a list

Correct Answer: B. Employees who do not respond differ systematically from those who respond

Question 8

A researcher selects HR managers because they have direct experience of implementing a new appraisal system. This is:

A. Purposive sampling

B. Simple random sampling

C. Systematic sampling

D. Proportionate stratification

Correct Answer: A. Purposive sampling

Question 9

Which practice best reduces social desirability pressure in a sensitive employee survey?

A. Allowing line managers to collect completed forms

B. Using leading questions

C. Providing confidentiality safeguards and neutral wording

D. Reporting individual responses to senior management

Correct Answer: C. Providing confidentiality safeguards and neutral wording

Question 10

What is the principal purpose of cognitive interviewing during a pilot?

A. To calculate the final population size

B. To examine how participants understand and answer items

C. To guarantee a high response rate

D. To replace ethical review

Correct Answer: B. To examine how participants understand and answer items