Hypothesis Testing and Comparative Analysis

Introduction

Hypothesis testing is a disciplined method for using sample evidence to evaluate claims about a population. In human resource management, it supports decisions about recruitment methods, training effectiveness, employee satisfaction, absenteeism, compensation, turnover, and workplace interventions. Comparative analysis extends this logic by examining whether two or more groups, periods, or conditions differ in a statistically meaningful way.

A sample rarely provides certainty about a population. Instead, statistical testing quantifies how compatible observed evidence is with a stated assumption. Good hypothesis testing therefore combines mathematics with judgment: the researcher must define the question, select an appropriate test, assess assumptions, interpret uncertainty, and distinguish statistical significance from practical importance.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain null and alternative hypotheses, significance levels, p-values, and errors.
2. Select appropriate tests for means, proportions, variances, and group comparisons.
3. Perform and interpret one-sample, two-sample, paired-sample, and categorical tests.
4. Explain confidence intervals, effect sizes, statistical power, and practical significance.
5. Apply comparative analysis to HR decisions while recognizing ethical and methodological limitations.

1. The Logic of Statistical Hypothesis Testing

A statistical hypothesis is a claim about a population parameter. A parameter may be a population mean (μ), proportion (p), variance (σ²), or difference between parameters. A statistic, such as a sample mean (x̄), is calculated from observed data and used to learn about the parameter.

The conventional procedure begins with two competing statements:

• Null hypothesis (H₀): the reference claim, commonly stating no difference, no association, or no effect.
• Alternative hypothesis (H₁ or Hₐ): the claim supported if the evidence is sufficiently inconsistent with H₀.

For a training programme, a two-sided test might be:

H₀: μ_after - μ_before = 0

H_a: μ_after - μ_before ≠ 0

A one-sided alternative is justified only when a directional claim was specified before examining the data. For example, if management expects training to increase productivity:

H_a: μ_after - μ_before > 0

The test statistic measures how far the sample result is from the null value relative to sampling variability. A generic form is:

The standard error estimates the typical sampling fluctuation of the statistic. A large absolute test statistic indicates that the observed result is relatively unusual under H₀.

Significance Level and Decision Rule

The significance level, α, is the maximum probability of rejecting H₀ when H₀ is actually true. Common choices are 0.05 and 0.01. It must be selected before testing. If the p-value is less than or equal to α, the result is conventionally called statistically significant and H₀ is rejected. If p > α, H₀ is not rejected.

“Fail to reject” does not mean “prove true.” A non-significant result may reflect no real effect, a small effect, insufficient sample size, or noisy measurement.

p-Values

A p-value is the probability, assuming H₀ is true, of obtaining a result at least as extreme as the observed result in the direction specified by the alternative hypothesis. It is not the probability that H₀ is true, nor is it the probability that the finding occurred by chance in a general sense.

For a two-sided test, evidence in both tails of the sampling distribution is considered. For a one-sided test, only the prespecified relevant tail is considered. Choosing one-sided testing after seeing the data inflates the false-positive risk.

Type I and Type II Errors

A Type I error occurs when H₀ is rejected although it is true. Its long-run probability is controlled by α. A Type II error occurs when H₀ is not rejected although H₁ is true. Its probability is denoted β. Statistical power is:

Power increases when the true effect is larger, variability is smaller, sample size is larger, or α is less restrictive. Increasing power is important in HR research because failing to detect an effective intervention can be costly.

2. Sampling Distributions and Standard Errors

Hypothesis tests depend on the sampling distribution of a statistic. If independent observations are sampled from a population with mean μ and standard deviation σ, the sample mean has standard error:

where n is sample size. When σ is unknown, it is estimated by the sample standard deviation s:

The Central Limit Theorem states that, under broad conditions, the sampling distribution of x̄ becomes approximately normal as n increases, even if individual observations are not normally distributed. For small samples, especially when the population is plausibly normal, t-distributions are generally used.

A standardised score is:

when σ is known, and:

when σ is estimated. The t statistic follows a t-distribution with n − 1 degrees of freedom under the usual one-sample assumptions.

3. One-Sample Tests

One-Sample Mean Test

A one-sample test evaluates whether a population mean differs from a specified benchmark μ₀. The t statistic is:

Assumptions include independent observations, a quantitative outcome, and either an approximately normal population or a sufficiently large sample. Severe outliers can invalidate the result.

Worked Example: Average Absence

An organisation believes that average annual absence is 8 days. A random sample of 25 employees has x̄ = 9.2 days and s = 3 days. Test H₀: μ = 8 against Hₐ: μ ≠ 8 at α = 0.05.

Degrees of freedom are 24. The two-sided 5% critical value is approximately ±2.064. Since 2.00 is smaller than 2.064, H₀ is not rejected. The sample suggests higher absence, but it does not provide sufficient evidence at the 5% level to conclude that the population mean differs from 8 days.

A 95% confidence interval is:

Thus, the interval is approximately 7.96 to 10.44 days. The benchmark of 8 lies inside the interval, consistent with the test decision.

One-Sample Proportion Test

For a population proportion p, the sample proportion is p̂ = x ÷ n, where x is the number possessing the characteristic and n is sample size. Testing H₀: p = p₀ uses:

The normal approximation is generally appropriate when both np₀ and n(1 − p₀) are sufficiently large; exact binomial methods are preferable for small counts.

4. Comparing Two Groups

Comparative analysis asks whether an observed difference is larger than expected from sampling variation. The groups may be independent, such as employees in two departments, or paired, such as the same employees measured before and after training.

Independent Two-Sample Means

Let x̄₁ and x̄₂ be sample means, and let μ₁ and μ₂ be population means. The null hypothesis is often H₀: μ₁ − μ₂ = 0. The widely recommended Welch t-test does not assume equal population variances:

Here Δ₀ is the hypothesised difference, usually 0; s₁ and s₂ are sample standard deviations; and n₁ and n₂ are sample sizes. Welch degrees of freedom are estimated by the Welch–Satterthwaite formula:

The observations should be independent within and between groups. A quantitative outcome and reasonably well-behaved distributions are also required, particularly for small samples.

Paired-Sample t-Test

A paired test analyses within-person or matched differences. Define dᵢ = afterᵢ − beforeᵢ. Then test H₀: μ_d = 0 using:

where d̄ is the mean difference, s_d is the standard deviation of differences, and μd0 is usually 0. Pairing removes some person-to-person variation and may increase power, but the pairs must be meaningfully matched and the differences should be approximately normal.

Worked Example: Training Evaluation

Ten employees complete a skills assessment before and after training. Their mean improvement is d̄ = 6 points, with s_d = 8 points. Test whether the mean improvement differs from zero.

With 9 degrees of freedom, the two-sided 5% critical value is approximately 2.262. Since 2.37 exceeds 2.262, H₀ is rejected. The evidence supports a non-zero mean improvement. However, the result does not by itself establish that training caused the improvement; practice effects, workload changes, or external events may also contribute.

5. Tests for Categorical Data

HR data often consist of categories: promoted/not promoted, retained/left, or satisfied/unsatisfied. A chi-square test of independence examines whether two categorical variables are associated.

Expected frequency for cell i,j is:

The test statistic is:

where O is the observed frequency and E is the expected frequency. For an r × c table, degrees of freedom are:

The observations must be independent, categories mutually exclusive, and expected counts sufficiently large for the chi-square approximation. Fisher’s exact test is suitable for small 2 × 2 tables.

A significant association does not establish causation. It also does not indicate the strength or managerial importance of the relationship. Cramér’s V can describe association strength:

where N is total sample size.

6. Comparing More Than Two Means: ANOVA

Analysis of variance (ANOVA) tests whether three or more population means are equal. Its null hypothesis is H₀: μ₁ = μ₂ = ... = μ_k. The alternative is that at least one mean differs.

ANOVA partitions total variability into between-group and within-group variability:

The F statistic is:

where MSB = SSB ÷ (k − 1) and MSW = SSW ÷ (N − k). Here k is the number of groups and N is the total number of observations. Under H₀ and standard assumptions, F follows an F-distribution with k − 1 and N − k degrees of freedom.

ANOVA assumes independent observations, approximately normal errors within groups, and reasonably similar variances. If the omnibus test is significant, post hoc comparisons such as Tukey’s procedure can identify which groups differ while controlling the familywise error rate. Conducting many unadjusted t-tests increases the chance of false positives.

7. Confidence Intervals and Effect Sizes

A confidence interval gives a range of parameter values compatible with the data and method. For a mean difference, a general interval is:

A 95% confidence procedure produces intervals that contain the true parameter in approximately 95% of repeated samples under the model. It does not mean there is a 95% probability that one fixed interval contains the parameter.

Statistical significance depends strongly on sample size. Effect size addresses magnitude. For a standardised mean difference, Cohen’s d is:

where s_p is a pooled standard deviation when equal-variance assumptions are appropriate. A common pooled estimate is:

Rules such as 0.2, 0.5, and 0.8 for small, medium, and large effects are rough conventions, not universal standards. HR professionals should interpret effects in relation to cost, feasibility, fairness, and organisational objectives.

8. Choosing a Test

The choice must follow the research design and measurement scale, not merely the software menu. Transformations, robust methods, or nonparametric methods may be appropriate when assumptions are seriously violated. Nonparametric tests are not automatically tests of means; their interpretation often concerns ranks or distributions.

9. A Complete Testing Workflow

1. Define the decision problem. Specify the population, outcome, comparison, and practical threshold.
2. State hypotheses before analysis. Identify whether the test is one- or two-sided.
3. Plan sampling and power. Determine a sample size that can detect a meaningful effect.
4. Inspect the data. Check missing values, coding, outliers, distributions, and independence.
5. Select the method. Match the test to design, scale, and assumptions.
6. Calculate the statistic and p-value. Report degrees of freedom where applicable.
7. Report an interval and effect size. These are essential for interpretation.
8. Make a contextual decision. Statistical evidence is one part of managerial judgment.
9. Document limitations. Record exclusions, multiple tests, missing data, and deviations from the plan.

Multiple Comparisons and Reproducibility

When many hypotheses are tested, at least one small p-value may occur by chance. Researchers may control the familywise error rate using Bonferroni adjustment, α* = α/m, where m is the number of tests, or use false discovery rate procedures when many exploratory tests are conducted. Adjustments should be planned and transparently reported.

Reproducible analysis requires a clear data dictionary, versioned code or documented procedures, preserved outputs, and an audit trail. Statistical software calculates results efficiently but cannot correct biased sampling, poor measurement, or an inappropriate research question.

10. Illustrative HR Case: Evaluating a Flexible-Work Pilot

Background. An organisation introduces flexible working in one division. HR wants to know whether the pilot changes monthly absenteeism without disadvantaging employees in other divisions.

Problem. The pilot division has 48 employees and a comparison division has 52. Absence days are measured for three months before and three months after implementation. Because employees are observed repeatedly, a simple comparison of post-pilot means could be misleading.

Analysis. HR first examines the change for each employee and compares the mean changes between divisions using a Welch two-sample test. The null hypothesis is that the average change is equal in both divisions. HR also reports a confidence interval for the difference in changes and an effect size. It checks whether baseline absence, job roles, seasonal workload, and employee turnover differ between divisions.

If the divisions were not randomly assigned, the analysis is observational. A significant result may reflect pre-existing differences or simultaneous organisational changes. A stronger design would use a difference-in-differences model, comparing change over time in the pilot group with change over time in the comparison group, while checking the plausibility of parallel pre-intervention trends.

Approach and solution. HR combines quantitative results with interviews and reviews whether absence recording changed after the pilot. It avoids using absence results alone for individual disciplinary action. Results are reported in aggregate, with safeguards against identifying small groups.

Outcome. Suppose the estimated reduction is 0.8 absence days per employee, with a 95% confidence interval from 0.1 to 1.5 days and p = 0.03. The evidence is statistically significant at 5%, but management must assess whether a reduction of 0.8 days justifies implementation costs and whether effects differ fairly across caregiving responsibilities, disability status, job type, or gender.

Learning points. The case illustrates that design precedes testing, statistical significance is not the same as practical value, association is not automatically causation, and ethical HR analysis must consider privacy and distributive fairness.

11. Ethics, Governance, and Contemporary Practice

HR data may contain sensitive information about health, disability, ethnicity, pay, performance, or absence. Analysts should collect only necessary data, restrict access, protect confidentiality, and comply with applicable data-protection and employment law. Small-cell reporting can reveal identities even when names are removed.

Fairness requires checking whether an intervention or decision produces different effects across relevant groups. A statistically significant group difference may reflect discrimination, unequal opportunity, measurement bias, or legitimate job-related conditions; statistical testing alone cannot determine which explanation is correct.

AI and automated analytics can support power calculations, data validation, and pattern detection, but automated systems may reproduce historical bias. HR professionals should require explainability appropriate to the decision, human review, documented validation, and monitoring for drift. P-values should not be used as automatic hiring or termination rules.

Best practice includes preregistering major analyses where feasible, distinguishing confirmatory from exploratory work, reporting all important outcomes, avoiding p-hacking, and presenting estimates with intervals and effect sizes. A responsible report might state: “The estimated reduction was 0.8 days per employee (95% CI 0.1–1.5; p = 0.03), subject to non-random assignment and possible seasonal confounding.”

Challenges and Limitations

Hypothesis tests depend on assumptions that may be difficult to satisfy in organisational data. Employees are often clustered within teams, observations may be repeated, and missing data may be systematic. Ignoring clustering can make standard errors too small. Repeated testing, selective reporting, low power, and unreliable survey measures can produce misleading conclusions.

Statistical significance also varies with sample size: a trivial effect may be significant in a very large sample, while an important effect may be non-significant in a small sample. Managers should therefore consider confidence intervals, effect size, costs, risks, generalisability, and the quality of the research design.

Chapter Summary

Hypothesis testing evaluates population claims using sample evidence. The null hypothesis provides a reference model, while the alternative expresses the research claim. Test statistics compare an observed estimate with its null value relative to standard error. p-values measure compatibility with H₀, whereas α controls the long-run Type I error rate. Type II error and power explain the risk of missing genuine effects.

One-sample, independent two-sample, paired-sample, proportion, chi-square, and ANOVA procedures address different research designs. Confidence intervals and effect sizes complement p-values by showing uncertainty and practical magnitude. Correct analysis requires attention to independence, distributional assumptions, sample size, multiple comparisons, and measurement quality. In HRM, statistical evidence must be combined with causal reasoning, ethical governance, privacy protection, fairness analysis, and managerial judgment.

Key Terms

Hypothesis: A testable claim about a population or relationship.

Null hypothesis: The reference claim tested against sample evidence.

Alternative hypothesis: The claim supported when evidence is sufficiently inconsistent with the null.

Parameter: A numerical characteristic of a population.

Statistic: A numerical characteristic calculated from a sample.

Significance level: The chosen maximum probability of a Type I error.

p-value: Probability, under H₀, of results at least as extreme as the observed result.

Type I error: Rejecting a true null hypothesis.

Type II error: Not rejecting a false null hypothesis.

Statistical power: Probability of rejecting H₀ when a specified alternative is true.

Standard error: Estimated sampling standard deviation of a statistic.

Confidence interval: An interval produced by a method with a stated long-run coverage rate.

Welch t-test: A two-sample mean test that does not require equal variances.

Paired t-test: A test based on differences within matched observations.

Chi-square test: A test for categorical frequency discrepancies or association.

ANOVA: Analysis of variance for comparing three or more means.

Effect size: A measure of the magnitude of a difference or relationship.

Statistical significance: Evidence judged sufficiently inconsistent with H₀ under a chosen rule.

Practical significance: Importance of a result in real-world terms.

Cramér’s V: A standardised measure of association for contingency tables.

Review Questions

Short Answer Questions

1. Distinguish a null hypothesis from an alternative hypothesis.
2. What does a p-value mean, and what does it not mean?
3. Define Type I error and Type II error.
4. Why is a paired t-test appropriate for before-and-after employee measurements?
5. Why should effect sizes and confidence intervals accompany p-values?

Long Answer Questions

1. Explain the complete hypothesis-testing process, including assumptions, decision rules, and reporting.
2. Compare independent two-sample, paired-sample, and one-sample t-tests.
3. Explain the logic of ANOVA and why post hoc procedures may be required.
4. Discuss statistical significance, practical significance, power, and sample size in HR decision-making.
5. Evaluate the ethical and governance issues involved in testing employee data.

Analytical / Application Questions

1. An HR department claims that average recruitment time is 30 days. A random sample of 36 vacancies has a mean of 33 days and standard deviation of 9 days. Formulate a two-sided test, calculate the test statistic, and explain what additional information is required for a final decision.
2. A training study records performance for the same employees before and after training. Explain why analysing individual differences is preferable to treating the two samples as independent, and identify assumptions that should be checked.
3. An organisation compares promotion rates across four departments. Propose an appropriate categorical analysis, describe the expected-frequency calculation, and explain why a significant result would not by itself prove discriminatory intent.

Multiple Choice Questions with Answers

Question 1

Which statement correctly describes a p-value?

A. The probability that the null hypothesis is true

B. The probability of observing a result at least as extreme, assuming the null hypothesis is true

C. The probability that the alternative hypothesis is false

D. The proportion of observations that support the sample mean

Correct Answer: B. The probability of observing a result at least as extreme, assuming the null hypothesis is true

Question 2

If the significance level is α = 0.05 and the p-value is 0.02, what is the conventional decision?

A. Reject the null hypothesis

B. Accept the null hypothesis as proven

C. Increase the sample size automatically

D. Use a one-sided test instead

Correct Answer: A. Reject the null hypothesis

Question 3

Which test is most appropriate for comparing before-and-after scores from the same employees?

A. Independent two-sample t-test

B. Paired-sample t-test

C. Chi-square test of independence

D. One-proportion z-test

Correct Answer: B. Paired-sample t-test

Question 4

In a one-sample t-test, if x̄ = 54, μ₀ = 50, s = 10, and n = 25, what is the test statistic?

A. 0.40

B. 1.00

C. 2.00

D. 4.00

Correct Answer: C. 2.00

Question 5

What does statistical power represent?

A. The probability of rejecting a true null hypothesis

B. The probability of rejecting the null hypothesis when a specified alternative is true

C. The probability that a sample is normally distributed

D. The percentage of missing observations

Correct Answer: B. The probability of rejecting the null hypothesis when a specified alternative is true

Question 6

For a chi-square test of independence, the expected frequency in a cell equals:

A. The grand total divided by the number of cells

B. The row total multiplied by the column total, divided by the grand total

C. The observed frequency minus the sample mean

D. The column total divided by the row total

Correct Answer: B. The row total multiplied by the column total, divided by the grand total

Question 7

What is the primary purpose of ANOVA?

A. To compare three or more population means

B. To compare two population proportions only

C. To estimate one population variance without assumptions

D. To calculate a sample median

Correct Answer: A. To compare three or more population means

Question 8

Which statement about a non-significant result is correct?

A. It proves that there is no effect

B. It proves that the null hypothesis is true

C. It indicates insufficient evidence to reject the null under the selected procedure

D. It means the sample was collected incorrectly

Correct Answer: C. It indicates insufficient evidence to reject the null under the selected procedure

Question 9

Why are multiple unadjusted pairwise tests problematic?

A. They always reduce statistical power to zero

B. They increase the chance of at least one false-positive finding

C. They eliminate sampling variation

D. They make confidence intervals narrower without changing assumptions

Correct Answer: B. They increase the chance of at least one false-positive finding

Question 10

Which measure describes the magnitude of a categorical association in a contingency table?

A. Cramér’s V

B. The sample mean

C. The standard error of a mean

D. The t statistic

Correct Answer: A. Cramér’s V