10. Nonparametric, Robust and Resampling Methods

Introduction and Context

Many statistical procedures are developed under assumptions about the population from which data arise. Common assumptions include normality, equal variances, independence, linearity, and the measurement of variables on an interval or ratio scale. These assumptions are useful because they make probability calculations tractable and often lead to powerful tests and efficient estimators. However, real data frequently contain outliers, skewed distributions, ordinal measurements, small samples, dependence, or complex sampling structures. In such situations, a classical method may produce misleading conclusions even when its calculations are technically correct.

This chapter develops three related responses to these difficulties. Nonparametric methods reduce reliance on a specified population distribution, often by replacing raw measurements with ranks or by constructing a reference distribution directly from the observed data. Robust methods seek estimates and procedures that remain stable when the data contain contamination, extreme observations, or departures from ideal assumptions. Resampling methods repeatedly reuse the available sample to approximate sampling distributions, standard errors, confidence intervals, or null distributions. Permutation tests, the bootstrap, and cross-validation are especially important in data science because they can be applied to statistics and predictive models for which simple theoretical formulas are unavailable.

These methods are not assumption-free. Every method depends on assumptions concerning independence, exchangeability, sampling, or the statistic being estimated. The central professional skill is therefore not to choose a “distribution-free” method automatically, but to match the method to the data-generating process, the research question, and the consequences of error.

Learning Objectives

After studying this chapter, the student should be able to:

1. Explain why nonparametric, robust, and resampling methods are needed.

2. Select and interpret rank-based tests for common study designs.

3. Distinguish robustness from distribution-free inference.

4. Construct and interpret permutation tests and bootstrap confidence intervals.

5. Explain the logic of cross-validation and identify sources of data leakage.

6. Evaluate assumptions, limitations, and ethical implications in practical applications.

7. Use software outputs critically rather than treating computational results as automatic evidence.

When Classical Assumptions Become Problematic

A normality assumption concerns the distribution of observations or, in regression, often the distribution of errors conditional on predictors. It does not mean that every variable in a data set must be normally distributed. Nevertheless, strong skewness, heavy tails, zero-inflation, censoring, or a small sample can make normal-theory procedures unreliable. A single extreme observation can substantially alter a mean, standard deviation, correlation, or regression coefficient.

Measurement scale also matters. A satisfaction response recorded as “very dissatisfied” through “very satisfied” is ordinal: the ordering is meaningful, but the numerical distance between adjacent categories need not be equal. Applying a t-test to such data may be defensible in some large-sample settings, but it should not be treated as automatically appropriate. Rank methods are often better aligned with the information actually present in the measurements.

A useful diagnostic sequence is:

• What is the estimand: a mean, median, probability of superiority, association, prediction error, or another quantity?
• Are observations independent, paired, clustered, or repeated over time?
• Are extreme values errors, valid rare cases, or evidence of a mixture of populations?
• Does the proposed method require exchangeability, symmetry, equal distributions, or only independence?
• Would a transformation, robust estimator, nonparametric method, or resampling approach best address the problem?

A method should be chosen because its assumptions fit the scientific question—not merely because a normality test has produced a small or large p-value.

Rank-Based Inference

Ranks and their interpretation

To rank observations, order them from smallest to largest and replace each value with its position. Tied values receive their average rank. Ranking discards some information about the distances between observations but preserves order and reduces the influence of extreme numerical values. A rank-based result commonly concerns whether one group tends to produce larger observations than another, rather than whether its arithmetic mean is larger.

The Mann–Whitney U test, also called the Wilcoxon rank-sum test, compares two independent groups. All observations are ranked together, and the ranks belonging to one group are summed. Under the null hypothesis of identical distributions, group labels are exchangeable. If the two distributions have a similar shape, the test is often interpreted as a test of location or median difference. Without that shape condition, it is more accurately interpreted as evidence that one group tends to yield larger values than the other.

For two paired or matched samples, the Wilcoxon signed-rank test ranks the absolute nonzero differences within pairs and uses their signs. It tests whether the distribution of paired differences is symmetric around zero under its usual interpretation. It is not simply a nonparametric substitute for every paired t-test: symmetry of differences is relevant, and zero differences affect the effective sample size.

For three or more independent groups, the Kruskal–Wallis test ranks all observations and compares the rank sums across groups. A significant result indicates that at least one group differs in distribution. It does not identify which groups differ; post hoc pairwise comparisons require multiplicity control, such as the Holm procedure. For repeated measurements across three or more conditions, the Friedman test ranks observations within each subject or block and tests whether conditions differ systematically.

Association through ranks

The Spearman rank correlation is the Pearson correlation calculated on ranks. It measures monotonic association: as one variable increases, the other tends to increase or decrease. It can detect a strong curved monotonic relationship that Pearson correlation may summarize poorly, but it will not necessarily detect a nonmonotonic relationship. Kendall’s tau is based on concordant and discordant pairs and often has a useful probabilistic interpretation, especially for ordinal data. Neither coefficient establishes causation, and both require attention to ties, dependence, and confounding.

Effect sizes and reporting

A p-value from a rank test does not communicate the size or practical importance of an effect. Reports should include a suitable effect size, confidence interval where available, sample sizes, the direction of the effect, and a description of the data. Examples include the Hodges–Lehmann estimate of a location shift, rank-biserial correlation, probability of superiority, or Kendall’s tau. A result such as “the probability that a randomly selected treated observation exceeds a randomly selected control observation is estimated at 0.68” is often more informative than a statement that the groups are statistically different.

Robust Estimation and Robust Models

A procedure is robust when its performance changes relatively little under plausible violations of its assumptions or the presence of a limited number of unusual observations. Robustness is not equivalent to being nonparametric. A robust regression model may be parametric, and a rank test can be sensitive to distributional differences even though it does not specify a normal population.

The median has a breakdown point of approximately 50 percent: nearly half the observations must be contaminated before it can be driven arbitrarily far. The mean has a breakdown point of zero because one arbitrarily large observation can change it without bound. The trimmed mean removes a fixed proportion of observations from both tails before averaging and can provide a useful compromise between the mean and median. The winsorized mean replaces extreme observations with less extreme retained values rather than deleting them.

The ordinary standard deviation is highly sensitive to outliers. The median absolute deviation (MAD) is a robust scale estimator based on the median of absolute deviations from the median. For approximately normal data, a consistency adjustment is often applied when using MAD as an estimate comparable to the standard deviation. The interquartile range (IQR), the difference between the third and first quartiles, is another robust measure of spread.

In regression, least squares minimizes squared residuals, so large residuals receive disproportionate influence. Alternatives include least absolute deviations, which minimizes absolute residuals; M-estimation, which uses a less rapidly increasing loss function; and quantile regression, which models conditional quantiles such as the median rather than the conditional mean. A robust method should not be used to hide genuine influential observations. Analysts must investigate whether an unusual value represents data error, fraud, a rare but important event, or a different subgroup.

Robustness can also be examined through sensitivity analysis. A responsible analysis may compare the mean and median, ordinary and robust regression, or results with and without a verified data error. If the substantive conclusion changes, that instability should be reported rather than concealed.

Permutation Tests: Building a Null Distribution from the Data

A permutation test evaluates a statistic by repeatedly rearranging labels or signs in ways permitted by the null hypothesis. Suppose two independent groups are compared using the difference in means, a difference in medians, or a rank statistic. Under the null hypothesis that group membership is exchangeable, the observed labels can be shuffled many times. For each shuffle, the statistic is recalculated. The resulting values form an empirical null distribution.

If the observed statistic is unusually extreme relative to this distribution, the null hypothesis is questioned. With B random permutations, a conservative Monte Carlo p-value is commonly calculated as

p = (1 + number of permuted statistics at least as extreme as the observed statistic) / (B + 1).

The added one avoids reporting a zero p-value merely because no sampled permutation was as extreme. If all possible permutations are enumerated, the test is exact conditional on the observed data and the exchangeability assumption. Random permutations provide an approximation whose precision improves with B.

The permutation scheme must reflect the design. In a paired experiment, signs of within-pair differences may be permuted rather than observations being freely mixed. In a blocked or clustered design, labels should be shuffled within appropriate blocks, or entire clusters should be resampled or permuted. Time-series observations generally cannot be freely permuted because temporal dependence destroys exchangeability. In machine learning, permutation tests must also preserve the train-test structure.

Permutation tests are attractive because the statistic can be chosen to match the practical question, including a complex model-performance difference. However, a permutation test does not prove that groups are identical in every respect; its validity depends on the null hypothesis and permissible rearrangements.

Bootstrap Inference

The bootstrap estimates sampling uncertainty by treating the observed sample as an approximation to the population. A nonparametric bootstrap repeatedly draws samples of size n with replacement from the observed n observations. For each resample, the statistic is computed. The empirical distribution of these bootstrap statistics approximates the sampling distribution.

The bootstrap can estimate a standard error, bias, or confidence interval for statistics such as a median, regression coefficient, prediction metric, or ratio. A basic bootstrap confidence interval is based on the estimated standard error, while a percentile interval uses bootstrap quantiles directly. The bias-corrected and accelerated (BCa) interval adjusts for bias and skewness and can perform better in some settings, although it requires careful implementation. A confidence interval is not a guarantee that the parameter lies inside the calculated range; it is a procedure intended to achieve a stated long-run coverage under relevant assumptions.

The bootstrap is not universally reliable. With a very small sample, the empirical population may not represent the population tails. It can perform poorly for extreme quantiles, maxima, non-smooth statistics, highly dependent observations, and parameters near boundaries. For time series, a block bootstrap preserves local dependence. For clustered data, resampling should generally occur at the cluster level. In stratified surveys, the design should be reflected in the resampling plan. A model-based or parametric bootstrap may be appropriate when a credible probability model is available.

A practical workflow is to define the statistic before resampling, choose a design-consistent resampling unit, generate a sufficiently large number of replicates, inspect the bootstrap distribution, calculate an appropriate interval, and perform sensitivity checks. The number of replicates reduces Monte Carlo error but cannot repair a biased sample or an inappropriate resampling design.

Cross-Validation and Resampling for Prediction

Cross-validation estimates how a predictive method may perform on unseen data. In k-fold cross-validation, observations are divided into k folds. The model is trained on k−1 folds and evaluated on the remaining fold; this is repeated until every observation has served as validation data. The average loss estimates predictive performance for data generated in a similar way. Repeated or stratified k-fold procedures can reduce variability or preserve class proportions.

Leave-one-out cross-validation uses n folds and trains n models, each leaving out one observation. It uses data efficiently but can be computationally expensive and may have high variance. A separate test set is preferable when sufficient data are available. In nested cross-validation, an inner loop selects hyperparameters and an outer loop estimates performance. This prevents optimistic bias caused by tuning a model on the same validation information used to report its performance.

Cross-validation is a data-analysis procedure, not merely a software option. All preprocessing that learns from data—imputation, scaling, feature selection, dimensionality reduction, and target encoding—must be fitted within each training fold. Otherwise, information from the validation fold leaks into the model. For time-dependent data, training must precede validation chronologically. For multiple records from the same person, all records should normally remain in one fold. The split must reflect the deployment setting.

Performance should be accompanied by uncertainty estimates and relevant metrics. Accuracy can be misleading in class-imbalanced problems; precision, recall, F1 score, area under the precision–recall curve, calibration, or cost-sensitive loss may be more appropriate. A small difference in cross-validated performance may not justify adopting a more complex or less interpretable model.

An Illustrative Case: Evaluating a Digital Rehabilitation Programme

Background and problem

A hospital introduces a mobile rehabilitation programme for patients recovering from knee surgery. Recovery time, measured in days until a clinically defined mobility milestone, is recorded for 34 patients using the programme and 31 patients receiving standard advice. The recovery times are right-skewed, and two programme users have unusually long recoveries because of unrelated complications. The research team wants to estimate whether the programme is associated with faster recovery and whether a predictive model can identify patients likely to require extended support.

Analysis

A comparison of means is strongly affected by the two long recoveries. The team first displays the distributions and reports medians, IQRs, and individual observations. Because the groups are independent and the scientific question concerns whether recovery tends to be faster, the Mann–Whitney test is suitable as a primary rank-based analysis. The team also reports a Hodges–Lehmann location estimate and a confidence interval obtained through a design-appropriate bootstrap. It does not describe the test as a pure median test without checking whether the group distributions have comparable shapes.

To assess the robustness of the conclusion, analysts compare the ordinary mean difference, a 20 percent trimmed mean difference, and a robust regression adjusted for age and baseline mobility. The two unusually long recoveries are clinically verified rather than deleted. Their inclusion is therefore part of the primary analysis, while an analysis excluding complications is clearly labelled as sensitivity analysis.

For prediction, the team develops a model for extended recovery using age, baseline mobility, comorbidities, and early app activity. All imputation, scaling, and feature selection occur inside the training folds. Because some patients contribute multiple app records, splitting is performed at the patient level. Nested cross-validation is used to tune the model and estimate performance. Calibration and recall are reported because missing a patient who needs additional support is more consequential than generating an extra follow-up alert.

Outcome and learning points

Suppose the rank-based analysis indicates a tendency toward shorter recovery in the programme group, while the mean-based estimate is less stable. The appropriate conclusion is not that the programme has been proven effective, but that the observed data provide evidence of an association whose magnitude should be interpreted with its interval and study design. If allocation was not randomized, unmeasured confounding remains possible.

The case illustrates five principles: identify the estimand; inspect the data before selecting a test; use robustness checks without deleting inconvenient observations; respect the sampling and dependence structure during resampling; and separate predictive validation from causal claims.

Limitations, Risks, and Professional Judgement

Nonparametric tests may have lower power than well-specified parametric tests and may answer a different question from the one originally intended. Rank procedures can lose information through ties and may be affected by unequal distributional shapes. Robust procedures can downweight observations that are scientifically important. A bootstrap cannot create information absent from a biased or tiny sample. Permutation tests fail when the proposed rearrangements violate exchangeability. Cross-validation can be optimistic when repeated measurements, temporal ordering, or preprocessing leakage is ignored.

Multiple testing is another important risk. Searching across many statistics, subgroups, model specifications, and resampling choices increases the chance of a seemingly impressive result by chance. The analysis plan should identify primary outcomes where possible, control multiplicity when appropriate, and disclose exploratory analyses. Reproducible code, fixed random seeds where practical, documentation of the resampling design, and reporting of sensitivity analyses improve credibility.

Ethically, robust methods must not be used to erase disadvantaged or unusual cases. Outliers may represent safety incidents, rare diseases, or groups poorly served by a system. In predictive analytics, subgroup performance should be assessed separately, and validation data must be protected from unauthorized use. Automated model selection does not remove responsibility for fairness, privacy, interpretability, or appropriate human oversight.

Emerging Computational Practice

Modern statistical software makes thousands of resamples and repeated model fits feasible. Parallel computing, automatic differentiation, and probabilistic programming support complex bootstrap and simulation workflows. Libraries for cross-validation can automate fold construction, but default settings may not understand clusters, time order, or leakage. Analysts should inspect the generated splits rather than assuming that a software default is valid.

Simulation is increasingly used to compare procedures under plausible data-generating conditions. For example, an analyst can simulate skewed distributions, contamination, unequal variances, or clustered observations and compare bias, coverage, power, and prediction error. Such simulations do not replace empirical evidence, but they make the consequences of methodological choices visible. Contemporary practice is moving toward transparent computational notebooks, preregistered analysis plans, and reporting that distinguishes sampling uncertainty from model uncertainty.

Best-Practice Workflow

A defensible workflow begins with the research question and estimand. Next, the analyst examines measurement scale, dependence, missingness, outliers, and the intended deployment or inference population. A method is selected by its assumptions and target, not by a desire to avoid all parametric structure. The primary analysis is complemented by an effect size and uncertainty interval, and important assumptions are checked through graphical and substantive diagnostics.

For resampling, the unit of resampling must match the design. Permutation schemes must preserve the null hypothesis; bootstrap schemes must preserve dependence and sampling structure; cross-validation splits must reproduce future use. Preprocessing and tuning must occur inside training data. Finally, analysts should report the number of replicates, randomization or fold procedure, performance metric, uncertainty method, sensitivity analyses, and limitations in language accessible to decision-makers.

Chapter Summary

Nonparametric methods use ranks or empirical reference distributions to reduce reliance on specified population forms. Mann–Whitney, Wilcoxon signed-rank, Kruskal–Wallis, Friedman, Spearman, and Kendall procedures address common independent, paired, multi-group, repeated-measures, and association questions. Their interpretation depends on the estimand, ties, distributional shape, and study design.

Robust statistics reduce sensitivity to outliers and assumption violations through medians, trimmed means, MAD, robust regression, and quantile regression. Robustness is distinct from being distribution-free, and unusual observations must be investigated rather than automatically removed. Permutation tests construct null distributions by valid rearrangements, while bootstrap methods approximate sampling uncertainty by resampling with replacement. Both require careful attention to exchangeability and dependence. Cross-validation estimates predictive performance, but leakage, inappropriate splitting, and poor metric selection can invalidate it. Across all methods, sound statistical judgement requires transparent assumptions, effect sizes, uncertainty, sensitivity analysis, and ethical attention to the consequences of decisions.

Key Terms

Rank: The position assigned to an observation after ordering the data.

Nonparametric method: A method that does not require a fully specified parametric population distribution.

Mann–Whitney U test: A rank-based test for comparing two independent groups.

Wilcoxon signed-rank test: A rank-based test for paired differences.

Kruskal–Wallis test: A rank-based test for comparing three or more independent groups.

Friedman test: A rank-based test for repeated or blocked measurements across several conditions.

Spearman correlation: Pearson correlation computed on ranks to measure monotonic association.

Kendall’s tau: An association measure based on concordant and discordant pairs.

Robustness: Relative stability of a procedure under departures from ideal assumptions.

Trimmed mean: A mean calculated after removing specified proportions from both tails.

Median absolute deviation: A robust scale measure based on absolute deviations from the median.

M-estimation: Robust estimation using a loss function designed to limit the influence of extreme residuals.

Permutation test: A test that constructs a null distribution by valid rearrangement of observations or labels.

Exchangeability: The condition that observations or labels may be rearranged without changing their joint distribution under a null hypothesis.

Bootstrap: Repeated sampling with replacement from observed data to approximate sampling uncertainty.

BCa interval: A bootstrap confidence interval adjusted for bias and acceleration.

Cross-validation: Repeatedly training and evaluating a model on separate data partitions.

Data leakage: Unauthorized transfer of information from validation or test data into model training.

Nested cross-validation: An outer validation loop combined with an inner tuning loop.

Probability of superiority: The probability that a randomly selected observation from one group exceeds one from another.

Review Questions

Short Answer Questions

1. Why can a rank-based method be preferable for ordinal data?

2. Distinguish robustness from being distribution-free.

3. What does exchangeability mean in a permutation test?

4. Why is the median less sensitive to extreme observations than the mean?

5. What is data leakage in cross-validation?

Long Answer Questions

1. Explain the assumptions, interpretation, and appropriate applications of the Mann–Whitney U, Wilcoxon signed-rank, Kruskal–Wallis, and Friedman tests.

2. Compare the bootstrap and permutation test, including their purposes, procedures, and assumptions.

3. Discuss robust location and scale measures and explain when each may be useful.

4. Describe how nested cross-validation prevents optimistic estimates of predictive performance.

5. Explain why dependence, clustering, and time ordering must influence a resampling design.

Analytical / Application Questions

1. A company compares customer waiting times under two service systems. The data are heavily right-skewed, contain several verified long waits, and are collected from the same branches over several days. Propose an analysis plan and justify the test, effect measure, robustness checks, and resampling strategy.

2. A classifier is tuned using feature selection before ten-fold cross-validation, producing an unusually high accuracy. Diagnose the methodological problem and redesign the validation workflow.

3. A hospital reports a significant Kruskal–Wallis result for four treatments but provides no pairwise comparisons or effect size. Explain what can and cannot be concluded, and recommend a complete reporting strategy.

22. Multiple Choice Questions with Answers

Question 1

Which interpretation is most appropriate for a Mann–Whitney U test when the two group distributions have different shapes?

A. It is always a test of equality of means.

B. It is evidence about whether one group tends to produce larger observations than the other.

C. It proves that the group medians are different.

D. It tests whether both groups are normally distributed.

Correct Answer: B. It is evidence about whether one group tends to produce larger observations than the other.

Question 2

Which procedure is most suitable for comparing three treatments measured repeatedly on the same participants?

A. Kruskal–Wallis test

B. Mann–Whitney U test

C. Friedman test

D. Independent permutation of all observations

Correct Answer: C. Friedman test

Question 3

Why is the median generally more robust than the mean?

A. It uses every numerical distance between observations.

B. It is unaffected by sample size.

C. It is less influenced by extreme observations.

D. It assumes the data are normally distributed.

Correct Answer: C. It is less influenced by extreme observations.

Question 4

In a permutation test comparing two independent groups, labels are shuffled because under the null hypothesis:

A. The observations are necessarily normally distributed.

B. Group labels are exchangeable under the specified null.

C. The sample sizes must be equal.

D. The statistic must be a mean.

Correct Answer: B. Group labels are exchangeable under the specified null.

Question 5

A bootstrap sample of size n is usually generated by:

A. Sampling n observations without replacement from a new population.

B. Permuting the original observations exactly once.

C. Sampling n observations with replacement from the observed sample.

D. Generating observations from a normal distribution regardless of the data.

Correct Answer: C. Sampling n observations with replacement from the observed sample.

Question 6

Which situation most clearly requires a special bootstrap design rather than an ordinary individual-observation bootstrap?

A. Independent observations with no grouping

B. A clustered sample in which several records come from each person

C. A data set with a numerical response

D. A data set containing a median

Correct Answer: B. A clustered sample in which several records come from each person

Question 7

A model’s feature scaling is fitted using the entire data set before cross-validation. This creates:

A. Underfitting

B. Data leakage

C. Exchangeability

D. A robust estimator

Correct Answer: B. Data leakage

Question 8

What is the principal purpose of nested cross-validation?

A. To guarantee zero prediction error

B. To remove the need for a test metric

C. To separate hyperparameter tuning from performance estimation

D. To ensure that all models are parametric

Correct Answer: C. To separate hyperparameter tuning from performance estimation

Question 9

Which statistic is generally most resistant to a single extremely large observation?

A. Arithmetic mean

B. Sample variance

C. Median

D. Least-squares slope

Correct Answer: C. Median

Question 10

A researcher obtains a significant Kruskal–Wallis test for four groups. What is the correct next conclusion?

A. Every pair of groups differs significantly.

B. At least one group differs, but further appropriate comparisons are needed to locate differences.

C. The group means are necessarily unequal.

D. The result proves a causal treatment effect.

Correct Answer: B. At least one group differs, but further appropriate comparisons are needed to locate differences.