Sampling, Partitioning and Analytical Dataset Design

Introduction

Data preparation is not merely the mechanical cleaning of records. It is also the design of the evidence on which an analysis or model will depend. A dataset may contain millions of observations and still be misleading if some populations are systematically absent, if repeated individuals occur in multiple partitions, or if the outcome is observed only for selected cases. Sampling, partitioning, weighting, resampling and dataset design therefore belong to the methodological core of data science.

Sampling determines which units are observed and how observations represent a target population. Partitioning determines which observations are used to learn, tune and evaluate a model. Analytical dataset design determines how rows, variables, labels, time periods and exclusions are organised so that the resulting analysis is valid, reproducible and operationally useful.

A central distinction is between the target population, about which conclusions are desired, and the accessible population, from which observations can actually be obtained. The difference between these populations is a major source of selection bias.

Learning Objectives

By the end of this chapter, a student should be able to:

• distinguish probability and non-probability sampling designs;
• calculate inclusion probabilities, sampling weights and basic estimators;
• design stratified, clustered and systematic samples;
• create leakage-resistant training, validation and test partitions;
• diagnose and address class imbalance;
• explain selection bias, non-response and covariate shift;
• design model-ready analytical datasets with appropriate grain, labels and metadata;
• evaluate sampling and partitioning decisions ethically and statistically.

1. The Sampling Problem

Suppose a population contains N units, indexed by i = 1, …, N, and a sample contains n observed units. A sampling frame is the operational list or mechanism from which units are selected. A frame may be a customer register, a patient list, a transaction table or a web-recruitment platform. It may omit eligible units, include duplicates, or contain outdated information.

The sampling unit is the entity selected at a particular stage. It may be a person, household, store, account, event or geographical area. The unit of analysis is the entity represented by one analytical row. These are not always identical. For example, households may be sampled, while individuals within households become units of analysis.

A sample is representative not because it resembles the population on every visible characteristic, but because the sampling design and subsequent adjustments support valid inference for the intended population. Representativeness is therefore a design and estimation property, not merely a visual impression.

1.1 Probability and non-probability sampling

In probability sampling, every eligible unit has a known, non-zero probability of selection. Let πᵢ denote the inclusion probability of unit i:

The probability may differ between units. In simple random sampling without replacement, every unit has the same probability:

Probability sampling supports design-based estimation because the selection mechanism is known. In non-probability sampling, inclusion probabilities are unknown or not controlled, as in convenience, quota, purposive and voluntary-response samples. Such samples can be useful for exploratory work, rapid experimentation or populations without a usable frame, but they do not automatically support unbiased population estimates.

1.2 Sampling weights

A basic design weight is the inverse of the inclusion probability:

A unit selected with probability 0.10 represents approximately ten population units under the sampling design. For a population total of a variable y, the Horvitz–Thompson estimator is:

Here, s is the realised sample, yᵢ is the observed value for unit i, and πᵢ is known and non-zero. The estimator is design-unbiased under the specified probability-sampling design when the relevant inclusion probabilities are correctly defined.

For a population mean, a commonly used ratio or Hájek estimator is:

The numerator is a weighted sum and the denominator is the sum of weights, so units of the estimator remain the same as units of y. In practical survey work, weights may be adjusted for non-response and calibrated to known population totals. Very large weights can produce unstable estimates; trimming may reduce variance but introduces a bias–variance trade-off.

Worked example: stratified weighting

A firm samples customers from two regions. Region A contains 8,000 customers and Region B contains 2,000. It selects 80 customers from A and 60 from B. The inclusion probabilities are:

Region B is oversampled to obtain more precise regional estimates. If the sample mean purchase value is £40 in A and £70 in B, the population-weighted mean is:

The unweighted sample mean, (80×40 + 60×70)/140 ≈ £52.86, overemphasises Region B. This illustrates why the analytical objective determines whether weighting is required.

2. Stratification, Clustering and Resampling

2.1 Stratification

A stratum is a mutually exclusive subgroup defined before sampling. Good strata are internally similar with respect to important outcomes and collectively cover the target population. Common stratification variables include geography, age band, customer tier, institution type and disease status.

Stratified sampling can guarantee sufficient observations in small but important groups. Proportional allocation selects in proportion to stratum size. Disproportional allocation deliberately oversamples small or high-priority strata, followed by weighting when population-level estimates are required.

Stratification must be based on variables available before the outcome is measured. Using a future outcome to construct strata would cause leakage and invalidate evaluation.

2.2 Cluster sampling

In cluster sampling, naturally occurring groups such as schools, hospitals or stores are selected, and units within selected clusters are observed. Observations in the same cluster tend to be correlated. If m observations per cluster have intra-cluster correlation ρ, a rough design effect is:

The formula assumes approximately equal cluster sizes and a common intra-cluster correlation. When ρ > 0, the effective information is lower than that of m independent observations. Cluster structure must therefore be retained in standard errors, cross-validation and deployment evaluation.

2.3 Resampling

Resampling creates new samples from observed data. The bootstrap samples n observations with replacement from n observed units and estimates sampling variability by repeating the procedure. It is useful for confidence intervals and model stability, but it does not repair systematic selection bias in the original data.

The jackknife repeatedly omits one observation or one cluster and recalculates an estimate. It can diagnose influence and approximate variance. In predictive modelling, resampling includes random undersampling, oversampling and synthetic methods such as SMOTE. These must be applied within the training data only; applying them before splitting can duplicate or interpolate information into validation and test sets.

3. Selection Bias and Missing Representation

Selection bias occurs when inclusion in the analytical dataset is related to the outcome, predictors, or both, in a way that changes the relationship of interest. Important mechanisms include:

• coverage error: eligible units are absent from the frame;
• non-response bias: selected units do not provide data, and responders differ from non-responders;
• voluntary-response bias: participation is driven by motivation or dissatisfaction;
• survivorship bias: only units that remain active or observable are included;
• collider bias: conditioning on a variable influenced by two other variables creates a misleading association;
• label-selection bias: outcomes are recorded only for cases receiving a test, review or intervention.

Increasing sample size does not eliminate these problems. A very large biased sample can provide a highly precise estimate of the wrong quantity.

3.1 Adjustment strategies

Possible responses include improving the frame, increasing response rates, collecting auxiliary variables, post-stratifying, calibrating weights, modelling response propensity and conducting sensitivity analyses. A response propensity model estimates the probability that a selected unit responds, conditional on observed variables. Inverse-propensity weighting may then be used, but it relies on a strong assumption: after conditioning on observed variables, response is independent of the outcome or estimand-relevant information. Unmeasured differences remain a threat.

For predictive modelling, the relevant question is often not “Is the sample representative of the historical population?” but “Does the evaluation distribution resemble the future deployment distribution?” This requires explicit definition of the deployment population and prediction time.

4. Designing Training, Validation and Test Partitions

A partition is a mutually exclusive subset of data assigned a specific role:

• training data estimate model parameters;
• validation data support model selection, hyperparameter tuning, threshold choice and feature decisions;
• test data provide a final, minimally used estimate of generalisation.

A test set is not a ceremonial requirement. If analysts repeatedly inspect test performance and modify the model, the test set becomes part of the development process and is no longer an unbiased final evaluation.

4.1 Random and stratified splitting

For independent, identically distributed observations, random splitting may be appropriate. When the outcome is imbalanced, stratified splitting preserves approximate class proportions in each partition. If yᵢ is a class label, stratification seeks similar estimates of:

where n_k is the number of observations in class k and n is the total number of observations.

However, stratification is not a substitute for independence. If several rows belong to one patient, customer or device, all rows from that entity should normally be placed in one partition. Otherwise the model may recognise the entity rather than learn transferable patterns.

4.2 Grouped and temporal partitions

Group splitting assigns entire groups to a partition. It is appropriate for repeated measurements, households, accounts, locations and devices. A grouped split estimates performance on unseen groups.

Temporal splitting trains on earlier observations and evaluates on later observations. It is essential when the operational task is forecasting or future prediction. A typical arrangement is:

The dates T₁ and T₂ should be selected to reflect deployment, seasonality, policy changes and the latency between feature observation and label availability. Randomly mixing future records into training can create temporal leakage.

4.3 Leakage

Data leakage occurs when information unavailable at prediction time influences training or evaluation. Examples include:

• a feature recorded after the outcome;
• a data-cleaning rule calculated using the entire dataset;
• target encoding computed before partitioning;
• duplicate or near-duplicate cases across partitions;
• imputation parameters estimated from validation or test data;
• oversampling performed before splitting.

The safe pattern is: split according to the deployment design, fit preprocessing transformations on training data, apply them unchanged to validation and test data, and freeze the test set until final evaluation.

5. Class Imbalance and Evaluation

Class imbalance occurs when one class is much more frequent than another. Accuracy can then be misleading. If only 2% of transactions are fraudulent, a classifier that predicts “not fraud” for every transaction achieves 98% accuracy but has zero recall for fraud.

For a binary classifier, define:

• TP: true positives;
• TN: true negatives;
• FP: false positives;
• FN: false negatives.

Important measures include:

These denominators must be non-zero. Precision focuses on the reliability of positive alerts; recall focuses on the proportion of actual positives detected. The appropriate measure depends on the costs of errors. For strongly imbalanced problems, precision–recall curves may be more informative than ROC curves because they emphasise performance on the positive class.

5.1 Remedies

Possible remedies include:

1. Class weighting: assign greater loss to minority-class errors. A common balanced heuristic is w_k = n/(K n_k), where n is the training size, K is the number of classes and n_k is the count in class k. This is a heuristic, not a universal optimum.
2. Undersampling: remove some majority examples, reducing computation but potentially discarding information.
3. Oversampling: replicate minority examples, which may increase overfitting.
4. Synthetic sampling: generate minority examples using neighbourhood structure. Synthetic points must remain plausible and must not cross group or temporal boundaries.
5. Threshold adjustment: change the decision threshold after estimating probabilities. This is often preferable when the model is adequate but error costs are asymmetric.
6. Anomaly detection or ranking: useful when positives are extremely rare and labels are incomplete.

All choices should be assessed using a validation design that reflects the expected class prevalence and decision costs. Resampling the training set should not change the natural prevalence of the untouched test set unless the evaluation estimand explicitly requires a reweighted population.

6. Analytical Dataset Design

A model-ready dataset begins with a clear unit of observation. Each row should represent one well-defined entity or event at a stated time. A data dictionary should specify the grain, variable meaning, type, permissible values, missing-value codes, source, collection time and transformation history.

A useful analytical design records:

• entity identifier and event identifier;
• observation timestamp and prediction timestamp;
• label definition, label window and censoring rules;
• feature availability time;
• inclusion and exclusion criteria;
• population and sampling frame;
• partition assignment;
• sampling or adjustment weight;
• provenance and version information.

6.1 Labels and observation windows

A label must be defined independently of information that would not be available when the prediction is made. For example, a churn label may be “account becomes inactive during the 90 days after the prediction date.” Features must be drawn from a look-back window ending no later than the prediction date. The period between feature cutoff and label observation is a label window.

Censoring occurs when the observation period ends before the label window is complete. Such cases should not automatically be treated as negative labels. Survival analysis, delayed labelling or explicit exclusion may be more appropriate.

6.2 Sampling versus weighting for modelling

Survey weights are designed to estimate population quantities, whereas machine-learning weights may alter the loss function to reflect costs, prevalence or a target deployment distribution. These purposes should not be confused. A weighted model can improve population calibration or minority-class sensitivity, but the final performance metric must correspond to the intended operational population.

6.3 Reproducibility and data lineage

A robust pipeline stores the random seed, sampling algorithm, frame version, split boundaries, group identifiers, transformation parameters and software versions. The same raw data and configuration should reproduce the same partition. Randomness without recorded state is not a reproducible methodology.

7. Case Study: Predicting Hospital Readmission

This is an illustrative case.

Background

A hospital wants to predict whether an adult patient will be readmitted within 30 days of discharge. The available data include admissions, diagnoses, laboratory results, prescriptions and demographic information.

Problem

An initial team randomly split 200,000 rows into training and test sets. Because patients could have multiple admissions, the same patient appeared in both sets. In addition, several laboratory variables were updated after discharge, and the minority readmission class represented only 7% of records.

Analysis

The random row split inflated performance because patient-specific patterns crossed the partition boundary. Post-discharge variables created direct leakage. The dataset also overrepresented patients treated at the hospital compared with the regional population, so hospital-only performance could not automatically be generalised to all residents.

Approach and solution

The team first defined the row grain as one eligible discharge. The prediction time was discharge time, and the label window was the following 30 days. All features were restricted to information available by discharge. Patients were assigned wholly to one partition, and a temporal test set consisting of the most recent six months was reserved. Training and validation data used grouped, approximately stratified resampling, while the final test set retained natural prevalence.

The team evaluated recall, precision, area under the precision–recall curve, calibration and performance by age, sex, ethnicity and hospital ward. Class weighting was applied only during training. Missing laboratory values were imputed using parameters learned from training data. Model selection occurred using the validation set; the test set was opened once for final assessment.

Outcome

The apparent recall fell after leakage removal, but the estimate became operationally credible. The model was particularly weak for patients from wards with sparse data. Rather than claiming universal clinical validity, the hospital documented the target population, monitoring plan and need for prospective evaluation.

Learning points

The most important improvement did not come from a more complex algorithm. It came from defining the unit, prediction time, label window, group structure and deployment population correctly. A lower but valid estimate is more valuable than an impressive estimate produced by leakage.

8. Governance, Ethics and Contemporary Practice

Sampling decisions can distribute benefits and errors unevenly. Oversampling a subgroup for precision may be statistically justified but can increase privacy risk or burden. Excluding records with missing values may disproportionately remove disadvantaged groups. Class weighting may improve minority recall while increasing false alarms for another group.

Analysts should document the rationale for inclusion, exclusion, weighting and partitioning. Sensitive attributes may be needed for bias auditing even when they are not used as predictive features. Access controls, minimisation, retention limits and lawful processing are essential when datasets contain health, financial or demographic information.

Modern data platforms increasingly support feature stores, versioned datasets, data contracts and automated lineage. These tools improve reproducibility but do not replace methodological judgement. Automated random splitting remains unsafe for grouped, longitudinal or intervention data. Synthetic data can assist testing and privacy protection, but synthetic records may fail to preserve rare combinations, causal relationships or subgroup risks.

Best practice is to maintain a dataset card or equivalent record describing purpose, population, sources, sampling, missingness, known limitations, partition strategy, intended use and prohibited use. Monitoring after deployment should examine population drift, label drift, subgroup performance and changes in class prevalence.

Chapter Summary

Sampling defines which units contribute evidence and whether conclusions can be related to a target population. Probability designs provide known inclusion probabilities and support design-based estimation; non-probability designs may be practical but require caution about generalisation. Stratification can improve subgroup precision, while clustering reduces cost but introduces dependence. Inverse-probability weights, post-stratification and calibration can correct some observable imbalances, although they cannot guarantee removal of unmeasured selection bias.

Training, validation and test partitions must reflect the deployment setting. Random splitting is suitable only when observations are approximately independent and identically distributed. Grouped and temporal partitions are necessary when repeated entities or time ordering matter. Leakage prevention requires fitting preprocessing and resampling procedures inside the training workflow.

Class imbalance should be addressed through appropriate metrics, class weighting, resampling, threshold selection and cost-sensitive evaluation. Analytical dataset design begins with a precise row grain, prediction time, label window and population definition. Ethical and professional practice requires documenting assumptions, protecting privacy, evaluating subgroup effects and monitoring performance after deployment.

Key Terms

Sampling frame: Operational list or mechanism from which eligible units are selected.

Target population: Population about which conclusions are intended.

Accessible population: Population that can actually be reached through the available frame.

Inclusion probability: Probability that a particular unit enters the sample.

Sampling weight: Weight commonly based on the inverse of a unit’s inclusion probability.

Probability sampling: Sampling in which inclusion probabilities are known and non-zero.

Non-probability sampling: Sampling in which inclusion probabilities are unknown or uncontrolled.

Stratification: Division of a population into subgroups before sampling.

Cluster sampling: Selection of groups followed by observation of units within groups.

Design effect: Inflation of variance caused by a sampling design relative to simple independent sampling.

Resampling: Repeated construction of samples from observed data.

Selection bias: Systematic distortion caused by the mechanism determining inclusion.

Partition: Assignment of records to training, validation or test roles.

Data leakage: Use of information unavailable at prediction time or improperly shared across partitions.

Class imbalance: Unequal frequency of outcome classes.

Precision: Proportion of predicted positives that are true positives.

Recall: Proportion of actual positives that are detected.

Unit of observation: Entity or event represented by one analytical row.

Label window: Future period used to determine an outcome after prediction time.

Covariate shift: Change in the distribution of predictors between development and deployment populations.

Review Questions

Short Answer Questions

1. Distinguish a target population from an accessible population.
2. What is an inclusion probability, and why is it important?
3. Why can a random row split be invalid for repeated patient records?
4. Define data leakage and give one example.
5. Why may accuracy be misleading for a highly imbalanced classification problem?

Long Answer Questions

1. Compare simple random, stratified, cluster and systematic sampling, including one strength and one limitation of each.
2. Explain design weights, non-response adjustment and calibration, and discuss the assumptions required for their use.
3. Describe how training, validation and test partitions should be designed for a longitudinal prediction problem.
4. Discuss the advantages and risks of class weighting, undersampling, oversampling and threshold adjustment.
5. Explain how unit of observation, prediction time, label window and feature availability determine analytical dataset validity.

Analytical / Application Questions

1. A dataset contains 5,000 customers, but 40% of rows belong to 100 highly active customers. Propose a partitioning strategy and explain why a random row split is inappropriate.
2. A fraud dataset has 1% positive cases. Design an evaluation plan that preserves realistic prevalence while allowing effective model training. Specify suitable metrics.
3. A survey oversamples a rural subgroup that represents 20% of the population but 50% of the sample. Explain how this affects an unweighted estimate and propose an appropriate weighting strategy.

Multiple Choice Questions with Answers

Question 1

Which condition defines probability sampling?

A. Every unit has a known, non-zero probability of selection

B. Every subgroup has exactly the same sample size

C. Participants volunteer through an open website

D. The analyst selects the most informative cases

Correct Answer: A. Every unit has a known, non-zero probability of selection

Question 2

If a unit has inclusion probability πᵢ = 0.20, what is its basic design weight?

A. 0.20

B. 1.20

C. 5

D. 20

Correct Answer: C. 5

Question 3

Which partitioning method is most appropriate when future performance is the deployment objective?

A. Random row splitting across all dates

B. Temporal splitting with later observations held out

C. Oversampling the test set

D. Putting duplicate entities in every partition

Correct Answer: B. Temporal splitting with later observations held out

Question 4

Which action is most likely to create data leakage?

A. Fitting an imputer on training data only

B. Holding out the latest six months as a test period

C. Applying SMOTE to the full dataset before splitting

D. Keeping all records for one patient in one partition

Correct Answer: C. Applying SMOTE to the full dataset before splitting

Question 5

Recall is calculated as which quantity?

A. TP/(TP+FP)

B. TN/(TN+FP)

C. TP/(TP+FN)

D. 2TP/(2TP+FP+FN)

Correct Answer: C. TP/(TP+FN)

Question 6

What is a principal reason for using stratified sampling?

A. To guarantee that every selected unit is independent

B. To ensure important subgroups receive adequate representation

C. To eliminate all non-response bias

D. To avoid recording inclusion probabilities

Correct Answer: B. To ensure important subgroups receive adequate representation

Question 7

If ρ = 0.10 and the average cluster size m = 11, the approximate design effect is:

A. 0.10

B. 1.10

C. 2.00

D. 11.00

Correct Answer: C. 2.00

Question 8

Which statement about a test set is correct?

A. It should be repeatedly used for hyperparameter tuning

B. It should normally retain the natural evaluation prevalence

C. It must always be balanced by oversampling

D. It should contain duplicated training observations

Correct Answer: B. It should normally retain the natural evaluation prevalence

Question 9

Which item best defines the unit of observation?

A. The algorithm used to train the model

B. The entity or event represented by one analytical row

C. The number of variables in the dataset

D. The probability of a positive label

Correct Answer: B. The entity or event represented by one analytical row

Question 10

A sample includes only customers who remained active until the study date. Which bias is especially plausible?

A. Survivorship bias

B. Measurement-unit bias

C. Randomisation bias caused by stratification

D. Variance reduction from clustering

Correct Answer: A. Survivorship bias