3. Data Understanding, Quality and Preparation

Introduction and Context

Machine learning systems do not learn directly from business questions, organisational goals, or social phenomena. They learn from representations of those phenomena in the form of data. The quality of a predictive model is therefore inseparable from the quality of the data used to construct it. A sophisticated algorithm trained on incomplete, biased, incorrectly labelled, or contaminated data may produce results that are precise in a technical sense but unreliable in practice.

Data preparation is consequently not a preliminary clerical activity. It is an analytical and modelling activity through which the analyst determines what a variable means, which observations are trustworthy, how different data sources can be combined, and what information may legitimately be used for prediction. Decisions made during preparation often have a greater effect on model performance and validity than the choice between two advanced algorithms.

This chapter presents a systematic approach to understanding, profiling, assessing, and preparing structured and unstructured data. It addresses missing values, outliers, noise, class imbalance, data leakage, sampling bias, and reproducibility. The emphasis is not merely on performing transformations, but on choosing transformations that are defensible, transparent, and appropriate to the intended use of the model.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Describe the role of data understanding in the machine-learning lifecycle.

2. Distinguish data types, measurement scales, sources, and levels of data quality.

3. Profile datasets using descriptive statistics, visualisation, and integrity checks.

4. Diagnose and address missing values, outliers, noise, imbalance, and inconsistent records.

5. Explain data leakage and sampling bias and identify their consequences.

6. Select suitable transformations for structured, textual, image, and time-dependent data.

7. Construct a reproducible preprocessing workflow using training and validation principles.

8. Evaluate the ethical, organisational, and governance implications of data preparation.

Data Understanding as an Analytical Activity

Data understanding begins with the question: what do the observations actually represent? A row in a table may represent a customer, an account, a transaction, a hospital visit, or a daily measurement. These units must not be confused. If a customer appears in several rows, randomly splitting rows between training and test sets can make the model appear more accurate because it has indirectly encountered the same customer during training.

A useful data inventory records the source, owner, collection method, time period, unit of observation, update frequency, access restrictions, intended use, and known limitations of each dataset. The analyst should also identify the target variable and establish when that target becomes known. This temporal question is essential: a variable recorded after an event may be strongly predictive but unavailable when a real prediction must be made.

Data may be classified in several complementary ways. Structured data are organised into fields and records, such as relational tables. Semi-structured data contain regular features but flexible schemas, as in JSON documents or XML. Unstructured data include free text, images, audio, video, and documents. Data may also be quantitative or qualitative, cross-sectional or longitudinal, static or streaming, transactional or observational.

Measurement scale affects the transformations that are legitimate. Nominal variables identify categories without order, such as region or product type. Ordinal variables possess order but not necessarily equal intervals, such as satisfaction ratings. Interval variables have meaningful differences but no true zero, whereas ratio variables have both meaningful differences and a meaningful zero. Treating a nominal category as a numerical quantity can impose a false ordering; treating a genuinely ordered variable as unrelated categories can discard useful information.

A data dictionary should define every field, including its business meaning, data type, permitted values, units, missing-value codes, source system, and relationship to other fields. Codes such as -99, 999, “unknown,” and blank strings should not automatically be interpreted as genuine measurements. They may represent missingness, refusal, system failure, or a value that was not applicable. These meanings can have different implications for modelling.

Profiling a Dataset Before Modelling

Profiling is a structured examination of the dataset before substantive modelling begins. It combines automated checks with domain interpretation. Basic profiling includes the number of records and variables, data types, unique-value counts, frequency distributions, summary statistics, duplicate records, missingness, and the range of numerical fields.

For a numerical variable, the mean, median, standard deviation, minimum, maximum, quartiles, and selected percentiles provide an initial picture. The mean may be distorted by extreme values, while the median is more robust. The interquartile range, calculated as Q3 minus Q1, describes the spread of the middle half of the observations. For categorical variables, frequency tables reveal rare categories, dominant categories, and unexpected values. Visualisations such as histograms, box plots, bar charts, scatter plots, and correlation displays help identify patterns that summaries may conceal.

Univariate analysis examines one variable at a time. Bivariate analysis studies relationships between two variables, such as the association between income and expenditure or between a measured feature and the target. Multivariate analysis considers interactions and dependencies among several variables. A high correlation between two predictors may indicate redundancy, but correlation alone does not establish causality and does not detect all forms of dependence.

Quality assessment should address several dimensions:

• Accuracy: whether values represent the true or intended phenomenon.
• Completeness: whether required values and records are present.
• Consistency: whether values agree across fields, systems, and time.
• Validity: whether values conform to rules, formats, and permitted ranges.
• Uniqueness: whether records are duplicated inappropriately.
• Timeliness: whether data are current and available at the required point in time.

These dimensions may conflict. A dataset can be highly complete but systematically inaccurate, or accurate for a historical purpose but too old for operational prediction. Data quality is therefore contextual rather than absolute.

Missing Data: Diagnosis Before Treatment

Missingness is not a single problem. A value may be missing because it was not collected, the question did not apply, a respondent refused to answer, a sensor failed, or a value was deliberately suppressed. The pattern and mechanism of missingness should guide the response.

Three commonly used conceptual mechanisms are missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). Under MCAR, missingness is unrelated to observed and unobserved values. Under MAR, missingness can be explained by other observed variables. Under MNAR, the probability of missingness depends on the unobserved value itself or on unobserved factors. These categories are assumptions about a data-generating process, not labels that can always be proven from the dataset.

Deletion is appropriate only when the amount of missingness is small, the mechanism is defensible, and the deletion will not distort the sample. Imputation replaces missing values using a rule or model. Common simple methods include mean, median, and mode imputation. Median imputation is often preferable to mean imputation for skewed numerical variables, but both reduce natural variability and can weaken relationships.

More advanced methods include regression imputation, k-nearest-neighbour imputation, multiple imputation, and model-based methods. Multiple imputation creates several plausible completed datasets and combines the results, thereby reflecting uncertainty more appropriately than a single deterministic replacement. Missingness indicators can also be added when the fact that a value is missing may itself contain information. Such indicators require careful interpretation because they may encode administrative practices or social disadvantage rather than a causal attribute.

Imputation must be fitted using training data only. If the overall dataset mean is computed before splitting, information from the validation or test set enters the training process. Even when this produces only a small numerical difference, it violates the logic of honest evaluation.

Outliers and Noise

An outlier is an observation that is unusually distant from the general pattern of a variable or group. It may be a data-entry error, a measurement failure, a rare but valid event, or evidence that the population contains distinct subgroups. Outliers should not be removed simply because they reduce model performance.

Univariate detection methods include the interquartile-range rule, in which values below Q1 − 1.5 IQR or above Q3 + 1.5 IQR are flagged, and standardised-score rules based on distance from the mean. Multivariate methods consider combinations of variables; an observation may be ordinary on each variable separately but unusual in combination. Distance-based, density-based, and isolation-based methods can assist with this task.

Possible responses include correcting an identifiable recording error, retaining a valid extreme value, transforming a heavily skewed variable, winsorising extreme values, using robust estimators, or creating a separate category for a meaningful exceptional condition. Winsorisation limits extreme values to selected percentiles, but it changes the data and should be documented. Logarithmic or other monotonic transformations can reduce skewness, although they do not turn an error into a valid measurement.

Noise refers to random error or irrelevant variation in measurements and labels. In supervised learning, label noise is particularly serious because the model is trained to reproduce incorrect outcomes. Noise may be reduced through improved measurement, repeated observations, validation rules, expert review, or robust modelling. Smoothing a time series can be useful for description, but indiscriminate smoothing may remove genuine signals needed for forecasting.

Transforming Variables for Learning

Preprocessing transforms raw variables into representations suitable for a learning algorithm. Numerical scaling is important for distance-based, gradient-based, and regularised models. Standardisation transforms a variable using its training-set mean and standard deviation. Min–max scaling maps values to a chosen interval. Robust scaling uses the median and interquartile range and is less sensitive to extreme observations. Tree-based methods are generally less dependent on scaling, although consistent preprocessing may still be useful in a broader pipeline.

Categorical variables may be represented using one-hot encoding, ordinal encoding, target encoding, hashing, or learned embeddings. One-hot encoding is appropriate for nominal variables with a manageable number of categories. Ordinal encoding should be used only when category order has substantive meaning. Target encoding can be powerful for high-cardinality categories but is highly vulnerable to leakage; it must be calculated within training folds and with suitable smoothing.

Text preparation may involve tokenisation, normalisation, removal of irrelevant symbols, stemming or lemmatisation, and conversion into numerical representations. Traditional representations include term frequency–inverse document frequency. Contemporary systems frequently use pretrained language-model embeddings, but these representations still require attention to domain shift, privacy, and information leakage. Removing all punctuation or stop words is not universally beneficial; the appropriate treatment depends on the task.

Images may require resizing, normalisation, augmentation, and quality checks. Audio may require sampling-rate conversion, noise handling, and feature extraction. Data augmentation can improve robustness by creating plausible variations, but transformations must preserve the target meaning. For example, flipping an image may be inappropriate when left and right have different meanings.

Feature construction uses existing fields to create more informative variables, such as a customer’s average transaction value or the time since a previous event. Feature selection removes irrelevant, redundant, or unstable variables. These operations should be guided by domain knowledge and evaluated within the modelling workflow. A feature that is predictive in historical data may be inappropriate if it will not be available at deployment.

Sampling, Imbalance, and Data Partitioning

A sample is useful only to the extent that it represents the population or decision context to which the model will be applied. Sampling bias occurs when the mechanism by which observations enter the dataset is related to the outcome, predictors, or intended population. Sources include convenience sampling, non-response, survivorship, undercoverage, and changes in data-collection procedures.

Random sampling is not always sufficient. Stratified sampling preserves important subgroup proportions, while cluster sampling selects groups rather than individual units. In time-dependent problems, chronological splitting is usually more realistic than random splitting. In grouped data, all observations from a person, household, institution, or machine should generally remain in the same partition.

The usual partitions are training, validation, and test sets. The training set is used to fit parameters; the validation set supports model and hyperparameter decisions; the test set is reserved for final assessment. Cross-validation provides repeated training and validation splits, but it does not eliminate leakage if preprocessing is performed before the folds are created.

Class imbalance occurs when one outcome is much less frequent than another. Accuracy can then be misleading. A classifier that always predicts the majority class may achieve high accuracy while failing to identify the minority class. Useful measures include precision, recall, specificity, F1 score, balanced accuracy, and area under appropriate ranking curves. The selection of a metric should reflect the costs of false positives and false negatives.

Responses to imbalance include class-weighted learning, threshold adjustment, careful over-sampling, under-sampling, and synthetic methods such as SMOTE. Resampling must be performed only within the training portion of each fold. Applying synthetic oversampling before splitting can create near-duplicate information in both training and test data and lead to optimistic results. The actual deployment prevalence and decision costs should guide final threshold selection.

Data Leakage and Temporal Validity

Data leakage occurs when information unavailable at prediction time influences model training or evaluation. It is one of the most damaging preparation errors because it can produce impressive validation results that collapse in use.

Leakage may be direct, such as using a variable recorded after the target event. It may be indirect, such as computing a customer’s lifetime average using transactions that occurred after the prediction date. It may also result from preprocessing the complete dataset before splitting, duplicate entities across partitions, target-derived features, or inappropriate feature selection.

A practical safeguard is to define a prediction timestamp and an information-availability rule for every feature. The analyst should ask: when would this value have been known, by whom, and with what delay? Time-aware feature engineering and chronological validation are essential in credit risk, demand forecasting, fraud detection, clinical prediction, and predictive maintenance.

Reproducible Preprocessing Pipelines

A preprocessing pipeline expresses data preparation as an ordered, executable sequence. A typical pipeline may validate schema, separate the target, split the data, impute numerical variables, scale selected variables, encode categorical variables, construct features, and pass the resulting representation to an estimator. The fitting of every learned transformation must occur only on the training data.

Pipelines improve reproducibility by recording parameters, software versions, random seeds where appropriate, feature definitions, and the order of operations. They also reduce the risk that training and production systems apply different transformations. A pipeline should be tested with unseen records, missing fields, unexpected categories, extreme values, and changes in data types.

Reproducibility is more than obtaining the same numerical result twice. It includes traceability: the ability to explain which source data, code, assumptions, transformations, and model version produced an outcome. Version control, data snapshots, metadata catalogues, automated validation, and model registries support this objective. Reproducibility must be balanced with privacy; copies of sensitive data should not be created casually merely to support experimentation.

Illustrative Case Study: Predicting Hospital Readmission

Background

A regional hospital develops a model to identify patients at high risk of readmission within thirty days of discharge. The purpose is to offer additional follow-up and care coordination, not to deny treatment.

Situation and Problem

The initial dataset contains admission records, diagnoses, laboratory results, discharge information, follow-up appointments, and readmission outcomes. A preliminary model reports excellent discrimination. However, clinicians observe that the model is unreliable when used immediately after discharge.

Analysis

An audit shows that some variables were recorded after discharge, including whether a patient attended a follow-up appointment and whether a post-discharge care-management referral was completed. These variables are strongly associated with readmission but are not available at the moment the prediction is intended to be made. The dataset also contains multiple admissions for the same patient, and random row-level splitting has placed records from the same patients in both training and test sets.

Missing laboratory values are more common among patients transferred from smaller facilities. Simply imputing the overall median obscures this operational pattern. The outcome is relatively uncommon, so accuracy has concealed poor recall for readmitted patients. Finally, the hospital’s historical sample underrepresents patients who receive care outside its network.

Approach and Strategy

The team defines the prediction time as the planned discharge decision. Every feature is checked against this timestamp, and post-discharge variables are removed. A patient-level grouped split prevents records from the same patient appearing in different partitions. A time-based evaluation is added to test performance on more recent patients.

Numerical imputation and categorical encoding are placed inside a cross-validated pipeline. Missingness indicators are retained for selected laboratory fields because missingness reflects differences in care pathways, but the indicators are reviewed for fairness and clinical interpretation. Class-weighted learning and recall-oriented threshold selection are used because failing to identify a high-risk patient has greater intervention cost than reviewing an additional low-risk patient.

The team reports subgroup performance by age, sex, language, and transfer status. It also documents that the model estimates risk rather than causes readmission. Clinicians participate in reviewing features and setting an escalation protocol.

Outcome

The revised evaluation produces a lower but more credible performance estimate. The model is incorporated into a controlled pilot in which predictions trigger optional care-coordination review. Performance is monitored for drift, changes in referral practices, and unequal error rates. The hospital decides not to deploy the model autonomously and establishes a periodic governance review.

Learning Points

The case demonstrates that leakage control, grouping, temporal validity, missingness analysis, imbalance-aware evaluation, and fairness assessment are connected activities. A lower score obtained under realistic evaluation is more valuable than an inflated score produced by flawed preparation.

Professional and Ethical Considerations

Data preparation can reproduce or amplify institutional inequality. Missingness may be concentrated among disadvantaged groups, labels may reflect unequal access to services, and historical outcomes may encode biased decisions. Removing sensitive attributes does not necessarily remove their influence because proxy variables may remain. Fairness analysis should therefore examine both data-generating processes and model outcomes.

Organisations should establish responsibility for data definitions, access control, retention, consent, and permitted uses. Personally identifiable information should be minimised, protected, and processed according to applicable law and organisational policy. De-identification reduces risk but does not guarantee anonymity, especially when datasets can be linked.

Documentation should record assumptions, exclusions, imputation rules, sampling decisions, known limitations, and intended use. Analysts should avoid presenting cleaned data as objective or complete when substantial judgement has been applied. When a transformation changes the meaning or distribution of a variable, that change should be visible to reviewers and users.

Emerging Developments

Modern data platforms increasingly support automated data-quality checks, schema monitoring, lineage tracking, and feature stores. Automated tools can detect drift, anomalous values, and unexpected category changes, but they cannot determine whether a value is substantively correct. Human domain knowledge remains necessary.

Large language models and multimodal systems have expanded the use of text, images, audio, and mixed-format data. Their preparation challenges include annotation consistency, confidential information, provenance, copyright, representation gaps, and model-generated synthetic data. Synthetic data can support testing and privacy protection, but it may reproduce the biases and statistical weaknesses of the source data.

Data-centric machine learning places greater emphasis on improving labels, coverage, consistency, and representativeness rather than continually changing algorithms. In production environments, monitoring now extends beyond model accuracy to data drift, concept drift, missingness changes, subgroup performance, and the operational consequences of predictions.

Best-Practice Workflow

A defensible workflow follows these broad principles:

1. Define the prediction task, population, unit of analysis, target, and prediction time.

2. Inventory sources and create a data dictionary.

3. Profile distributions, missingness, duplicates, validity, and relationships.

4. Investigate quality problems with domain experts rather than relying only on automatic rules.

5. Split data according to time, groups, and deployment conditions.

6. Fit imputers, encoders, scalers, feature selectors, and resampling methods within the training workflow.

7. Evaluate using metrics that reflect decision costs and subgroup performance.

8. Document transformations, assumptions, versions, and limitations.

9. Test the pipeline on realistic unseen data.

10. Monitor the data and model after deployment and revise the process when conditions change.

Chapter Summary

Data understanding and preparation establish the validity of machine-learning analysis. Analysts must first determine what records represent, how variables were measured, when information became available, and whether the sample reflects the intended population. Profiling examines distributions, missingness, duplicates, validity, consistency, and relationships.

Missing values require diagnosis of their causes and patterns; outliers may be errors or valuable rare events; noise can arise from measurement or labels; and class imbalance demands suitable metrics and training strategies. Sampling bias limits generalisation, while leakage produces deceptively strong evaluation results. Transformations such as imputation, scaling, encoding, feature construction, and resampling must be fitted within reproducible training pipelines.

Professional preparation also requires attention to fairness, privacy, governance, traceability, temporal validity, and monitoring. A reliable model is therefore the product not only of an algorithm, but of a carefully designed data-generating, preparation, evaluation, and operational process.

Key Terms

Data dictionary: A documented description of variables, meanings, formats, units, and permitted values.

Data profiling: Systematic examination of data structure, distributions, quality, and relationships.

Data quality: The degree to which data are accurate, complete, consistent, valid, unique, and timely for a purpose.

Structured data: Data organised into defined fields, records, or relational tables.

Unstructured data: Data such as free text, images, audio, or video without a fixed tabular schema.

Missing completely at random: Missingness unrelated to observed or unobserved data values.

Missing at random: Missingness explainable by other observed variables.

Missing not at random: Missingness related to the unobserved value or unobserved causes.

Imputation: Replacement of missing values using a rule, statistic, or predictive method.

Outlier: An observation unusually distant from the general pattern of the data.

Noise: Random error, irrelevant variation, or incorrect information in measurements or labels.

Feature engineering: Creation or transformation of variables to represent predictive information.

Data leakage: Use of information in training or evaluation that would not be available at prediction time.

Sampling bias: Systematic difference between the observed sample and the intended population.

Class imbalance: Unequal representation of outcome classes in a dataset.

Cross-validation: Repeated partitioning of training data for model assessment or selection.

Data drift: Change in the distribution of input data over time.

Concept drift: Change in the relationship between predictors and the target over time.

Reproducible pipeline: An executable, documented sequence of preparation and modelling steps that can be repeated consistently.

Target encoding: Representation of a category using information derived from the target variable.

Review Questions

Short Answer Questions

1. Why is the unit of observation important when preparing data for machine learning?

2. Distinguish between data validity and data completeness.

3. What is the difference between an outlier and a data error?

4. Why should imputation parameters be learned from training data only?

5. Define data leakage and provide one example.

Long Answer Questions

1. Explain a systematic process for profiling a newly acquired dataset before modelling.

2. Compare deletion, simple imputation, and multiple imputation as approaches to missing data.

3. Discuss how sampling bias and class imbalance affect model evaluation and deployment.

4. Explain the role of reproducible preprocessing pipelines in professional machine-learning practice.

5. Describe the ethical and governance issues that can arise during data preparation.

Analytical / Application Questions

1. A fraud dataset has 98% legitimate transactions. A model achieves 98% accuracy but identifies very few fraudulent cases. Analyse why this result is inadequate and propose a more appropriate evaluation and training strategy.

2. A demand-forecasting model uses a variable containing the monthly total sales for the month being forecast. Identify the likely leakage problem and redesign the feature using only information available at the forecast time.

3. A hospital dataset contains repeated admissions, substantial missing laboratory values, and a patient population that changes over time. Design a data partitioning, preparation, and monitoring plan that would provide a credible estimate of deployment performance.

22. Multiple Choice Questions with Answers

Question 1

Which action most directly prevents information from the test set entering a preprocessing transformation?

A. Fit the transformation on the complete dataset before splitting

B. Fit the transformation on the training set within a pipeline

C. Remove all categorical variables

D. Use accuracy as the evaluation metric

Correct Answer: B. Fit the transformation on the training set within a pipeline

Question 2

A variable is recorded only after a customer defaults, although the model is intended to predict default beforehand. Using this variable is an example of:

A. Class imbalance

B. Sampling bias

C. Data leakage

D. Robust scaling

Correct Answer: C. Data leakage

Question 3

Which measure is generally more robust than the mean when a numerical variable contains extreme values?

A. Median

B. Maximum

C. Variance

D. Range

Correct Answer: A. Median

Question 4

Why can random row-level splitting be inappropriate when a dataset contains several records for each person?

A. It always increases class imbalance

B. It can place records from the same person in both training and test sets

C. It prevents numerical scaling

D. It removes all missing values

Correct Answer: B. It can place records from the same person in both training and test sets

Question 5

Which encoding is most appropriate for a nominal categorical variable when categories have no meaningful order and cardinality is manageable?

A. Ordinal encoding

B. One-hot encoding

C. Mean imputation

D. Standardisation

Correct Answer: B. One-hot encoding

Question 6

A classifier performs well on the majority class but poorly on a rare class. Which response is most appropriate?

A. Report accuracy alone

B. Ignore the rare class

C. Use metrics such as recall and class-weighted learning where appropriate

D. Delete all majority-class observations without evaluation

Correct Answer: C. Use metrics such as recall and class-weighted learning where appropriate

Question 7

Which statement best describes an outlier?

A. It is always an invalid record

B. It is an observation unusually distant from the general pattern

C. It is any missing value

D. It is a categorical variable with many levels

Correct Answer: B. It is an observation unusually distant from the general pattern

Question 8

Why must target encoding be performed carefully?

A. It cannot represent categories

B. It may leak target information into validation or test data

C. It requires all values to be numerical originally

D. It always increases missingness

Correct Answer: B. It may leak target information into validation or test data

Question 9

For a forecasting problem, which validation strategy is usually most realistic?

A. A chronological split that preserves the direction of time

B. A split created after randomly mixing all future and past records

C. A test set containing duplicated training records

D. A split based only on alphabetical order

Correct Answer: A. A chronological split that preserves the direction of time

Question 10

Which statement about missingness is correct?

A. All missing values have the same meaning

B. Missingness may reflect a process such as refusal, inapplicability, or system failure

C. Missing values should always be replaced by zero

D. Missingness never contains useful information

Correct Answer: B. Missingness may reflect a process such as refusal, inapplicability, or system failure