Data Acquisition, Understanding, and Preparation

Introduction

Machine-learning systems learn from data, but data rarely arrives in a form that can be used directly. It may be incomplete, duplicated, inconsistent, incorrectly labelled, biased, or stored across multiple systems. Data preparation is therefore not a minor preliminary task; it is a central engineering and analytical activity that strongly influences model accuracy, fairness, reliability, and maintainability.

A useful principle is “garbage in, garbage out.” A sophisticated algorithm cannot reliably compensate for poorly collected or poorly understood data. This chapter presents a practical data workflow: acquiring data, understanding its structure and meaning, profiling its quality, cleaning and integrating sources, transforming and encoding variables, scaling numerical features, splitting datasets, and identifying risks such as leakage, sampling bias, class imbalance, and outliers.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish structured, semi-structured, and unstructured data;
• identify appropriate data sources and acquisition methods;
• profile datasets using descriptive and quality checks;
• handle missing values, duplicates, inconsistent formats, and outliers;
• integrate data from multiple sources while preserving meaning and provenance;
• transform, encode, and scale features appropriately;
• create training, validation, and test sets without leakage;
• recognise sampling bias, class imbalance, and other threats to data quality;
• apply ethical, legal, and governance principles to data preparation.

1. Types and Sources of Data

1.1 Structured data

Structured data is organised according to a predefined schema, usually in rows and columns. Examples include customer tables, bank transactions, sensor readings, and examination records. Each column has an intended meaning and often a specified type such as integer, decimal, date, Boolean, or text category.

Structured data is comparatively easy to query and analyse, but it is not automatically reliable. A database may contain invalid dates, duplicated customer identifiers, incorrect units, or values entered under inconsistent coding schemes.

1.2 Semi-structured and unstructured data

Semi-structured data has some organisational features but does not conform to a rigid table. JSON documents, XML files, application logs, and email headers are common examples. Fields may be optional, nested, or different across records.

Unstructured data lacks a consistent tabular schema. Images, audio, video, free-form documents, social-media posts, and scanned forms are examples. Machine-learning applications often convert such data into usable representations, such as image pixels, audio features, document tokens, or vector embeddings. The original data should nevertheless be preserved because later methods may require a different representation.

1.3 Common acquisition sources

Data may be acquired from internal operational databases, sensors and Internet-of-Things devices, surveys, experiments, public repositories, application programming interfaces, transaction systems, and third-party providers. Acquisition decisions should consider relevance, authority, cost, update frequency, licensing, privacy, and technical compatibility.

A data catalogue should record the source, owner, collection date, update schedule, intended use, definitions, access restrictions, and known limitations. Provenance—the history of where data came from and how it was changed—is essential for reproducibility and accountability.

2. Defining the Data Problem Before Collection

Data collection should begin with a precise analytical question. A classification project may ask whether a transaction is fraudulent, whereas a regression project may estimate delivery time. These objectives determine the target variable, the required observations, the time period, and acceptable sources.

Before collecting data, document:

1. the unit of analysis, such as a customer, transaction, image, or time interval;
2. the target variable and its operational definition;
3. the observation period and geographic coverage;
4. the population to which conclusions should apply;
5. inclusion and exclusion criteria;
6. ethical, legal, security, and consent requirements.

A frequent error is confusing the unit of analysis. For example, combining several transactions into one customer record may be appropriate for customer-level churn prediction but inappropriate for transaction-level fraud detection. The unit must be consistent with the intended prediction.

3. Data Profiling and Understanding

Data profiling is the systematic examination of a dataset before modelling. It combines automated statistics with domain knowledge. A basic profile includes the number of rows and columns, data types, unique values, missingness, minimum and maximum values, central tendency, dispersion, and likely identifier fields.

For numerical variable x, the arithmetic mean is:

where n is the number of observations and xᵢ is observation i. The mean is sensitive to extreme values. The median, the middle ordered observation, is often more representative for skewed data. The range is the maximum minus the minimum, while standard deviation describes typical dispersion around the mean. These measures are meaningful only when the variable is numerical and observations are interpreted consistently.

Profiling should also inspect:

• frequency tables for categorical variables;
• histograms or box plots for distributions;
• scatter plots for relationships and unusual points;
• correlations, interpreted cautiously because correlation does not establish causation;
• dates, time zones, and ordering;
• impossible values, such as negative age or a temperature outside the instrument’s valid range;
• duplicate records and duplicate identifiers;
• target distribution and subgroup representation.

A data dictionary should define each field, its type, unit, permissible values, source, and business meaning. A column named “income” is not sufficiently defined unless the documentation states the currency, period, gross or net status, and treatment of missing or zero values.

4. Data Quality Dimensions

Data quality is multidimensional. Important dimensions include:

• Accuracy: values correspond to the real-world condition.
• Completeness: required values are present.
• Consistency: related records and fields do not contradict one another.
• Validity: values conform to rules, types, ranges, and formats.
• Uniqueness: records are not unintentionally duplicated.
• Timeliness: data is sufficiently current for the use case.
• Relevance: collected variables support the stated objective.

Quality checks should be expressed as explicit rules. Examples include “order date cannot be after delivery date,” “customer identifier must be unique in the customer table,” and “age must be within the permitted domain.” Automated checks should run whenever new data is ingested, while exceptions should be reviewed rather than silently discarded.

5. Cleaning Data

5.1 Missing values

Missingness can arise because a question was not asked, a sensor failed, a value was unavailable, or a respondent deliberately withheld information. Missingness is informative in some contexts; for example, failure to report income may correlate with economic circumstances.

Common strategies include:

• removing rows when only a small, plausibly random portion is missing;
• removing a variable when it is mostly absent and has little value;
• replacing numerical values with a mean or, more robustly, a median;
• replacing categorical values with the mode or an explicit “unknown” category;
• using model-based or nearest-neighbour imputation;
• adding a missingness indicator when absence itself may contain information.

Imputation must be fitted using the training data only. If the median of the complete dataset is used before splitting, information from the validation or test set can leak into the model-development process. The method should also be recorded because different imputations can affect fairness and interpretation.

5.2 Duplicates and inconsistent records

Exact duplicates may result from repeated uploads. Near-duplicates may differ only in punctuation, spelling, or formatting. Deduplication requires a defined rule, often based on a stable identifier and relevant attributes. Blindly removing duplicates can be harmful: two identical purchases may be legitimate separate transactions.

Standardisation may include converting text to a common case, trimming whitespace, harmonising date formats, converting units, and mapping synonymous categories. Such operations must preserve meaning. Converting miles to kilometres requires a documented conversion, not a simple renaming of the unit.

5.3 Outliers

An outlier is an observation that is unusually distant from the rest of the data. It may represent measurement error, fraud, a rare but valid event, or a different population. It should not automatically be deleted.

The interquartile range method defines IQR = Q₃ − Q₁, where Q₁ and Q₃ are the first and third quartiles. A common screening rule marks values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening convention, not proof that an observation is erroneous. The rule is most useful for continuous variables and can be unsuitable for highly discrete or strongly structured data.

Possible responses are correcting a verified error, retaining the value, transforming the variable, using a robust model, or investigating the observation separately. The choice should reflect the real application and the cost of false removal.

6. Integrating Data Sources

Integration combines records from different systems. It may involve joining customer information to transactions, merging sensor streams, or linking labelled and unlabelled observations.

Important issues include:

• Keys: whether identifiers are stable, unique, and correctly matched;
• Granularity: whether one table is customer-level and another transaction-level;
• Time alignment: whether measurements refer to the same period;
• Units and definitions: whether fields with similar names mean the same thing;
• Conflicts: which source is authoritative when values disagree;
• Missing matches: whether unmatched records should be retained;
• Lineage: how every derived field was produced.

A one-to-many join can unintentionally multiply rows and distort counts. For example, joining one customer record to five transactions creates five joined rows. Aggregation may be required before modelling, depending on the unit of analysis.

7. Transformation and Feature Engineering

Transformation changes data into a form suitable for analysis. Typical operations include extracting year or day-of-week from dates, aggregating transactions, converting text to tokens, creating ratios, and applying logarithms to strongly right-skewed positive variables.

Feature engineering should be guided by domain reasoning, not indiscriminate trial and error. A feature must be available at the time a prediction is made. A hospital model predicting admission risk cannot use a diagnosis recorded after admission.

Categorical variables can be encoded in several ways:

• Ordinal encoding assigns ordered numbers and is appropriate only when categories have a genuine order.
• One-hot encoding creates a binary indicator for each category and avoids imposing a false order. With k categories, it commonly creates k indicator columns, although one column may be omitted in some linear-model designs to avoid perfect redundancy.
• Frequency encoding replaces a category with its observed frequency, but may lose semantic distinctions.
• Target encoding uses target-related statistics and therefore must be computed within training folds to avoid leakage.

High-cardinality categories require care. One-hot encoding may create thousands of sparse columns, while grouping rare categories or using learned representations may be more practical.

Scaling numerical variables

Scaling is important for distance-based methods, nearest-neighbour algorithms, support-vector machines, neural networks, and many optimisation procedures. It is often less important for tree-based models.

Min–max scaling maps a value to a chosen interval, commonly [0, 1]:

Here x_min and x_max are calculated from the training data. The denominator must be non-zero. New values outside the training range may produce values below 0 or above 1, which is not necessarily an error.

Standardisation produces a variable with training mean approximately 0 and training standard deviation approximately 1:

Here μ is the training mean and σ is the training standard deviation, which must be non-zero. Standardisation does not make data normally distributed; it only changes location and scale.

Robust scaling based on the median and interquartile range can be preferable when extreme values are present. All transformation parameters must be learned from training data and then applied unchanged to other partitions.

8. Sampling, Imbalance, and Bias

8.1 Sampling bias

Sampling bias occurs when the collected sample systematically differs from the target population. Sources include convenience sampling, non-response, undercoverage, survivorship bias, and changes in data-collection procedures. A model trained mostly on urban users may perform poorly for rural users even if its average test score is high.

Mitigation includes defining the population explicitly, using probability sampling where feasible, stratifying important groups, monitoring representation, collecting additional data, and reporting performance separately by subgroup. Weighting can help in some settings, but it does not automatically repair poor measurement or missing populations.

8.2 Class imbalance

In a rare-event problem, one class may greatly outnumber another. A classifier that predicts “not fraud” for every transaction could achieve high accuracy while detecting no fraud. Therefore, precision, recall, specificity, balanced accuracy, and area-under-curve measures may be more informative than accuracy alone.

Resampling methods include oversampling the minority class, undersampling the majority class, and generating synthetic examples. Resampling must occur within the training data or within each training fold; otherwise duplicated or synthetic information can enter evaluation data. Class weights and decision-threshold adjustment are alternative approaches. The appropriate metric depends on the cost of false positives and false negatives.

9. Splitting Data and Preventing Leakage

A standard supervised-learning workflow separates data into training, validation, and test partitions. The training set is used to fit model parameters and preprocessing parameters. The validation set supports model selection and tuning. The test set is reserved for final, approximately unbiased evaluation.

For independent, identically distributed observations, a random split may be suitable. Classification often uses stratification so that class proportions are approximately preserved. Time-dependent data should generally be split chronologically, because using future observations to predict the past is unrealistic. Grouped data, such as several records from one patient, should be split by group so that the same person does not appear in both training and test sets.

Data leakage occurs when information unavailable at prediction time, or information derived from the evaluation data, influences training. Examples include:

• imputing or scaling before splitting;
• selecting features using the complete dataset;
• including a post-outcome field;
• allowing the same person or device in both training and test sets;
• computing target encoding using validation labels;
• using future time periods in historical prediction.

A reproducible pipeline should place preprocessing inside the training workflow, record random seeds where relevant, version datasets and code, and preserve the untouched test set until final evaluation.

10. Applied Case: Predicting Loan Default

Background

The following is an illustrative case. A financial institution wants to estimate whether an applicant will default within twelve months. The goal is to support responsible credit assessment, not to automate decisions without human oversight.

Problem

The institution combines application data, repayment history, income information, and external credit records. Initial inspection reveals missing income values, inconsistent occupation labels, duplicate applications, and a strong class imbalance because most applicants do not default.

Analysis

The team first defines the unit as one application and establishes the prediction point as the date of application. Any field recorded after that date is excluded. A data dictionary identifies currency, time period, source, and permitted values for each variable. Applications from the same individual are grouped during splitting to prevent identity leakage.

Missing income values are analysed by applicant group. A training-only median imputation is used for numerical income, accompanied by a missingness indicator because non-reporting may be informative. Occupation labels are standardised, rare categories are grouped, and one-hot encoding is fitted only on the training data.

The team discovers that a “collection action” field is created only after missed payments. It is removed because it reveals future information. The data is split chronologically so that earlier applications train the model and later applications test its future performance. Class weights are considered, and recall, precision, and subgroup performance are reported alongside overall measures.

Outcome

The resulting evaluation is more realistic than the initial random split. Performance is lower but more trustworthy. The institution introduces monitoring for population drift, missingness changes, and unequal error rates. Human review remains necessary, and applicants receive appropriate information about decisions according to applicable law and policy.

Learning points

The case demonstrates that data preparation involves domain definitions, temporal reasoning, privacy, fairness, quality controls, and operational monitoring. A higher score obtained through leakage would not represent a better system.

11. Governance, Ethics, and Professional Practice

Data preparation can affect people’s access to employment, credit, healthcare, education, and public services. Practitioners should collect only necessary data, establish lawful authority and appropriate consent, protect sensitive information, restrict access, and apply retention and deletion policies. Direct identifiers should be removed or protected where possible, but de-identification is not always irreversible.

Fairness requires more than deleting protected attributes. Other variables may act as proxies, and historical data may encode unequal treatment. Teams should evaluate subgroup performance, document intended and prohibited uses, involve domain and affected communities where feasible, and provide mechanisms for review and correction.

A professional data workflow should maintain version control, data lineage, quality-test results, preprocessing specifications, model cards or equivalent documentation, and an audit trail of decisions. Reproducibility is both a scientific value and a governance safeguard.

12. Best-Practice Workflow

A practical workflow is:

1. define the question, population, target, and unit of analysis;
2. identify lawful and reliable sources;
3. preserve raw data and document provenance;
4. profile structure, distributions, missingness, and quality;
5. clean using explicit, reviewable rules;
6. integrate sources while checking keys, granularity, units, and time;
7. split data according to the deployment scenario;
8. fit imputers, encoders, scalers, and feature selectors on training data only;
9. address imbalance and evaluate relevant subgroups;
10. test for leakage, bias, and robustness;
11. document the pipeline and monitor data after deployment.

Chapter Summary

Data preparation converts raw observations into trustworthy analytical inputs. Structured, semi-structured, and unstructured data require different forms of interpretation and preparation. Profiling reveals types, distributions, missingness, duplicates, invalid values, and possible bias. Cleaning includes principled treatment of missing values, inconsistent records, and outliers rather than automatic deletion. Integration requires attention to keys, granularity, time, units, conflicts, and provenance.

Transformation includes feature engineering, categorical encoding, and numerical scaling. Parameters for these operations must be learned from training data only. Sampling bias, class imbalance, and leakage can produce misleadingly strong results, so splitting must reflect the real deployment setting, including chronological or group-based constraints. Finally, data preparation has ethical and governance consequences; privacy, fairness, documentation, security, and monitoring are essential parts of professional AI practice.

Key Terms

Data acquisition: The process of obtaining data from one or more sources.

Structured data: Data organised in a predefined tabular schema.

Semi-structured data: Data with organisational features but no rigid table structure.

Unstructured data: Data without a consistent predefined schema, such as images or free text.

Data profiling: Systematic examination of data structure, content, and quality.

Data dictionary: Documentation describing fields, types, units, and meanings.

Data provenance: The recorded origin and transformation history of data.

Missing value: An observation for which the required value is absent or unavailable.

Imputation: Replacing missing values using a defined statistical or domain-based method.

Outlier: An unusually distant or atypical observation.

Interquartile range: The difference between the third and first quartiles.

Feature engineering: Creating or modifying input variables for modelling.

One-hot encoding: Representing categories using binary indicator variables.

Scaling: Rescaling numerical variables to comparable numerical ranges.

Sampling bias: Systematic difference between a sample and its target population.

Class imbalance: Unequal representation of classes in a labelled dataset.

Data leakage: Use of information that should not be available during model training or evaluation.

Stratification: Sampling or splitting while approximately preserving subgroup proportions.

Data quality: The fitness of data for its intended purpose across dimensions such as accuracy and completeness.

Data lineage: A record of how data moves and changes through a system.

Review Questions

Short Answer Questions

1. What distinguishes structured data from unstructured data?
2. Why should preprocessing parameters normally be fitted using training data only?
3. Give two possible causes of missing values.
4. What is the purpose of one-hot encoding?
5. Why can accuracy be misleading for a highly imbalanced classification problem?

Long Answer Questions

1. Explain a complete data-profiling process and identify the quality checks it should include.
2. Compare mean, median, deletion, and model-based approaches to missing-value treatment.
3. Explain how data from multiple sources can be integrated without creating duplicate or misleading records.
4. Discuss the difference between sampling bias, class imbalance, and data leakage.
5. Describe how ethical and governance requirements influence data acquisition and preparation.

Analytical / Application Questions

1. A hospital dataset contains multiple records per patient, and the target indicates readmission within 30 days. Design a train-test splitting strategy and explain how you would prevent leakage.
2. A fraud dataset contains 99.5% legitimate transactions. Recommend an evaluation and preparation strategy, including suitable metrics and safeguards during resampling.
3. A dataset contains income values in different currencies, missing occupation labels, and a field recorded after loan approval. Identify the preparation actions required and justify each one.

Multiple Choice Questions with Answers

Question 1

Which characteristic best describes structured data?

A. It has no recurring organisation or schema

B. It is organised according to a predefined schema, commonly rows and columns

C. It must consist only of images

D. It cannot be stored in a database

Correct Answer: B. It is organised according to a predefined schema, commonly rows and columns

Question 2

Which action is most appropriate before modelling a dataset?

A. Delete every unusually large value

B. Profile variables, missingness, types, ranges, and duplicates

C. Scale the target using the test set

D. Randomly change category labels

Correct Answer: B. Profile variables, missingness, types, ranges, and duplicates

Question 3

Why should an imputer normally be fitted on the training partition only?

A. To make the training set smaller

B. To prevent information from evaluation data entering model development

C. Because validation data cannot contain missing values

D. Because imputation works only on categorical variables

Correct Answer: B. To prevent information from evaluation data entering model development

Question 4

What does one-hot encoding generally do?

A. Converts each category into an indicator variable

B. Orders categories from smallest to largest

C. Removes all categorical variables

D. Replaces missing numerical values with zero

Correct Answer: A. Converts each category into an indicator variable

Question 5

Using the interquartile-range screening rule, a potential high outlier is commonly above which boundary?

A. Q₁ − 1.5IQR

B. Q₁ + 1.5IQR

C. Q₃ + 1.5IQR

D. Q₃ − 1.5IQR

Correct Answer: C. Q₃ + 1.5IQR

Question 6

Which situation is an example of data leakage?

A. Splitting patients by patient identifier

B. Using a post-admission diagnosis to predict admission risk

C. Grouping rare categories before encoding

D. Recording the source of each field

Correct Answer: B. Using a post-admission diagnosis to predict admission risk

Question 7

Which split is generally most appropriate for forecasting future events?

A. A chronological split that trains on earlier data and tests on later data

B. A split that trains on future data and tests on past data

C. A split that duplicates every rare event in the test set

D. A split based only on alphabetical order

Correct Answer: A. A chronological split that trains on earlier data and tests on later data

Question 8

Why can accuracy be unsuitable for detecting rare fraud?

A. Accuracy cannot be calculated for classification

B. A model can obtain high accuracy by predicting the majority class almost always

C. Fraud datasets contain no labels

D. Accuracy measures only numerical variables

Correct Answer: B. A model can obtain high accuracy by predicting the majority class almost always

Question 9

In min–max scaling, what is the usual purpose of using training-set minimum and maximum values?

A. To ensure evaluation information is not used to fit preprocessing

B. To remove all outliers automatically

C. To convert categories into text

D. To guarantee a normal distribution

Correct Answer: A. To ensure evaluation information is not used to fit preprocessing

Question 10

Which issue is most directly associated with a convenience sample?

A. Sampling bias caused by systematic underrepresentation of parts of the population

B. Perfect preservation of population proportions

C. Automatic prevention of missing values

D. Guaranteed causal inference

Correct Answer: A. Sampling bias caused by systematic underrepresentation of parts of the population