Foundations of Data Preparation and Analysis

Introduction

Data science is often described as the process of turning data into insight. A more accurate description is that it is a disciplined process of turning a decision need into an evidence-supported action. Data preparation and analysis occupy the central position in this process. Poorly defined questions, inappropriate units of analysis, biased samples, ambiguous variables, and misunderstood measurement scales can produce results that are computationally precise but substantively wrong.

This chapter establishes the conceptual foundations required for later work in data cleaning, exploratory analysis, statistical modelling, machine learning, and data-driven decision-making. It introduces the data-to-decision lifecycle; explains how analytical problems are formulated; distinguishes populations, samples, observations, and units of analysis; examines variables and measurement scales; and describes the data-generating process. It also compares three influential process frameworks: Knowledge Discovery in Databases (KDD), CRISP-DM, and OSEMN.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the stages through which data supports a decision.
2. Translate a business, policy, or research concern into an analytical problem.
3. Identify the unit of analysis, population, sample, and observation in a dataset.
4. Classify variables and distinguish measurement scales.
5. Explain how data-generating processes affect validity and bias.
6. Compare KDD, CRISP-DM, and OSEMN and select an appropriate framework.
7. Evaluate data projects critically, including their ethical and governance implications.

1. The Data-to-Decision Lifecycle

A data project begins with a decision context, not with a spreadsheet. A manager may need to decide which customers to contact, a hospital may need to allocate beds, or a government agency may need to identify households requiring support. The analytical task is to produce evidence that is relevant to that decision, sufficiently reliable, and available at the right time.

A useful lifecycle is:

Decision context → problem formulation → data generation and acquisition → preparation → exploration and analysis → interpretation → action → monitoring and learning.

These stages are iterative rather than strictly sequential. Analysis may reveal that the original question was poorly specified; exploration may expose missing data; monitoring may show that a model has degraded because the population has changed.

Decision context

The decision context identifies the decision-maker, available actions, constraints, consequences, and time horizon. A technically excellent prediction is not useful if no action can be taken from it. For example, predicting customer churn is valuable only if the organization can offer a retention intervention before the customer leaves.

Problem formulation

Problem formulation converts a broad concern into a precise analytical question. It should specify:

• the target population;
• the unit of analysis;
• the outcome or quantity of interest;
• the predictors or explanatory factors;
• the prediction or decision time;
• the required level of accuracy and acceptable error;
• the intended use and users of the result.

Data generation and acquisition

Data may be generated by transactions, sensors, experiments, surveys, administrative systems, applications, or digital behaviour. Acquisition includes locating sources, obtaining permissions, recording provenance, and defining inclusion and exclusion rules.

Preparation

Preparation includes profiling, integration, validation, handling missing values, resolving inconsistencies, transforming variables, and documenting decisions. Preparation is not merely cosmetic. It determines what the data means and which observations are represented.

Exploration and analysis

Exploratory analysis examines distributions, relationships, anomalies, and subgroups. Confirmatory analysis evaluates specified hypotheses or estimates. Predictive analysis estimates unknown or future outcomes, while descriptive analysis summarizes observed data. These purposes should not be confused.

Interpretation and action

Results must be translated into implications, uncertainty, limitations, and recommended actions. A correlation may support further investigation but does not automatically justify a causal intervention. Decisions should also consider cost, fairness, operational feasibility, and legal constraints.

Monitoring

After deployment, data quality, population composition, performance, and unintended effects should be monitored. A model trained on historical data may become unreliable when behaviour, policy, technology, or economic conditions change.

2. Formulating an Analytical Problem

A vague question such as “How can we improve sales?” is not directly analyzable. It can be refined into questions such as:

• Which customers are likely to purchase within the next 30 days?
• What is the expected effect of a discount on repeat purchases?
• Which product attributes are associated with returns?

These questions differ in objective, outcome, time horizon, data requirements, and suitable methods.

Descriptive, diagnostic, predictive, and prescriptive questions

Descriptive analysis asks what happened: for example, how monthly revenue varied by region. Diagnostic analysis asks why a pattern may have occurred, using comparisons and relationships while recognizing that association is not necessarily causation. Predictive analysis estimates an unknown or future outcome. Prescriptive analysis evaluates possible actions under constraints.

A sound problem statement often follows this form:

For [population and unit], estimate or explain [outcome] over [time horizon], using [available information], to support [decision], subject to [constraints].

For example: “For active subscription customers, estimate the probability of cancellation within 30 days using information available at the end of the current month, so that the retention team can prioritize outreach, subject to a fixed contact capacity.”

Target leakage and temporal validity

Information used as a predictor must be available at the time the prediction or decision is made. A hospital discharge code may strongly predict readmission, but it is unavailable at admission and therefore creates leakage if used in an admission-time model. Temporal validity requires a clear separation between information available before the prediction point and the outcome observed afterward.

Estimand and operational metric

An estimand is the precise quantity an analysis seeks to learn, such as the mean waiting time, a population proportion, or an average treatment effect. The operational metric is the measure used to judge practical success, such as cost saved, response rate, or service-level compliance. A model metric and a business metric are not interchangeable.

3. Units of Analysis, Populations, and Samples

The unit of analysis is the entity about which a conclusion is made. It may be a person, household, transaction, product, account, session, patient visit, or region. The unit of analysis is distinct from the physical source of data. A transaction table may contain multiple rows for one customer; treating rows as independent customers would distort the analysis.

An observation is a recorded instance of a unit. A dataset can contain repeated observations for the same unit, such as monthly records for an employee. The identifier distinguishes units, whereas a timestamp identifies when an observation was recorded. These concepts must be inspected before aggregation or modelling.

The population is the complete set of units about which a study intends to make claims. The target population is the conceptual group of interest, while the accessible population is the group from which data can actually be obtained. A sample is the subset observed.

If the sample differs systematically from the target population, sampling bias can occur. For instance, an online customer survey may exclude customers with limited internet access. A large sample does not automatically correct systematic undercoverage.

Sampling and generalization

Probability sampling gives units a known, non-zero selection probability and supports formal generalization under appropriate assumptions. Simple random sampling selects units independently with equal probability. Stratified sampling divides the population into meaningful groups and samples within each. Cluster sampling selects groups, such as schools or branches, and observes units within selected groups.

Non-probability samples—such as convenience, voluntary-response, or judgment samples—may be useful operationally but require caution when making population claims. Weighting, calibration, and post-stratification can reduce some discrepancies, but they cannot guarantee removal of unmeasured selection bias.

Ecological and atomistic errors

An ecological error occurs when relationships observed for groups are assumed to hold for individuals. Conversely, an atomistic error occurs when individual-level relationships are assumed to describe groups. Aggregated regional unemployment and health outcomes, for example, cannot by themselves establish relationships for individual residents.

4. Variables and Data Structures

A variable is a characteristic that can take different values across units. A constant has one value within the relevant dataset or analysis. Variables may be classified by role and by type.

Numeric variables may be discrete, such as number of claims, or continuous, such as mass or duration. Categorical variables may be binary, nominal, or ordinal. A date is often stored as text but represents a structured temporal variable; extracting month or elapsed time should be justified by the analytical question.

A tidy rectangular dataset generally has one observation per row, one variable per column, and one value per cell. This is a useful organization, not a universal law. Network, document, spatial, and event-stream data may require other representations.

5. Measurement Scales

Measurement scales determine which comparisons and operations are meaningful.

The distinction between interval and ratio scales is important. A temperature of 20 °C is not twice as hot as 10 °C because Celsius has an arbitrary zero. In contrast, 20 kg is twice 10 kg because zero kilograms represents absence of mass and ratios are meaningful.

Measurement scale does not mechanically determine every permissible method. An ordinal variable may be modelled using ordinal regression, represented with indicator variables, or summarized using proportions, depending on assumptions and purpose. Treating categories as numeric scores can impose unjustified equal spacing.

Reliability and validity

Reliability concerns consistency of measurement. Validity concerns whether the measurement represents the intended construct. A scale can be reliable but invalid: a faulty instrument may produce highly consistent but incorrect readings. Construct validity is especially important for concepts such as well-being, trust, or skill, which are not directly observed.

6. The Data-Generating Process

A dataset is an output of a process, not a neutral mirror of reality. The data-generating process includes the underlying phenomenon, measurement instruments, human decisions, system rules, missingness mechanisms, and recording practices.

Suppose an organization records complaints. The observed complaint count depends not only on service quality but also on awareness of the complaint channel, willingness to complain, accessibility, staff recording practices, and whether complaints are resolved or removed. Treating the count as a direct measure of poor service may therefore be misleading.

A useful conceptual distinction is:

• Signal: systematic information relevant to the phenomenon or decision.
• Noise: random variation or irrelevant detail.
• Bias: systematic deviation caused by selection, measurement, processing, or modelling.

Missing data

Missingness may be completely at random, related to observed variables, or related to unobserved values. If high-income respondents are less likely to report income, missingness is not harmless. Deleting incomplete records may alter the target population and reduce precision. Imputation can be useful, but imputed values are estimates and should not be treated as observed facts without qualification.

Selection and measurement bias

Selection bias occurs when inclusion depends on factors related to the analysis. Measurement bias occurs when variables are recorded inaccurately or differently across groups. Algorithmic systems may amplify historical inequities when labels reflect unequal access, differential scrutiny, or institutional decisions rather than the underlying construct.

Causal interpretation

Association describes how variables vary together. Causal claims require a defensible identification strategy, such as randomization, a natural experiment, a carefully justified quasi-experimental design, or strong assumptions in an observational study. A predictive feature can be useful without being causal, and a causal factor may have limited predictive power.

7. Frameworks for Data Work

Knowledge Discovery in Databases (KDD)

KDD is commonly described as an iterative process comprising selection, preprocessing, transformation, data mining, and interpretation/evaluation. It emphasizes discovering useful and understandable patterns in large datasets. Data mining is one stage within KDD, not a synonym for the entire process.

Its strength is conceptual clarity about the relationship between raw data, transformation, pattern discovery, and interpretation. Its limitation is that organizational deployment and business understanding may receive less explicit attention than in managerial frameworks.

CRISP-DM

CRISP-DM—Cross-Industry Standard Process for Data Mining—contains six iterative phases:

1. Business understanding
2. Data understanding
3. Data preparation
4. Modelling
5. Evaluation
6. Deployment

CRISP-DM begins with business objectives and repeatedly returns to them. It is valuable because it connects technical work to organizational goals and deployment. However, it does not prescribe a specific statistical method, governance model, or fairness procedure; these must be added by the project team.

OSEMN

OSEMN is a concise workflow: Obtain, Scrub, Explore, Model, and iNterpret. It is memorable and practical for analysts working through a dataset. “Scrub” includes cleaning and restructuring, while “interpret” includes communicating findings and implications. OSEMN is less explicit than CRISP-DM about business-case definition, deployment, and monitoring.

These frameworks are complementary rather than competing doctrines. A project may use CRISP-DM for governance, KDD concepts for discovery, and OSEMN as an analyst’s daily workflow.

8. An Illustrative Case: Predicting Missed Clinic Appointments

This is a fictional illustrative case.

Background and problem

A public clinic experiences missed appointments, reducing capacity and delaying care. It proposes a model to identify appointments requiring reminder support. The decision is not simply “predict no-shows”; it is whether a reminder should be sent, through which channel, and when.

Unit, population, and outcome

The unit of analysis is a scheduled appointment, not a patient. The target population is appointments scheduled during the next quarter. The outcome is whether the appointment is missed without timely cancellation. A patient may contribute multiple appointments, so dependence between observations must be considered.

Data and data-generating process

Available information includes appointment lead time, clinic, appointment type, prior attendance, reminder history, and communication preference. Prior attendance is partly influenced by access barriers and previous reminder policies. Communication preference may be missing more often for newly registered patients, creating a potential selection issue.

Preparation and analysis

The team defines a prediction time: 48 hours before the appointment. Only information available then is included. Historical records are split chronologically rather than randomly to better simulate future use. Duplicate appointments are reconciled, impossible dates are investigated, and missingness patterns are reported. The team evaluates discrimination and calibration, but also examines reminder burden and error rates across language and accessibility groups.

Decision and outcome

Instead of automatically denying or changing appointments, the model prioritizes supportive reminders. Staff retain oversight, and patients can update communication preferences. Monitoring tracks performance drift, opt-out rates, and whether reminders improve attendance rather than merely identify likely no-shows.

Learning points

The case demonstrates that the unit of analysis affects the design; temporal availability prevents leakage; data-generating processes affect fairness; and a model is only one component of a decision system. Evaluation must include operational and ethical outcomes, not only predictive accuracy.

9. Governance, Ethics, and Contemporary Practice

Data preparation involves judgments about inclusion, exclusion, representation, and acceptable error. Important safeguards include data minimization, purpose limitation, access control, retention rules, documentation, and secure handling of personal information. Legal requirements vary by jurisdiction, but ethical responsibility extends beyond minimum legal compliance.

Teams should maintain data dictionaries, provenance records, transformation logs, versioned datasets, and model cards or equivalent documentation. Sensitive attributes may need protection, but omitting them from analysis does not remove inequity; it may make disparities harder to detect. Fairness assessment should be connected to the actual decision, affected groups, and available remedies.

Modern data platforms and generative AI can accelerate profiling, coding, and documentation. They can also introduce fabricated explanations, privacy leakage, automation bias, and unverified transformations. AI-assisted preparation requires human review, reproducibility, access controls, and explicit testing against the original data and requirements.

Best Practices

1. Start with the decision and define the action that analysis will support.
2. Write down the unit of analysis and prediction time before modelling.
3. Distinguish the target population from the accessible sample.
4. Create a data dictionary describing meaning, type, units, and provenance.
5. Inspect missingness, selection, measurement, and temporal processes.
6. Separate exploratory discovery from confirmatory claims.
7. Prevent target leakage and evaluate on data representing future use.
8. Report uncertainty, limitations, subgroup performance, and operational consequences.
9. Preserve reproducible records of preparation decisions.
10. Monitor deployed analyses because populations and processes change.

Chapter Summary

Data preparation and analysis form an iterative data-to-decision lifecycle. Reliable analysis begins with a well-defined decision and analytical problem, including the population, unit of analysis, outcome, predictors, time horizon, and intended action. Populations and samples must be distinguished, and generalization depends on how observations are selected. Variables have substantive roles and measurement scales that govern meaningful comparisons. Data-generating processes create signal, noise, missingness, selection effects, and measurement bias; therefore, data must be interpreted in context rather than treated as a neutral representation of reality. KDD emphasizes knowledge discovery, CRISP-DM emphasizes end-to-end project development, and OSEMN provides a practical analyst workflow. Ethical governance, documentation, temporal validity, fairness, and monitoring are essential components of professional data practice.

Key Terms

Analytical problem: A precisely specified question linking data to a decision or research objective.

Bias: Systematic deviation from the intended representation or quantity.

CRISP-DM: An iterative framework of business understanding, data understanding, preparation, modelling, evaluation, and deployment.

Data-generating process: The substantive, technical, and institutional process that produces recorded data.

Estimand: The precisely defined quantity an analysis seeks to learn.

Feature: An input variable used in analysis or modelling.

KDD: Knowledge Discovery in Databases, a framework for selecting, preprocessing, transforming, mining, and interpreting data.

Measurement scale: The structure governing meaningful comparisons among variable values.

Missingness: The absence of an expected data value.

Observation: A recorded instance of a unit of analysis.

OSEMN: Obtain, Scrub, Explore, Model, and iNterpret.

Population: The complete set of units about which claims are intended.

Sample: The subset of units actually observed.

Selection bias: Systematic distortion caused by how units enter or remain in a dataset.

Target leakage: Use of information unavailable at the time a prediction or decision would be made.

Unit of analysis: The entity about which an analytical conclusion is made.

Validity: The extent to which a measure or analysis represents what it is intended to represent.

Variable: A characteristic that can take different values across units.

Review Questions

Short Answer Questions

1. What is the difference between a population and a sample?
2. Define the unit of analysis and explain why it matters.
3. Distinguish nominal, ordinal, interval, and ratio measurement scales.
4. What is target leakage?
5. Name the five stages of OSEMN.

Long Answer Questions

1. Explain the data-to-decision lifecycle and show why it is iterative.
2. Compare KDD, CRISP-DM, and OSEMN in terms of purpose, strengths, and limitations.
3. Discuss how the data-generating process can create selection and measurement bias.
4. Explain how a vague business question can be transformed into a defensible analytical problem.
5. Discuss why reliability, validity, and measurement scale matter in data analysis.

Analytical / Application Questions

1. A retailer has one row per purchase but wants to predict customer churn. Identify the likely unit-of-analysis problem and propose a suitable restructuring.
2. A hospital uses information recorded after discharge to predict readmission at admission. Explain the validity problem and design a leakage-resistant evaluation approach.
3. A survey of employee well-being has substantially more missing responses from temporary workers. Identify possible data-generating mechanisms and recommend analyses or safeguards before generalizing results.

Multiple Choice Questions with Answers

Question 1

Which entity is the unit of analysis in a study of monthly sales revenue by store?

A. Each individual transaction

B. Each store-month observation

C. The company’s annual report

D. Each product advertisement

Correct Answer: B. Each store-month observation

Question 2

Which measurement scale has ordered categories but does not necessarily have equal intervals?

A. Nominal

B. Ordinal

C. Interval

D. Ratio

Correct Answer: B. Ordinal

Question 3

Which framework explicitly includes business understanding and deployment as named phases?

A. CRISP-DM

B. OSEMN

C. Only descriptive statistics

D. A data dictionary

Correct Answer: A. CRISP-DM

Question 4

Why is using a discharge diagnosis to predict readmission at hospital admission problematic?

A. It is a nominal variable

B. It creates target leakage

C. It reduces the sample size to zero

D. It guarantees causal inference

Correct Answer: B. It creates target leakage

Question 5

Which statement best describes a population?

A. The variables in a dataset

B. The complete set of units about which claims are intended

C. Only the observations with no missing values

D. A model’s training error

Correct Answer: B. The complete set of units about which claims are intended

Question 6

A temperature measured in degrees Celsius is generally treated as which scale?

A. Nominal

B. Ordinal

C. Interval

D. Ratio

Correct Answer: C. Interval

Question 7

In OSEMN, what does the letter S represent?

A. Sample

B. Scrub

C. Standardize

D. Simulate

Correct Answer: B. Scrub

Question 8

Which situation most directly illustrates selection bias?

A. A sensor records measurements in kilograms

B. A survey excludes people without internet access

C. A variable is renamed in a data dictionary

D. A date is converted to a month number

Correct Answer: B. A survey excludes people without internet access

Question 9

What is the primary purpose of a data dictionary?

A. To replace the analytical question

B. To document variable meaning, type, units, and provenance

C. To guarantee that missing data are random

D. To select a machine-learning algorithm automatically

Correct Answer: B. To document variable meaning, type, units, and provenance

Question 10

Which statement about association and causation is correct?

A. Every association is causal

B. Prediction always proves causation

C. Causal claims require a defensible identification strategy

D. Causation can be established from a larger sample alone

Correct Answer: C. Causal claims require a defensible identification strategy