AI-Assisted Preparation and Emerging Practices

Introduction

Data preparation has traditionally been treated as a sequence of manual activities: inspecting files, correcting errors, joining tables, transforming variables, and documenting decisions. Contemporary data environments have made this approach insufficient. Organisations now collect high-volume, high-velocity and heterogeneous data from applications, sensors, transactions, documents, images and external platforms. The preparation process must therefore be scalable, reproducible, observable and sufficiently intelligent to identify problems before they affect analysis or automated decisions.

Artificial intelligence (AI) and machine learning increasingly support data profiling, anomaly detection, schema matching, imputation, feature engineering, synthetic-data generation and natural-language interaction with data platforms. These capabilities can reduce repetitive work, but they do not eliminate the need for human judgement. Automated systems may reproduce historical bias, infer incorrect meanings, introduce leakage, or produce transformations that are statistically convenient but operationally inappropriate.

This chapter examines AI-assisted data preparation as a socio-technical process. It evaluates automation techniques, establishes principles for human oversight and validation, and considers emerging practices including data-centric AI, active learning, data observability, real-time analytics, semantic layers and autonomous data workflows.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain automated data profiling and AI-assisted cleaning.
2. Evaluate automated feature engineering and synthetic-data methods.
3. Assess the reliability, fairness and security of generative-AI assistance.
4. Design human-in-the-loop validation and governance controls.
5. Explain data-centric AI, active learning, observability and semantic layers.
6. Analyse a contemporary data-preparation architecture and its limitations.

1. From Rule-Based Preparation to Intelligent Preparation

Earlier preparation systems relied mainly on deterministic rules: trim whitespace, cast a column to a date type, replace a known code, or reject a record that violated a constraint. These rules remain valuable because they are transparent and repeatable. AI-assisted preparation adds probabilistic capabilities. A model may infer that “NY”, “New York” and “New York State” refer to related geographic entities, or identify an unusual transaction without an explicitly coded threshold.

The distinction between automation and intelligence is important. Automation executes a known procedure. AI assistance proposes, ranks or learns procedures from data. In professional settings, proposed actions should be recorded as hypotheses until they pass validation.

A useful preparation pipeline contains six logical stages:

1. Ingestion and identification: locate sources, files, tables and streams.
2. Profiling: measure structure, completeness, distributions, uniqueness and relationships.
3. Transformation and cleaning: standardise, validate, deduplicate and reconcile values.
4. Feature construction: create variables suitable for analysis or modelling.
5. Validation and monitoring: test quality, drift, leakage, fairness and reproducibility.
6. Documentation and release: publish datasets, metadata, lineage and quality evidence.

AI may support every stage, but accountability remains with the data and model owners.

2. Automated Data Profiling

Data profiling is the systematic examination of a dataset’s structure and content. A modern profiler commonly reports data types, null rates, distinct counts, frequency distributions, quantiles, outliers, candidate keys, correlations, dependencies and schema changes.

For a variable X with n observations, the missingness rate is:

where m is the number of missing observations and n is the total number of observations. The result is dimensionless and lies between 0 and 1. It describes absence, not the reason for absence. Missingness may be completely at random, related to observed variables, or related to the unobserved value itself. Consequently, an AI system should not automatically interpret missing values as errors.

For a categorical value v, its empirical frequency is:

A profiler can use rare-value thresholds to identify possible spelling errors, but rarity is not proof of invalidity. A rare diagnosis, high-value customer or unusual sensor reading may be legitimate.

A useful distribution comparison is the population stability index (PSI):

Here, Eᵢ is the expected or reference proportion in bin i, Aᵢ is the current proportion, and the sum covers all bins. Both proportions must be positive; zero values require a documented smoothing convention. PSI is a screening measure rather than a universal test of drift. Its interpretation depends on binning, sample size and the business context.

AI-enhanced profiling can infer semantic types such as email address, telephone number, latitude, currency or personal identifier. It can also identify potential relationships between tables by comparing value patterns. These inferences should be treated as suggestions because semantic ambiguity is common: an integer may represent age, a product code or a count.

Profiling Outputs That Matter Professionally

A useful profile should answer more than “what values occur?” It should identify:

• whether columns have stable meanings across sources;
• whether candidate keys are unique and non-null;
• whether values meet domain and range constraints;
• whether relationships between tables are plausible;
• whether the data differs across time, geography or demographic groups;
• whether changes are attributable to a source-system revision;
• whether the data is suitable for a particular decision.

A profile is therefore both a statistical report and a risk assessment.

3. AI-Assisted Cleaning and Entity Resolution

3.1 Anomaly Detection

Anomaly detection identifies observations that differ substantially from a reference pattern. Methods include robust statistical rules, clustering, isolation-based algorithms, autoencoders and domain-specific constraints. An anomaly is not synonymous with an error. It may represent fraud, a new operating condition, a data-entry mistake or a rare but important event.

A classical standardised score is:

where x is an observation, μ is the reference mean and σ is the reference standard deviation. This score is most interpretable when the distribution is reasonably stable and σ is non-zero. For skewed data or data containing extreme values, robust alternatives based on the median and median absolute deviation may be preferable.

The correct action after detecting an anomaly depends on context: retain and flag it, correct it using verified evidence, exclude it with a documented reason, or investigate it operationally.

3.2 Deduplication and Entity Resolution

Entity resolution determines whether records refer to the same real-world entity. It is used for customers, suppliers, patients, products and addresses. Exact matching is simple but fragile. Probabilistic matching combines evidence such as name similarity, address similarity, telephone agreement and temporal consistency.

A practical system should distinguish:

• match: evidence is sufficiently strong to merge;
• non-match: evidence supports distinct entities;
• review: evidence is ambiguous and requires human assessment.

Over-merging creates false identities, while under-merging fragments an entity. Both errors can distort revenue, risk and fairness analysis. Sensitive fields should be protected during matching, and matching decisions should be auditable.

3.3 Imputation

AI systems can estimate missing values through regression, nearest-neighbour methods, multiple imputation, matrix factorisation or generative models. Imputation should preserve uncertainty. A single predicted value can falsely imply certainty and may reduce natural variability.

For inferential analysis, multiple imputation is often preferable. It creates several plausible completed datasets, analyses each one, and combines results so that uncertainty due to missingness is represented. Imputation must be fitted using training data only when used in predictive modelling; otherwise information from validation or test data may leak into the model.

4. Automated Feature Engineering

Feature engineering converts raw observations into variables that represent useful structure for a task. AI-assisted systems may generate ratios, time-window aggregates, interaction terms, text embeddings, date components, geographic features and transformations selected through search.

For a transaction value y observed over a customer’s previous k transactions, a lagged mean can be defined as:

The index t denotes the current event and yₜ₋ⱼ denotes earlier observations. The feature must use only information available before or at the prediction time. Using future transactions creates temporal leakage.

Automated feature engineering offers speed and can discover interactions that analysts overlook. However, feature quality is not determined only by predictive accuracy. A feature should also be valid, stable, available at deployment, legally permissible and interpretable enough for its use case.

Important controls include:

• time-aware splitting for temporal problems;
• pipeline isolation, so transformations are learned only from training data;
• feature lineage, recording source columns and code;
• availability checks, confirming that a feature exists at decision time;
• stability analysis, examining performance across periods and groups;
• complexity limits, preventing unmanageable feature libraries.

A feature that improves an offline score but cannot be generated in production is not a successful feature.

5. Synthetic Data

Synthetic data is artificially generated data intended to reproduce selected properties of a real dataset without directly releasing the original records. It may be generated using statistical models, generative adversarial networks, variational autoencoders, diffusion models or language models.

Synthetic data has several uses:

• software development without exposing production records;
• simulation of rare events;
• privacy-preserving data sharing;
• testing pipelines and failure conditions;
• augmenting under-represented classes;
• education and prototyping.

Quality has at least three dimensions:

1. Fidelity: Does the synthetic data reproduce relevant distributions, dependencies and temporal structure?
2. Utility: Does it support the intended analysis or model-building task?
3. Privacy: Does it reduce disclosure risk, including membership inference or memorisation?

These dimensions can conflict. A generator that reproduces real records too closely may have high utility but poor privacy. A generator that adds excessive noise may be private but analytically useless. Synthetic data is not automatically anonymous; privacy must be evaluated against a defined threat model.

When synthetic data is used for model training, researchers should compare models trained on synthetic, real and mixed data; inspect subgroup performance; test rare-event behaviour; and document the generator, random seed, training data and limitations.

6. Generative-AI Assistance in Preparation

Generative AI can translate natural-language requests into SQL, Python, validation rules, documentation and transformation proposals. It can suggest mappings between schemas, explain anomalies and summarise profiling results. This creates a more accessible interface for analysts and domain experts.

However, generated code may contain incorrect joins, unsafe operations, invented columns, insecure queries or silent assumptions. A fluent explanation is not evidence of correctness. Every generated artefact should pass code review, tests and data-quality checks.

A controlled workflow is:

1. define the task and constraints precisely;
2. provide only authorised and minimised data to the model;
3. request an explicit plan and assumptions;
4. generate code in a sandbox;
5. test row counts, keys, distributions and known examples;
6. review privacy, security and fairness implications;
7. approve, version and monitor the result.

Organisations should establish policies for confidential data, prompt retention, model providers, intellectual property, audit logs and human approval. Retrieval-augmented systems may improve access to internal definitions, but retrieved documents can themselves be outdated or malicious. Source authority and version must therefore be checked.

7. Human Oversight and Validation

Human oversight is not merely a final approval step. It should be designed throughout the preparation lifecycle. The level of review should be proportional to risk. A formatting correction in a low-risk internal report requires less scrutiny than an imputation affecting credit, healthcare or employment decisions.

A strong validation framework combines four layers:

For a binary classification assistant that proposes data corrections, precision and recall may be useful evaluation measures:

TP denotes correctly identified true corrections, FP denotes incorrect proposed corrections, and FN denotes missed corrections. Precision is important when false corrections are costly; recall is important when undetected errors are costly. The appropriate balance depends on the application.

Human reviewers should see the evidence behind a recommendation, not only the recommendation itself. Review interfaces should show source values, proposed changes, confidence, comparable records, applicable rules and downstream consequences. Review decisions should be captured as labelled feedback for subsequent improvement.

8. Data-Centric AI and Active Learning

Data-centric AI focuses on systematically improving the data used by models rather than concentrating exclusively on model architecture. Activities include correcting labels, increasing coverage of difficult cases, balancing representation, improving consistency and documenting provenance.

Active learning selects the most informative or uncertain examples for human labelling. If a classifier estimates a probability p for a class, uncertainty is often highest near p = 0.5, although uncertainty sampling is not always sufficient. A robust strategy may also seek diversity, rare cases and examples from changing environments.

Active learning can reduce labelling effort, but it may reinforce existing blind spots if the selection mechanism repeatedly ignores unrepresented populations. Evaluation must therefore include random samples, targeted subgroup samples and newly emerging cases.

9. Data Observability and Real-Time Analytics

Data observability extends data quality monitoring into continuous operational awareness. It commonly covers freshness, volume, distribution, schema, lineage and pipeline health. A useful observability system links an alert to affected datasets, jobs, dashboards and models.

Real-time analytics introduces additional constraints: event time differs from processing time, records may arrive late, streams may be duplicated, and schemas may evolve. Preparation must define watermarks, deduplication keys, replay behaviour and late-arrival policies. A transformation that is correct in a batch setting may produce different results in a stream because the complete window is not yet available.

Monitoring should distinguish data drift from concept drift. Data drift concerns changes in input distributions. Concept drift concerns changes in the relationship between inputs and the target. A stable input distribution does not guarantee stable predictive performance.

10. Semantic Layers and Autonomous Workflows

A semantic layer provides governed business meanings for measures, dimensions, entities and relationships. For example, “net revenue” should have an approved definition rather than being independently calculated by every dashboard. Semantic layers improve consistency for analysts and allow natural-language interfaces to generate queries against trusted concepts.

Autonomous data workflows combine agents, orchestration, metadata, tests and monitoring. An agent may detect a schema change, identify downstream assets, propose a transformation and open a review request. Fully automatic remediation is appropriate only for bounded, reversible and low-risk actions. High-impact decisions require approval, traceability and rollback.

The future is therefore more likely to be semi-autonomous than entirely autonomous. Human expertise will shift from editing every record to defining policies, evaluating evidence, managing exceptions and governing consequences.

11. Illustrative Case: Retail Returns Intelligence

Background

A national retailer receives transaction data from stores, websites and third-party marketplaces. Product identifiers, customer records and return reasons differ across systems. The organisation wants to predict excessive returns and improve inventory planning.

Problem

A manually prepared dataset contained duplicate customers, inconsistent product codes, missing return reasons and a feature called “customer lifetime returns” that accidentally included transactions after the prediction date. Offline model performance was high, but performance fell sharply after deployment.

Analysis

Automated profiling detected conflicting data types, a rising null rate in marketplace records and a sudden shift in product-code frequencies. Entity-resolution models produced match, non-match and review categories rather than merging all similar names. A temporal audit identified leakage in the lifetime-return feature.

Approach

The team implemented a governed pipeline. Product mappings were validated by merchandising specialists. Missing return reasons were retained as an explicit category when absence itself was informative, while other fields were multiply imputed for an analytical report. Features were generated using only events available before each prediction timestamp. A semantic layer defined common measures such as gross sales, returned units and net revenue.

A data-observability service monitored freshness, schema changes, null rates and distribution shifts. Generative AI assisted with test generation and documentation, but code execution required review and automated validation. Sensitive customer data was excluded from prompts.

Outcome

The revised model had lower but more realistic offline performance and substantially more stable deployment performance. Analysts gained consistent definitions, while ambiguous entity matches were routed to review. The organisation also discovered that return behaviour differed across product categories and fulfilment channels, leading to operational investigation rather than indiscriminate customer targeting.

Learning Points

The case illustrates that better data preparation may reduce apparent model performance while improving validity. Leakage control, semantic consistency, temporal reasoning and governance were more important than maximising an initial score. AI accelerated discovery, but domain experts remained essential for interpretation and approval.

12. Challenges, Ethics and Regulation

AI-assisted preparation can create several risks:

• bias amplification, when historical data reflects unequal treatment;
• proxy discrimination, when apparently neutral variables encode protected attributes;
• privacy loss, through exposure, linkage or memorisation;
• automation bias, when users over-trust system recommendations;
• opacity, when transformations cannot be reconstructed;
• model collapse or feedback effects, when generated data is repeatedly reused;
• security vulnerabilities, including prompt injection and data exfiltration;
• vendor dependence, when proprietary tools limit portability and auditability.

Governance should include purpose limitation, data minimisation, access control, retention limits, lineage, documented decisions, impact assessment and mechanisms for correction or appeal. Regulatory obligations vary by jurisdiction and application, but high-impact uses generally demand stronger evidence, transparency, human oversight and accountability.

Fairness evaluation should not be reduced to one metric. Different fairness criteria can conflict, especially when base rates differ between groups. Organisations should define the relevant harm, affected populations and acceptable trade-offs with domain and legal expertise.

13. Best-Practice Framework

A mature AI-assisted preparation programme should:

1. begin with a clearly defined analytical or operational purpose;
2. maintain a business glossary and data contracts;
3. profile data at ingestion and continuously thereafter;
4. separate detection from correction;
5. preserve raw data and version every transformation;
6. test for leakage, drift, subgroup performance and reproducibility;
7. require human review for ambiguous or high-impact actions;
8. evaluate synthetic data for fidelity, utility and privacy;
9. use least-privilege access and minimise prompt data;
10. monitor deployed pipelines and models, not only training datasets;
11. provide rollback and incident-response procedures;
12. measure success through quality, reliability, fairness and business value.

Chapter Summary

AI-assisted preparation combines automated profiling, anomaly detection, entity resolution, imputation, feature engineering, synthetic-data generation and generative-AI interfaces. These tools improve scale and efficiency, but statistical plausibility does not guarantee semantic correctness, fairness, privacy or operational suitability. Automated features must be protected against leakage and evaluated for availability and stability. Synthetic data requires separate evaluation of fidelity, utility and disclosure risk.

Human oversight should be risk-based, continuous and evidence-oriented. Data-centric AI emphasises improving labels, coverage and consistency; active learning directs human effort toward informative examples; observability monitors data and pipeline behaviour; real-time systems require explicit treatment of event time and late data; and semantic layers provide governed meanings for analytical concepts. Emerging autonomous workflows should remain bounded, auditable and reversible. The central professional responsibility is not to automate preparation indiscriminately, but to build trustworthy systems in which automation, expertise and governance reinforce one another.

Key Terms

Active learning: A method that selects informative examples for human labelling.

Anomaly detection: Identification of observations that differ substantially from an expected pattern.

Data contract: An explicit agreement about schema, meaning, quality and delivery expectations.

Data-centric AI: An approach that improves data quality, labels, coverage and representation as primary routes to better AI.

Data drift: Change in the distribution of input data over time.

Data observability: Continuous monitoring of data health, freshness, lineage and pipeline behaviour.

Entity resolution: Determining whether records refer to the same real-world entity.

Feature leakage: Use of information unavailable at the time a prediction would be made.

Feature engineering: Construction or transformation of variables for analytical or predictive use.

Generative AI: AI that produces content such as text, code, data or transformations.

Human-in-the-loop: A design in which human judgement reviews, guides or approves automated actions.

Imputation: Estimation or replacement of missing values.

Population stability index: A measure used to screen for distributional change between reference and current data.

Semantic layer: A governed representation of business concepts, measures and relationships.

Synthetic data: Artificially generated data designed to reproduce selected properties of real data.

Temporal leakage: Leakage caused by using future information in a historical prediction task.

Validation: Evidence-based assessment that data or transformations satisfy defined requirements.

Workflow autonomy: The degree to which a system can plan and execute preparation tasks without direct human action.

Review Questions

Short Answer Questions

1. What is the difference between automated profiling and automated cleaning?
2. Why is an anomaly not necessarily an error?
3. Define feature leakage and explain why it is dangerous.
4. What are the three main dimensions for evaluating synthetic data?
5. What purpose does a semantic layer serve?

Long Answer Questions

1. Explain how AI can support the complete data-preparation lifecycle, and identify appropriate human controls at each stage.
2. Compare deterministic data-quality rules with probabilistic anomaly detection and entity-resolution methods.
3. Discuss the benefits and limitations of generative-AI assistance for SQL, code generation and data documentation.
4. Explain how data observability differs from conventional data-quality testing.
5. Critically evaluate the claim that future data workflows will be fully autonomous.

Analytical / Application Questions

1. A customer dataset has a sudden increase in missing values after a source-system migration. Design an investigation using profiling, lineage, domain review and monitoring.
2. A model trained on synthetic healthcare data performs well overall but poorly for a small demographic group. Propose tests and remediation steps.
3. Design a human-in-the-loop workflow for automatically generated feature engineering in a high-impact lending application. Specify approval gates, leakage tests, fairness checks and rollback procedures.

Multiple Choice Questions with Answers

Question 1

Which statement best describes automated data profiling?

A. It permanently corrects every detected error without review.

B. It systematically examines structure, content, distributions and relationships in data.

C. It replaces the need for a data dictionary.

D. It guarantees that a dataset is fit for every analytical purpose.

Correct Answer: B. It systematically examines structure, content, distributions and relationships in data.

Question 2

In the missingness-rate formula r_m = m / n, what does m represent?

A. The number of variables

B. The number of missing observations

C. The mean of the variable

D. The number of distinct values

Correct Answer: B. The number of missing observations

Question 3

Why can an anomaly not automatically be treated as an error?

A. Anomalies are always caused by missing values.

B. Anomalies cannot be detected algorithmically.

C. An anomaly may be a legitimate rare event or a genuine data problem.

D. Anomalies occur only in categorical data.

Correct Answer: C. An anomaly may be a legitimate rare event or a genuine data problem.

Question 4

Which practice most directly prevents temporal feature leakage?

A. Using all available records when calculating every feature

B. Generating features only from information available at prediction time

C. Removing all date variables

D. Replacing future values with zero without documentation

Correct Answer: B. Generating features only from information available at prediction time

Question 5

Which is not one of the three principal evaluation dimensions of synthetic data identified in the chapter?

A. Fidelity

B. Utility

C. Privacy

D. Screen brightness

Correct Answer: D. Screen brightness

Question 6

A system correctly proposes 90 corrections, of which 72 are actually correct. What is its precision?

A. 0.20

B. 0.72

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 7

What is the primary purpose of active learning?

A. To label every available example randomly and equally

B. To select informative examples for human labelling

C. To eliminate the need for labelled data

D. To guarantee equal class frequencies

Correct Answer: B. To select informative examples for human labelling

Question 8

Which capability is most closely associated with data observability?

A. Monitoring freshness, schema, distributions and lineage

B. Creating only synthetic records

C. Replacing all business definitions with model embeddings

D. Encrypting every value using the same public key

Correct Answer: A. Monitoring freshness, schema, distributions and lineage

Question 9

What is a semantic layer designed to provide?

A. A governed representation of business meanings and relationships

B. A randomisation method for anonymising all data

C. A replacement for source-system controls

D. A mechanism that guarantees causal inference

Correct Answer: A. A governed representation of business meanings and relationships

Question 10

Which action is most appropriate for a high-impact, ambiguous automated data correction?

A. Apply it silently because the model has high average accuracy.

B. Delete the record to avoid uncertainty.

C. Route it for human review with supporting evidence and an audit trail.

D. Use the correction only in the test set.

Correct Answer: C. Route it for human review with supporting evidence and an audit trail.