Engineering Data for Analytics and AI

Introduction

Modern organisations do not obtain value from data merely by collecting large volumes of it. Value emerges when data is made discoverable, trustworthy, timely, interpretable and usable by decision-makers and computational models. Business intelligence requires governed measures and dimensions; machine learning requires correctly labelled training examples and stable features; generative AI requires well-structured documents, reliable metadata and retrieval systems that can provide relevant evidence to a language model.

Data engineering for analytics and AI therefore extends beyond extraction, transformation and loading. It includes semantic modelling, data quality, feature engineering, feature serving, document processing, vector search, evaluation, lineage, security and operational monitoring. The central engineering challenge is to create a dependable path from raw observations to decisions or model outputs while preventing leakage, bias, inappropriate access and silent degradation.

This chapter examines that path. It treats datasets and features as governed products rather than temporary technical artefacts, and connects data engineering with MLOps, retrieval-augmented generation (RAG) and responsible AI.

Learning Objectives

By the end of this chapter, a student should be able to:

1. distinguish analytical datasets, features, embeddings and governed data products;
2. design batch and streaming feature pipelines for machine-learning use cases;
3. explain the purpose and architecture of offline and online feature stores;
4. assess vector databases and RAG pipelines, including document-quality risks;
5. identify leakage, drift, bias, lineage and reproducibility problems;
6. integrate data pipelines with model training, deployment and monitoring;
7. propose governance controls for analytics, machine learning and generative AI.

1. From Operational Data to Decision-Ready Data

Operational systems are designed to execute transactions: orders are placed, payments are processed and support tickets are updated. Analytical systems are designed to answer questions across time and entities. AI systems add further requirements: the data must represent the prediction context, labels must be correctly defined, and the transformation used during training must be reproducible at inference time.

A useful conceptual chain is:

source events → validated data → curated entities and measures → features or knowledge units → model or dashboard → decision → observed outcome

Each transition can introduce error. A timestamp may use local time in one system and UTC in another. A customer identifier may be duplicated after a merger. A revenue measure may be calculated differently by finance and marketing. A document parser may remove table headings, making a retrieved passage misleading. Data engineering must therefore make meaning explicit.

1.1 Analytical datasets

An analytical dataset is a structured collection of observations designed for a defined analytical purpose. Its design should specify:

• Grain: what one row represents, such as one order, one customer-day or one sensor reading.
• Keys: attributes that identify rows and relationships.
• Time semantics: event time, processing time, effective time and expiry time.
• Measures: numerical quantities such as revenue, quantity or latency.
• Dimensions: descriptive attributes such as region, product category or customer segment.
• Labels: outcomes used for supervised learning, including the time at which they become known.
• Provenance: source systems, transformations, owners and quality evidence.

A dataset with an unclear grain is especially dangerous. If a customer table is joined directly to an order table, a customer with five orders appears five times. A subsequent average or count may then be distorted. Correct joins require an explicit relationship and, often, aggregation to a common grain before joining.

1.2 Data products and contracts

A data product is a maintained dataset or service with identified users, an owner, documented semantics, quality expectations and an access interface. A data contract formalises expectations between producers and consumers. It may specify schema, allowed nulls, units, valid ranges, update frequency, compatibility rules and responsibility for breaking changes.

Contracts do not eliminate change; they make change visible. Schema evolution should distinguish additive changes, which may be backward-compatible, from breaking changes, such as changing a field from currency to a free-text description. Contract tests can reject invalid records before they contaminate downstream assets.

2. Data Quality as an Engineering Discipline

Data quality is multidimensional. Common dimensions include completeness, validity, accuracy, consistency, uniqueness, timeliness and fitness for purpose. A dataset may be complete but inaccurate, or valid according to a schema but semantically wrong.

For a field, a simple completeness rate is:

Here, n_non-null is the number of non-null observations and n_total is the total number of observations in the assessment period. The rate is dimensionless and applies when null status is an appropriate proxy for presence. It does not prove that present values are correct.

A validity rule can be represented as:

For example, a valid order quantity might satisfy q > 0, while a valid percentage might satisfy 0 ≤ p ≤ 1 when stored as a fraction. Quality thresholds should be tied to business impact. A 99.9% completeness target may be essential for regulatory reporting but insufficient if the missing 0.1% contains all high-value transactions.

Quality checks should occur at several stages:

1. Ingestion: file presence, encoding, schema and transport integrity.
2. Transformation: key uniqueness, referential integrity, valid ranges and reconciliation totals.
3. Publication: freshness, distribution changes, row counts and semantic tests.
4. Consumption: suitability for a particular model, report or decision.

Data observability extends monitoring from pipeline availability to freshness, volume, schema, distribution and lineage. Alerts should be actionable and should identify the affected data products and consumers.

3. Feature Engineering for Machine Learning

A feature is an input representation used by a model. It may be a raw attribute, a transformation, an aggregate, a categorical encoding, a text embedding or a signal derived from several sources. Feature engineering is not merely a preprocessing step: it expresses assumptions about what information is relevant and what will be available when a prediction is made.

3.1 Time-aware feature construction

Suppose a model predicts whether an account will default at time t₀. A feature such as the number of missed payments in the previous 90 days is valid only if it uses records available by t₀. A future payment reversal, even if present in a warehouse at training time, must not be included.

For an entity e, a count over a historical window can be defined as:

Here, t_i is the event time of an event associated with e, W is the look-back duration, and 1(·) is an indicator equal to 1 when the condition is true and 0 otherwise. The interval convention must be documented; using t_i < t₀ instead of t_i ≤ t₀ may be necessary when events occurring at the prediction instant are not yet available.

For a mean transaction value, the corresponding feature is:

where a_i are eligible historical amounts and N is their count. The feature is undefined when N = 0 unless a justified missing-value policy is specified. Replacing it with zero can be misleading because “no transactions” and “transactions averaging zero” have different meanings.

3.2 Training-serving consistency

Training-serving skew occurs when a feature is computed differently for historical training data and live inference. Typical causes include different code paths, different time-zone logic, inconsistent null handling and use of warehouse-only fields. A robust design defines features once, tests them against historical and real-time inputs, and records their point-in-time semantics.

A feature specification should include:

3.3 Feature selection and leakage

Feature selection should consider predictive usefulness, stability, cost, interpretability, fairness and availability. A feature with strong historical correlation may be unsuitable if it is a proxy for a protected attribute or is unavailable at decision time.

Target leakage occurs when a feature contains information caused by, or recorded after, the target outcome. For example, using a “case closure reason” to predict whether a case will be escalated is leakage if the closure occurs after escalation. Leakage commonly produces excellent validation scores and poor production performance.

Point-in-time correct joins address this problem by joining each training observation only with records whose effective time and availability time precede the observation time. This is more demanding than an ordinary relational join because it models historical knowledge, not merely current warehouse state.

4. Feature Stores

A feature store is a platform or architectural pattern for managing feature definitions, historical feature values and serving interfaces. Its primary purpose is consistency and reuse across model development and production.

4.1 Offline and online stores

The offline store contains historical, often columnar, feature values used for training, backtesting and analysis. It supports large scans and point-in-time extraction. The online store provides low-latency access to current feature values during prediction, commonly through a key-value or low-latency database.

A typical flow is:

1. events arrive through batch files or streams;
2. transformations compute validated features;
3. historical values are written to the offline store;
4. the latest eligible values are materialised in the online store;
5. training jobs retrieve point-in-time correct data;
6. deployed models request features by entity key;
7. serving logs record feature versions and prediction context.

The two stores create an engineering trade-off. Offline systems optimise throughput and history; online systems optimise latency and availability. Materialisation delays can cause a newly computed value to be absent online. Systems should expose freshness and last-update timestamps rather than implying that every value is current.

4.2 Architecture and operating choices

A feature store may be centralised, federated or embedded in a broader data platform. Centralisation improves discoverability and governance but can become a bottleneck. Federated ownership preserves domain expertise but requires strong contracts and common metadata.

Important design decisions include:

• batch, micro-batch or event-driven computation;
• event time versus processing time;
• feature backfills and correction of historical values;
• idempotency, so replaying an event does not duplicate its effect;
• point-in-time retrieval;
• versioning and deprecation;
• encryption, access control and audit logging;
• monitoring of freshness, null rates, ranges and distributions.

Feature stores do not automatically create good features. They can efficiently operationalise a flawed definition, encode bias at scale or make leakage easier to reuse. Governance must therefore cover both the platform and the feature semantics.

5. Preparing Unstructured Data

Unstructured data includes documents, emails, web pages, images, audio and video. It is not truly structureless; rather, its structure is implicit, heterogeneous or difficult to represent. Preparation for analytics and AI requires preservation of meaning, provenance and access constraints.

A document pipeline commonly includes:

1. acquisition and source authentication;
2. malware scanning and file-type validation;
3. text extraction or optical character recognition;
4. layout, table and heading detection;
5. language identification and encoding normalisation;
6. removal of duplicated boilerplate;
7. segmentation into meaningful chunks;
8. metadata enrichment;
9. access-control propagation;
10. quality assessment and indexing.

Chunking is a semantic decision. Chunks that are too short lose context; chunks that are too long reduce retrieval precision and may exceed model context limits. A chunk should generally preserve a coherent proposition, procedure or section. Overlap can reduce boundary loss, but excessive overlap increases storage and duplicate retrieval.

Quality controls should test OCR confidence, extraction completeness, language, document dates, page references, duplicate rates and table preservation. A system that retrieves an incorrect number from a badly parsed table may produce a fluent but dangerous answer.

6. Embeddings, Vector Databases and Retrieval

An embedding maps an object such as a text segment into a numerical vector. Semantically related objects are intended to occupy nearby regions of the embedding space. The embedding model determines what similarity means; a general-purpose model may perform poorly on legal clauses, scientific formulas or multilingual content without suitable evaluation.

A common similarity measure is cosine similarity:

Here, u and v are non-zero vectors, u · v is their dot product, ||u|| and ||v|| are their Euclidean norms, and θ is the angle between them. Cosine similarity is scale-invariant and is applicable when directional similarity is meaningful. If vectors are normalised to unit length, cosine similarity equals their dot product.

A vector database stores embeddings with identifiers, metadata and an index for approximate or exact nearest-neighbour search. Metadata filtering is essential. A query for a policy document should be restricted by tenant, document permissions, effective date and possibly jurisdiction before or during similarity search.

Approximate nearest-neighbour methods trade a small amount of recall for lower latency and memory use. Index parameters should be evaluated against a representative workload rather than selected solely from vendor defaults. Useful retrieval metrics include:

• Recall@k: proportion of relevant items appearing among the first k results;
• precision@k: proportion of the first k results that are relevant;
• latency: time to return results under realistic concurrency;
• filter correctness: whether access and metadata constraints are respected.

Retrieval quality is not equivalent to generation quality. A language model may ignore relevant evidence, misinterpret it or answer beyond it. Evaluation should therefore separate retrieval, grounding, answer correctness, citation completeness and safety.

7. Retrieval-Augmented Generation

RAG combines retrieval from an external knowledge source with generation by a language model. A simplified pipeline is:

query → query normalisation or rewriting → permission-aware retrieval → reranking → context assembly → generation → citation and validation

RAG is useful when information changes frequently, must be traceable or is too large to place in model parameters. It does not guarantee truth. If the index is stale, the parser is faulty or retrieval misses the governing clause, generation remains unreliable.

A production RAG design should address:

• document version and effective-date selection;
• tenant and user authorisation;
• prompt-injection content inside retrieved documents;
• citation mapping to source pages or record identifiers;
• context-window budgeting;
• answer abstention when evidence is insufficient;
• protection of confidential prompts and retrieved passages;
• evaluation on difficult, ambiguous and adversarial queries.

Prompt injection is a data-security problem as well as a model problem. Retrieved content should be treated as untrusted input. Instructions found in a document must not automatically override system policies or user permissions.

8. MLOps Integration and the Data Lifecycle

MLOps connects data, code, models and operations. A mature lifecycle includes:

1. Problem definition: decision, population, prediction horizon and success metric.
2. Data discovery: sources, ownership, consent, quality and representativeness.
3. Dataset construction: reproducible extraction, labels, point-in-time joins and splits.
4. Experimentation: versioned code, data references, feature definitions and parameters.
5. Validation: statistical performance, robustness, fairness, security and leakage tests.
6. Deployment: packaging, approvals, rollback and infrastructure controls.
7. Monitoring: data quality, drift, performance, latency, cost and incidents.
8. Retirement: deprecation, archival, deletion and documentation.

Random train-test splitting is not always valid. For temporal prediction, a chronological split better represents deployment: earlier observations train the model and later observations test it. For entities appearing multiple times, splitting individual rows can place the same entity in both sets and overstate generalisation. Group-based splitting may be required.

Reproducibility requires recording dataset versions, source snapshots, transformation code, feature definitions, model artefacts, environment dependencies and random seeds where relevant. Exact reproducibility may still be difficult when upstream data changes or hardware-dependent operations are nondeterministic; such limitations should be documented rather than concealed.

9. Drift, Leakage and Monitoring

Data drift is a change in the distribution of input data. Concept drift is a change in the relationship between inputs and outcomes. Label drift is a change in outcome prevalence. These phenomena have different remedies.

For a binary outcome, prevalence can be monitored as:

where n_positive is the number of positive outcomes and n_total is the number of labelled observations. A change in p may reflect population change, policy change or labelling change. It does not by itself prove model deterioration.

A simple standardised difference for a numerical feature is:

where μ₁ and μ₀ are means in two periods or populations, and s_p is a pooled standard deviation. This measure is useful for comparison when the variable is approximately numerical and the variance is interpretable, but it should not replace distribution-sensitive tests or domain review.

Monitoring should connect a signal to an action. For example, a sudden rise in missing customer income may trigger source investigation; a gradual change in product mix may require recalibration; a decline in recall may require retraining or a retrieval-index update. Alerts without ownership and runbooks create noise rather than control.

10. Governance, Ethics and Regulation

Data and AI governance should be risk-based and proportional to the use case. Core controls include:

• purpose limitation and documented lawful basis;
• data minimisation and retention schedules;
• role-based or attribute-based access;
• encryption in transit and at rest;
• pseudonymisation where identity is not required;
• lineage, audit logs and approval records;
• correction, deletion and access procedures where applicable;
• fairness assessment and human oversight;
• incident response and vendor accountability.

Privacy risk can arise from direct identifiers, quasi-identifiers, embeddings and retrieved text. Removing names does not necessarily prevent re-identification. Sensitive data may also leak through model outputs, logs, prompts or debug traces.

Fairness assessment should examine the decision context, protected groups, measurement quality and possible disparate effects. A single aggregate accuracy measure can conceal unequal error rates. Technical mitigation, such as reweighting or threshold adjustment, cannot substitute for questioning whether the decision should be automated or whether the target itself is appropriate.

Governance must also cover intellectual property, copyright, records management, accessibility and explainability. For generative systems, citations and evidence traces support review but do not guarantee that an answer is legally or factually correct.

11. Illustrative Case: Customer-Service Knowledge Assistant

Background

A telecommunications company operates a customer-service centre. Agents consult product manuals, tariff documents, outage procedures and regulatory notices. The documents are stored in multiple repositories and frequently revised.

Problem

Agents report inconsistent answers and spend substantial time searching. A proposed RAG assistant must retrieve the correct policy for the customer’s region and contract date. The company must prevent one customer’s information from appearing in another customer’s session.

Analysis

An initial prototype extracts text from PDFs and retrieves the top five chunks using embeddings. Evaluation reveals three weaknesses. First, tables containing price bands are extracted in the wrong reading order. Second, retrieval returns obsolete policies because effective dates are not indexed. Third, access filtering is applied after retrieval, allowing sensitive passages to influence ranking and potentially appear in logs.

Approach and solution

The engineering team establishes a document contract containing document identifier, owner, jurisdiction, effective start, effective end, confidentiality class, version and source location. A parser preserves headings, table row labels and page references. Each chunk inherits the document’s permissions and temporal metadata.

Retrieval first applies tenant, region, role and effective-date filters, then performs vector search and lexical reranking. The generation prompt requires citations and an explicit abstention when no retrieved passage supports the answer. Test sets contain ordinary, ambiguous, obsolete-policy and prompt-injection queries. Retrieval recall, citation correctness, answer accuracy, latency and unauthorised-disclosure rate are measured separately.

Outcome

The assistant is released initially as an agent-support tool rather than an autonomous customer-facing system. Human agents can inspect source passages and report incorrect or missing documents. Monitoring tracks parser failures, index freshness, retrieval quality and policy changes. When a regulation changes, the affected documents are reprocessed and the old versions remain available for historical audit but are excluded from current-date retrieval.

Learning points

The case demonstrates that RAG quality depends on document engineering, metadata and access control as much as on the language model. Temporal and permission-aware retrieval are data-engineering requirements. Human review, abstention and evaluation on adverse cases are essential operational controls.

12. Best-Practice Design Checklist

Before releasing an analytical or AI data product, ask:

1. Is the row or document grain unambiguous?
2. Are owners, consumers, lineage and quality expectations documented?
3. Are event time, availability time and time zones explicit?
4. Can every feature be reproduced point-in-time correctly?
5. Are training and serving transformations consistent?
6. Are schema, freshness, distribution and access checks automated?
7. Are unstructured documents parsed, deduplicated and versioned correctly?
8. Does retrieval enforce permissions before evidence reaches generation?
9. Are leakage, drift, bias and representativeness tested?
10. Are model, data, feature and index versions linked for audit and rollback?
11. Is there an owner for every alert and incident response procedure?
12. Can the system abstain or fall back safely when evidence is inadequate?

Contemporary Developments

Current engineering practice is moving toward unified lakehouse architectures, declarative data quality, real-time feature computation, multimodal retrieval and smaller domain-specific models. Data contracts are increasingly treated as interfaces between autonomous teams. Metadata systems are becoming active components of governance rather than passive catalogues.

Synthetic data can support testing and privacy protection, but it may reproduce source bias or fail to represent rare cases. Foundation models can reduce feature-engineering effort for text and images, yet their embeddings may be difficult to interpret and may change across model versions. Agentic systems introduce additional concerns because they can call tools, modify records or initiate workflows. Their data and permission boundaries must be evaluated as carefully as their language outputs.

The enduring principle is separation of concerns with traceability: data preparation, retrieval, inference and action should be individually observable and jointly accountable.

Chapter Summary

Engineering data for analytics and AI requires more than moving data into a warehouse or training a model. A dependable data product has a defined grain, semantics, owner, lineage, quality controls and access policy. Feature engineering must respect event time and prediction-time availability; point-in-time correctness and prevention of target leakage are fundamental. Feature stores coordinate historical offline analysis with low-latency online serving, but they do not guarantee sound feature definitions.

Unstructured-data pipelines must preserve layout, metadata, provenance, permissions and version history. Vector databases support semantic retrieval through embeddings, while RAG combines retrieval with language generation. Retrieval quality, grounding, citation correctness and security must be evaluated separately. MLOps integrates data, features, models, deployment and monitoring through reproducible lifecycle controls. Drift, leakage, bias, privacy, prompt injection and poor document quality are operational risks requiring explicit governance. The strongest systems connect technical design with business ownership, human oversight and measurable outcomes.

Key Terms

Feature: An input representation used by a predictive or analytical model.

Feature store: A system for defining, storing, governing and serving features.

Offline store: Historical feature storage optimised for training and analysis.

Online store: Low-latency storage used to serve current feature values.

Data contract: An agreement specifying a data product’s schema, semantics and quality expectations.

Data product: A maintained dataset or data service with users, ownership and governance.

Data lineage: The recorded origin, transformation and movement of data.

Point-in-time correctness: Use of only information available at the relevant historical prediction time.

Target leakage: Inclusion of information unavailable at prediction time or caused by the outcome.

Training-serving skew: Difference between feature computation during training and production inference.

Embedding: A numerical vector representation of an object, often designed to preserve semantic relationships.

Vector database: A database supporting storage and similarity search over vectors and metadata.

Cosine similarity: A directional similarity measure based on the angle between two non-zero vectors.

Retrieval-augmented generation: Generation grounded by retrieved external information.

Data drift: Change in the distribution of input data over time or across populations.

Concept drift: Change in the relationship between inputs and the target outcome.

Data observability: Monitoring of data freshness, volume, schema, distribution and lineage.

MLOps: Practices integrating machine learning development, deployment, operation and governance.

Prompt injection: Untrusted instructions in input or retrieved content that attempt to manipulate model behaviour.

Governance: Policies, controls, accountability and oversight for responsible data use.

Review Questions

Short Answer Questions

1. Why must the grain of an analytical dataset be documented?
2. Distinguish an offline feature store from an online feature store.
3. What is target leakage, and why does it produce misleading validation results?
4. State two quality risks specific to unstructured documents.
5. Why should retrieval quality and generation quality be evaluated separately?

Long Answer Questions

1. Explain how point-in-time correct feature construction supports reliable machine-learning training.
2. Describe the architecture and governance requirements of a production feature store.
3. Explain the stages of a secure retrieval-augmented generation pipeline.
4. Discuss how MLOps integrates data versioning, model deployment and monitoring.
5. Analyse the ethical and regulatory issues involved in governing analytical and AI data products.

Analytical / Application Questions

1. A fraud model has excellent validation performance but fails after deployment. Investigation shows that a feature was calculated using a transaction status updated after fraud investigations. Diagnose the problem and propose a corrected data and validation design.
2. Design a metadata schema and retrieval strategy for a legal-document assistant that must respect jurisdiction, effective date, document version and user permissions.
3. A feature’s missingness rises from 2% to 18% while its mean remains stable. Explain what additional analyses are required before deciding whether to retrain the model or repair the source pipeline.

Multiple Choice Questions with Answers

Question 1

Which property specifies what one row in an analytical dataset represents?

A. Grain

B. Latency

C. Embedding

D. Recall

Correct Answer: A. Grain

Question 2

Which practice most directly prevents target leakage in historical training data?

A. Increasing the embedding dimension

B. Using point-in-time correct joins

C. Adding more dashboard filters

D. Applying model quantisation

Correct Answer: B. Using point-in-time correct joins

Question 3

What is the primary purpose of an online feature store?

A. Long-term archival of raw documents

B. Low-latency serving of current feature values

C. Manual annotation of training labels

D. Rendering business-intelligence charts

Correct Answer: B. Low-latency serving of current feature values

Question 4

If two vectors are normalised to unit length, cosine similarity is equal to their:

A. Euclidean distance

B. Dot product

C. Arithmetic mean

D. Variance

Correct Answer: B. Dot product

Question 5

Which control is essential for a permission-aware RAG system?

A. Ignoring document metadata

B. Applying access constraints to retrieved evidence

C. Removing all document versions

D. Using only random train-test splits

Correct Answer: B. Applying access constraints to retrieved evidence

Question 6

A feature’s distribution changes over time, but the relationship between the feature and outcome has not yet been assessed. This is best described as:

A. Data drift

B. Target leakage

C. Prompt injection

D. Schema normalisation

Correct Answer: A. Data drift

Question 7

Which document-processing problem can cause a numerical table to be misinterpreted by a language model?

A. Correct page numbering

B. Incorrect reading order during extraction

C. Strong access control

D. Stable effective dates

Correct Answer: B. Incorrect reading order during extraction

Question 8

Why are chronological splits often preferable for temporal prediction problems?

A. They imitate deployment using earlier data to predict later data

B. They always produce higher accuracy

C. They remove the need for labels

D. They guarantee fairness

Correct Answer: A. They imitate deployment using earlier data to predict later data

Question 9

Which metric measures the proportion of relevant items among the first k retrieved items?

A. Precision@k

B. Completeness

C. Freshness

D. Concept drift

Correct Answer: A. Precision@k

Question 10

A document contains instructions attempting to override the system prompt. In a RAG pipeline, those instructions should be treated as:

A. Authoritative system policy

B. Untrusted retrieved content

C. A new training label

D. A database schema change

Correct Answer: B. Untrusted retrieved content