Data Pipelines, Evaluation and Experimentation

Introduction

Deep-learning performance depends not only on architecture and optimization. It also depends on how data are collected, represented, split, transformed, evaluated, and documented. A powerful model trained on contaminated or poorly defined data may produce an impressive but invalid result. Conversely, a modest architecture supported by a rigorous pipeline can provide reliable and reproducible performance.

This chapter treats the data pipeline and experimental process as a single scientific system. The focus is multimodal learning involving, for example, images, text, audio, tabular attributes, and metadata. Particular attention is given to leakage prevention, class imbalance, metric selection, reproducibility, error analysis, sensitivity analysis, and ablation studies.

Learning Objectives

By the end of this chapter, you should be able to:

1. Design a reliable pipeline for multimodal data.
2. Distinguish legitimate preprocessing from data leakage.
3. Select evaluation metrics appropriate to the task and its risks.
4. Address class imbalance without corrupting evaluation.
5. Design reproducible and statistically responsible experiments.
6. Conduct error, sensitivity, and ablation analyses.
7. Communicate model results with appropriate technical and governance safeguards.

1. The Data-to-Decision Pipeline

A modern deep-learning system commonly contains the following stages:

1. Problem specification: Define the prediction target, unit of prediction, population, time horizon, and decision that the model supports.
2. Data acquisition: Gather raw observations and record provenance, permissions, timestamps, and collection conditions.
3. Data validation: Check schema, missingness, duplicates, corrupted files, inconsistent labels, and impossible values.
4. Partitioning: Create training, validation, and test sets according to the deployment setting.
5. Preprocessing: Normalize, tokenize, resize, augment, or otherwise transform inputs using rules learned only from permitted data.
6. Representation and fusion: Encode each modality and combine representations through early, intermediate, or late fusion.
7. Training: Optimize model parameters using the training set and use validation data for model selection.
8. Evaluation: Apply a frozen model to an untouched test set and report uncertainty and subgroup results.
9. Deployment monitoring: Track drift, calibration, failures, fairness, and changes in data quality.

A useful principle is that every transformation must have an explicitly stated scope. A transformation may be fitted globally, per training fold, per subject, per time period, or per sample. The scope determines whether information has crossed an evaluation boundary.

Figure: A leakage-aware data and evaluation pipeline. The validation set supports development, while the test set is reserved for final assessment.

2. Preparing Multimodal Data

2.1 Define the observation and alignment unit

Before joining modalities, define what constitutes one example. It might be a patient visit, customer account, video clip, document, or time window. Every modality must be aligned to that unit. For a patient-level task, splitting individual images from the same patient across training and test sets creates dependence and often produces overly optimistic results.

For temporal applications, features must be available before the prediction time. A laboratory result recorded after a clinical decision, for example, cannot be used as a pre-decision feature merely because it is present in the database.

2.2 Modality-specific preparation

Images: Verify file integrity, color conventions, spatial resolution, orientation, and acquisition device. Resize or crop consistently. Augmentations should represent plausible changes and must not alter the label unintentionally.

Text: Record the tokenizer and vocabulary version. Remove or mask identifiers where appropriate. Be cautious with truncation: a label or future outcome may appear near the end of a document. Language normalization must preserve clinically, legally, or technically meaningful distinctions.

Audio: Standardize sampling rate and channel convention, inspect silence and clipping, and use windows appropriate to the phenomenon. Speaker identity may create leakage if the same speaker occurs in multiple partitions.

Tabular data: Specify units, valid ranges, missing-value codes, and categorical vocabularies. Imputation, scaling, feature selection, and dimensionality reduction must be fitted within the training data of each split.

Metadata: Metadata can be predictive without being causal. A device identifier, hospital code, or file naming convention may encode site or label collection practices. Such variables require explicit justification and robustness testing.

2.3 Fusion strategies

In early fusion, modality features or raw representations are concatenated before substantial task-specific processing. It can capture cross-modal interactions but requires compatible alignment and careful scaling.

In intermediate fusion, each modality is encoded separately and the learned representations are combined. This is common when modalities have different structures, such as a convolutional image encoder and a transformer text encoder.

In late fusion, separate models produce predictions that are combined, for example by averaging probabilities or learning a meta-classifier. Late fusion is modular and can tolerate missing modalities, but it may miss fine-grained interactions.

Missing modalities should be treated as a design condition, not silently converted into arbitrary zeros. Useful strategies include modality masks, modality dropout during training, imputation with uncertainty indicators, and models explicitly designed for partial observations.

3. Splitting Data and Preventing Leakage

3.1 What is leakage?

Data leakage occurs when information unavailable at deployment, or information from the evaluation data, influences training or model selection. Leakage can be direct, such as including the target as a feature, or indirect, such as computing normalization statistics using the complete dataset.

Common forms include:

• Duplicate leakage: identical or near-identical records occur in different splits.
• Group leakage: observations from one person, household, device, or site appear in both training and test sets.
• Temporal leakage: future observations influence a past prediction.
• Preprocessing leakage: scaling, imputation, feature selection, or vocabulary construction uses validation or test data.
• Label leakage: a proxy recorded after the outcome or generated from the outcome is used as an input.
• Model-selection leakage: repeated examination of the test set influences architecture or hyperparameter decisions.

The correct split unit is usually the unit at which dependence exists. Grouped splitting is required when examples within a group are correlated. Time-based splitting is required when deployment predicts future cases from past data. In multimodal data, all modalities belonging to one unit must move together.

3.2 Transformation discipline

Suppose a feature x is standardized using mean μ and standard deviation σ. The transformation is:

Here x is an observed feature value, μ is the mean, σ is the standard deviation, and z is the standardized value. For a valid evaluation, μ and σ must be estimated from the training portion only, and the same frozen values must then be applied to validation and test data. The condition σ > 0 is required; constant features need separate handling.

In k-fold cross-validation, this fitting process occurs independently within each training fold. A pipeline library that combines fitting and transformation can reduce accidental leakage.

3.3 Leakage audit

A serious project should maintain a leakage register. For each variable and transformation, record its source, timestamp, availability at prediction, fitting scope, and rationale. Attempted leakage tests are also useful: if a supposedly difficult task is solved almost perfectly, inspect duplicates, identifiers, timestamps, label-generation code, and split logic before celebrating the result.

4. Class Imbalance and Sampling

Let TP denote true positives, TN true negatives, FP false positives, and FN false negatives. In an imbalanced classification problem, the majority class may dominate accuracy even when the minority class is poorly detected.

Possible interventions include:

1. Data-level methods: oversampling the minority class, undersampling the majority class, or generating synthetic examples.
2. Loss-level methods: class-weighted loss or focal loss.
3. Decision-level methods: threshold selection based on operational costs.
4. Data-collection methods: targeted collection of rare or difficult cases.

Oversampling must occur only within the training data. Otherwise, duplicated or synthetic versions of a test example can contaminate evaluation. In grouped data, oversampling must not place related records across partitions.

For binary classification, weighted cross-entropy can be written as:

Here y ∈ {0,1} is the true label, p is the predicted probability of class 1, and w₁ and w₀ are non-negative class weights. The logarithm is natural unless otherwise specified. The formula applies to probabilistic binary classification; numerical implementations use stable forms to avoid log(0). Class weights alter the optimization emphasis, but they do not automatically make predicted probabilities calibrated.

A reasonable starting point is inverse-frequency weighting, but weights should be treated as hyperparameters and validated against the real objective. If false negatives cost more than false positives, a cost-sensitive threshold may be more appropriate than aggressive resampling.

5. Evaluation Metrics and Their Interpretation

5.1 Classification metrics

Accuracy is:

It is appropriate when class proportions and error costs are reasonably stable and balanced. It can be misleading under severe imbalance.

Precision and recall are:

Precision measures the proportion of predicted positives that are correct; recall, also called sensitivity, measures the proportion of actual positives detected. If the denominators are zero, the metric requires a stated convention, such as reporting it as undefined rather than silently assigning a favorable value.

The F1 score is the harmonic mean:

It is useful when precision and recall both matter, but it ignores true negatives and does not encode explicit business or clinical costs.

Specificity is TN ÷ (TN + FP). Balanced accuracy is the mean of sensitivity and specificity:

For probabilistic classifiers, the receiver operating characteristic curve examines true-positive rate against false-positive rate across thresholds. The precision–recall curve is often more informative when the positive class is rare. Area under a curve summarizes ranking performance, not necessarily performance at the operating threshold.

5.2 Calibration and decision utility

A model is calibrated if among cases assigned probability p, approximately a proportion p has the outcome. Calibration can be assessed using reliability diagrams, calibration curves, and proper scoring rules such as the Brier score:

Here n is the number of cases, pᵢ is the predicted probability, and yᵢ ∈ {0,1} is the outcome. Lower values are better. The Brier score combines calibration and discrimination, so it should not be interpreted as calibration alone.

Threshold selection should reflect consequences. A threshold that maximizes F1 may be unsuitable when false negatives are much more harmful than false positives. Report the threshold, validation procedure, prevalence, and the resulting confusion matrix.

5.3 Regression and ranking

For regression, mean absolute error is:

Root mean squared error is:

MAE is in the target’s units and is less sensitive to large errors than RMSE. RMSE emphasizes large errors and is also in the target’s units. Neither should be used without considering the consequences of extreme cases and the target distribution.

For ranking and retrieval, metrics such as mean average precision, recall at k, and normalized discounted cumulative gain may be more appropriate than accuracy. For generative systems, automatic metrics are incomplete; human assessment, factuality checks, toxicity screening, and task-specific success measures are often required.

6. Reproducible Experimental Design

A reproducible experiment is more than setting a random seed. It requires versioned data, fixed preprocessing code, recorded environment details, explicit split files, configuration management, model checkpoints, and evaluation scripts. Seeds control some randomness but cannot guarantee determinism across hardware, kernels, or distributed systems.

A strong experiment record includes:

• research question and primary metric;
• dataset version, inclusion criteria, and split mechanism;
• model architecture and parameter count;
• preprocessing and augmentation policy;
• optimizer, learning-rate schedule, batch size, and stopping rule;
• random seeds and hardware/software versions;
• number of runs and selection procedure;
• complete test results, confidence intervals, and subgroup results.

Do not tune repeatedly on the test set. If a test result motivates a change, the resulting number is no longer a clean final estimate. Use a validation set, nested cross-validation, or a new holdout test set.

Uncertainty and repeated runs

For a proportion p̂ estimated from n independent cases, a simple approximate standard error is:

This approximation is most defensible when observations are independent and n is sufficiently large. In clustered data, the effective information is smaller; bootstrap resampling should respect groups. Confidence intervals can be obtained using bootstrap procedures, exact methods, or model-based methods, depending on the metric and sampling design.

For neural networks, report variation across independent training runs where feasible. A single best seed is not evidence of typical performance. When comparing two models, paired evaluation on the same test cases is generally more informative than comparing unrelated averages.

7. Error, Sensitivity, and Ablation Analysis

7.1 Error analysis

Error analysis is a structured investigation of failures, not a collection of anecdotes. Begin with a confusion matrix and stratify errors by meaningful factors: class, subgroup, modality availability, acquisition device, time period, difficulty, and confidence. Inspect false positives and false negatives, but also inspect correct predictions to identify shortcuts and fragile patterns.

A useful error-analysis table records the case identifier, predicted probability, predicted and true labels, available modalities, relevant metadata, error category, suspected cause, and proposed intervention. Categories might include annotation error, distribution shift, occlusion, ambiguous language, missing modality, poor audio quality, shortcut feature, or model capacity limitation.

7.2 Sensitivity analysis

Sensitivity analysis asks how conclusions change when assumptions or conditions change. Vary image resolution, missingness rate, decision threshold, class weights, noise level, temporal window, or subgroup composition. The goal is not to find a favorable setting but to characterize robustness.

For a parameter θ and performance measure m(θ), a local sensitivity can be approximated by:

This finite-difference quantity is interpretable only for sufficiently small ∆θ and comparable evaluation conditions. It is not a causal effect and should not be extrapolated beyond the tested range.

7.3 Ablation studies

An ablation removes or changes one component while keeping other conditions fixed. Examples include removing text, replacing learned fusion with concatenation, disabling augmentation, or removing a loss term. A credible ablation changes one theoretically motivated component at a time, uses the same data splits and evaluation metric, and repeats training when randomness is material.

Ablation results should report the absolute difference as well as the relative change. If a baseline score is m₀ and an ablated score is m₁, the absolute change is ∆m = m₁ − m₀. A relative percentage change, when meaningful and m₀ ≠ 0, is 100 × (m₁ − m₀) ÷ |m₀|. For metrics where lower is better, interpret signs carefully.

8. Illustrative Case: Multimodal Hospital Readmission Prediction

Background

Consider an illustrative postgraduate project that predicts whether a patient will be readmitted within 30 days. The data contain structured admission variables, clinical notes, and chest images. The deployment decision occurs at discharge.

Problem

An initial model reports 96% accuracy. Investigation shows that the dataset is highly imbalanced and that images from the same patient appear in several partitions. Some notes also contain discharge summaries written after the readmission risk was assessed.

Analysis

The prediction unit is the discharge episode, but patient-level dependence remains. The team therefore groups all episodes from a patient into one partition and removes text written after the decision time. It creates a temporal test set from later admissions to approximate deployment. Image preprocessing statistics and text vocabulary are fitted using training data only.

The team reports sensitivity, specificity, precision, negative predictive value, area under the precision–recall curve, calibration, and subgroup performance rather than accuracy alone. Class weighting is used during training, while the decision threshold is selected on validation data according to the operational cost of missed high-risk patients.

Approach and outcome

Three systems are compared: structured data alone, image plus structured data, and all three modalities. The full model improves recall but is poorly calibrated. Temperature scaling is fitted on validation data, and calibration is then assessed on the untouched temporal test set. Error analysis reveals that performance falls for hospitals using a different imaging device and for patients with missing notes. The final system therefore includes a missing-modality indicator and a monitoring plan for site and device drift.

The outcome is not merely a higher score. The revised study provides a more credible estimate, identifies limitations, and defines conditions under which the system should defer to human review.

Learning points

1. A high accuracy value can conceal serious failure under imbalance.
2. Group and temporal splitting must reflect deployment.
3. Post-decision text is label leakage even if it appears in the database.
4. Calibration and subgroup performance matter when predictions support clinical decisions.
5. Robust deployment requires explicit handling of missing modalities and distribution shift.

9. Limitations, Governance, and Contemporary Practice

Evaluation metrics are conditional on the dataset, sampling design, prevalence, and decision context. A model can rank cases well while producing unreliable probabilities. Aggregate performance can hide subgroup disparities. Removing a sensitive attribute does not remove bias if correlated proxies remain.

Data governance should address consent, lawful use, privacy, retention, access control, auditability, and documentation of data provenance. Images, voices, and free text may contain identifying information. De-identification is not automatically irreversible. Model cards, datasheets, risk assessments, and incident procedures support responsible deployment.

Contemporary pipelines increasingly use data-centric quality monitoring, dataset versioning, automated validation, foundation-model embeddings, synthetic data, and multimodal foundation models. These tools introduce additional risks: pretrained-model licensing constraints, hidden training-data overlap, representation bias, prompt or retrieval leakage, and uncertain transfer across domains. Claims of improvement should therefore be supported by contamination checks, external validation, and task-relevant human evaluation.

Best practice is to define an evaluation protocol before inspecting final results, preserve a sealed test set, preregister primary comparisons when feasible, report negative findings, and distinguish exploratory analyses from confirmatory claims.

Chapter Summary

Reliable deep learning begins with a well-defined prediction unit, provenance-aware data preparation, and partitions that reproduce deployment dependence. Multimodal data require careful alignment, modality-specific quality checks, and explicit treatment of missing inputs. Leakage may arise through duplicates, groups, time, preprocessing, labels, or repeated test-set use.

Imbalance should be addressed through training, sampling, threshold, or collection strategies without contaminating evaluation. Metric selection must reflect prevalence, costs, ranking needs, probability quality, and operational decisions. Reproducible experiments require versioned data and code, documented configurations, repeated runs, uncertainty estimates, and disciplined model selection. Error analysis reveals failure mechanisms; sensitivity analysis tests robustness; ablation analysis evaluates the contribution of system components. Finally, governance, subgroup analysis, privacy, calibration, and monitoring are essential when models influence consequential decisions.

Key Terms

Data leakage: Use of information unavailable at deployment or reserved for evaluation.

Prediction unit: The real-world entity or event represented by one example.

Grouped split: A partition that keeps related observations in the same dataset split.

Temporal split: A split that trains on earlier data and evaluates on later data.

Multimodal learning: Learning from two or more data modalities.

Early fusion: Combining modalities near the input or feature-construction stage.

Late fusion: Combining predictions from separate modality-specific models.

Class imbalance: Unequal representation of target classes.

Precision: Proportion of predicted positives that are true positives.

Recall: Proportion of actual positives detected by the model.

Specificity: Proportion of actual negatives correctly identified.

Calibration: Agreement between predicted probabilities and observed frequencies.

Brier score: Mean squared error of probabilistic binary predictions.

Data augmentation: Label-preserving transformation used to expand training variation.

Reproducibility: Ability to obtain consistent findings using documented data, code, and procedures.

Error analysis: Systematic investigation of model failures and their causes.

Sensitivity analysis: Study of performance changes under altered assumptions or conditions.

Ablation study: Controlled removal or modification of a system component.

Distribution shift: Change in data-generating conditions between development and deployment.

Data provenance: Record of the origin, processing, ownership, and history of data.

Review Questions

Short Answer Questions

1. Why must the split unit sometimes be a patient, speaker, device, or household rather than an individual record?
2. Distinguish precision from recall.
3. Give two examples of preprocessing leakage.
4. Why is accuracy potentially misleading in an imbalanced classification problem?
5. What is the purpose of an ablation study?

Long Answer Questions

1. Explain how you would design a leakage-resistant multimodal pipeline for a clinical prediction task.
2. Compare early, intermediate, and late fusion, including their advantages and limitations.
3. Discuss how metric choice should reflect prevalence, error costs, calibration, and deployment decisions.
4. Describe the components of a reproducible deep-learning experiment.
5. Explain how error analysis, sensitivity analysis, and ablation analysis complement one another.

Analytical / Application Questions

1. A fraud model has high accuracy but low recall for fraudulent transactions. Propose a revised evaluation and training strategy, and justify each metric and intervention.
2. A dataset contains multiple recordings from each speaker and is used for emotion recognition. Design an appropriate split, leakage audit, and subgroup evaluation plan.
3. A multimodal model performs well when all modalities are present but fails when text is missing. Design an experiment to determine whether the problem is representation, training distribution, calibration, or data quality.

Multiple Choice Questions with Answers

Question 1

Which practice most directly prevents group leakage when several records belong to the same patient?

A. Randomly split individual records

B. Place all records from a patient in one partition

C. Duplicate minority records before splitting

D. Normalize features using the complete dataset

Correct Answer: B. Place all records from a patient in one partition

Question 2

In a binary classifier, recall is defined as:

A. TP ÷ (TP + FP)

B. TN ÷ (TN + FP)

C. TP ÷ (TP + FN)

D. (TP + TN) ÷ total cases

Correct Answer: C. TP ÷ (TP + FN)

Question 3

Which transformation must be fitted using training data only?

A. Test-set labeling

B. Feature standardization

C. Final reporting

D. Manual inspection of one prediction

Correct Answer: B. Feature standardization

Question 4

Which metric is especially useful for assessing probability quality?

A. Brier score

B. Accuracy only

C. Number of parameters

D. Training loss only

Correct Answer: A. Brier score

Question 5

Why can accuracy be misleading in severe class imbalance?

A. It ignores all true negatives

B. It can be high while minority-class detection is poor

C. It cannot be computed from a confusion matrix

D. It always requires calibrated probabilities

Correct Answer: B. It can be high while minority-class detection is poor

Question 6

Where should oversampling generally be applied?

A. To the sealed test set

B. To training data within each training procedure

C. After final test evaluation

D. Equally across training and test sets

Correct Answer: B. To training data within each training procedure

Question 7

What is the primary purpose of sensitivity analysis?

A. To guarantee causal interpretation

B. To test how conclusions change under altered conditions

C. To replace the validation set

D. To remove all model parameters

Correct Answer: B. To test how conclusions change under altered conditions

Question 8

An ablation study typically involves:

A. Changing every component simultaneously

B. Removing or modifying one component under controlled conditions

C. Evaluating only on training data

D. Increasing the test-set size by duplication

Correct Answer: B. Removing or modifying one component under controlled conditions

Question 9

For a future-prediction deployment setting, which split is usually most realistic?

A. A random split that mixes all dates

B. A temporal split using earlier data for training and later data for testing

C. A split based only on file size

D. A split made after test predictions are inspected

Correct Answer: B. A temporal split using earlier data for training and later data for testing

Question 10

Which statement about a random seed is correct?

A. It guarantees identical results on all hardware

B. It removes the need to version data

C. It controls some randomness but does not guarantee complete reproducibility

D. It prevents distribution shift

Correct Answer: C. It controls some randomness but does not guarantee complete reproducibility