Data Engineering, Evaluation and Experimentation

Introduction

Computer vision systems are often described through model architectures—convolutional networks, vision transformers, detectors, segmenters, and multimodal models. Yet model quality depends at least as much on the data and evaluation process surrounding the model. A highly sophisticated network trained on biased, noisy, duplicated, or poorly labelled images may perform worse than a simpler model trained on a carefully engineered dataset.

Data engineering for computer vision includes acquiring data, defining annotation policies, cleaning and validating labels, transforming images, controlling dataset versions, and monitoring changes after deployment. Evaluation includes selecting metrics appropriate to the task, designing reliable train–validation–test splits, analysing errors, testing robustness, and quantifying uncertainty. Experimentation connects these activities through controlled comparisons such as ablation studies.

This chapter presents a practical and analytical framework for building trustworthy vision datasets and evaluating models responsibly. The discussion applies to image classification, object detection, semantic and instance segmentation, and related applications.

Learning Objectives

By the end of this chapter, students should be able to:

1. Design an image acquisition and annotation workflow.

2. Explain dataset leakage, imbalance, domain shift, and dataset versioning.

3. Select suitable metrics for classification, detection, and segmentation.

4. Design ablation studies and reproducible experiments.

5. Perform statistical error analysis, robustness testing, and uncertainty analysis.

6. Identify ethical, privacy, and governance risks in vision datasets.

1. The Dataset as an Engineered System

A dataset is not simply a collection of images. It is an engineered system with a purpose, population, schema, provenance, quality controls, and version history. Before collecting data, the team should specify:

• the prediction task and intended users;
• the target population and operating environment;
• the unit of analysis, such as an image, video frame, object, or pixel;
• the label vocabulary and annotation rules;
• acceptable error rates and safety constraints; and
• legal, ethical, and privacy requirements.

A useful dataset card records the dataset’s motivation, composition, collection process, annotation procedure, known limitations, recommended uses, and prohibited uses. This documentation makes assumptions visible and supports later auditing.

Acquisition and Sampling

Images may come from cameras, sensors, public repositories, licensed providers, synthetic generators, or user submissions. Sampling should reflect the deployment distribution rather than merely the easiest data to obtain. For example, a road-scene dataset collected only in daylight cannot support strong claims about night-time driving.

Important acquisition variables include location, time, weather, illumination, camera hardware, viewpoint, resolution, demographic characteristics, and environmental context. These variables should be logged as metadata when possible. Metadata enables stratified analysis and helps detect domain shift.

A major danger is selection bias: the observed sample differs systematically from the population in which the system will operate. Convenience samples may overrepresent urban scenes, common objects, high-quality images, or users with reliable internet access. Sampling plans should therefore define coverage targets and monitor the number of examples in relevant strata.

Data Splitting and Leakage

Training, validation, and test sets serve different purposes. The training set is used to fit parameters; the validation set supports model and hyperparameter selection; the test set provides a final estimate of generalisation. The test set should remain isolated until the evaluation protocol is fixed.

Random image-level splitting is unsafe when images are correlated. Frames from the same video, images of the same patient, or photographs from the same physical location may appear in multiple partitions. The model can then recognise background, identity, or capture conditions rather than learn the intended task. Grouped splitting assigns all related examples to one partition. In temporal applications, training on earlier data and testing on later data may better represent deployment.

Leakage also occurs when preprocessing uses test information, when labels are indirectly encoded in filenames, or when repeated hyperparameter tuning is performed against the test set. A leakage audit should be part of dataset validation.

2. Annotation and Quality Assurance

Annotation Schemas

The annotation format must match the task. Classification assigns one or more labels to an image. Detection normally represents each object by a class and a bounding box. Semantic segmentation assigns a class to each pixel, while instance segmentation additionally distinguishes separate objects of the same class. Keypoint tasks represent anatomical or structural landmarks.

Annotation guidelines should define class boundaries, occlusion rules, truncation, difficult examples, ambiguous cases, and the treatment of unknown objects. A short example-based handbook is often more useful than a long abstract specification. Guidelines must be versioned because changes in definitions can make labels from different periods incomparable.

Agreement and Adjudication

Human disagreement is informative. If two annotators frequently disagree, the class may be visually ambiguous or the guidelines may be inadequate. For categorical labels, Cohen’s kappa is sometimes used for agreement between two annotators, although its interpretation depends on prevalence and assumptions. For bounding boxes, intersection over union (IoU) can compare spatial overlap:

IoU = area of (A intersection B) / area of (A union B).

Here A and B are two regions. IoU ranges from 0 to 1, where 1 denotes identical regions. It is useful for evaluating boxes and masks, but it does not by itself measure whether the class labels agree.

A robust workflow uses independent annotation, automatic checks, targeted re-annotation, and adjudication by a senior annotator. Agreement statistics should not replace examination of disagreement cases. Rare classes can produce misleading aggregate agreement, so per-class analysis is important.

Quality Checks

Automated checks can identify corrupt files, unsupported formats, invalid dimensions, duplicate hashes, missing labels, impossible box coordinates, empty masks, and inconsistent class identifiers. Visual inspection remains essential. A random sample and a sample selected from unusual metadata or model errors should be reviewed.

Quality is multidimensional. A dataset may have accurate labels but poor coverage, broad coverage but inconsistent labels, or excellent labels but severe duplication. Data quality should therefore be reported by dimension rather than compressed into one score.

3. Augmentation, Synthetic Data, and Versioning

Data augmentation creates modified training examples through transformations such as cropping, horizontal flipping, colour changes, blur, noise, scaling, and geometric distortion. Augmentation should represent plausible variation in the deployment environment. A horizontal flip may be appropriate for many natural objects but inappropriate when left and right have different meanings, as in traffic signs or medical anatomy.

Transformations must be applied consistently to the image and its annotation. A crop requires corresponding adjustment of bounding boxes and masks. Augmentation should normally be applied only to training data. Validation and test images should retain a stable evaluation distribution, apart from explicitly defined test-time augmentation experiments.

Synthetic data can increase coverage of rare conditions, but generated images may contain unrealistic textures, simplified backgrounds, or generator-specific artefacts. Synthetic-to-real transfer should be tested rather than assumed. A useful strategy is to combine synthetic data with real examples and evaluate separately on a real-world test set.

Dataset Versioning and Lineage

Each dataset release should have an immutable identifier and record:

• source files and licences;
• preprocessing and augmentation code;
• annotation version;
• split assignments;
• removed or corrected examples;
• metadata schema; and
• checksums or content hashes.

Tools such as DVC, Git-based repositories, object-storage versioning, and experiment-tracking platforms can support this process. The essential principle is reproducibility: a reported result should be traceable to the exact data, code, configuration, and random seeds that produced it.

A dataset version is not necessarily better merely because it is larger. Adding low-quality or distributionally inappropriate data can reduce performance. Changes should therefore be evaluated using a fixed benchmark and documented release notes.

4. Imbalance and Domain Shift

Class Imbalance

Imbalance occurs when some classes or conditions have many more examples than others. Accuracy can then conceal poor minority-class performance. Remedies include collecting additional data, targeted augmentation, over-sampling, under-sampling, class-weighted losses, focal loss, threshold adjustment, and cost-sensitive decision rules.

Each remedy has limitations. Over-sampling may cause memorisation; under-sampling discards information; class weights can increase false positives for rare classes; and synthetic augmentation may not add genuinely new information. The appropriate method depends on the operational cost of errors.

Evaluation should report per-class precision, recall, support, and confusion matrices. Macro-averaging gives each class equal weight, whereas micro-averaging aggregates individual predictions and is dominated by common classes. Neither is universally superior.

Domain Shift

Domain shift occurs when the deployment distribution differs from the training distribution. Covariate shift changes the input distribution while the conditional relationship may remain approximately stable. Label shift changes class proportions. Concept shift changes the relationship between inputs and labels.

Examples include a camera upgrade, geographic transfer, seasonal changes, new medical equipment, different compression, or changes in annotation policy. A model may achieve high in-distribution accuracy while failing under these shifts.

Mitigation includes diverse sampling, domain-specific validation sets, normalization, fine-tuning, domain adaptation, augmentation, test-time monitoring, and recalibration. Evaluation should include a shift matrix—for example, performance by location, weather, camera, and time period—rather than only one aggregate score.

5. Task-Specific Evaluation Metrics

No single metric fully describes a vision system. Metrics should be selected from the task, class distribution, decision threshold, and consequences of errors.

Classification

For binary classification, let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives. Accuracy is:

Accuracy = (TP + TN) / (TP + TN + FP + FN).

Precision and recall are:

Precision = TP / (TP + FP),

Recall = TP / (TP + FN).

Precision measures the reliability of positive predictions, while recall measures the proportion of actual positives detected. The F1 score is the harmonic mean:

F1 = 2PR / (P + R),

where P is precision and R is recall. It is defined when P + R is non-zero.

Specificity, also called the true-negative rate, is TN/(TN + FP). The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate over thresholds. The area under this curve, AUROC, measures ranking discrimination across thresholds. Precision–recall curves are often more informative for rare positive classes because they focus on positive prediction quality.

Calibration concerns whether predicted probabilities correspond to observed frequencies. For example, among cases assigned probability near 0.7, approximately 70% should be positive if the model is well calibrated. Discrimination and calibration are different properties; a model can rank examples well but produce poorly calibrated probabilities.

Detection

Object detection commonly uses IoU to decide whether a predicted box matches a ground-truth box, subject to class agreement and one-to-one matching. Average precision (AP) summarises the precision–recall curve for one class at a specified IoU criterion. Mean average precision (mAP) averages AP across classes and, depending on the benchmark, across IoU thresholds.

A reported detection score is incomplete unless the IoU threshold, matching protocol, class averaging, confidence threshold, and treatment of difficult or ignored objects are specified. Small objects and crowded scenes often require separate analysis because their errors are structurally different.

Segmentation

For masks, intersection over union is also called the Jaccard index. The Dice coefficient is:

Dice = 2|A intersection B| / (|A| + |B|),

where |A| and |B| are the numbers of pixels in the predicted and reference masks. Dice and IoU are related but not identical. Boundary F-scores, pixel accuracy, and class-wise IoU may be useful for different applications. Large background regions can make pixel accuracy misleading.

Metric Selection as a Decision Problem

Metric choice should reflect consequences. In a screening system, missed positives may be more costly than false alarms, so recall at a specified minimum precision may be appropriate. In an automated rejection system, excessive false positives may be unacceptable. A deployment specification should define thresholds, acceptable subgroup performance, latency, memory, and failure-handling behaviour in addition to a headline metric.

6. Experimental Design and Ablation Studies

A fair experiment changes one meaningful factor at a time while controlling other conditions. Important controls include dataset version, split, preprocessing, training budget, optimizer, number of epochs, random seeds, and evaluation code.

An ablation study removes or modifies components to estimate their contribution. For example, a vision pipeline may compare:

1. baseline model;

2. baseline plus augmentation;

3. baseline plus class weighting;

4. baseline plus augmentation and class weighting; and

5. the complete proposed system.

The comparison should use the same test set and report both aggregate and subgroup results. If components interact, single-component ablations may not explain all effects; factorial designs or carefully chosen combinations may be necessary.

Repeated runs with different random seeds reveal training variability. Report the mean and standard deviation, but do not treat standard deviation as a confidence interval without appropriate assumptions. A confidence interval for a metric requires an estimation method, such as bootstrap resampling, and must account for dependence between examples where relevant.

Statistical significance is not the same as practical significance. A tiny improvement may be statistically detectable with a large test set but irrelevant operationally. Conversely, a meaningful improvement may be uncertain with a small sample. Effect size, uncertainty, computational cost, and deployment risk should all be considered.

7. Error, Robustness, and Uncertainty Analysis

Error Taxonomy

Error analysis begins with a confusion matrix or ranked list of failures. Errors can be categorised as annotation errors, missing objects, wrong class, localization error, background confusion, occlusion, blur, scale, lighting, domain mismatch, or threshold error. Each category suggests a different intervention. More training data will not fix a systematic labelling error, and a better detector may not solve a camera-placement problem.

Slice-based analysis evaluates performance for meaningful subgroups, such as skin tone, age range, geographic region, object size, weather, or device type. Slices must be defined carefully and used in a privacy-preserving manner. Small slices produce uncertain estimates and should not be overinterpreted.

Robustness

Robustness testing measures performance under plausible perturbations and distribution changes. These may include noise, blur, compression, occlusion, altered illumination, viewpoint changes, weather, and sensor shifts. Corruption tests should be tied to realistic failure modes; arbitrary perturbations can produce misleading conclusions.

Stress tests should report the perturbation type and severity, not just a single robustness score. A model that performs well under Gaussian noise may still fail under glare or motion blur. Robustness can also involve operational constraints such as latency, memory, missing metadata, and intermittent connectivity.

Uncertainty

Predictive uncertainty is particularly important when a model can defer uncertain cases to a human. Aleatoric uncertainty reflects irreducible ambiguity or measurement noise; epistemic uncertainty reflects limited knowledge and may be reduced with more representative data. In practice, uncertainty estimates can be approximated using ensembles, Monte Carlo dropout, probabilistic models, or calibrated confidence scores.

A confidence score is not automatically a valid probability. Calibration methods such as temperature scaling can adjust scores using a validation set, but calibration may degrade under domain shift. Selective prediction evaluates risk when the model is allowed to abstain. A useful system may achieve lower error on retained cases while sending difficult examples for review.

8. Worked Example: Evaluating an Imbalanced Detector

Suppose a pedestrian detector is evaluated on 1,000 annotated pedestrians and 9,000 background windows after a chosen decision threshold. It produces TP = 900, FN = 100, FP = 300, and TN = 8,700.

Precision is 900/(900 + 300) = 0.75, or 75%. Recall is 900/(900 + 100) = 0.90, or 90%. Accuracy is (900 + 8,700)/10,000 = 0.96, or 96%.

The contrast is important: 96% accuracy may appear excellent, but 100 pedestrians are missed and one quarter of positive predictions are false alarms. The F1 score is:

F1 = 2(0.75)(0.90)/(0.75 + 0.90) ≈ 0.818.

If the application is a safety warning system, the team may prefer the current high recall. If alerts are expensive, the threshold might be raised to improve precision, but the resulting recall must be measured. The team should also report performance by night/day, camera, pedestrian size, and weather. Aggregate results alone cannot establish deployment suitability.

9. Illustrative Case Study: Regional Deployment of a Crop-Disease Classifier

Background and Problem

An agricultural technology company develops a classifier for leaf photographs collected by mobile phones. The initial dataset contains 40,000 images from one region, mostly captured in bright daylight. The model reaches 94% test accuracy on a randomly split test set but performs poorly during a pilot in another region.

Analysis

An audit finds that multiple frames from the same field appear in different partitions. The original test set therefore overestimates generalisation. The new region has different cultivars, soil backgrounds, phone cameras, and lighting. Disease classes are also imbalanced: one common disease represents 60% of images, while two important diseases each represent less than 5%.

Approach

The team creates grouped, field-level splits and a temporally separated test set. It expands collection across regions, cultivars, phones, and lighting conditions. Annotators receive revised guidelines for visually similar diseases, and difficult examples are adjudicated. Training uses moderate photometric augmentation and class-balanced sampling. Evaluation reports macro-F1, per-class recall, calibration, and performance by region and camera. A low-confidence abstention option sends uncertain cases to agricultural advisers.

Outcome

Overall accuracy falls on the corrected benchmark, but minority-class recall and performance in the new region improve substantially. The apparent decrease is not a failure: it reflects a more realistic test. The system is released initially as decision support rather than autonomous diagnosis, with monitoring for new cultivars and periodic review of errors.

Learning Points

The case demonstrates that realistic splitting can lower headline performance while improving scientific validity. It also shows why data diversity, subgroup metrics, annotation governance, calibration, and human escalation are parts of one engineering problem.

10. Ethical, Legal, and Governance Considerations

Images may contain faces, biometric information, homes, licence plates, medical conditions, or culturally sensitive content. Collection requires a lawful basis, appropriate consent where applicable, data minimisation, access control, retention limits, and secure storage. De-identification reduces risk but may not guarantee anonymity, especially when images can be linked with external information.

Datasets should document licences and restrictions on commercial use, redistribution, derived models, and geographic transfer. Annotators should receive fair working conditions and support when content is disturbing. Automated systems should not be evaluated only on average performance if errors are distributed unequally across protected or vulnerable groups.

Governance should define who approves data use, who can access raw images, how incidents are reported, and when a model must be withdrawn or retrained. Documentation should distinguish measured facts from assumptions and should state known non-applications.

11. Best-Practice Workflow

A practical end-to-end workflow is:

1. Define the task, population, risks, and success criteria.

2. Design a sampling plan that covers deployment conditions.

3. Acquire data with provenance, consent, and metadata controls.

4. Write and version annotation guidelines.

5. Annotate with quality checks, agreement review, and adjudication.

6. Remove corruption, duplicates, leakage, and invalid records.

7. Create grouped or time-aware splits before model tuning.

8. Version data, code, labels, and preprocessing together.

9. Establish a simple baseline and predefine evaluation metrics.

10. Run controlled experiments and ablations with repeated seeds.

11. Report aggregate, per-class, subgroup, robustness, and uncertainty results.

12. Monitor the deployed distribution and establish a retraining policy.

Contemporary Developments

Current practice increasingly treats data-centric improvement as a first-class engineering activity. Active learning selects informative or uncertain samples for annotation. Foundation models can provide initial labels or embeddings, but their outputs require human verification and may reproduce hidden biases. Automated data-quality tools detect duplicates, near-duplicates, outliers, and label conflicts. Synthetic data and simulation support rare-event coverage, while privacy-preserving learning and federated approaches reduce the need to centralise sensitive images.

These techniques do not eliminate the need for representative sampling and independent evaluation. Automation can accelerate annotation and testing, but it can also propagate systematic errors at scale.

Chapter Summary

Reliable computer vision begins with engineered data rather than model selection alone. Acquisition should reflect the deployment population, and grouped or temporal splitting is essential when examples are correlated. Annotation schemes require explicit rules, quality checks, agreement analysis, and version control. Augmentation and synthetic data should represent plausible variation and must transform labels consistently.

Imbalance requires per-class reporting and interventions matched to operational costs. Domain shift should be anticipated through diverse sampling, subgroup evaluation, and monitoring. Classification, detection, and segmentation require different metrics, and no metric should be reported without its protocol and assumptions. Ablation studies and repeated experiments establish which components matter, while error taxonomies, robustness tests, calibration, and uncertainty analysis reveal limitations hidden by aggregate scores.

Finally, privacy, fairness, licensing, worker welfare, documentation, and accountability are engineering requirements. A trustworthy vision system is reproducible, measured against realistic conditions, transparent about uncertainty, and designed to fail safely.

Key Terms

Dataset leakage: Unintended use of information from validation or test data during training or model selection.

Annotation schema: Formal definition of labels and their representation for a task.

Intersection over union (IoU): Overlap area divided by union area for two regions.

Data augmentation: Transformation of training examples to represent plausible variation.

Dataset versioning: Recording and identifying changes to data, labels, splits, and preprocessing.

Class imbalance: Unequal representation of classes or conditions.

Domain shift: Difference between training and deployment data distributions.

Precision: Fraction of positive predictions that are correct.

Recall: Fraction of actual positives that are detected.

F1 score: Harmonic mean of precision and recall.

Calibration: Agreement between predicted probabilities and observed frequencies.

Average precision: Summary of a detection precision–recall curve under a specified protocol.

Ablation study: Controlled experiment that removes or changes system components.

Error taxonomy: Structured classification of observed failure types.

Robustness: Ability to maintain performance under plausible perturbations or shifts.

Aleatoric uncertainty: Uncertainty arising from irreducible ambiguity or measurement noise.

Epistemic uncertainty: Uncertainty associated with limited model knowledge or data coverage.

Selective prediction: Prediction with the option to abstain or defer uncertain cases.

Data provenance: Record of where data originated and how it was transformed.

Review Questions

Short Answer Questions

1. Why can random image-level splitting produce leakage in video-based datasets?

2. Distinguish precision from recall.

3. What is the purpose of dataset versioning?

4. Give two examples of domain shift in computer vision.

5. Why should validation and test images normally not receive random training augmentation?

Long Answer Questions

1. Explain how an annotation guideline, agreement analysis, and adjudication process can improve label quality.

2. Compare accuracy, macro-F1, and precision–recall analysis for an imbalanced classification problem.

3. Describe how to design a reproducible ablation study for a vision model.

4. Explain the difference between robustness testing and uncertainty estimation.

5. Discuss the ethical, privacy, and governance issues involved in collecting facial images.

Analytical / Application Questions

1. A classifier has 980 true positives, 20 false negatives, 180 false positives, and 8,820 true negatives. Calculate precision, recall, accuracy, and F1, and explain which metric would be most important for a safety-critical screening task.

2. A detector performs well on a laboratory test set but poorly after deployment on mobile cameras. Design an investigation covering leakage, domain shift, subgroup evaluation, and possible interventions.

3. Propose an experimental plan to determine whether augmentation, class weighting, or a new architecture produces the greatest practical improvement. Include controls, metrics, repeated runs, and uncertainty reporting.

Multiple Choice Questions with Answers

Question 1

Which splitting strategy best prevents frames from the same video appearing in both training and test sets?

A. Random pixel-level splitting

B. Grouped splitting by video

C. Sorting images by filename

D. Applying stronger test augmentation

Correct Answer: B. Grouped splitting by video

Question 2

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0.40

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 3

Which metric is generally most informative for detecting rare positive cases across thresholds?

A. Precision–recall analysis

B. Pixel accuracy only

C. Training loss only

D. Image resolution

Correct Answer: A. Precision–recall analysis

Question 4

What does IoU measure?

A. The number of image channels

B. The overlap divided by the union of two regions

C. The probability calibration error only

D. The number of classes in a dataset

Correct Answer: B. The overlap divided by the union of two regions

Question 5

Which is an example of domain shift?

A. Changing a model’s variable name

B. Using a different camera in deployment

C. Increasing the batch size during training

D. Removing a duplicate file before training

Correct Answer: B. Using a different camera in deployment

Question 6

What is the main purpose of an ablation study?

A. To encrypt the dataset

B. To measure the contribution of system components

C. To increase image resolution automatically

D. To replace the test set with training data

Correct Answer: B. To measure the contribution of system components

Question 7

Which statement about calibration is correct?

A. It measures only inference speed

B. A probability near 0.7 should correspond approximately to a 70% event frequency when calibrated

C. It guarantees high recall

D. It is identical to IoU

Correct Answer: B. A probability near 0.7 should correspond approximately to a 70% event frequency when calibrated

Question 8

Which augmentation is inappropriate when left–right orientation carries essential meaning?

A. Horizontal flipping

B. Moderate brightness adjustment

C. Sensor noise simulation

D. Resolution standardisation

Correct Answer: A. Horizontal flipping

Question 9

What does aleatoric uncertainty represent?

A. Uncertainty from irreducible ambiguity or measurement noise

B. Only uncertainty caused by software bugs

C. The number of training examples

D. The proportion of background pixels

Correct Answer: A. Uncertainty from irreducible ambiguity or measurement noise

Question 10

Which practice best supports reproducibility?

A. Changing the test set after every experiment

B. Reporting only the best random seed

C. Recording dataset, code, configuration, splits, and seeds

D. Using undocumented manual corrections

Correct Answer: C. Recording dataset, code, configuration, splits, and seeds