AI/ML Evaluation, Reproducibility, and Research Analytics

Introduction

An artificial intelligence or machine learning system is not established as useful merely because it produces predictions. A credible research result must demonstrate that the system performs well for an appropriate task, generalizes beyond the data used for development, behaves acceptably across relevant groups and conditions, and can be independently examined or reproduced. Evaluation is therefore both a technical and a research activity.

In AI/ML research, evaluation includes selecting suitable metrics, designing valid validation procedures, quantifying uncertainty, studying errors and robustness, examining fairness and explainability, and documenting the complete computational process. These activities support scientific validity, responsible deployment, and effective communication with technical and non-technical stakeholders.

This chapter presents a practical framework for evaluating AI/ML systems and producing reproducible research. It is intended for undergraduate students who are beginning to design, implement, and report empirical studies.

Learning Objectives

By the end of this chapter, students should be able to:

1. Select evaluation metrics that match the task, data, and decision context.

2. Design appropriate training, validation, and test procedures.

3. Explain uncertainty using confidence intervals, variability estimates, and statistical comparisons.

4. Conduct error, robustness, sensitivity, fairness, and explainability analyses.

5. Identify sources of irreproducibility in AI/ML research.

6. Construct a reproducible workflow using version control, experiment tracking, fixed seeds, environment specifications, and documentation.

7. Communicate evaluation results accurately and critically.

1. Evaluation as an Evidence-Building Process

Evaluation answers the question: “How well does a model perform for a clearly defined purpose?” This question has several dimensions:

• Predictive performance: Are predictions accurate or useful?
• Generalization: Does performance remain satisfactory on unseen data?
• Reliability: Are performance estimates stable across samples and runs?
• Robustness: Does the model withstand realistic changes or disturbances?
• Fairness: Are errors distributed acceptably across relevant groups?
• Interpretability and explainability: Can users understand important factors behind outputs?
• Operational suitability: Does the system satisfy requirements for latency, cost, safety, and maintainability?

No single metric captures all these dimensions. A high accuracy score can coexist with poor minority-group performance, severe calibration errors, or vulnerability to distribution shift. Evaluation must therefore be designed before results are observed, rather than selected after experimentation to produce favorable conclusions.

1.1 Define the evaluation target

A research report should specify:

1. The prediction target and its operational meaning.

2. The population to which conclusions are intended to apply.

3. The unit of analysis, such as a patient, image, transaction, or user.

4. The consequences of false positives and false negatives.

5. The deployment threshold, if predictions are converted into decisions.

6. Primary and secondary evaluation metrics.

7. The comparison or baseline system.

A metric is meaningful only relative to a task. For example, accuracy may be acceptable when classes are balanced and error costs are similar, but misleading for rare-event detection.

2. Metrics for Common AI/ML Tasks

2.1 Classification metrics

For binary classification, predictions can be represented using:

• True positive (TP): a positive instance correctly predicted as positive.
• True negative (TN): a negative instance correctly predicted as negative.
• False positive (FP): a negative instance incorrectly predicted as positive.
• False negative (FN): a positive instance incorrectly predicted as negative.

The confusion matrix summarizes these four outcomes.

Accuracy is

\[

Accuracy = \frac{TP+TN}{TP+TN+FP+FN}.

\]

It measures the proportion of correct predictions. It is most informative when class proportions and error costs are reasonably balanced.

Precision, also called positive predictive value, is

\[

Precision = \frac{TP}{TP+FP},

\]

provided that \(TP+FP>0\). It answers: among predicted positives, how many are actually positive?

Recall, or sensitivity, is

\[

Recall = \frac{TP}{TP+FN},

\]

provided that \(TP+FN>0\). It answers: among actual positives, how many are detected?

The F1 score is the harmonic mean of precision and recall:

\[

F1 = \frac{2PR}{P+R},

\]

where \(P\) is precision and \(R\) is recall. It is useful when both types of performance matter and a single summary is required. It should not be interpreted as an average of precision and recall in the ordinary arithmetic sense.

For imbalanced data, balanced accuracy is often preferable to ordinary accuracy:

\[

Balanced\ Accuracy = \frac{Sensitivity+Specificity}{2},

\]

where specificity is \(TN/(TN+FP)\). Balanced accuracy gives equal weight to the positive and negative classes.

The Matthews correlation coefficient (MCC) is

\[

MCC=\frac{TP\cdot TN-FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}.

\]

It is defined when the denominator is nonzero and ranges from −1 to 1. It considers all entries of the confusion matrix and can be useful for imbalanced binary classification.

2.2 Thresholds and ranking metrics

Many classifiers output a score or probability rather than a final class. Changing the decision threshold changes precision, recall, and other measures. A threshold should be selected using validation data and justified by application costs, not chosen using the test set.

The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate over thresholds. Its area, AUROC, measures ranking ability across thresholds. The precision–recall curve is often more informative when the positive class is rare because it focuses on positive predictive performance and detection.

AUC values summarize ranking, not necessarily the quality of probabilities or the performance at the operational threshold. Consequently, a study should report threshold-specific results when a real decision threshold exists.

2.3 Probability quality and calibration

A model is calibrated if, among cases assigned a probability of approximately 0.7, roughly 70% are positive over an appropriate population. Calibration is different from discrimination: a model may rank cases correctly while producing probabilities that are systematically too high or too low.

Calibration can be studied using reliability diagrams, calibration curves, calibration error measures, and the Brier score. For binary outcomes, the Brier score is the mean squared difference between predicted probability \(p_i\) and observed outcome \(y_i\in\{0,1\}\):

\[

Brier = \frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2.

\]

Lower values are better. The score is sensitive to both calibration and, to some extent, refinement of predictions.

2.4 Regression metrics

For continuous outcomes, common metrics include:

\[

MAE=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat y_i|

\]

and

\[

RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2}.

\]

Here, \(y_i\) is the observed value, \(\hat y_i\) is the prediction, and \(n\) is the number of observations. MAE is expressed in the original units and treats errors linearly. RMSE also uses the original units but penalizes large errors more heavily because errors are squared before averaging.

The coefficient of determination is

\[

R^2=1-\frac{\sum_i(y_i-\hat y_i)^2}{\sum_i(y_i-\bar y)^2},

\]

where \(\bar y\) is the sample mean of observed values. The usual interpretation of the denominator as total variation assumes an intercept-based comparison to the mean. An \(R^2\) value can be negative on test data when predictions are worse than the mean baseline; it is not automatically a percentage of correct predictions.

2.5 Ranking, recommendation, and information retrieval

Recommendation and retrieval systems often evaluate the top \(k\) results. Precision@k is the proportion of the first \(k\) returned items that are relevant. Recall@k is the proportion of all relevant items retrieved in the first \(k\). Mean average precision and normalized discounted cumulative gain give greater attention to ranking quality and item position. The chosen metric should reflect whether users value finding any relevant item, finding most relevant items, or placing the best items first.

3. Validation and Experimental Design

3.1 Holdout evaluation

A basic design separates data into training, validation, and test sets. The training set estimates model parameters. The validation set supports model selection, hyperparameter tuning, threshold selection, and feature decisions. The test set provides a final estimate after choices are complete.

The test set must not influence iterative development. Repeatedly inspecting test performance and modifying the model creates test-set leakage and produces an optimistically biased estimate.

Splits should be performed at the correct independence level. If several records belong to the same patient, customer, household, video, or device, records from one entity should generally not appear in both training and test sets. Otherwise, the model may exploit identity-specific information rather than learn general patterns.

3.2 Cross-validation

In \(k\)-fold cross-validation, the training data are divided into \(k\) folds. The model is trained on \(k-1\) folds and evaluated on the remaining fold, repeating until every fold has served as validation data. The scores are then summarized, commonly using their mean and standard deviation.

Cross-validation is useful when data are limited, but it does not replace a final untouched test set when an unbiased final estimate is required. For time-dependent data, random folds may be invalid; chronological or rolling-window validation is more appropriate. For grouped data, group-aware splitting is required. In nested cross-validation, an inner loop tunes hyperparameters and an outer loop estimates generalization, reducing selection bias.

3.3 Baselines and ablation studies

A sophisticated model should be compared with meaningful baselines, such as majority-class prediction, a mean predictor, logistic regression, a decision tree, or an existing operational method. Baselines reveal whether complexity provides practical benefit.

An ablation study removes or changes one component at a time, such as a feature group, data augmentation method, loss term, or architectural module. It supports claims about which components contribute to performance. Ablations must preserve a fair experimental budget and use the same evaluation protocol.

Worked Example: Classification Metrics

Suppose a test set contains 100 cases with TP = 30, TN = 50, FP = 10, and FN = 10. Then:

\[

Accuracy=\frac{30+50}{100}=0.80.

\]

Precision is \(30/(30+10)=0.75\), and recall is \(30/(30+10)=0.75\). Therefore, the F1 score is \(2(0.75)(0.75)/(0.75+0.75)=0.75\). Specificity is \(50/(50+10)=0.833\), so balanced accuracy is \((0.75+0.833)/2\approx0.792\). The example shows why multiple metrics provide a more informative picture than accuracy alone.

4. Quantifying Uncertainty

A reported score is an estimate based on a sample, not an unquestionable property of the model. Uncertainty can arise from finite sample size, sampling variation, random initialization, data splitting, labeling variation, and model-selection decisions.

A confidence interval describes a procedure that, under its assumptions, captures the corresponding population quantity at a stated long-run rate. It should not be casually described as the probability that a fixed parameter lies in one particular interval.

For complex metrics, bootstrap intervals are often practical. A bootstrap procedure repeatedly samples observations with replacement, recalculates the metric, and uses the resulting empirical distribution to form an interval. Resampling must respect dependence: for grouped data, resample groups rather than individual records.

Repeated runs with different random seeds can quantify variation due to initialization and stochastic training. Results should report the number of runs, mean or median, dispersion measure, and evaluation protocol. A seed improves repeatability but does not remove uncertainty or guarantee exact replication across hardware and software versions.

When comparing models, paired evaluation is usually preferable: evaluate both models on the same test cases and analyze per-case differences. Statistical significance should not replace practical significance. A small improvement may be statistically detectable but operationally irrelevant, whereas a meaningful improvement may require more data to establish confidently.

5. Error, Robustness, and Sensitivity Analysis

5.1 Error analysis

Aggregate metrics conceal systematic failure. Error analysis should categorize incorrect predictions by factors such as class, data source, lighting, language, document type, severity, or confidence. A useful procedure is:

1. Collect representative false positives and false negatives.

2. Inspect examples with domain expertise.

3. Assign error categories using a documented coding scheme.

4. Quantify the frequency and severity of categories.

5. Identify possible remedies.

6. Re-evaluate after changes.

The sample used for qualitative inspection should not be selected only for dramatic examples. Random samples and stratified samples reduce selection bias.

5.2 Robustness

Robustness concerns performance under plausible changes, including noise, missing values, image transformations, domain shifts, changed prevalence, adversarial manipulation, and changes in data collection. Robustness tests should represent realistic threats rather than arbitrary perturbations.

A robustness experiment should state the perturbation, its magnitude, why it is plausible, the affected cases, and the performance change. If deployment data may come from a different hospital, country, sensor, or time period, external or temporal validation is more informative than synthetic noise alone.

5.3 Sensitivity analysis

Sensitivity analysis examines how conclusions change when assumptions, parameters, features, thresholds, or preprocessing choices vary. Examples include changing a classification threshold, removing a feature family, varying missing-data treatment, or comparing alternative definitions of the outcome.

Sensitivity analysis is especially important when research decisions are contestable. If a conclusion disappears under a reasonable alternative specification, the report should state that limitation rather than present the result as definitive.

6. Fairness Analysis

Fairness analysis begins by identifying relevant groups and the social context of the decision. Groups may be defined by legally protected attributes, access-related variables, geography, language, age, disability, or other factors. Researchers must consider privacy, consent, and the risk of re-identification.

Common group metrics include:

• Demographic parity: prediction rates are similar across groups.
• Equal opportunity: true-positive rates are similar across groups.
• Equalized odds: both true-positive and false-positive rates are similar across groups.
• Group calibration: predicted probabilities have comparable outcome meaning across groups.

These criteria are not generally simultaneously achievable when groups have different base rates, except under special conditions. Fairness is therefore not a matter of selecting the universally correct formula. It requires a justified account of harms, benefits, legal requirements, and stakeholder priorities.

Researchers should report group sample sizes, uncertainty intervals, missing attribute data, intersectional groups where feasible, and both absolute and relative differences. A fairness analysis should also examine data quality and label validity: unequal performance may result from biased labels, unequal measurement quality, or differences in access rather than from the model alone.

7. Explainability and Interpretability

Interpretability often refers to the degree to which a model is understandable in itself, whereas explainability commonly refers to methods that provide information about a model’s behavior after or around prediction. Examples include feature importance, partial dependence, counterfactual explanations, saliency maps, and local surrogate models.

Explanations must be evaluated rather than treated as automatically truthful. A visually persuasive explanation may be unstable, incomplete, or unrelated to the model’s actual causal reasoning. Useful checks include:

• Fidelity: does the explanation accurately represent the model?
• Stability: does it remain similar under small irrelevant changes?
• Robustness: does it resist manipulation?
• Human usefulness: does it improve appropriate user understanding or decisions?
• Actionability: does it identify factors that a user can influence?

Feature importance is not proof of causation. An explanation should not be used to claim that changing a feature will change the outcome unless a causal design supports that conclusion.

8. Reproducibility and Research Transparency

Reproducibility means that the same materials and procedures can produce consistent results. Replicability is often used for obtaining consistent findings in a new study, although terminology varies across disciplines. Both are important.

8.1 Versioning

Use a version-control system such as Git for source code, configuration files, documentation, and small metadata files. Record commit identifiers in experiment outputs. Large datasets and model artifacts should be managed using suitable data-versioning or artifact-storage systems. A data version should include a checksum or immutable identifier where possible.

8.2 Data documentation

A dataset description should state its origin, collection period, population, inclusion and exclusion criteria, labeling process, known missingness, licensing, privacy protections, and intended use. Datasheets or data cards can provide a structured format. Researchers must document transformations, deduplication, filtering, and train–test splitting.

8.3 Code, configuration, and environments

A reproducible project should separate code from configuration. Hyperparameters, preprocessing options, random seeds, data paths, and model versions should be recorded rather than embedded invisibly in scripts. Environment files should specify programming-language versions, libraries, operating-system assumptions, hardware requirements, and relevant drivers. Containerization can improve portability, although it does not solve data access, nondeterministic hardware operations, or undocumented external services.

8.4 Randomness and determinism

Record all relevant random seeds, including those used for data splitting, model initialization, augmentation, and sampling. Set deterministic modes when feasible and document performance trade-offs. Exact bitwise reproducibility may be impossible across different GPUs, parallel libraries, compiler versions, or numerical implementations. The appropriate goal may therefore be reproducible conclusions within a documented tolerance.

8.5 Experiment tracking

Each experiment record should include:

• unique run identifier;
• code and data version;
• configuration and hyperparameters;
• seed and hardware;
• training and validation curves;
• final metrics and confidence intervals;
• model artifact location;
• warnings, failures, and elapsed time;
• notes explaining deviations from the plan.

Experiment-tracking platforms or structured logs reduce reliance on memory and make comparison auditable. A research report should distinguish planned experiments from exploratory analyses.

9. Research Analytics and Reporting

Research analytics concerns the systematic organization and interpretation of experimental evidence. A results table should report the metric definition, test population, sample size, point estimate, uncertainty, and baseline. Rounding should not imply greater precision than the data support.

Plots should have labeled axes, units, sample sizes where relevant, and uncertainty indicators. Bar charts without uncertainty can exaggerate small differences. For learning curves, report the selection rule and whether curves represent one run or aggregated runs.

Avoid common analytical errors:

• comparing metrics computed on different test sets;
• tuning on the test set;
• reporting only the best random seed;
• using inappropriate random splits for temporal or grouped data;
• interpreting correlation as causation;
• treating a statistically significant difference as practically important;
• hiding failed experiments or exclusions;
• claiming generalization from a narrow benchmark.

A strong technical report links each claim to evidence and identifies limitations. The abstract, tables, figures, and conclusion should not make stronger claims than the design supports.

10. Case Study: Reproducible Screening for Diabetic Retinopathy

This is an illustrative case, not a report of a real study.

Background

A university research team develops an image classifier to identify referable diabetic retinopathy from retinal photographs. The intended use is to prioritize patients for specialist review, not to make an autonomous diagnosis.

Problem

The initial model achieves 94% accuracy on a randomly split image dataset. However, several patients contribute multiple images, and images from the same clinic appear in both training and test sets. The research team must determine whether the result is credible and whether performance is equitable.

Analysis

The team identifies patient and clinic as grouping variables. It creates a patient-level split and reserves images from two clinics for external evaluation. Because missing a referable case is considered more harmful than referring an additional case, recall is designated as a primary metric, with precision, specificity, AUROC, calibration, and workload as secondary measures.

The team performs five-fold group-aware cross-validation on the development data and uses a separate external test set. It calculates bootstrap confidence intervals by resampling patients. It also repeats training with ten recorded seeds and reports mean performance and run-to-run variation.

Error analysis reveals that poor-quality images, unusual camera models, and advanced disease produce many errors. Performance is reported by age group, sex, ethnicity where ethically and legally appropriate, clinic, and image-quality category. The team evaluates threshold choices because the operational workload depends strongly on the threshold.

Approach and solution

The project uses Git for code, a versioned data manifest, configuration files, an environment lockfile, and an experiment-tracking system. Each run records the data-manifest hash, commit identifier, seed, hardware, and metrics. A model card documents intended use, limitations, subgroup performance, and prohibited uses.

The team compares the model with a simple baseline and conducts an ablation excluding image-quality preprocessing. Explanations are treated as supporting tools for review rather than proof of clinical reasoning. A clinician panel examines whether highlighted regions are relevant and whether explanations improve review decisions.

Outcome

The original 94% accuracy falls after patient-level splitting, demonstrating leakage in the initial evaluation. Recall remains acceptable at the selected threshold, but precision is lower in one clinic and calibration is poorer for a subgroup with fewer training examples. The team decides that prospective monitoring and additional data collection are required before deployment.

Learning points

1. A high score can be invalid when the split violates independence.

2. Metric selection should reflect the consequences of errors.

3. External, subgroup, and calibration analyses reveal problems hidden by aggregate accuracy.

4. Reproducibility requires more than publishing source code.

5. A responsible conclusion may be that further evidence is needed, not that deployment is ready.

11. Governance, Ethics, and Contemporary Practice

Evaluation data may contain personal, medical, financial, or biometric information. Researchers must follow applicable privacy and research-ethics requirements, minimize data access, protect identifiers, and document consent or lawful data use. Releasing code or models can create risks if artifacts permit re-identification, enable misuse, or expose security vulnerabilities.

AI systems may change after deployment because populations, policies, sensors, and behavior change. Monitoring should include data drift, performance drift, subgroup disparities, calibration, abstention or referral rates, and incident reporting. A model should have defined ownership, escalation procedures, and criteria for retraining or withdrawal.

Contemporary practice increasingly emphasizes model cards, data cards, reproducibility checklists, benchmark governance, dataset statements, synthetic-data evaluation, uncertainty-aware prediction, and responsible foundation-model assessment. Large models also require evaluation for contamination, prompt sensitivity, hallucination, harmful outputs, and performance across languages and domains. These tools support accountability but do not replace domain judgment or institutional governance.

Best-Practice Checklist

Before submitting an AI/ML study, ask:

1. Is the research question and intended population explicit?

2. Are primary metrics justified by the decision context?

3. Is the test set untouched during development?

4. Are splits appropriate for groups, time, and deployment conditions?

5. Are baselines, ablations, and uncertainty estimates included?

6. Have errors, robustness, sensitivity, fairness, and calibration been examined where relevant?

7. Are data, code, configurations, environments, seeds, and artifacts versioned?

8. Can another researcher reconstruct the experiment from the documentation?

9. Are limitations and negative results reported honestly?

10. Are privacy, safety, legal, and governance responsibilities addressed?

Chapter Summary

AI/ML evaluation is a multidimensional evidence-building process. Classification, regression, ranking, calibration, and operational metrics must be selected according to the task and consequences of error. Valid validation requires appropriate data splitting, protection of the test set, group and temporal awareness, meaningful baselines, and, when needed, nested cross-validation. Uncertainty should be quantified through confidence intervals, repeated runs, and paired comparisons.

Error analysis identifies systematic failure; robustness and sensitivity analyses test behavior under plausible changes; fairness analysis examines group outcomes and social context; and explainability analysis evaluates whether explanations are faithful, stable, and useful. Reproducibility depends on versioned code and data, documented environments, recorded seeds, configuration management, experiment tracking, and transparent reporting. Ethical governance, privacy, monitoring, and responsible deployment are integral parts of technical evaluation rather than optional additions.

Key Terms

Evaluation metric: A quantitative measure used to assess model performance.

Confusion matrix: A table of true and false predictions by class.

Precision: The proportion of predicted positives that are positive.

Recall: The proportion of actual positives that are detected.

Specificity: The proportion of actual negatives correctly identified.

Calibration: Agreement between predicted probabilities and observed frequencies.

Brier score: Mean squared error between predicted probabilities and binary outcomes.

Cross-validation: Repeated training and validation across partitions of data.

Nested cross-validation: Cross-validation with separate inner tuning and outer assessment loops.

Data leakage: Unintended use of information unavailable at prediction time.

Baseline: A reference method used for comparison.

Ablation study: An experiment removing or changing a model component.

Robustness: Ability to maintain performance under plausible disturbances or shifts.

Sensitivity analysis: Examination of conclusions under changed assumptions or settings.

Fairness: Evaluation of whether outcomes and errors are acceptable across groups.

Explainability: Methods that provide information about model behavior.

Reproducibility: Ability to obtain consistent results using the same materials and procedures.

Experiment tracking: Systematic recording of configurations, artifacts, metrics, and run information.

Model card: Documentation describing a model’s intended use, performance, limitations, and risks.

Data leakage: Information from validation, test, or future data improperly influencing development.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading for an imbalanced classification problem?

2. Distinguish precision from recall.

3. What is the purpose of an untouched test set?

4. Why should patient- or customer-level grouping be considered during data splitting?

5. Name four items that should be recorded for a reproducible experiment.

Long Answer Questions

1. Explain how metric selection should reflect the consequences of false positives and false negatives. Use a suitable application example.

2. Compare holdout validation, cross-validation, and nested cross-validation, including their advantages and limitations.

3. Describe a complete error and robustness analysis for an image-classification system.

4. Explain why fairness criteria may conflict and how researchers should report group performance responsibly.

5. Discuss the relationship among version control, environment management, random seeds, experiment tracking, and reproducibility.

Analytical / Application Questions

1. A classifier produces TP = 45, TN = 35, FP = 15, and FN = 5. Calculate accuracy, precision, recall, F1 score, and balanced accuracy. Interpret the results for a screening application.

2. A fraud-detection dataset contains many transactions from the same customers across several months. Design a validation strategy that avoids leakage and reflects deployment over time.

3. A model has higher overall AUROC than its baseline but lower recall for a small demographic group and poor calibration in a new hospital. Recommend additional analyses and decide whether immediate deployment is justified.

Multiple Choice Questions with Answers

Question 1

Which metric is the harmonic mean of precision and recall?

A. Accuracy

B. F1 score

C. Specificity

D. Brier score

Correct Answer: B. F1 score

Question 2

Which dataset should generally remain untouched during model development?

A. Training set

B. Augmentation set

C. Test set

D. Feature-engineering set

Correct Answer: C. Test set

Question 3

What does calibration primarily assess?

A. Whether probabilities correspond to observed frequencies

B. Whether training is faster than testing

C. Whether classes have equal sample sizes

D. Whether explanations are visually attractive

Correct Answer: A. Whether probabilities correspond to observed frequencies

Question 4

For strongly time-dependent data, which validation approach is usually most appropriate?

A. Randomly shuffle all records

B. Use chronological or rolling-window splits

C. Duplicate the newest records

D. Use only the training score

Correct Answer: B. Use chronological or rolling-window splits

Question 5

If TP=30, TN=50, FP=10, and FN=10, what is accuracy?

A. 0.60

B. 0.70

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 6

Which practice most directly helps identify the contribution of a model component?

A. Ablation study

B. Data anonymization

C. Increasing font size in figures

D. Deleting failed runs

Correct Answer: A. Ablation study

Question 7

What is a key risk of using records from the same patient in both training and test sets?

A. The model becomes deterministic

B. The evaluation may be optimistically biased by leakage

C. Precision is mathematically forced to zero

D. Calibration becomes impossible to calculate

Correct Answer: B. The evaluation may be optimistically biased by leakage

Question 8

Which item is essential for reconstructing a computational experiment?

A. Only the final accuracy

B. Only the model name

C. Code, data version, configuration, and environment information

D. A screenshot of the training window

Correct Answer: C. Code, data version, configuration, and environment information

Question 9

What does recall measure in binary classification?

A. The proportion of predicted positives that are correct

B. The proportion of actual positives detected

C. The proportion of actual negatives detected

D. The average squared probability error

Correct Answer: B. The proportion of actual positives detected

Question 10

Why are bootstrap confidence intervals useful in ML evaluation?

A. They eliminate sampling variation

B. They estimate uncertainty in a statistic using repeated resampling

C. They guarantee fairness across groups

D. They replace the need for a test set

Correct Answer: B. They estimate uncertainty in a statistic using repeated resampling