Chapter 12: Reproducibility, Open Science, and Research Analytics

1. Introduction

Information technology research increasingly depends on complex software, distributed infrastructures, large datasets, machine-learning models, cloud platforms, and rapidly changing computational environments. These characteristics create significant opportunities for discovery, but they also create risks. A result may be difficult to reproduce because source code is unavailable, data are poorly documented, software dependencies have changed, or analytical decisions were made after examining the results. In doctoral research, the credibility of a contribution depends not only on originality but also on the transparency and reliability of the process through which the contribution was produced.

Reproducibility, open science, and research analytics provide an integrated response to these challenges. Reproducibility concerns whether an analysis can be repeated using the same data, code, and procedures. Replication concerns whether a finding can be obtained in a new study, often with new data or in a different context. Open science promotes appropriate openness in methods, materials, data, code, publications, and peer review. Research analytics uses structured evidence to evaluate research processes, outputs, influence, and quality.

This chapter presents these ideas as components of responsible research management rather than as isolated technical practices. It explains preregistration, version control, data and code management, FAIR principles, replication, sensitivity analysis, transparent reporting, research software quality, and responsible research metrics. The central argument is that reproducibility must be designed into a project from its earliest planning stage.

2. Learning Objectives

After studying this chapter, the reader should be able to:

1. Distinguish reproducibility, replicability, repeatability, and generalizability.

2. Design a preregistration and research data-management plan.

3. Apply version control and structured workflows to research code and documents.

4. Evaluate research data and software using FAIR principles and quality criteria.

5. Plan direct, conceptual, and computational replication studies.

6. Conduct sensitivity analysis and report analytical decisions transparently.

7. Select responsible research metrics and avoid misleading evaluation practices.

8. Assess ethical, legal, security, and governance issues in open research.

9. Develop a practical reproducibility framework for doctoral IT research.

3. Meaning and Concept

Reproducibility is the ability to obtain consistent analytical results when the same data, procedures, code, and computational environment are used. For example, if a researcher publishes a machine-learning experiment with the original dataset, preprocessing script, model configuration, and execution instructions, another researcher should be able to generate the reported performance values.

Repeatability usually refers to obtaining consistent results under the same conditions, often by the same team. Reproducibility may involve a different researcher using the same research materials. Replicability generally refers to obtaining consistent findings through a new investigation using new data, a new sample, or a different implementation. These terms are not used identically in every discipline, so researchers should define them explicitly in a thesis or article.

Open science is a broad approach to making the research process as accessible, transparent, and reusable as ethically and legally possible. It can include open-access publications, open methods, open-source software, open data, preregistration, registered reports, open educational resources, and open peer review. Openness is not absolute. Personal data, security-sensitive code, confidential industrial information, and culturally restricted knowledge may require controlled or prohibited access.

Research analytics refers to the systematic collection and interpretation of information about research activities and outcomes. Examples include citation patterns, software downloads, dataset reuse, collaboration networks, review times, replication outcomes, and societal or organizational use. Responsible research analytics uses metrics to support judgment rather than replace it.

4. Fundamental Principles

The first principle is transparency. A reader should be able to understand what was planned, what was changed, why changes were made, and how conclusions follow from evidence. Transparency does not require exposing every exploratory idea as if it had been predicted in advance. Instead, confirmatory and exploratory analyses should be clearly distinguished.

The second principle is traceability. Each important result should be connected to its source data, transformation steps, analytical code, software version, and reporting artifact. A traceable workflow makes it possible to identify where an error occurred and to update results when data or code change.

The third principle is proportional openness. Research materials should be shared at the greatest level of openness consistent with privacy, intellectual property, security, contractual obligations, and participant consent. If complete sharing is impossible, researchers should provide metadata, documentation, synthetic data, access procedures, or controlled repositories.

The fourth principle is preservation. Research objects should remain available and interpretable over time. A link to a personal computer or a temporary file-sharing service is not an adequate preservation strategy. Repositories, persistent identifiers, archival formats, and clear licensing support long-term access.

The fifth principle is quality assurance. Reproducibility is not achieved merely by publishing code. Code must be tested, data must be validated, documentation must be sufficient, and the environment must be specified. A reproducible workflow can still produce a biased or invalid conclusion; reproducibility supports reliability but does not guarantee scientific truth.

The sixth principle is responsible measurement. Metrics should be interpreted within disciplinary and institutional contexts. Citation counts, journal rankings, download numbers, and social-media attention can be informative, but each is incomplete and potentially distorted.

5. Major Components / Elements

Preregistration

Preregistration records research questions, hypotheses, outcomes, sampling plans, exclusion rules, and analysis procedures before access to outcome data or before outcome analysis begins. It reduces undisclosed flexibility, selective reporting, and hindsight bias. A preregistration can be brief or highly detailed. It should identify which elements are confirmatory and which remain exploratory. Amendments are permissible, but they should be dated and explained rather than silently replacing the original plan.

A registered report extends preregistration by obtaining peer review before data collection. If the protocol is accepted in principle, publication depends primarily on following the approved design and transparently reporting results rather than on obtaining statistically significant findings.

Version Control

Version control records changes to code, documents, configuration files, and sometimes data-processing scripts. A repository should use meaningful commits, descriptive messages, branches where appropriate, and tagged releases corresponding to submitted or published versions. Version control also supports collaboration, review, rollback, and auditability.

Large or sensitive data should not necessarily be stored directly in a public code repository. Instead, the repository can contain data dictionaries, acquisition instructions, checksums, scripts, and references to an approved data archive. Secrets such as passwords, API keys, and private certificates must never be committed.

Data Management

A data-management plan describes how data will be collected, organized, documented, stored, protected, backed up, shared, and eventually preserved or destroyed. It should address file naming, folder structures, data formats, metadata, quality checks, access permissions, retention periods, and responsibilities. Data dictionaries should define variables, units, coding schemes, missing-value conventions, and transformations.

The distinction between raw, processed, and analysis-ready data is essential. Raw data should be preserved as an immutable source where possible. Processing scripts should transform raw data into documented intermediate and analysis datasets. Manual modifications should be minimized and recorded.

Code and Research Software Management

Research code should be treated as software, not merely as supplementary text. Good practice includes modular design, automated tests, dependency specification, continuous integration, code review, documentation, licensing, and release management. A computational environment may be captured through a lockfile, container image, virtual machine, workflow specification, or detailed installation instructions.

FAIR Principles

FAIR means Findable, Accessible, Interoperable, and Reusable. Findable objects have persistent identifiers, rich metadata, and searchable descriptions. Accessible objects can be retrieved through standard protocols, with authentication where necessary. Interoperable objects use documented formats, controlled vocabularies, and clear relationships to other objects. Reusable objects include provenance, quality information, licensing, and sufficient context for appropriate reuse.

FAIR does not mean that all data must be openly available. Sensitive health data may be FAIR through a controlled-access repository with clear discovery metadata and an application process.

Replication

A direct replication attempts to reproduce the methods of an earlier study as closely as possible. A conceptual replication tests the same theoretical proposition using different measures, populations, technologies, or procedures. A computational replication reruns existing analyses using the same or reconstructed computational materials. Replication strengthens confidence when results persist, but a failed replication does not automatically prove that the original study was fraudulent or worthless. Differences in context, implementation, statistical power, or measurement may explain divergent outcomes.

Sensitivity Analysis

Sensitivity analysis examines whether conclusions change when reasonable assumptions, model specifications, preprocessing choices, thresholds, missing-data methods, or influential observations are varied. In IT research, a sensitivity analysis might compare alternative train-test splits, feature-selection methods, hyperparameters, privacy-preserving transformations, or definitions of system failure. Researchers should report why the tested alternatives are plausible and how conclusions respond to them.

Transparent Reporting

Transparent reporting identifies the research design, data sources, inclusion criteria, exclusions, missing data, instruments, software, statistical models, limitations, deviations from the protocol, and conflicts of interest. In computational studies, it should also report hardware, operating-system details when relevant, random seeds, evaluation metrics, baseline methods, and resource requirements.

6. Process / Framework / Methodology

A practical reproducibility framework can be organized into eight stages.

First, define the research claim and evidence requirements. Specify what would count as support, what alternative explanations exist, and which elements are confirmatory or exploratory.

Second, preregister the protocol where appropriate. State hypotheses, variables, sampling or data-acquisition procedures, analysis models, stopping rules, and exclusion criteria. If the study is exploratory, register the broad purpose without pretending that specific findings were predicted.

Third, create a data-management and governance plan. Classify data by sensitivity, define access roles, select storage and backup arrangements, establish retention rules, and document consent and legal restrictions.

Fourth, establish a version-controlled project repository. Separate source data, derived data, code, documentation, figures, manuscripts, and configuration files. Use a reproducible directory structure and meaningful release tags.

Fifth, automate the workflow. A workflow manager can express dependencies between data preparation, analysis, visualization, and report generation. Automation reduces accidental omission and makes it easier to rerun the full pipeline.

Sixth, verify quality. Use data validation, unit tests, integration tests, code review, independent analytical checks, and environment testing. Compare generated outputs with expected results and investigate discrepancies.

Seventh, conduct robustness and sensitivity analyses. Test reasonable alternatives and report whether the substantive conclusion remains stable. Avoid presenting only the specification that produces the preferred result.

Eighth, publish and preserve research objects. Deposit the article, protocol, code, metadata, documentation, and permitted data in suitable repositories. Assign persistent identifiers, specify licenses, state limitations, and provide a reproduction guide. After publication, monitor issues, correct errors transparently, and preserve updated versions.

7. Important Theories and Models

The Open Science Framework model treats research as a connected lifecycle in which planning, data collection, analysis, sharing, and preservation are linked. Its importance lies in integrating rather than isolating research objects.

The FAIR model provides a stewardship framework for data and other digital objects. It is principally about technical and organizational conditions for discovery and reuse, not about unrestricted public release.

The credibility spectrum distinguishes levels of evidence, from undocumented claims to fully specified, independently reproduced findings. It emphasizes that reproducibility is gradual. A study may have open code but inaccessible data, or open data but poorly documented preprocessing. Evaluation should therefore examine multiple dimensions.

The research lifecycle model views research as a sequence of planning, acquisition, processing, analysis, dissemination, and reuse. At each stage, provenance and quality controls should be maintained. This model is especially useful for research managers because it assigns responsibilities and resources across the project rather than treating openness as an activity performed only at publication.

The logic of responsible metrics is associated with principles such as responsible interpretation, transparency, diversity of indicators, and recognition of qualitative judgment. Metrics should inform decisions, not become automatic substitutes for peer assessment.

8. Practical Applications

In a software-engineering experiment, a doctoral researcher can preregister the defect-prediction hypotheses, define evaluation datasets, specify performance measures, publish preprocessing code, and report results for all tested models. A containerized environment can reduce dependency problems.

In cybersecurity research, open release must be balanced against misuse. Researchers may publish detection algorithms, synthetic logs, threat models, and evaluation protocols while restricting exploit code or operational data. A controlled-access process can allow qualified researchers to verify results without increasing public risk.

In information-systems research, survey instruments, anonymized codebooks, analysis scripts, and recruitment procedures can be shared. If organizational identities are sensitive, researchers can provide a synthetic dataset and a secure audit process rather than the original responses.

In artificial-intelligence research, reproducibility requires reporting data provenance, labeling practices, model architecture, training procedures, random seeds, compute budgets, evaluation populations, and known failure modes. A single headline accuracy score is insufficient for responsible evaluation.

9. Examples

Consider a study claiming that a new recommender algorithm improves user engagement. A weak report provides only the final percentage improvement. A stronger report provides the dataset construction, temporal split, treatment of inactive users, baseline algorithms, hyperparameters, random seeds, code, evaluation scripts, confidence intervals, and results under alternative engagement definitions. It also discusses whether improved engagement might reflect undesirable addictive behavior.

As another example, a cloud-performance study may find that a scheduling policy reduces response time. Reproduction requires the workload traces, infrastructure configuration, monitoring definitions, warm-up period, resource allocation, and scripts used to calculate latency. If a cloud provider changes its hardware or pricing model, a later replication may produce different absolute values while still testing the same conceptual claim.

10. Technology and Digital Applications

Git and comparable distributed version-control systems support change tracking, collaboration, branching, and release tagging. Repository platforms can add issue tracking, code review, access control, and continuous integration.

Workflow tools such as Snakemake, Nextflow, or computational notebooks can connect data-processing and analysis steps. Notebooks are useful for exploration and communication, but production analyses should separate reusable functions from informal exploratory cells and should record execution order.

Containers, including Docker-compatible technologies, can package software dependencies and system libraries. They improve portability but do not guarantee identical hardware behavior, randomness, external services, or data access. Container images should be versioned and scanned for vulnerabilities.

Repository services such as institutional archives, disciplinary repositories, and general-purpose data repositories support persistent identifiers, metadata, access control, and preservation. Object-storage systems can support large datasets, but they require explicit governance, backup, encryption, and lifecycle policies.

Research information-management systems can connect projects, outputs, grants, datasets, software, and researcher identifiers. Bibliometric and usage dashboards may support portfolio analysis, but their results require contextual and ethical interpretation.

11. Strategic / Managerial Importance

For universities and research organizations, reproducibility reduces waste. Time and funding are lost when teams cannot locate data, reconstruct analyses, or maintain obsolete software. Standardized templates, repository policies, training, data stewards, research-software engineers, and secure infrastructure create institutional capacity.

Reproducible research also strengthens external credibility. Industry partners, regulators, funders, and public agencies are more likely to use findings that can be inspected and validated. In doctoral education, these practices develop transferable skills in project management, software engineering, documentation, and evidence-based decision-making.

Managers should avoid imposing openness without resources. Data curation, secure storage, software testing, and documentation require time and expertise. Evaluation systems should recognize high-quality datasets, software, protocols, replications, and negative results rather than rewarding only papers in prestigious outlets.

12. Detailed Case Study

Background

The following fictional case is illustrative. A doctoral research group, UrbanSense Lab, develops an edge-computing system for predicting failures in municipal traffic sensors. The project combines sensor streams, weather data, maintenance records, and a machine-learning model.

Problem

The initial paper reports high predictive accuracy, but an external team cannot reproduce the result. The dataset is not clearly defined, preprocessing was performed manually, the source code depends on an undocumented library version, and the reported evaluation uses a random split that may allow information from future events to enter the training set. Several sensor locations were excluded without explanation.

Analysis

An internal audit finds that the published accuracy was calculated on a subset of sensors with relatively complete records. Missing values were imputed using information from the entire observation period. The code repository contains several experimental notebooks, but no clear indication of which notebook generated the paper’s tables. The team also used different random seeds during development and did not retain the final environment.

The audit distinguishes three problems. First, there is a provenance problem: the relationship between raw data, processed data, code, and reported outputs is unclear. Second, there is a validity problem: temporal leakage may inflate performance. Third, there is a governance problem: maintenance records include identifiable worker information and cannot be publicly released without additional protection.

Strategy / Solution

The team creates a preregistered protocol for a corrective evaluation. It defines a chronological training-validation-test design, sensor inclusion rules, missing-data procedures, baseline models, primary metrics, and sensitivity analyses. All deviations from the original study are documented.

Raw data are preserved in a restricted institutional repository. A data dictionary, metadata record, synthetic example dataset, and access procedure are released publicly. Personal identifiers are removed or transformed, and the data-access committee reviews legitimate requests.

The team moves code into a version-controlled repository, tags the release used for the corrected analysis, pins dependencies, and builds a container image. A workflow automatically validates inputs, performs preprocessing, trains models, generates figures, and creates a report. Tests verify that future observations are not used in training and that missing-data rules operate as intended.

The researchers perform sensitivity analyses using alternative imputation methods, time windows, model classes, and sensor subsets. They also invite an independent researcher to run the workflow and review the documentation before resubmission.

Outcome

Under the leakage-free temporal design, the claimed improvement falls from 18 percent to 7 percent, but it remains positive across most reasonable specifications. Performance is substantially lower for sensors installed in older environments. The revised paper reports both the original and corrected analyses, explains the discrepancy, and provides open code, metadata, a synthetic dataset, and controlled access to the protected data.

The result is more credible even though the effect is smaller. The municipality adopts the system only for high-risk sensor prioritization rather than fully automated maintenance decisions. The research group also creates a laboratory reproducibility checklist for future projects.

Learning Points

The case demonstrates that transparency can weaken an attractive claim while strengthening the research contribution. It shows that reproducibility requires data provenance, temporal validity, software management, and governance. It also illustrates that controlled access and synthetic data can support openness when full public release is inappropriate.

13. Advantages / Benefits

Reproducible and open practices improve error detection, enable independent verification, increase the reuse of data and code, reduce duplicated effort, and make collaboration more efficient. They support cumulative science because later researchers can extend rather than reconstruct prior work. They also improve doctoral supervision by making research progress visible and reviewable.

For institutions, these practices enhance reputation, compliance, funder confidence, and the practical transfer of research. For researchers, well-documented software and datasets can become recognized scholarly outputs and can create collaborations beyond the original publication.

14. Limitations / Challenges

Reproducibility has costs. Documentation, curation, testing, repository maintenance, and secure access require funding and specialized labor. Some experiments depend on proprietary platforms, unavailable hardware, confidential data, or rapidly changing online services. Exact environmental replication may therefore be impossible.

Open materials can also be misunderstood or misused. Poorly documented datasets may be reused in inappropriate contexts, and public security tools may facilitate attacks. Repositories can disappear, links can decay, and software dependencies can become obsolete. Finally, reproducible analysis can still reflect biased sampling, invalid constructs, weak theory, or flawed causal reasoning.

15. Ethical and Governance Considerations

Researchers must respect informed consent, privacy, data-protection law, intellectual-property rights, contractual restrictions, and community expectations. Data minimization and purpose limitation should guide collection and reuse. De-identification is not always irreversible, particularly when datasets can be linked with external sources.

Governance should define who may access sensitive data, for what purpose, under what conditions, and with what accountability. Security research requires dual-use assessment and responsible disclosure. Artificial-intelligence studies should address fairness, representational harms, model misuse, and the environmental cost of computation.

Metric governance is also ethical. Institutions should not use citation counts or publication volume as the sole basis for hiring, promotion, funding, or doctoral evaluation. Researchers should disclose conflicts of interest, analytic deviations, funding sources, and relevant limitations.

16. Current and Emerging Trends

Registered reports and results-blind review are expanding because they reduce publication bias. Reproducibility badges and open-science statements increasingly signal the availability of materials, although badges should not replace substantive assessment.

Machine-actionable metadata allow repositories and software tools to discover relationships among datasets, code, protocols, publications, and researchers. Persistent identifiers for people, organizations, datasets, and software improve attribution and provenance.

Artificial intelligence is being used to assist documentation, code testing, metadata generation, and literature analysis. However, AI-generated code and summaries require verification, disclosure, and provenance because they can introduce hidden errors or licensing problems.

Privacy-preserving computation, federated learning, synthetic data, secure enclaves, and differential privacy may expand responsible data sharing. Reproducible computational notebooks are increasingly complemented by workflow systems, containers, and executable papers. Research evaluation is gradually moving toward portfolios that recognize software, datasets, societal contribution, mentoring, and replication.

17. Best Practices

1. Define reproducibility and replication requirements at project initiation.

2. Preregister hypotheses, outcomes, exclusions, and analysis plans where suitable.

3. Distinguish confirmatory analysis from exploration.

4. Use version control for code, protocols, manuscripts, and configuration files.

5. Preserve raw data and document every transformation.

6. Maintain a data dictionary, README, changelog, and reproducibility guide.

7. Never store credentials or confidential data in public repositories.

8. Use persistent identifiers, open formats, clear licenses, and rich metadata.

9. Automate analyses and use tests, code review, and independent verification.

10. Record software versions, dependencies, random seeds, and computational resources.

11. Conduct and report sensitivity analyses.

12. Share restricted materials through controlled-access mechanisms when needed.

13. Report null results, deviations, limitations, and conflicts of interest.

14. Use multiple responsible metrics and interpret them with expert judgment.

15. Plan for preservation, maintenance, correction, and eventual deprecation of research objects.

18. Future Scope

Future research will increasingly address reproducibility across heterogeneous cloud, edge, quantum, and AI environments. Automated provenance systems may record data transformations and model decisions at machine-readable levels. Digital research objects may become executable, citable, and continuously versioned.

Important future questions concern the reproducibility of foundation models, proprietary APIs, autonomous systems, and adaptive software. Researchers will need new standards for documenting training data, model updates, energy use, emergent behavior, and safety constraints. Institutions will also need sustainable funding models for research software and data stewardship.

19. Chapter Summary

Reproducibility, open science, and research analytics are central to credible doctoral IT research. Reproducibility depends on traceable data, code, environments, and decisions; replication tests whether findings persist in new studies. Preregistration reduces undisclosed flexibility, version control preserves research history, FAIR principles improve discovery and reuse, and sensitivity analysis tests robustness. Transparent reporting must include limitations and deviations, while research software requires engineering-quality practices. Openness must be balanced with privacy, security, intellectual property, and participant rights. Finally, responsible metrics should support—not replace—expert evaluation. A reproducibility framework is therefore both a technical workflow and a governance strategy.

20. Key Terms

1. Reproducibility — Obtaining consistent results using the same data, code, methods, and environment.

2. Repeatability — Obtaining consistent results under the same conditions, often by the same team.

3. Replicability — Obtaining consistent findings through a new study or implementation.

4. Preregistration — Publicly recording a research plan before testing outcomes.

5. Registered report — A peer-reviewed research protocol accepted before results are known.

6. Version control — A system for recording and managing changes to digital research materials.

7. Data provenance — The documented history and origin of data and transformations.

8. Data-management plan — A document specifying how research data will be handled throughout its lifecycle.

9. FAIR principles — Standards requiring research objects to be Findable, Accessible, Interoperable, and Reusable.

10. Direct replication — Repeating a prior study as closely as possible.

11. Conceptual replication — Testing the same theoretical claim using different methods or contexts.

12. Sensitivity analysis — Testing whether conclusions change under reasonable alternative assumptions.

13. Transparent reporting — Complete disclosure of methods, decisions, results, deviations, and limitations.

14. Research software — Code developed or adapted to collect, process, analyze, or communicate research.

15. Persistent identifier — A durable reference to a digital object, person, or organization.

16. Controlled access — A governed process through which approved users obtain restricted materials.

17. Research metric — A quantitative indicator of research activity, output, influence, or use.

18. Computational environment — The hardware, operating system, libraries, dependencies, and settings used for analysis.

21. Review Questions

Short Answer Questions

1. Distinguish reproducibility from replicability.

2. What is the purpose of preregistration?

3. Explain each element of the FAIR framework.

4. Why should research code be treated as software?

5. What is sensitivity analysis, and why is it important?

Long Answer Questions

1. Develop a complete reproducibility plan for a doctoral machine-learning project.

2. Explain how version control, data provenance, and workflow automation work together.

3. Discuss the ethical limits of open data in IT research.

4. Compare direct, conceptual, and computational replication.

5. Evaluate the benefits and risks of using bibliometric and online attention metrics in research assessment.

Analytical / Application Questions

1. A published cybersecurity experiment cannot be reproduced because the original traffic data contain confidential information. Design a FAIR and ethically responsible sharing strategy.

2. An AI study reports superior accuracy using a random data split from a time-dependent dataset. Identify possible validity threats and propose a sensitivity-analysis plan.

3. A university proposes evaluating doctoral researchers primarily by publication counts and citation scores. Critically assess the proposal and recommend a responsible research-analytics framework.

22. Multiple Choice Questions with Answers

Question 1

Which practice most directly reduces undisclosed analytical flexibility before results are examined?

A. Preregistration

B. Post-publication promotion

C. Increasing the sample after analysis

D. Removing all exploratory analysis

Correct Answer: A. Preregistration

Question 2

A researcher wants to preserve the exact code and dependencies used for a published experiment. Which combination is most appropriate?

A. A personal folder and an informal email description

B. Version control, a tagged release, and an environment specification

C. A screenshot of the final output

D. Only the executable results table

Correct Answer: B. Version control, a tagged release, and an environment specification

Question 3

Which statement best describes FAIR data?

A. All data must be publicly downloadable without restrictions

B. Data must be Findable, Accessible, Interoperable, and Reusable

C. Data may be reused without metadata or licensing

D. FAIR applies only to quantitative survey data

Correct Answer: B. Data must be Findable, Accessible, Interoperable, and Reusable

Question 4

A study tests whether a model remains effective when different reasonable imputation methods are used. This is an example of:

A. Sensitivity analysis

B. Preregistration

C. Direct replication

D. Citation analysis

Correct Answer: A. Sensitivity analysis

Question 5

What is the main purpose of a direct replication?

A. To test a different theory with unrelated measures

B. To repeat an earlier study as closely as possible

C. To conceal deviations from an earlier protocol

D. To maximize the number of publications

Correct Answer: B. To repeat an earlier study as closely as possible

Question 6

Which practice provides the strongest protection against accidentally using future observations in a predictive model?

A. Randomly shuffling all records before splitting

B. Using a chronological train-validation-test design

C. Deleting records with missing values after modeling

D. Reporting only the highest-performing model

Correct Answer: B. Using a chronological train-validation-test design

Question 7

A dataset cannot be openly released because it contains identifiable personal information. Which response best supports responsible openness?

A. Publish the identifiable dataset without consent

B. Destroy all documentation and provide no explanation

C. Provide metadata, a synthetic dataset, and controlled-access procedures

D. Publish only a claim that the data exist

Correct Answer: C. Provide metadata, a synthetic dataset, and controlled-access procedures

Question 8

Which item is most important for transparent reporting of a computational experiment?

A. Only the final accuracy value

B. The preferred result without alternative specifications

C. Data-processing procedures, software details, evaluation design, and limitations

D. The researcher’s personal opinion about the model

Correct Answer: C. Data-processing procedures, software details, evaluation design, and limitations

Question 9

Why should research metrics not replace expert judgment?

A. Metrics are always impossible to calculate

B. Metrics can be incomplete, discipline-dependent, and susceptible to gaming

C. Experts cannot interpret quantitative evidence

D. Research outputs should never be evaluated

Correct Answer: B. Metrics can be incomplete, discipline-dependent, and susceptible to gaming

Question 10

In the UrbanSense case, the corrected performance decreased after temporal leakage was removed. What does this most strongly demonstrate?

A. Transparency can reveal weaker but more credible findings

B. All machine-learning models are invalid

C. Replication is unnecessary when results are significant

D. Open data always eliminates bias

Correct Answer: A. Transparency can reveal weaker but more credible findings