10. Reproducibility, Open Science, and Research Software Engineering

Introduction and context

Contemporary information-technology research increasingly depends on software, computational infrastructure, data pipelines, simulation, and automated analysis. A published claim may therefore be supported not only by a manuscript and dataset, but also by source code, configuration files, dependency specifications, execution logs, notebooks, workflow definitions, and records of analytical decisions. When these elements are unavailable, undocumented, or impossible to execute, readers may be unable to determine how a result was produced or whether it can be independently verified.

Reproducibility is consequently more than a technical convenience. It is an epistemic and ethical requirement. A reproducible study makes its evidential pathway sufficiently transparent for qualified others to inspect, rerun, challenge, and extend it. Open science broadens this principle by encouraging responsible sharing of methods, data, code, materials, and outputs. Research software engineering provides the professional practices needed to build dependable computational artefacts rather than treating research code as disposable support material.

The objective is not to demand that every study release every file. Privacy, security, intellectual property, contractual restrictions, and potential misuse may justify controlled access. The standard is responsible transparency: researchers should disclose what can be shared, explain what cannot, and provide enough documentation for an informed assessment of the work.

Learning objectives

By the end of this chapter, learners should be able to:

• distinguish reproducibility, replicability, repeatability, and generalizability;
• design an auditable research repository using version control and persistent identifiers;
• explain the roles of testing, workflow automation, containers, and dependency management;
• compare preregistration and registered reports with conventional publication practices;
• evaluate ethical, legal, security, and governance constraints on open research;
• develop a practical reproducibility plan for a computational research project; and
• critically assess the limitations of claims that a computational result is “reproducible.”

The evidential meaning of reproducibility

Terminology varies across disciplines, so definitions should be stated explicitly. In this chapter, repeatability refers to obtaining consistent results when the same researcher uses the same procedures and environment. Reproducibility commonly refers to obtaining consistent results when another researcher uses the same data, code, and documented procedures. Replicability usually concerns whether an independent study addressing the same question, often with new data or an independently implemented method, reaches compatible conclusions. Generalizability concerns whether findings transfer to other populations, settings, datasets, or conditions. These concepts overlap but are not interchangeable.

A computational result has several layers. The numerical output may be reproducible while the scientific interpretation is not; a script may run successfully while the data-processing decisions remain opaque; and a repository may be publicly accessible while its dependencies are obsolete. Reproducibility therefore has technical, procedural, interpretive, and social dimensions. It includes the ability to identify the relevant artefact, understand its provenance, execute it, inspect its assumptions, and judge whether the resulting conclusion is warranted.

A useful model is an audit chain:

1. Question and design: What was planned, and why is the design appropriate?

2. Inputs: What data, instruments, files, or external services were used?

3. Transformation: Which cleaning, feature-engineering, exclusion, and analytical operations occurred?

4. Execution: Which code, software versions, hardware, parameters, and random seeds were involved?

5. Outputs: What tables, models, visualizations, logs, and reports were generated?

6. Interpretation: How were results translated into claims, and what uncertainties remain?

An auditable project connects each link. A final figure without provenance is a weak evidential object; a figure linked to a versioned script, input manifest, test results, and documented interpretation is considerably stronger.

Open science as a research governance model

Open science is an approach to research that promotes transparency, accessibility, collaboration, and reuse throughout the research lifecycle. It includes open access to publications, open or appropriately governed data, open-source software, open methods, open peer review in some settings, and sharing of educational or infrastructural resources. Openness is not identical to unrestricted disclosure. A responsible model distinguishes between open, controlled, restricted, and non-shareable materials.

The principle of “as open as possible, as closed as necessary” requires justification. For example, personally identifiable health records should not be placed in a public repository merely to demonstrate openness. Instead, researchers may publish a data dictionary, synthetic example data, de-identification methods, access criteria, and code that operates on a compatible protected dataset. Similarly, vulnerability research may require coordinated disclosure rather than immediate publication of exploit details.

Open methods are particularly important in software-dependent research. A methods section should identify data acquisition, preprocessing, analytical choices, software libraries, parameter values, hardware requirements, and known failure conditions. Statements such as “standard preprocessing was applied” are inadequate when preprocessing can materially affect the result.

Preregistration and registered reports

Preregistration records research questions, hypotheses, outcomes, analysis plans, sampling rules, and decision criteria before analysis begins. It separates confirmatory work, in which tests are planned in advance, from exploratory work, in which patterns are investigated and hypotheses may be generated after examining data. Exploratory research is legitimate; the ethical problem arises when exploratory decisions are presented as if they had been specified beforehand.

A registered report extends preregistration into the publication process. The study design and analysis plan are peer reviewed before data collection or analysis, and an in-principle acceptance may be granted before results are known. This reduces incentives to select methods or conclusions based on the eventual result. It does not eliminate poor design, implementation errors, or questionable interpretation, but it makes the decision process more visible.

Preregistration must be sufficiently precise to be useful without pretending that all contingencies can be predicted. Deviations are acceptable when documented, justified, and clearly labelled. A strong report distinguishes planned analyses, deviations from the plan, sensitivity analyses, and post hoc explorations.

Research software engineering: from scripts to engineered artefacts

Research software engineering applies software-engineering discipline to scientific and scholarly computation. Its concern is not to turn every researcher into a commercial software developer, but to ensure that code supporting a claim is understandable, testable, maintainable, and appropriately controlled.

Version control and provenance

A version-control system records changes to code and related files. Each commit should represent a coherent change and include a meaningful message. Branches can isolate experimental work, while tags or releases identify the exact state associated with a manuscript, dataset, or published result. A repository should contain a clear README, licence, citation information, contribution guidance where relevant, and a change history.

Version control does not automatically provide reproducibility. A repository containing only the latest script may omit earlier analytical decisions, generated configuration, data provenance, or system dependencies. Large or sensitive files may require dedicated storage and access controls. The relationship between repository versions and persistent archival releases should be explicit.

Documentation and literate research

Documentation operates at several levels. Conceptual documentation explains the research question and design. Technical documentation describes installation, inputs, commands, configuration, expected outputs, and troubleshooting. Code documentation explains interfaces, assumptions, units, and exceptional cases. Data documentation includes provenance, schema, missing-value conventions, transformations, licences, and ethical restrictions.

Literate programming and computational notebooks combine narrative, code, output, and interpretation. They can make reasoning visible, but notebooks also create risks: hidden state, execution out of order, embedded credentials, excessive manual editing, and dependence on an interactive session. A robust project converts critical notebook logic into tested modules or scripts and records the execution order and environment.

Testing and quality assurance

Testing provides evidence that software behaves as intended. Unit tests examine small functions; integration tests assess interactions among components; system or workflow tests evaluate an end-to-end pipeline; and regression tests detect unintended changes in previously working behaviour. Property-based tests examine general properties across many inputs, while validation tests compare outputs with known benchmarks or independently calculated results.

Testing does not prove that a scientific model is true. It can show that an implementation conforms to its specification, reproduces a known result, rejects invalid input, and remains stable after modification. Scientific validation must additionally consider construct validity, measurement error, statistical assumptions, data leakage, bias, and sensitivity to analytical choices.

Automation and workflow management

Manual execution is a major source of irreproducibility. Workflow systems represent analyses as directed graphs of inputs, processes, and outputs. They can rerun only tasks affected by changed inputs, record dependencies, parallelize computation, and preserve provenance. Continuous-integration services can automatically install dependencies, run tests, check formatting, build documentation, and verify that a minimal example executes.

Automation should be transparent. A workflow that silently downloads changing data, relies on an external API, or uses the current date without recording it may be automated yet non-auditable. Effective workflows pin important inputs, record checksums, expose parameters, and fail loudly when assumptions are violated.

Environments, containers, and dependency management

A computational environment consists of operating-system components, programming languages, libraries, system packages, hardware, environment variables, and external services. Dependency specifications such as lockfiles or environment manifests record required versions. Containers package an application and many of its dependencies in an isolated image, improving portability between a researcher’s workstation, a server, and a cloud platform.

Containers are not complete time capsules. They depend on a host kernel, hardware, registries, credentials, external data, and sometimes nondeterministic parallel operations. Images may contain security vulnerabilities or confidential material. They should be built from trusted sources, scanned, pinned where appropriate, documented, and archived with a digest or other immutable identifier. Virtual machines, package environments, workflow systems, and hardware-specific documentation may be preferable in some contexts.

Randomness must also be managed. Seeds should be recorded, but a seed alone does not guarantee identical results across libraries, processors, or parallel execution models. Researchers should document random-number generators, deterministic settings where feasible, numerical tolerances, and the acceptable range of variation.

Persistent identifiers and research objects

A persistent identifier provides a stable reference to an object despite changes in location. Digital object identifiers can identify articles, datasets, software releases, and archived materials; researcher identifiers distinguish authors with similar names; repository URLs and version tags connect working development to citable releases. A software citation should identify the software name, authors, version, repository or archive, and persistent identifier when available.

The scholarly object is increasingly a network rather than a single paper. A publication may link to a preregistration, protocol, dataset, code repository, container image, workflow definition, computational notebook, and software release. Metadata should describe relationships among these objects. Citation of software is also a matter of credit: researchers whose libraries or infrastructure enable a result deserve recognition, while reuse must comply with licences and attribution requirements.

A practical reproducibility architecture

A mature project can be organized into four layers. The research layer contains questions, hypotheses, protocols, and preregistration. The data layer contains raw-data manifests, provenance records, schemas, transformations, and access rules. The software layer contains source code, tests, workflows, configuration, and dependency specifications. The evidence layer contains generated outputs, logs, reports, figures, and links to the exact versions that produced them.

A practical lifecycle is as follows:

• Define the research question, estimand, primary outcomes, and governance constraints.
• Create a project structure and repository before substantial analysis begins.
• Record provenance and access conditions for every important input.
• Separate raw, intermediate, and derived data; never overwrite irreplaceable raw material.
• Implement analysis as parameterized, version-controlled code rather than undocumented manual operations.
• Add tests for transformations, edge cases, expected dimensions, and key scientific calculations.
• Specify the environment and automate execution through a workflow.
• Run the workflow in a clean environment before release.
• Archive a release with documentation, metadata, licence information, and a persistent identifier.
• Publish a reproducibility statement that reports what was shared, what was restricted, known limitations, and how the results can be evaluated.

This architecture supports both open and restricted research. Where data cannot be released, the computational pathway can often still be released with synthetic or de-identified examples, controlled-access instructions, and a clear explanation of the difference between code reproducibility and result reproducibility.

Illustrative case study: an auditable machine-learning study

Background

The following fictional case illustrates common decisions in doctoral research. A university research group develops a machine-learning model to identify deterioration risk in patients admitted to a hospital. The project uses electronic health records from two institutions and aims to support clinical research, not automated clinical decision-making.

Situation and problem

An initial manuscript reports high predictive performance, but reviewers cannot reproduce the main table. The code is a collection of notebooks; preprocessing differs between training and evaluation; library versions are missing; some records were excluded manually; and the data-use agreement prohibits public release of patient-level data. The team must determine whether the result is trustworthy and how to make the study auditable without violating confidentiality.

Analysis

The group maps the audit chain. It discovers that an analyst changed an imputation rule after inspecting preliminary results, but the change was not recorded. A random split allowed records from the same patient to appear in both training and test sets, inflating performance. The model also depends on a feature recorded only after clinical deterioration, creating a form of target leakage. These problems are not solved by publishing the original notebooks.

Approach

The team preregisters a revised analysis plan for a new evaluation phase and labels the earlier work as exploratory. It creates a version-controlled repository containing modular preprocessing code, unit tests for date handling and patient-level splitting, integration tests for the feature pipeline, and a workflow that generates tables and figures from fixed configuration files. Data remain in a secure environment, while the repository includes a schema, synthetic records, a data dictionary, access requirements, and instructions for authorized researchers.

Containerized execution is used inside the secure environment, and the image digest, package manifest, hardware assumptions, and random-number settings are recorded. A continuous-integration process runs tests on synthetic data. The final manuscript links to a tagged software release and archived metadata. It reports the leakage discovered in the first analysis, the deviation from the original plan, sensitivity analyses, and the limits of transporting the model to other hospitals.

Outcome

The revised performance is lower but more credible. Independent authorized researchers can execute the workflow on a compatible dataset and inspect each transformation. The public cannot access patient records, but can evaluate the code, documentation, synthetic example, tests, and governance process. The group’s credibility improves because the correction is treated as part of responsible scholarship rather than concealed as an inconvenient result.

Learning points

Reproducibility begins with design and governance, not with uploading code at publication. Testing cannot correct a flawed estimand or leakage, but it can expose implementation problems. Controlled access can be ethically superior to public release. Finally, negative or corrective findings should be preserved in the project history because they explain how the final claim was reached.

Risks, limitations, and common failure modes

A frequent mistake is confusing availability with usability. A public repository may lack installation instructions, data access, a licence, or a working entry point. Another is relying on a single notebook whose hidden execution state cannot be reconstructed. Researchers also commonly fail to distinguish raw data from derived data, overwrite intermediate files, omit random seeds, use unpinned dependencies, or download mutable external resources without recording versions.

Reproducibility can be expensive. Legacy code may require obsolete software; large datasets may be costly to transfer; environmental differences may produce small numerical variations; and long-term maintenance requires resources. The pursuit of exact bitwise identity can also be misplaced when the scientific question concerns robust conclusions rather than identical floating-point output. Conversely, “minor” variation should not be dismissed if it changes a substantive conclusion.

Open research creates security and privacy risks. Repositories may accidentally expose passwords, access tokens, personal information, proprietary data, or exploitable vulnerabilities. Automated dependency installation can introduce malicious or compromised packages. Release procedures should include secret scanning, dependency review, least-privilege access, removal of sensitive history where necessary, and organizational incident-response procedures.

Reproducibility badges, checklists, and automated metrics are useful signals but imperfect measures. They may reward the presence of files rather than the quality of the reasoning. A project can be technically executable and scientifically invalid. Evaluation must therefore combine computational checks with methodological and interpretive review.

Ethical, professional, and governance considerations

Researchers have duties to participants, collaborators, institutions, funders, users, and future investigators. Consent materials should anticipate appropriate data sharing where possible. Data protection principles such as purpose limitation, minimization, security, and controlled access should guide the research object lifecycle. Anonymization should not be assumed merely because direct identifiers have been removed; re-identification risk depends on context and linkage possibilities.

Authorship and credit should reflect intellectual and technical contributions. Research software engineers, data stewards, maintainers, and infrastructure contributors should receive appropriate acknowledgement or authorship under applicable standards. Licences communicate permissions and obligations. A permissive licence, a copyleft licence, and a restricted data-use agreement create different conditions for reuse; researchers should not redistribute material in ways inconsistent with its licence.

Supervisors and institutions should treat reproducibility as research integrity infrastructure. Training, code review, secure storage, archival services, and time for documentation are organizational responsibilities, not merely individual preferences. A reproducibility plan should be proportionate to risk and significance: a small exploratory script and a safety-critical clinical system require different controls.

Emerging directions

Research practice is moving toward executable publications, machine-readable metadata, provenance standards, software citation, open peer review, and automated reproducibility checks. Cloud platforms can provide scalable execution, while workflow registries and archival services improve discoverability. Artificial-intelligence tools are increasingly used to generate code, documentation, tests, and data transformations. Their use introduces additional requirements: generated code must be reviewed and tested; prompts and model versions may need recording; training-data or licence constraints must be considered; and researchers remain responsible for errors, bias, insecure code, and fabricated documentation.

Long-term preservation is becoming more important as software ecosystems change. Future reproducibility may depend less on freezing one environment forever and more on preserving multiple forms of evidence: source code, dependency metadata, container images, workflow specifications, test fixtures, representative data, execution logs, and a clear account of known limitations. The goal is resilient understanding rather than a false promise of permanent execution.

Best-practice checklist

A doctoral researcher should be able to answer the following questions before submission:

• Can another qualified researcher identify the exact code and data versions behind each principal result?
• Are planned and exploratory analyses clearly distinguished?
• Can the workflow run from a clean environment using documented commands?
• Are dependencies, random settings, external resources, and hardware assumptions recorded?
• Do tests cover critical transformations and scientific calculations?
• Are sensitive materials protected, and are restrictions transparently explained?
• Does the release have a licence, metadata, persistent identifier, and software citation?
• Have independent colleagues attempted execution and reviewed the interpretation?

Chapter Summary

Reproducibility is an evidential property supported by transparent design, provenance, executable methods, and critical interpretation. Open science extends transparency across publications, data, code, methods, and scholarly infrastructure while recognizing legitimate privacy, security, legal, and intellectual-property constraints. Research software engineering supplies version control, documentation, testing, workflow automation, dependency management, containers, and release practices that transform fragile scripts into auditable research artefacts. Preregistration and registered reports clarify the difference between confirmatory and exploratory reasoning. Persistent identifiers connect papers with code, data, protocols, and environments. Yet technical execution alone cannot guarantee scientific validity; sound constructs, appropriate designs, careful governance, and honest reporting remain essential.

Key Terms

Reproducibility: The ability of others to obtain consistent results using the same data, code, and documented procedures.

Repeatability: Consistency of results when the same researcher repeats an analysis under the same conditions.

Replicability: Consistency of conclusions when an independent study uses new data or an independently implemented method.

Generalizability: The extent to which findings apply beyond the original data, population, or setting.

Open science: Research practice emphasizing responsible transparency, accessibility, sharing, and reuse.

Preregistration: A time-stamped public record of research questions and analysis plans made before the relevant analysis.

Registered report: A publication format in which the study plan is peer reviewed before results are known.

Provenance: A record of the origin, history, transformations, and custody of data or research outputs.

Version control: A system for recording and managing changes to files and code over time.

Persistent identifier: A durable reference that identifies a scholarly object across changes in location.

Research software engineering: Application of software-engineering practices to research code and computational infrastructure.

Unit test: A test that examines the behaviour of a small software component.

Workflow: An explicit sequence or graph of computational tasks and dependencies.

Container: An isolated, portable package containing software and many of its dependencies.

Dependency: An external library, tool, service, or system component required by software.

Data leakage: Inappropriate transfer of information from evaluation data or future events into model development.

Computational notebook: An interactive document combining executable code, narrative, and outputs.

Audit chain: The connected evidential pathway from research question and inputs to computation, outputs, and interpretation.

Controlled access: A governance arrangement in which approved users may access restricted research materials under specified conditions.

Review Questions

Short Answer Questions

1. Distinguish repeatability, reproducibility, and replicability.

2. Why does open science not require unrestricted release of every dataset?

3. What information should a dependency manifest provide?

4. How do unit tests differ from scientific validation?

5. What is the purpose of a persistent identifier for research software?

Long Answer Questions

1. Explain how version control, documentation, testing, and workflow automation jointly create an audit chain.

2. Compare preregistration with registered reports, including their benefits and limitations.

3. Discuss the role of containers in reproducible computational research and explain why they are not complete time capsules.

4. Analyze the ethical and governance issues involved in sharing sensitive research data.

5. Develop a reproducibility architecture for a doctoral machine-learning project.

Analytical / Application Questions

1. A published analysis runs only on the author’s laptop and uses an undocumented external API. Diagnose the reproducibility failures and propose a remediation plan.

2. A study cannot release participant-level data but wishes to support independent evaluation. Design an open and controlled-access strategy.

3. A machine-learning result changes substantially when a library version is updated. Explain how you would investigate the change and decide what should be reported.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes reproducibility from replicability?

A. Reproducibility uses new data, whereas replicability uses the original data.

B. Reproducibility commonly uses the same data and code, whereas replicability generally involves an independent study or implementation.

C. Reproducibility concerns ethics only, whereas replicability concerns software only.

D. They are always identical terms in every research discipline.

Correct Answer: B. Reproducibility commonly uses the same data and code, whereas replicability generally involves an independent study or implementation.

Question 2

A researcher cannot release patient-level data because of a data-use agreement. Which response best reflects responsible open science?

A. Release the data without identifiers because all de-identification is automatically safe.

B. Release nothing, including methods and code, to avoid any risk.

C. Share code, metadata, synthetic examples, access procedures, and a clear explanation of restrictions.

D. Replace the data with arbitrary numbers and claim full result reproducibility.

Correct Answer: C. Share code, metadata, synthetic examples, access procedures, and a clear explanation of restrictions.

Question 3

What is the primary purpose of a registered report?

A. To guarantee that the study will produce statistically significant findings.

B. To obtain peer review of the study plan before results are known.

C. To eliminate the need for data governance.

D. To prevent all exploratory research.

Correct Answer: B. To obtain peer review of the study plan before results are known.

Question 4

Which practice most directly reduces hidden-state problems in computational notebooks?

A. Adding more narrative text without rerunning code.

B. Executing cells in an undocumented order.

C. Moving critical logic into tested modules or scripts and recording execution procedures.

D. Deleting intermediate outputs before publication.

Correct Answer: C. Moving critical logic into tested modules or scripts and recording execution procedures.

Question 5

A container improves portability, but why does it not guarantee complete reproducibility?

A. Containers cannot contain any dependencies.

B. Containers may still depend on hosts, hardware, external data, credentials, and nondeterministic operations.

C. Containers automatically change source code during execution.

D. Containers prevent version control.

Correct Answer: B. Containers may still depend on hosts, hardware, external data, credentials, and nondeterministic operations.

Question 6

Which example is a regression test?

A. Checking whether a previously working calculation still produces its expected result after code changes.

B. Writing a research question after viewing results.

C. Assigning a persistent identifier to a paper.

D. Removing personal identifiers from a dataset.

Correct Answer: A. Checking whether a previously working calculation still produces its expected result after code changes.

Question 7

A model shows unusually high test performance because records from the same patient occur in both training and test sets. What problem is most directly indicated?

A. Software licensing conflict.

B. Data leakage caused by an inappropriate split.

C. Failure of persistent identification.

D. Excessive container isolation.

Correct Answer: B. Data leakage caused by an inappropriate split.

Question 8

Which item most strengthens the connection between a published figure and the computation that produced it?

A. A screenshot of the figure without source information.

B. A tagged software release, workflow, input manifest, and execution documentation.

C. A statement that standard analysis was used.

D. An unversioned notebook stored on a personal computer.

Correct Answer: B. A tagged software release, workflow, input manifest, and execution documentation.

Question 9

Why should exploratory analyses be labelled separately from preregistered analyses?

A. Exploratory analyses are always invalid.

B. The distinction clarifies which decisions were planned and which arose after examining data.

C. Preregistration makes exploratory work illegal.

D. Exploratory analyses never require documentation.

Correct Answer: B. The distinction clarifies which decisions were planned and which arose after examining data.

Question 10

Which claim about automated reproducibility checks is most defensible?

A. Successful execution proves that the scientific conclusion is true.

B. Automated checks can reveal technical failures but cannot replace methodological and interpretive evaluation.

C. Automated checks make ethical review unnecessary.

D. A passing test eliminates all dependency risks.

Correct Answer: B. Automated checks can reveal technical failures but cannot replace methodological and interpretive evaluation.