10. Reproducible Research and Software Engineering

Introduction and Context

Data science is often presented through models, visualisations, and statistical results. Yet the value of an analysis depends not only on whether a model produces an impressive output, but also on whether another competent person can inspect, rerun, test, and extend the work. A result that cannot be reproduced is difficult to trust, audit, maintain, or deploy. Reproducible research therefore connects analytical validity with software engineering discipline.

A typical data-science project combines raw data, external libraries, notebooks, scripts, configuration files, statistical procedures, visualisations, and written conclusions. Each component can change over time. A library update may alter a model's behaviour; a missing preprocessing step may make a result impossible to recreate; an undocumented manual edit may introduce an error; and a notebook containing hidden state may produce different results when cells are executed in another order.

This chapter examines the practices used to make analytical work reliable and maintainable. It covers Git-based version control, dependency and environment management, notebooks and scripts, package development, workflow orchestration, clean code, logging, exception handling, testing, code review, continuous integration, and reproducible reporting. The emphasis is not on treating data science as ordinary software development, but on adapting software-engineering principles to the special characteristics of data, experimentation, uncertainty, and computation.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain reproducibility, repeatability, provenance, and traceability in data-science projects.

2. Use Git branches, commits, tags, and pull requests appropriately.

3. Manage dependencies and create portable, locked computational environments.

4. Distinguish productive uses of notebooks, scripts, modules, and packages.

5. Design data workflows using explicit dependencies and orchestration tools.

6. Apply clean-code principles, structured logging, and robust exception handling.

7. Develop unit, integration, and end-to-end tests for analytical software.

8. Evaluate code-review and continuous-integration practices.

9. Produce reports whose results and figures are generated automatically from controlled inputs.

10. Identify technical, ethical, governance, and organisational risks affecting reproducibility.

Reproducibility as a Property of an Analytical System

Reproducibility means that an analysis can be regenerated from identified inputs, code, configuration, and computational conditions. Repeatability usually refers to obtaining consistent results when the same team, tools, and conditions are used again. Reproducibility is often used more broadly to include independent execution by another person or organisation. The distinction is useful: a researcher may repeat an analysis successfully on a laptop while failing to provide sufficient information for an independent researcher to reproduce it.

A reproducible project records at least five forms of provenance:

• Data provenance: where data came from, when it was obtained, and how it was transformed.
• Code provenance: which source files and revisions produced an output.
• Environment provenance: operating-system, language, library, and system dependencies.
• Parameter provenance: configuration values, random seeds, thresholds, and model settings.
• Result provenance: which workflow steps generated a table, figure, model, or report.

Reproducibility does not mean that every execution must produce byte-for-byte identical output. Randomised algorithms, parallel computation, floating-point arithmetic, changing external data, and nondeterministic hardware can cause legitimate variation. The appropriate objective may be deterministic execution, bounded variation, or a documented explanation of unavoidable differences.

A useful mental model is:

Output = f(data version, code version, environment, configuration, random state)

If any important input to this function is unknown, the output is difficult to explain or recreate. Good engineering makes these inputs visible and controlled.

Version Control with Git

Git is a distributed version-control system that records changes to files. A repository contains the project history, while a commit records a coherent snapshot with an author, message, and identifier. Version control provides more than backup. It enables comparison, rollback, collaboration, experimentation, and accountability.

A practical Git workflow begins by creating a repository with a meaningful project structure. Files that should not be committed—such as passwords, local environments, temporary files, and large generated outputs—are listed in `.gitignore`. A typical commit should represent one logical change, such as adding validation to a data loader or correcting a feature calculation. Messages should explain the change rather than merely stating that files were modified.

Branches isolate work. A short-lived feature branch allows a developer to change code without destabilising the main branch. After review and automated checks, the branch can be merged. Tags identify significant versions, such as a release used for a published report. Commit history is most useful when it is understandable; frequent, coherent commits are preferable to a single large commit made after weeks of unstructured work.

Git does not automatically version large datasets, database contents, secrets, or software environments. Data-science teams may use data-versioning tools, object storage, or content hashes. A repository should contain metadata and procedures sufficient to locate or reconstruct restricted data, even when the data itself cannot legally or ethically be distributed.

Collaboration and Code Review

A pull request is both a technical and social control. The author explains the purpose, design, tests, data implications, and known limitations. Reviewers examine correctness, readability, security, computational cost, and compatibility with project conventions. Effective review is specific and evidence-based. It should distinguish defects that must be corrected from suggestions that are matters of style.

Review is especially important for analytical code because a syntactically correct program may still implement an invalid research design. Reviewers should ask whether preprocessing was fitted only on training data, whether target leakage is possible, whether a join changes row counts unexpectedly, and whether evaluation metrics support the stated conclusion.

Dependency Management and Environment Locking

A dependency is an external component required by a project. It may be a Python package, an R package, a system library, a database driver, or a command-line program. Dependency management records what is required and, ideally, the exact versions that were used.

A project commonly distinguishes direct dependencies from transitive dependencies. Direct dependencies are explicitly selected by the project, such as pandas or data.table. Transitive dependencies are installed because a direct dependency requires them. A version range may provide flexibility, but it can also allow an untested update to change behaviour. For important analyses, a lock file records resolved versions and often integrity hashes.

In Python, a project may use `pyproject.toml` to declare metadata and dependencies, together with a lock file generated by a packaging or environment tool. In R, a project may use `renv` to record package versions and a lock file. Conda or container specifications can additionally record non-Python system libraries. The exact tool is less important than the principle: installation should be automated and reproducible rather than dependent on undocumented commands typed by one analyst.

Virtual environments isolate project dependencies from the global installation. Containers extend this idea by packaging application code, libraries, and operating-system components into a portable image. Containers improve consistency but do not solve every problem. A container may still access changing external data, depend on a remote service, or produce different results because of hardware and randomisation. Images should therefore be versioned, scanned for vulnerabilities, and rebuilt from controlled specifications.

Secrets such as access tokens and database passwords must never be committed to Git. They should be supplied through a secure secret manager or protected environment variables. A reproducible project records the interface required by a secret without revealing its value.

Notebooks, Scripts, Modules, and Packages

Notebooks are effective for exploration, visual explanation, and communication. They allow code, output, mathematical notation, and narrative to appear together. This makes them valuable for investigating unfamiliar data and presenting a guided analysis. Their weakness is hidden state: cells can be executed out of order, outputs can become stale, and intermediate objects may exist only in memory.

Scripts are better suited to repeatable execution. A script can accept arguments, return an exit status, write defined outputs, and run in a clean process. Reusable functions should be placed in modules rather than copied among notebooks. A package provides a structured distribution unit with importable code, metadata, tests, documentation, and often a controlled release process.

A productive pattern is to explore in a notebook, transfer stable logic into tested modules, execute the analysis through scripts or workflow tasks, and generate a clean report from the resulting outputs. The notebook may remain as an explanatory artefact, but it should not be the only source of production logic.

A notebook intended for publication should be restarted and executed from top to bottom in a clean environment. Its data inputs, package versions, random seeds, and output paths should be explicit. Large outputs and sensitive data should be handled carefully because notebooks can unintentionally store confidential information.

Workflow Orchestration and Data Lineage

A workflow is a set of tasks connected by dependencies. For example, downloading data must precede validation; validation must precede feature creation; feature creation must precede model fitting; and model fitting must precede report generation. An orchestration tool represents these relationships as a directed acyclic graph (DAG). A task can run when its prerequisites are complete, and unchanged tasks may be skipped or cached.

Workflow orchestration improves reliability by making execution order explicit. It can also support retries, scheduling, parallel execution, monitoring, and provenance. Tools differ in syntax and scale, but the underlying design principles are similar. Tasks should be as deterministic as practical, have clear inputs and outputs, and avoid hidden dependence on the analyst's current working directory or interactive memory.

A workflow should separate raw, intermediate, and final data. Raw data should be treated as immutable; transformations should create new artefacts rather than silently overwriting the source. File names, metadata, checksums, and schemas help detect accidental changes. A data-quality task may verify column types, acceptable ranges, uniqueness, missingness, and expected row counts before downstream modelling begins.

Clean Code for Data Science

Clean code is code whose purpose and behaviour can be understood and safely changed. In data science, clarity is particularly important because analytical mistakes often occur in apparently simple transformations. Descriptive names are preferable to abbreviations: `customer_churn_rate` communicates more than `ccr`. Functions should have focused responsibilities, explicit inputs, and predictable outputs.

Repeated logic should be extracted into a function or module. Configuration values should not be scattered as unexplained constants. A configuration file or command-line interface can distinguish parameters from code. Comments should explain reasoning, assumptions, or domain constraints—not restate obvious syntax.

A clean data transformation makes its assumptions visible. For example, a function that imputes missing values should specify whether the statistic is calculated globally or within a training partition. A function that joins two tables should state the expected key relationship and validate whether it is one-to-one, one-to-many, or many-to-many.

Readable code must also be computationally responsible. Vectorised operations may be preferable to unnecessary row-by-row loops, but optimisation should not obscure correctness. Profiling should guide performance work. Premature optimisation can create complex code without meaningful benefit.

Logging and Exception Handling

Logging provides an operational record of what a program did. A useful log may include the task name, timestamp, input identifier, row counts, model version, duration, and outcome. Log levels such as DEBUG, INFO, WARNING, and ERROR allow different audiences to select appropriate detail. Logging is preferable to scattered print statements because it can be formatted, filtered, redirected, and collected centrally.

Exception handling should distinguish expected data-quality problems from unexpected programming defects. A program may catch a known exception when it can provide a useful response, such as reporting a missing input file. It should not silently catch every exception and continue with invalid data. Broad exception handling can conceal the original failure and produce plausible but incorrect results.

Errors should contain actionable information without exposing confidential data. A failed workflow should normally return a non-zero status so that orchestration and continuous integration systems detect the failure. Warnings should be used when execution can continue safely, and fatal errors when the result cannot be trusted.

Testing Analytical Software

Testing provides evidence that code behaves as intended. It does not prove that a research question is well chosen or that a model is causally valid, but it can detect many implementation errors.

Unit tests examine small components in isolation. A unit test for a date parser may check valid dates, malformed values, time zones, and missing values. A feature-engineering function can be tested with a small hand-calculated example. Tests should include boundary cases and invalid inputs, not only typical cases.

Integration tests examine interactions among components. Examples include loading a fixture file, applying a transformation, writing a result, and reading it back; or testing communication with a database using a controlled test instance.

End-to-end tests execute a representative workflow from input to report or model output. They are slower but reveal failures in configuration, file paths, orchestration, and component interaction.

Tests should be deterministic where possible. Randomised algorithms should use controlled seeds in tests, while production experiments may use separately recorded seeds. Floating-point comparisons should use tolerances rather than exact equality. Test data should be small, synthetic, legally usable, and designed to expose meaningful edge cases.

For machine-learning systems, tests may check schema, feature ranges, absence of target leakage, prediction shape, and performance against a documented baseline. A model test should not require an unrealistic claim that performance will never change; it should define acceptable conditions and investigate meaningful degradation.

Continuous Integration and Delivery

Continuous integration (CI) automatically builds and tests changes when code is pushed or a pull request is opened. A CI pipeline may format and lint code, install dependencies from a lock file, run unit and integration tests, scan for secrets, build documentation, and package an application. CI provides rapid feedback and prevents the main branch from accumulating unverified changes.

A useful pipeline is layered. Fast checks run first, followed by more expensive tests. The pipeline should use a clean environment so that success does not depend on uncommitted local files. Reproducibility requires recording the CI configuration, tool versions, and relevant environment variables.

Continuous delivery means that validated changes are kept ready for release; continuous deployment automatically releases them under defined conditions. In research settings, deployment may mean publishing a report, registering a model, or updating a data product rather than serving a web application. Approval controls remain necessary when outputs affect people or regulated decisions.

Reproducible Reporting

A reproducible report is generated from controlled data and code rather than assembled by manually copying values into a document. Literate-programming tools such as Quarto, R Markdown, and notebook-to-document systems can combine narrative, executable code, tables, and figures. The report should state the data period, inclusion criteria, software environment, model settings, and limitations.

Automated reporting reduces transcription errors, but automation does not guarantee valid interpretation. Analysts must still inspect whether a chart is appropriate, whether a denominator changed, whether missing data bias the result, and whether uncertainty is communicated. A report should distinguish observed facts, model-based estimates, assumptions, and recommendations.

A strong publication or delivery process stores the report's source, code revision, data version, environment identifier, and generation date. Where data cannot be shared, a synthetic example, schema, documentation, or secure reproducibility package may be provided instead.

Illustrative Case Study: Rebuilding a Customer-Retention Analysis

Background

A fictional subscription company, Northstar Services, asks an analytics team to estimate monthly customer churn and identify factors associated with cancellation. An analyst previously produced a notebook showing a promising predictive model, but the notebook cannot be rerun because the source database has changed, package versions are unknown, and several cells depend on hidden variables.

Situation and Problem

Management receives different churn estimates from two analysts. One report uses a manually exported spreadsheet and excludes customers with missing contract dates; another uses a database query and includes them. Neither report records the precise query, extraction date, random seed, or model configuration. The organisation needs a dependable monthly process and an auditable report.

Analysis

The team first defines the analytical contract: the unit of analysis is an active customer at the beginning of a month; churn means cancellation within the following 30 days; and the observation period is explicitly recorded. They identify data sources, access permissions, expected keys, and data-quality rules.

The project is placed in Git. The old notebook is retained as historical evidence but its transformation logic is moved into tested Python modules. A locked environment records package versions. Credentials are supplied through the deployment platform rather than the repository. A workflow contains tasks for extraction, validation, feature construction, model fitting, metric calculation, and report generation.

Unit tests verify date windows, cancellation definitions, and missing-value treatment. Integration tests verify that the extraction output conforms to the expected schema and that feature generation preserves one row per customer-month. A data-quality task fails when duplicate customer identifiers or impossible dates appear. The model is evaluated using a time-based split rather than a random split, because future information must not enter the training data.

A pull request is reviewed by a data engineer and a domain specialist. The engineer examines pipeline reliability; the domain specialist checks whether the churn definition matches business practice. CI runs tests and lints the code. The final report includes the code revision, data-extraction date, environment lock, model metrics, confidence intervals where appropriate, and a statement that association does not establish causation.

Outcome

The company obtains a scheduled workflow that produces the same defined analysis whenever the same versioned inputs are used. When a source table later adds a new status value, schema validation stops the workflow and creates an actionable alert instead of publishing a misleading report. The process is slower to design than the original notebook but substantially easier to audit and maintain.

Learning Points

The case demonstrates that reproducibility is organisational as well as technical. Definitions, data access, code, environments, testing, review, and reporting must align. Version control alone would not have resolved the inconsistent churn definition, and a locked environment alone would not have prevented target leakage. Reliability emerges from the complete workflow.

Challenges, Risks, and Limitations

Reproducibility has costs. Tests, documentation, environment management, and review require time and expertise. Highly exploratory research may not justify production-level engineering for every temporary experiment. The appropriate level of control should reflect the consequences of error, expected project lifetime, collaboration needs, and regulatory exposure.

External data can change or disappear. APIs may alter response formats, websites may block access, and proprietary databases may not be shareable. Time-dependent analyses may be irreproducible if historical snapshots are unavailable. Computationally intensive models may be difficult to rerun without specialised hardware. These limitations should be documented rather than concealed.

Common mistakes include committing secrets, relying on absolute local paths, using `latest` container tags, overwriting raw data, storing only final figures, testing only the happy path, and treating a notebook's visible output as evidence that its code was executed in order. Another serious mistake is confusing reproducible computation with valid science. A perfectly repeatable analysis can still use biased data, an inappropriate metric, or an unsupported interpretation.

Ethical, Governance, and Professional Considerations

Reproducibility must be balanced with privacy, security, intellectual property, and research ethics. Personal data should be minimised, access-controlled, encrypted where appropriate, and retained only as long as justified. Public release of code does not authorise release of confidential data. De-identification may reduce risk but does not guarantee anonymity, especially when datasets can be linked.

A reproducible project should document data ownership, consent restrictions, permitted uses, retention requirements, and responsible contacts. Audit trails should support accountability without exposing unnecessary personal information. For models used in employment, credit, health, education, or public services, versioning and testing should be accompanied by fairness assessment, explainability appropriate to the context, human oversight, and procedures for contesting harmful decisions.

Professional responsibility includes admitting uncertainty and recording failed experiments when they affect interpretation. Selectively publishing only successful runs can create a misleading record. Reproducibility practices support scientific integrity, but they do not replace independent ethical review or institutional governance.

Current Developments and Emerging Practice

Modern data teams increasingly use infrastructure-as-code, container registries, managed workflow platforms, data contracts, and model registries. Data contracts specify expectations between data producers and consumers, including schemas, freshness, and quality rules. Data and model lineage systems connect sources to downstream reports and predictions.

Reproducible machine learning is developing toward automated experiment tracking, dataset versioning, feature stores, model cards, and monitoring for drift. Large language models introduce additional concerns: prompts, retrieved documents, model versions, sampling parameters, and safety filters can all affect outputs and should be recorded for important applications. Automated code-generation tools may accelerate development, but generated code requires review, testing, licence awareness, and security assessment.

The central trend is a move from reproducible individual analyses toward reproducible analytical platforms. This shift makes provenance, access control, observability, and governance part of the normal data-science architecture.

Best-Practice Workflow

A disciplined project can follow this sequence:

1. Define the research or business question, data contract, success criteria, and risks.

2. Create a Git repository with a clear structure and secure ignore rules.

3. Record data sources, versions, schemas, permissions, and acquisition dates.

4. Declare dependencies and create a locked, documented environment.

5. Explore in notebooks, but move stable logic into modules and tested scripts.

6. Design a workflow with explicit inputs, outputs, validation, and provenance.

7. Add unit tests first, then integration and representative end-to-end tests.

8. Use structured logging and fail loudly when outputs cannot be trusted.

9. Submit changes for review and run CI in a clean environment.

10. Generate reports automatically and archive the code, configuration, environment, and relevant data identifiers.

11. Monitor scheduled jobs, data quality, model performance, and changes in assumptions.

These practices should be scaled to the project. A classroom exercise may require only a small repository and a few tests, whereas a regulated production model requires formal approvals, immutable artefacts, security controls, and documented operational ownership.

Chapter Summary

Reproducible research treats an analytical result as the output of a controlled system involving data, code, environment, configuration, and random state. Git records code history and supports collaboration, while dependency management and environment locking reduce variation caused by software changes. Notebooks are valuable for exploration and communication, but scripts, modules, and packages provide stronger foundations for repeatable execution.

Workflow orchestration makes task dependencies, data lineage, validation, and scheduling explicit. Clean code, logging, and principled exception handling improve maintainability and operational visibility. Unit, integration, and end-to-end tests provide different forms of evidence about correctness. Code review and continuous integration create quality controls before changes reach users. Reproducible reports connect narrative, computation, and evidence, while ethical governance ensures that transparency does not violate privacy or misuse data. Together, these practices transform isolated analyses into dependable professional data products.

Key Terms

Reproducibility: The ability to regenerate an analysis from identified inputs, code, environment, and configuration.

Repeatability: The ability to obtain consistent results under substantially the same conditions.

Data provenance: A record of a dataset’s origin, transformations, and history.

Version control: A system for recording and managing changes to files over time.

Commit: A recorded snapshot of changes in a version-control repository.

Branch: An independent line of development within a Git repository.

Pull request: A proposed change submitted for review before integration.

Dependency: An external library, tool, or system component required by a project.

Lock file: A file recording resolved dependency versions and sometimes integrity hashes.

Virtual environment: An isolated environment containing project-specific software dependencies.

Container: A portable execution unit containing software and its required runtime components.

Notebook: An interactive document combining executable code, outputs, and narrative text.

Workflow orchestration: The coordination of dependent computational tasks and their execution.

Directed acyclic graph: A graph of tasks and dependencies containing no circular dependency.

Unit test: A test of a small software component in isolation.

Integration test: A test of interactions among multiple components.

Continuous integration: Automated validation of code changes in a clean environment.

Logging: The structured recording of program events and operational information.

Data contract: An explicit agreement about the schema, quality, and behaviour of exchanged data.

Target leakage: The unintended use of information unavailable at prediction time.

Review Questions

Short Answer Questions

1. Distinguish reproducibility from repeatability in data science.

2. Why should raw data generally be treated as immutable?

3. State two limitations of notebooks for production analytical workflows.

4. What is the purpose of a lock file?

5. Differentiate a unit test from an integration test.

Long Answer Questions

1. Explain how Git, dependency management, and environment locking work together to support reproducible research.

2. Discuss the appropriate roles of notebooks, scripts, modules, and packages in a data-science project.

3. Describe how a workflow orchestration system can improve data lineage and operational reliability.

4. Explain the importance of logging, exception handling, and testing in analytical software.

5. Discuss how continuous integration and code review contribute to trustworthy data products.

Analytical / Application Questions

1. A notebook produces different results each time it is opened because cells were executed in varying orders. Diagnose the problem and propose an engineering solution.

2. A model’s performance suddenly improves after a new feature is added. Design checks to determine whether the improvement results from legitimate information or target leakage.

3. Design a reproducible workflow for a monthly sales report, identifying its inputs, validation tasks, tests, versioned artefacts, and reporting process.

22. Multiple Choice Questions with Answers

Question 1

Which combination most completely specifies the conditions needed to reproduce an analytical output?

A. The analyst’s name and the final chart

B. Data version, code version, environment, configuration, and random state

C. The programming language and report title

D. The number of lines of code and notebook length

Correct Answer: B. Data version, code version, environment, configuration, and random state

Question 2

What is the main reason for moving stable notebook logic into tested modules or scripts?

A. To eliminate the need for documentation

B. To prevent all exploratory analysis

C. To reduce hidden state and support repeatable execution

D. To ensure that every model is deterministic

Correct Answer: C. To reduce hidden state and support repeatable execution

Question 3

A project’s direct dependency is upgraded, causing a transitive library to change as well. Which practice best reduces this risk in a published analysis?

A. Use a lock file recording resolved dependency versions

B. Install packages globally without version specifications

C. Delete the project environment before every run

D. Store only the package names in the report

Correct Answer: A. Use a lock file recording resolved dependency versions

Question 4

Which workflow design best protects against target leakage?

A. Fit preprocessing transformations using the entire dataset before splitting

B. Use future observations to fill missing historical values

C. Calculate features only from information available at prediction time

D. Randomly shuffle time-dependent observations before every evaluation

Correct Answer: C. Calculate features only from information available at prediction time

Question 5

What is the most appropriate behaviour when a required input file is missing?

A. Silently create an empty replacement file

B. Catch every error and continue with the previous output

C. Log an actionable error and fail the workflow

D. Ignore the problem if the final report can still be opened

Correct Answer: C. Log an actionable error and fail the workflow

Question 6

Which statement best describes continuous integration?

A. Manual publication of a report after an analyst approves it

B. Automated building and testing of changes in a clean environment

C. Replacing all human code review with an automated model

D. Running a notebook once on the developer’s laptop

Correct Answer: B. Automated building and testing of changes in a clean environment

Question 7

In the Northstar case, why was a time-based train-test split appropriate?

A. It guarantees a higher accuracy score

B. It avoids allowing future information to influence past predictions

C. It removes the need for model validation

D. It makes all customer records independent

Correct Answer: B. It avoids allowing future information to influence past predictions

Question 8

Which practice creates the greatest security risk?

A. Recording a database schema in project documentation

B. Using environment variables for deployment secrets

C. Committing an access token to a public Git repository

D. Scanning dependencies for known vulnerabilities

Correct Answer: C. Committing an access token to a public Git repository

Question 9

What is the principal value of a data contract?

A. It guarantees that a model is causally valid

B. It specifies expectations about exchanged data, such as schema and quality

C. It replaces the need for version control

D. It prevents all changes to an organisation’s database

Correct Answer: B. It specifies expectations about exchanged data, such as schema and quality

Question 10

A test compares two floating-point model metrics using exact equality and fails intermittently. What is the best correction?

A. Remove the test permanently

B. Compare values using an appropriate tolerance

C. Round all production data to integers

D. Catch the test exception and continue

Correct Answer: B. Compare values using an appropriate tolerance