Git for Data Science and Reproducibility

Introduction

Data science depends on more than a final model or a polished chart. A trustworthy result requires the ability to identify the data, code, configuration, environment, assumptions, and decisions that produced it. Version control provides the foundation for this traceability. Git records changes to text-based project files, supports collaboration, and enables a team to restore or compare earlier project states.

Data-science projects create special challenges. Notebooks combine executable code, explanatory text, outputs, and metadata. Datasets may be very large, confidential, or subject to licensing restrictions. Machine-learning models and generated reports can also be too large or unsuitable for ordinary Git repositories. A professional workflow therefore combines Git with appropriate practices for notebooks, secrets, data, models, and computational environments.

Reproducibility does not mean that every execution produces byte-for-byte identical output in every environment. It means that another authorised person can understand the process, obtain permitted inputs, recreate the computational environment sufficiently, and reproduce or explain the reported result.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain how Git supports reproducible data-science work.
2. Design a repository structure for notebooks, scripts, configuration, documentation, and tests.
3. Manage notebook diffs, outputs, metadata, and merge conflicts.
4. Identify secrets, personal data, proprietary information, and unsuitable datasets that must not be committed.
5. Distinguish Git, Git LFS, DVC, and artifact storage.
6. Create a practical workflow for experiments, review, releases, and recovery.
7. Evaluate the limitations and governance responsibilities of version-controlled data science.

1. Reproducibility as a Data-Science Practice

A result is reproducible when the steps needed to obtain it are recorded clearly enough for repetition. Four dimensions are especially important:

Git primarily versions text and project history. It does not automatically capture a database state, an installed library, a random-number generator state, or an external API response. Reproducibility therefore requires a connected system of records.

A useful provenance description can be expressed conceptually as:

Here, f represents the computational workflow. This is not a statistical equation; it is a provenance model. If any input changes, the result may change. The model also highlights why saving only a final notebook is inadequate.

Reproducibility and repeatability

Repeatability usually refers to obtaining consistent results by the same team, using the same equipment and procedures. Reproducibility commonly refers to an independent team obtaining consistent results using the documented materials and methods. In applied data science, both terms are used differently across disciplines, so a project should define its intended standard explicitly.

Exact numerical equality may be inappropriate for stochastic algorithms, distributed computation, changing external data, or different hardware. In such cases, the team should record tolerances, evaluation criteria, random seeds where meaningful, and the reasons exact equality is not expected.

2. A Professional Git Repository for Data Science

A repository should make the project understandable to a new contributor. A typical structure is:

project/
├── README.md
├── LICENSE
├── .gitignore
├── pyproject.toml or requirements.txt
├── environment.yml
├── src/
├── notebooks/
│   ├── 01_exploration.ipynb
│   └── 02_modeling.ipynb
├── tests/
├── configs/
├── docs/
├── data/
│   ├── README.md
│   └── .gitkeep
├── models/
├── reports/
└── .github/

The exact structure can differ, but responsibilities should be clear:

• README.md explains purpose, setup, data access, usage, and limitations.
• src/ contains reusable production or analysis code.
• notebooks/ contains exploratory analysis, demonstrations, and documented experiments.
• tests/ contains automated checks for functions and data transformations.
• configs/ contains non-secret parameters, preferably in readable YAML, TOML, or JSON.
• docs/ contains methods, data dictionaries, decisions, and operational guidance.
• data/ usually contains instructions, schemas, or small permitted samples rather than raw data.
• models/ and reports/ contain selected releases or pointers to externally stored artifacts, not uncontrolled output dumps.

A repository is a communication system. Names should be meaningful, files should have one clear purpose, and obsolete notebooks should be archived or removed rather than left as competing sources of truth.

A normal Git workflow

A basic workflow is:

git clone <repository-url>
cd project
git switch -c feature/validated-features
# edit files
git status
git diff
git add src/ tests/ README.md
git commit -m "Validate missing-value handling"
git push -u origin feature/validated-features

A commit should represent a coherent, reviewable change. Messages should describe the change and, where useful, its purpose. A branch and pull request workflow allows another person to inspect code, documentation, tests, and configuration before integration.

Do not use commits as a substitute for backups. A local repository may be lost, and a remote repository may have access or retention limitations. Important projects require authorised remote hosting, backup policies, and recovery tests.

3. Versioning Notebooks Effectively

Jupyter notebooks are JSON documents containing cells, outputs, execution counts, metadata, and source code. Git can version them, but ordinary line-based diffs are often difficult to interpret. A one-line code change can appear beside thousands of changed output lines.

Sources of notebook difficulty

1. Generated outputs: tables, plots, and rich display data can create large diffs.
2. Execution counts: running cells in a different order changes metadata without changing logic.
3. Hidden state: a notebook may depend on variables created in an earlier, unrelated cell.
4. Non-determinism: sampling, parallel processing, and model training may change results.
5. Mixed responsibilities: exploration, explanation, production code, and reporting may be combined.

Recommended practices

• Keep notebooks short and focused on a question or experiment.
• Move reusable transformations and model logic into tested scripts or modules.
• Restart the kernel and run all cells before committing a result notebook.
• Clear outputs when outputs are not part of the reviewed deliverable.
• Preserve outputs when they are essential evidence, but control their size and sensitivity.
• Record data versions, parameters, software versions, and execution date.
• Avoid relying on execution order that is not visible in the notebook.
• Use notebook-aware tools such as nbdime for more meaningful diffs.
• Consider Jupytext or a paired .py/Markdown representation for code review.
• Do not treat a notebook as the sole documentation of a production pipeline.

Clearing outputs is not an absolute rule. A teaching notebook or published report may intentionally include outputs. The decision should be documented and consistent with the review and release purpose.

Notebook merge conflicts

A conflict can occur when two people edit the same notebook or when one person changes code while another changes outputs. Resolve conflicts using a notebook-aware tool, or regenerate the notebook from a clean source where possible. Manually editing raw notebook JSON is error-prone. After resolution, restart and execute the entire notebook, inspect key outputs, and commit the verified result.

An effective design separates an exploratory notebook from a reproducible pipeline. The notebook may call functions in src/, while tests validate those functions independently. This reduces both merge complexity and the risk that an undocumented notebook state controls the result.

4. What Should and Should Not Be Committed?

The central rule is that a repository should contain files that are necessary, permitted, reviewable, and useful for reconstruction. It should not become an uncontrolled warehouse.

Usually suitable for Git

• Source code and tests
• Documentation and data dictionaries
• Small, legally distributable sample data
• Non-secret configuration templates
• Schemas and validation rules
• Environment specifications and lock files
• Experiment definitions and evaluation summaries
• Small plots or selected reports, when permitted

Usually unsuitable for ordinary Git

• Passwords, API keys, private certificates, and access tokens
• Personal or sensitive records without approved controls
• Large raw datasets and database exports
• Temporary caches and downloaded packages
• Unreviewed generated outputs
• Proprietary data that the repository audience is not authorised to access
• Credentials embedded in notebook outputs or logs
• Model files that exceed repository limits or contain restricted information

A .gitignore file prevents many accidental additions, but it is not a security boundary. It does not remove a file already committed, and it does not prevent a user from forcing an ignored file into Git.

Example:

.env
*.pem
*.key
__pycache__/
.ipynb_checkpoints/
.venv/
data/raw/
data/processed/
models/*.bin

Use a committed template such as .env.example with placeholder names but no real values. Secrets should be supplied through an approved secret manager or protected environment variables. If a secret is committed, removing it in a later commit is insufficient: assume it has been exposed, revoke or rotate it, investigate access, and rewrite history only under an approved procedure.

Data protection and governance

Before adding data, ask:

• Is collection and use lawful and authorised?
• Does the dataset contain personal, confidential, health, financial, or security-sensitive information?
• Does the license permit storage, processing, and redistribution?
• Can the data be minimised, anonymised, aggregated, or replaced by a synthetic sample?
• Who can access the repository and its backups?
• How long should the data be retained?

Anonymisation is not automatically irreversible. Combinations of fields may re-identify individuals. Data governance must therefore consider the threat model, access controls, retention, purpose limitation, and applicable institutional or national requirements.

5. Tracking Data, Models, and Artifacts

Git stores snapshots of files efficiently when they are text-based and reasonably sized. It is not designed to provide full lifecycle management for large binary datasets, trained models, video, or repeated experiment outputs.

Git Large File Storage (Git LFS)

Git LFS stores a small pointer file in the Git repository while the large object is stored in an LFS server. It is useful for versioned binary files that must behave like repository assets, such as a modest approved model or an image used in testing.

git lfs install
git lfs track "models/*.pt"
git add .gitattributes models/model.pt
git commit -m "Track approved model with Git LFS"

LFS is still subject to repository permissions, storage quotas, retention policies, and data governance. It does not automatically provide dataset lineage, pipeline metadata, or transformations. A large confidential dataset should not be uploaded merely because LFS accepts it.

Data Version Control (DVC)

DVC connects Git commits with data and model artifacts stored in local or remote storage. Git records small metadata files, while the actual objects are stored elsewhere. A DVC pipeline can describe dependencies, parameters, stages, and outputs.

A simplified workflow might be:

dvc init
dvc add data/approved_sample.csv
git add data/approved_sample.csv.dvc .gitignore
git commit -m "Register approved sample dataset"
dvc push

DVC is useful when a project needs dataset versions, reproducible stages, and links between code and artifacts. It introduces operational requirements: remote storage must be secured, objects must be retained, and users must understand which Git commit and DVC metadata identify the desired state.

Artifact stores and registries

Object storage, experiment trackers, package registries, and model registries are often better for large or operational artifacts. A registry can record model version, training data reference, evaluation metrics, approval status, owner, and deployment history. The Git repository should contain the configuration or immutable identifier needed to locate the approved artifact.

These tools are complementary, not interchangeable. The correct choice depends on size, access control, lineage, collaboration, cost, retention, and regulatory requirements.

6. Environments, Configuration, and Dependencies

A script may fail to reproduce because a library changed, an operating-system dependency is absent, or a path is specific to one computer. Record dependencies explicitly. Examples include:

• requirements.txt for Python package requirements;
• pyproject.toml for project metadata and dependency configuration;
• environment.yml for Conda environments;
• lock files for exact or resolved dependency versions;
• Dockerfiles or approved container specifications for stronger isolation.

Pinning every dependency can improve repeatability but may reduce maintainability and fail to capture operating-system or hardware behaviour. A practical policy distinguishes direct dependencies from transitive dependencies and records the tool used to create the environment.

Configuration should be external to reusable code. For example, a training script can receive a configuration file specifying a data reference, target column, seed, and model parameters. Do not place passwords in configuration files. Use separate development, test, and production configurations with controlled promotion.

Random seeds can make some experiments repeatable, but they are not a universal guarantee. Parallel execution, library implementation, floating-point behaviour, GPU kernels, and external services may still produce differences. Record the seed and execution conditions, and define acceptable evaluation tolerances.

7. Experiment Tracking and Commit Design

An experiment should have an identifiable relationship among code revision, data version, configuration, environment, and results. A useful experiment record includes:

• experiment identifier and date;
• Git commit or release identifier;
• data or DVC version;
• configuration and parameter values;
• environment specification;
• evaluation metrics and validation design;
• model or report artifact location;
• interpretation, limitations, and reviewer status.

Do not commit every generated file merely to show activity. Commit meaningful milestones: a validated preprocessing change, a documented baseline, a selected model, or a corrected evaluation procedure. Separate exploratory work from approved results so that an informal trial is not mistaken for a production decision.

Commit history is an audit aid, not proof that a result is correct. Code review, tests, data validation, statistical review, and appropriate approval remain necessary.

8. Illustrative Case: A Demand-Forecasting Project

Background

A small retailer develops a weekly demand-forecasting model for store-level replenishment. The team includes an analyst, a data engineer, and an operations manager. Sales data contain customer-related fields that are not needed for forecasting.

Problem

The first prototype is a notebook saved on one analyst’s laptop. It reads a changing spreadsheet, contains hard-coded paths, includes a token used to download data, and saves a large trained model inside the repository. A second analyst cannot reproduce the reported error metric.

Analysis

The team identifies five causes:

1. The input spreadsheet was replaced without a version record.
2. The notebook depended on cells being executed in a particular order.
3. The token was exposed in notebook source and output.
4. The environment and library versions were not recorded.
5. The evaluation split was changed during experimentation and not documented.

The team also determines that the customer-related fields are unnecessary and should not enter the modelling workflow.

Approach and solution

The team creates a repository with src/, notebooks/, tests/, configs/, and docs/. Reusable feature engineering moves into tested Python modules. The notebook becomes a thin, explanatory client of those modules. Raw sales data remain in approved object storage; DVC records the permitted dataset reference and its version. A non-secret configuration file records the target, forecast horizon, split policy, and random seed. Credentials are supplied through a secret manager.

The environment is specified with a lock file and a container definition. The team adds tests for date handling, missing values, and prevention of future-data leakage. A model registry stores approved models and records the Git commit, data version, metrics, and approval status. Notebook outputs are cleared for routine commits, while a final report notebook retains selected charts.

Outcome

The second analyst can recreate the pipeline from a documented commit, retrieve the authorised data version, build the environment, and obtain metrics within the agreed tolerance. The retailer also gains a reviewable history of changes and can distinguish exploratory models from approved operational models.

Learning points

The solution did not come from Git alone. Git supplied source history, but reproducibility required data lineage, dependency management, secret protection, tests, governance, and an artifact registry. The project also improved privacy by excluding unnecessary personal information.

9. Limitations, Ethics, and Contemporary Practice

Version control can preserve harmful code as efficiently as useful code. A repository may contain biased training data, discriminatory features, insecure dependencies, or a model whose limitations are not communicated. Review must therefore include fairness, privacy, security, licensing, and impact assessment where relevant.

Data deletion requests can be difficult when information exists in commits, caches, forks, backups, and artifact stores. Organisations should define retention and deletion procedures before sensitive data enters a repository. Access controls should follow least privilege, and audit logs should be protected.

Continuous integration can automatically run tests, linting, notebook execution checks, dependency scans, and secret detection. These tools reduce risk but produce false positives and cannot determine whether a feature is ethically appropriate or whether a metric is scientifically meaningful. Human review remains essential.

Modern AI-assisted coding tools make it easier to generate scripts and documentation, but they also create risks: unverified code, licensing uncertainty, accidental disclosure of proprietary prompts or data, and fabricated explanations. Generated contributions should be reviewed, tested, attributed according to institutional policy, and never trusted merely because they compile.

Best-Practice Checklist

Before committing:

• Is the change focused and understandable?
• Have tests and relevant notebooks been run from a clean state?
• Are data, outputs, and dependencies identified?
• Are secrets and personal data absent?
• Is the license and access permission clear?
• Can another contributor follow the README?
• Does the commit identify the appropriate data and artifact versions?

Before releasing a model or report:

• Record code, data, configuration, environment, and evaluation information.
• Store large artifacts in an approved system.
• Document limitations, intended use, and monitoring responsibilities.
• Obtain technical, governance, and domain approval where required.

Chapter Summary

Git provides a durable history for code, documentation, tests, configuration, and selected project assets. In data science, however, reproducibility requires more than source control. A reliable project links results to data versions, environments, parameters, execution procedures, and stored artifacts.

Notebooks require special handling because they combine code, outputs, metadata, and hidden state. Notebook-aware diffs, clean execution, modular source code, and tests improve reviewability. Secrets, personal data, proprietary datasets, caches, and unsuitable large files should not be committed. A .gitignore file helps prevent mistakes but cannot secure or erase exposed information.

Git LFS is appropriate for some large binaries, DVC supports data and pipeline lineage, and object storage or model registries serve large operational artifacts. These systems complement Git and must be governed through access control, retention, licensing, and security policies. The professional data scientist treats version control as one part of a broader reproducibility and responsible-innovation practice.

Key Terms

Git: A distributed version-control system that records changes to files.

Repository: A directory and metadata store containing a project’s files and history.

Commit: A recorded snapshot of selected changes with an explanatory message.

Branch: An independent line of development within a repository.

Pull request: A proposed change submitted for review and integration.

Reproducibility: The ability to recreate or explain a result using documented materials and procedures.

Provenance: Information describing the origin, transformations, and relationships of data or results.

Notebook: An interactive document combining executable code, text, metadata, and outputs.

Notebook diff: A comparison of notebook versions, ideally showing meaningful cell-level changes.

Configuration: Parameters and settings that control execution without changing core source code.

Secret: Sensitive authentication or encryption information, such as a password or API key.

Gitignore: A file specifying patterns for files Git should normally disregard.

Git LFS: Git Large File Storage, which keeps pointers in Git and large objects in separate storage.

DVC: Data Version Control, a system connecting Git history with data, models, and pipeline metadata.

Artifact: A generated object such as a dataset, model, report, or package.

Artifact registry: A controlled service for storing and cataloguing versioned generated objects.

Data lineage: A record of where data came from and how it was transformed.

Environment specification: A declaration of software dependencies and execution requirements.

Data leakage: The inappropriate use of information that would not be available at prediction time or that improperly influences evaluation.

Secret rotation: Replacing an exposed or ageing credential with a new credential and invalidating the old one.

Review Questions

Short Answer Questions

1. Why is Git alone insufficient to guarantee reproducibility in a data-science project?
2. Give three reasons why notebooks can produce difficult Git diffs.
3. Why should a secret be rotated if it was committed and later deleted?
4. Distinguish Git LFS from DVC.
5. What information should be linked to an approved machine-learning model?

Long Answer Questions

1. Design a repository structure for a team developing and documenting a machine-learning pipeline. Explain the purpose of each major directory.
2. Explain a professional procedure for reviewing and merging a notebook that has conflicting edits.
3. Discuss the advantages and limitations of storing datasets in Git, Git LFS, DVC, and object storage.
4. Explain how dependency specifications, configuration files, random seeds, and containers contribute to reproducibility.
5. Discuss the ethical and governance risks of versioning personal or confidential data.

Analytical / Application Questions

1. A repository contains a notebook, an API key, a 5 GB raw dataset, a small permitted sample, and a trained model. Classify each item and propose an appropriate storage or protection method.
2. Two analysts obtain different evaluation scores from the same project. Develop a diagnostic plan using Git history, data versions, configuration, environment records, and notebook execution checks.
3. Design a release record for a forecasting model that would allow an auditor to connect the deployed artifact to its code, data, environment, evaluation, approval, and limitations.

Multiple Choice Questions with Answers

Question 1

Which item is normally most appropriate to store directly in Git?

A. A password used to access a database

B. A large confidential customer database

C. A tested source-code module

D. A temporary package cache

Correct Answer: C. A tested source-code module

Question 2

Why can ordinary Git diffs be difficult to interpret for Jupyter notebooks?

A. Git cannot store notebook files

B. Notebooks contain outputs, metadata, and execution information as well as source code

C. Notebook code is always encrypted

D. Git automatically deletes notebook history

Correct Answer: B. Notebooks contain outputs, metadata, and execution information as well as source code

Question 3

What is the main purpose of a .gitignore file?

A. To encrypt repository contents

B. To prevent all previously committed files from being accessed

C. To specify files Git should normally disregard

D. To replace a secret-management system

Correct Answer: C. To specify files Git should normally disregard

Question 4

Which tool is specifically designed to connect Git commits with versions of datasets and pipeline outputs?

A. DVC

B. A text editor

C. A compiler

D. A password generator

Correct Answer: A. DVC

Question 5

What should normally happen after an API key is committed accidentally?

A. Only rename the file

B. Rotate or revoke the key and investigate the exposure

C. Add the key to .gitignore and take no further action

D. Publish the key in the README

Correct Answer: B. Rotate or revoke the key and investigate the exposure

Question 6

Which practice best reduces hidden notebook state?

A. Running cells in random order

B. Using longer notebook filenames

C. Restarting the kernel and executing all cells from a clean state

D. Deleting all source code after execution

Correct Answer: C. Restarting the kernel and executing all cells from a clean state

Question 7

Which system is generally suitable for storing a large operational model with approval and deployment metadata?

A. A model registry or controlled artifact store

B. A notebook execution counter

C. A .gitignore file

D. A commit message alone

Correct Answer: A. A model registry or controlled artifact store

Question 8

What does a dependency lock file primarily help record?

A. The legal identity of every user

B. Resolved software dependency versions

C. The physical location of a dataset’s subjects

D. The number of Git branches

Correct Answer: B. Resolved software dependency versions

Question 9

Why should reusable data transformations often be moved from notebooks into source modules?

A. To make testing and reuse easier

B. To prevent any documentation from being written

C. To guarantee that data never changes

D. To eliminate the need for version control

Correct Answer: A. To make testing and reuse easier

Question 10

Which statement about random seeds is most accurate?

A. A seed guarantees identical results across all hardware and libraries

B. A seed can support repeatability but does not eliminate every source of variation

C. Seeds are only used for storing passwords

D. Seeds replace data-version records

Correct Answer: B. A seed can support repeatability but does not eliminate every source of variation