The Essential Git Workflow

Introduction

Git is a distributed version control system used to record, inspect, compare, and coordinate changes to files. For data science students, Git is particularly valuable because projects contain not only source code, but also notebooks, configuration files, documentation, tests, environment specifications, and sometimes data-processing scripts. A disciplined Git workflow makes these artefacts traceable and recoverable.

Git does not automatically understand whether a change is good, correct, or scientifically valid. It records snapshots and relationships between snapshots. The quality of a repository therefore depends on human decisions: what to track, when to commit, how to describe changes, and how to prevent sensitive or unnecessary files from entering the repository.

This chapter develops the essential local workflow:

Working directory → Staging area → Repository history
       ↑                  │                  │
       └──── restore ─────┴──── inspect ─────┘

The staging area is an important design feature. It allows a user to select precisely which changes will form the next commit rather than committing every modification in the working directory.

Learning objectives

By the end of this chapter, you should be able to:

• inspect repository status and interpret its output;
• distinguish the working directory, staging area, and repository history;
• track files and stage selected changes;
• create meaningful, focused commits;
• review differences and inspect commit history;
• safely restore files and undo unsuitable changes;
• rename and remove tracked content correctly;
• design an effective .gitignore file;
• apply a reliable workflow to a small data-science project.

The three states of Git

A file in a Git project can exist in three practically important states.

A commit is not simply a save operation. It is a recorded snapshot with an author, timestamp, message, and parent relationship. A commit normally contains only the content placed in the staging area at commit time.

Suppose analysis.py is modified after it was staged. The staged version and the working version can then differ. This explains why git diff and git diff --staged answer different questions:

• git diff compares the working directory with the staging area.
• git diff --staged compares the staging area with the latest commit.

The distinction is central to controlled versioning.

Inspecting repository status

The first command in most Git tasks should be:

git status

A typical result may say that a file is modified but not staged, identify untracked files, or confirm that the working tree is clean. “Untracked” means that Git sees a file in the project directory but has not yet been instructed to include it in version control. “Modified” means that a tracked file differs from the version represented by the index or latest commit.

A useful routine is:

git status --short

The short format uses two columns. The first position describes the index, and the second describes the working directory. For example:

Status should be inspected before staging, before committing, and before changing branches or restoring content. It is a diagnostic command, not merely a progress display.

Tracking and staging files

To begin tracking a new file, use:

git add analysis.py

The command places the current content of analysis.py into the staging area. It does not create a commit. Several files can be staged explicitly:

git add analysis.py README.md requirements.txt

A directory can be staged recursively:

git add src/

The command git add . stages changes beneath the current directory. Although convenient, it should not be used thoughtlessly. It may stage generated files, large outputs, credentials, or unrelated edits. A professional workflow usually checks status and uses targeted staging when a project contains multiple kinds of files.

Interactive staging

When one file contains unrelated changes, interactive staging can separate them:

git add -p

Git presents portions, called hunks, and asks whether each should be staged. This supports focused commits such as “correct date parsing” rather than a mixed commit containing formatting, refactoring, and bug fixes.

Interactive staging is especially useful for notebooks or scripts that have accumulated several edits. However, notebook files may contain large output changes, making review difficult. Clearing unnecessary outputs and relying on reproducible execution can improve reviewability.

Unstaging without discarding work

If a file was staged accidentally, it can be removed from the index while retaining its working-directory edits:

git restore --staged analysis.py

The file remains modified, but it is no longer part of the next commit. On older Git versions, an equivalent command is commonly written as git reset HEAD -- analysis.py; git restore communicates the intended operation more clearly.

Creating meaningful commits

A commit should represent a coherent change that can be understood, reviewed, and, if necessary, reverted. After staging, create the commit with:

git commit -m "Handle missing values in customer data"

A useful commit message answers, briefly, what changed. Good messages are specific and use an imperative style, for example:

• Add validation for input dates
• Document model evaluation procedure
• Fix division by zero in feature scaling

Weak messages include stuff, changes, and final version, because they provide little information when reviewing history.

A commit should generally:

1. contain one logical purpose;
2. be small enough to review;
3. leave the project in a usable state where practical;
4. avoid secrets and unnecessary generated artefacts;
5. explain behaviour rather than merely naming a file.

A commit is not a substitute for a backup strategy. Local history can be lost through disk failure, and a repository containing sensitive information remains sensitive even if a file is later deleted. Remote hosting, access control, and secret-management practices are separate responsibilities.

Author identity

Git records an author identity. Configure it once on a machine:

git config --global user.name "Your Name"
git config --global user.email "[email protected]"

In an institutional or professional setting, the configured identity should comply with organisational policy. Commits create an accountability record, so identities must not be misrepresented.

Reviewing changes with git diff

Before committing, inspect the proposed work:

git diff

This shows unstaged changes. To inspect what is already selected for the next commit, use:

git diff --staged

A complete review commonly follows this sequence:

git status
git diff
git diff --staged

The review should ask:

• Is every changed line intentional?
• Are debug statements, passwords, tokens, or local paths present?
• Does the change belong in this commit?
• Are tests or documentation needed?
• Could the change alter analytical results?

Useful variations include:

git diff --stat
git diff HEAD

git diff --stat gives a compact summary. git diff HEAD compares the working directory and index together with the latest commit, showing all current changes whether staged or unstaged.

Reviewing history

The basic history command is:

git log

A compact, graph-oriented view is often more readable:

git log --oneline --graph --decorate --all

To inspect one commit and its patch:

git show <commit-id>

To identify a commit, Git accepts a complete or sufficiently distinctive commit identifier. A commit history can answer questions such as:

• When was a file changed?
• Which commit introduced a behaviour?
• What was the state before a change?
• Which person recorded the commit?

For file-specific history:

git log -- data/README.md

For line-level attribution:

git blame analysis.py

git blame identifies the commit associated with each line; it should be used for investigation, not personal criticism. A line may have been moved, generated, or changed under constraints not visible from the line alone.

Restoring files and undoing changes

Restoration must be chosen according to the state that should be preserved. Destructive commands should be preceded by status and diff inspection.

Discarding unstaged changes

To replace a working-directory file with its staged version:

git restore analysis.py

This discards unstaged modifications to that file. If the file has no staged version, Git restores the version from the latest commit. The discarded content may be difficult or impossible to recover, so this command should be used only when the edits are no longer needed.

Restoring a file from a commit

To retrieve a file as it existed in a particular commit:

git restore --source=<commit-id> -- analysis.py

This changes the working copy; it does not automatically create a commit. The resulting change should be reviewed and either staged or discarded.

Reverting a committed change

For a shared history, the safer way to undo a published commit is usually:

git revert <commit-id>

Git creates a new commit that reverses the selected commit. The original history remains visible. This is preferable to rewriting shared history.

Reset and history rewriting

git reset can move a branch reference and alter the staging area or working directory depending on its mode. For example, git reset --soft preserves changes staged, while git reset --hard can discard uncommitted work. Because reset can rewrite local history and --hard is destructive, it should be used only with a clear recovery plan and not casually on shared branches.

The practical rule is simple: use restore for file-state operations, revert for undoing published commits, and treat reset --hard as a high-risk command.

Renaming and removing content

Git can detect many renames by comparing file content, but explicitly recording a rename communicates intent:

git mv old_name.py new_name.py
git commit -m "Rename preprocessing module"

Similarly, to remove a tracked file from both the working directory and the index:

git rm obsolete.csv

To stop tracking a file while retaining it locally:

git rm --cached local_config.yml

This is useful when a file was tracked accidentally and should subsequently be ignored. It does not remove the file from earlier commits, so exposed secrets must be rotated and, where necessary, removed from repository history using specialised procedures.

Using .gitignore effectively

A .gitignore file contains patterns for files Git should not normally track. A typical data-science example is:

# Python caches
__pycache__/
*.py[cod]

# Virtual environments
.venv/
venv/

# Notebooks and local tools
.ipynb_checkpoints/
.vscode/

# Secrets and machine-specific settings
.env
*.local

# Generated outputs
reports/generated/
models/*.bin

Patterns may match files or directories. A trailing slash conventionally identifies a directory. The asterisk is a wildcard, so *.log matches files ending in .log.

.gitignore is itself normally committed because it communicates project policy to collaborators. It should not be used as a security mechanism: ignored files can still be deliberately added with git add -f, and files already tracked are not automatically untracked when a pattern is added.

If a tracked file should now be ignored:

git rm --cached settings.local

Then commit the .gitignore change and the removal from tracking.

Ignore rules should be selective. Source code, dependency specifications, documentation, small configuration templates, and reproducible project metadata generally belong in Git. Large raw datasets, secrets, personal credentials, temporary files, caches, and reproducible generated outputs generally do not. Whether a dataset is tracked also depends on licensing, privacy, size, and organisational policy.

A disciplined end-to-end workflow

A reliable local workflow can be expressed as follows:

# 1. Inspect
git status

# 2. Edit files using an editor or notebook

# 3. Inspect the edits
git diff

# 4. Stage only the intended files or hunks
git add src/analysis.py tests/test_analysis.py

# 5. Review the proposed snapshot
git diff --staged

# 6. Record a coherent commit
git commit -m "Validate analysis input schema"

# 7. Confirm the result
git status
git log --oneline -3

The workflow is iterative rather than strictly linear. A failed review may lead to more editing, selective unstaging, or restoration. The essential principle is that the user should know what is changing at each boundary.

Applied case: a reproducible student analysis project

The following is an illustrative case. A student team analyses bicycle rental demand. The repository contains src/, tests/, notebooks/, README.md, and a local virtual environment.

Background and problem

The team begins with a notebook that reads a downloaded dataset and produces charts. The first commit accidentally includes the virtual environment, notebook checkpoints, and a file containing an API token. Review becomes slow, and another team member cannot reproduce the analysis.

Analysis

The team inspects the repository:

git status --short
git ls-files

They identify three categories: project material, generated material, and sensitive local material. They also inspect previous commits to determine whether the token was committed. Adding the token to .gitignore alone would not remove it from earlier history.

Approach and solution

The team takes these steps:

1. Revokes and replaces the exposed API token.
2. Adds .venv/, .ipynb_checkpoints/, .env, and generated report paths to .gitignore.
3. Moves reusable transformations into src/cleaning.py.
4. Adds tests for date parsing and missing values.
5. Uses git add -p to separate refactoring from bug correction.
6. Reviews git diff --staged before each commit.
7. Adds a requirements.txt file and documents how to obtain the dataset legally.
8. Uses a history-cleaning procedure approved by the hosting platform if policy requires removing the token from old commits.

Example commits are:

Add reproducible cleaning pipeline
Test missing-date handling
Document dataset acquisition
Ignore local environments and generated reports

Outcome

The repository becomes smaller, easier to review, and more reproducible. A collaborator can create an environment from the dependency specification, obtain data through documented lawful means, run the pipeline, and distinguish source files from generated outputs.

Learning points

The case demonstrates that Git practice is also data governance. Version control supports reproducibility, but it does not by itself guarantee privacy, licensing compliance, scientific validity, or secure secret management. Those requirements must be addressed through project design and institutional policy.

Challenges, limitations, and professional practice

Git has a steep conceptual threshold because a file can simultaneously have staged and unstaged versions. Large binary files and frequently changing notebook outputs are difficult to review. Merge conflicts require human interpretation, and a clean commit does not prove that code is correct.

Data-science projects present additional challenges:

• Large data: use approved object storage or data-versioning systems when repositories are unsuitable.
• Privacy: do not commit identifiable or restricted data without lawful authority and appropriate controls.
• Reproducibility: record dependency versions, random seeds where appropriate, preprocessing decisions, and data provenance.
• Notebooks: keep outputs purposeful, clear unnecessary outputs, and move stable logic into tested modules.
• Secrets: use environment variables or a secrets manager; never treat .gitignore as protection after exposure.
• Collaboration: agree on commit conventions, review rules, branch policies, and ownership.

Automated tests, linters, and continuous integration can run after commits or when changes are proposed. These tools improve quality but do not replace critical review. In analytical work, a test may confirm that code executes while failing to establish that a model is unbiased or that the dataset is representative.

AI-assisted coding introduces additional governance concerns. Generated code should be reviewed, tested, licensed appropriately, and checked for leaked confidential data or unsupported claims. The Git history should document meaningful human decisions rather than become a record of unexamined generated fragments.

Best-practice checklist

Before committing, ask:

• Is the repository status understood?
• Are only relevant files staged?
• Has the staged diff been reviewed?
• Does the message describe one coherent change?
• Are secrets, private data, caches, and unnecessary outputs excluded?
• Can another person understand how to reproduce the work?
• If undoing a change, is restore, revert, or a history rewrite the appropriate operation?

Chapter Summary

Git organizes project work through a working directory, staging area, and repository history. git status reveals the current state; git add selects content for the next snapshot; git commit records a coherent change. git diff and git diff --staged provide complementary reviews, while git log and git show support historical investigation.

Restoration commands must be chosen carefully because some discard work. git restore manages file content, git revert safely reverses a published commit, and destructive history-rewriting commands require special care. git mv, git rm, and git rm --cached support content reorganisation. A committed .gitignore prevents routine tracking of unsuitable files, but it is not a security control and cannot erase secrets from history. In data science, effective Git practice supports reproducibility, collaboration, accountability, and responsible data governance.

Key Terms

Working directory: The checked-out project files currently available for editing.

Repository: Git’s database containing project history and metadata.

Staging area: The index containing content selected for the next commit.

Commit: A recorded snapshot with metadata and parent relationships.

Tracked file: A file known to Git through a commit or staging operation.

Untracked file: A file present in the directory but not yet recorded by Git.

git status: A command that reports working-directory and staging states.

git add: A command that stages file content.

git diff: A command that compares versions of content.

git log: A command that displays commit history.

git restore: A command for restoring file content or unstaging changes.

git revert: A command that creates a new commit reversing an earlier commit.

git reset: A command that can move references and alter index or working-tree state.

Rename: A change in a file’s path or name, often recorded with git mv.

.gitignore: A pattern file specifying files Git should normally ignore.

Hunk: A contiguous portion of a diff that can be staged independently.

Repository history: The sequence of connected commits recorded by Git.

Reproducibility: The ability to obtain consistent analytical results using documented inputs and procedures.

Secret: Sensitive authentication or access information, such as a token or password.

Review Questions

Short Answer Questions

1. What are the three principal states in the Git workflow?
2. What is the difference between git diff and git diff --staged?
3. Why should commits normally contain one coherent logical change?
4. What does git rm --cached do?
5. Why is .gitignore not a security mechanism?

Long Answer Questions

1. Explain how a file can have different versions in the working directory, staging area, and latest commit. Illustrate your explanation with commands.
2. Describe a professional workflow for reviewing and committing changes to a data-science project.
3. Compare git restore, git revert, and git reset, including the risks associated with each.
4. Explain how .gitignore should be designed for a Python-based data-science project.
5. Discuss how Git supports reproducibility while remaining insufficient by itself for privacy and scientific validity.

Analytical / Application Questions

1. A developer stages analysis.py, edits it again, and runs git commit. Which version is committed, and which command displays the later unstaged edit? Explain.
2. A repository contains a committed API token. The developer adds .env to .gitignore and commits the change. Analyse why this is insufficient and propose responsible next steps.
3. A team has a notebook containing model code, charts, cached outputs, and three unrelated fixes. Design a staging and commit strategy that produces reviewable, reproducible history.

Multiple Choice Questions with Answers

Question 1

Which command reports the current working-directory and staging-area state?

A. git status

B. git show

C. git blame

D. git mv

Correct Answer: A. git status

Question 2

What does git diff --staged compare?

A. Two remote branches

B. The staging area with the latest commit

C. The working directory with an untracked file

D. Two unrelated repositories

Correct Answer: B. The staging area with the latest commit

Question 3

Which command stages a file without creating a commit?

A. git log

B. git revert

C. git add

D. git show

Correct Answer: C. git add

Question 4

A file is staged and then edited again. Which statement is correct?

A. The later edit is automatically staged

B. The staged and working versions may differ

C. Git deletes the staged version

D. The file becomes untracked

Correct Answer: B. The staged and working versions may differ

Question 5

Which command is generally appropriate for undoing a published commit while preserving history?

A. git revert

B. git restore --hard

C. git clean

D. git init

Correct Answer: A. git revert

Question 6

What is the principal purpose of .gitignore?

A. Encrypt repository files

B. Prevent specified untracked files from normally being added

C. Delete files from all previous commits

D. Create a remote repository

Correct Answer: B. Prevent specified untracked files from normally being added

Question 7

Which command renames a tracked file and stages the operation?

A. git mv old.py new.py

B. git log old.py new.py

C. git diff old.py new.py

D. git blame old.py new.py

Correct Answer: A. git mv old.py new.py

Question 8

Which command removes a file from tracking but keeps the local file?

A. git rm file.txt

B. git rm --cached file.txt

C. git restore file.txt

D. git show file.txt

Correct Answer: B. git rm --cached file.txt

Question 9

What is a key advantage of interactive staging with git add -p?

A. It automatically tests every program

B. It permits selected hunks to form a focused commit

C. It encrypts sensitive files

D. It permanently deletes unstaged changes

Correct Answer: B. It permits selected hunks to form a focused commit

Question 10

Which practice best supports reviewable commit history?

A. Use vague messages such as “changes”

B. Commit generated caches with source code

C. Combine unrelated fixes into one commit

D. Use specific messages for coherent logical changes

Correct Answer: D. Use specific messages for coherent logical changes