2. Development Environments, Command Line and Version Control

Introduction and Context

Data science is often described through algorithms, statistical models, and visualisations. Yet the quality of analytical work also depends on the environment in which that work is created. A model may be mathematically sound but difficult to reproduce if its dependencies are undocumented, its data-processing steps are hidden in an interactive session, or its code is stored without a reliable history of change. Development environments, command-line tools, notebooks, package managers, and version-control systems provide the infrastructure for disciplined analytical work.

A development environment is the combination of hardware, operating system, software, libraries, configurations, data access mechanisms, and tools used to create and execute analytical programs. In professional data science, the environment must support more than successful execution on one person's computer. It should enable collaboration, testing, traceability, portability, security, and reproducibility.

This chapter examines the relationship between operating-system concepts, shells, package management, integrated development environments (IDEs), notebooks, Git, and collaborative repository platforms. These tools are not isolated conveniences. Together they form a workflow through which analysts transform ideas into controlled and maintainable computational artefacts.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain how operating systems, filesystems, processes, permissions, and environment variables affect data-science work.

2. Use a shell to navigate directories, manage files, run programs, and automate routine tasks.

3. Distinguish system-level, project-level, and environment-level dependencies.

4. Compare package managers, virtual environments, IDEs, and notebooks for different analytical purposes.

5. Apply Git concepts such as commits, branches, merges, remotes, and pull requests.

6. Design a reproducible project structure and dependency-management approach.

7. Identify technical, security, ethical, and collaborative risks in development workflows.

The Operating System as the First Analytical Platform

The operating system (OS) manages the relationship between applications and computer hardware. It allocates processor time and memory, manages files and devices, controls user access, and provides services through which programs execute. Linux, macOS, and Windows differ in implementation and conventions, but each provides a filesystem, processes, users, permissions, and mechanisms for installing and running software.

A filesystem organises information as files and directories. A path identifies the location of an item. Absolute paths begin at the filesystem root or a drive, whereas relative paths are interpreted from the current working directory. This distinction is important in data projects. A script that refers to `/home/analyst/project/data/file.csv` may work only on one computer, while a project-relative reference such as `data/file.csv` is more portable when the project is opened elsewhere.

A process is a running program with its own execution context. Data scientists encounter processes when launching a Python script, starting a notebook server, running a database client, or executing a distributed computation. Processes may consume substantial memory or CPU and may continue running after a terminal window appears inactive. Understanding process status and termination is therefore part of practical troubleshooting.

Operating systems also enforce permissions. A file may be readable, writable, or executable by its owner, a designated group, or other users. Poor permission settings can expose confidential datasets; excessive restrictions can prevent legitimate pipelines from running. Professional practice requires the principle of least privilege: users and programs should receive only the access necessary for their task.

Environment variables are named values supplied to programs through the operating system or shell. They may specify executable locations, configuration modes, service endpoints, or secret references. A program can use an environment variable such as `DATA_API_URL` without embedding the endpoint in source code. Secrets such as passwords and API tokens should not be committed to a repository or placed directly in notebooks.

Working Through the Command Line

A shell is a command interpreter that accepts text commands and invokes operating-system services or programs. Common shells include Bash, Z shell, and PowerShell. The command line is particularly valuable in data science because it supports repeatable operations, remote work, automation, and composability. A graphical interface is often convenient for inspecting files, but a command written in a script can be reviewed, repeated, scheduled, and tested.

A typical shell session has a current working directory and a prompt. Commands such as the following illustrate basic project navigation:

```bash

pwd

ls

cd project

mkdir -p data/raw data/processed notebooks src tests

cp config.example.env .env

```

The precise options vary across operating systems and shells. `pwd` prints the current directory, `ls` lists contents, `cd` changes location, and `mkdir` creates directories. File operations should be performed carefully, particularly commands that recursively delete or overwrite data. A command-line operation is not inherently safe merely because it is short.

Shell commands can be connected using pipes. For example, a command that produces text can pass its output to another command for filtering or counting. Redirection sends output to a file rather than the terminal. These facilities allow small utilities to be combined into a workflow. Shell scripts extend this idea by storing a sequence of commands in an executable file, often with checks that stop processing when an operation fails.

A useful distinction is that the shell is not the same as a programming language such as Python. The shell is principally concerned with launching programs, connecting their inputs and outputs, and managing the operating-system environment. Python or R is usually better for complex data transformations, statistical operations, and reusable domain logic. Mature projects use each tool for what it does best.

Command-line literacy also supports remote computing. Servers, cloud virtual machines, containers, and high-performance computing systems are frequently administered through secure terminal connections. An analyst who can inspect logs, check disk usage, monitor processes, and run a non-interactive pipeline is less dependent on a graphical desktop and better prepared for production environments.

Environments, Interpreters, and Dependencies

A dependency is a software component required by a project. Dependencies may include a programming-language interpreter, third-party libraries, operating-system packages, database drivers, external services, and system-level utilities. A project can fail on another computer even when its source code is identical because versions, platforms, or configuration values differ.

Package managers automate the discovery, installation, upgrading, and removal of software packages. Python projects commonly use tools such as `pip`, often with a virtual environment, while Conda can manage Python packages and some non-Python libraries. R projects use mechanisms such as `install.packages()` and project-oriented tools such as `renv`. JavaScript projects use package managers such as npm or pnpm. The principle is more important than the particular tool: dependencies should be declared and installed through a controlled process rather than remembered informally.

A virtual environment isolates project-specific packages from the global interpreter. For example:

```bash

python -m venv .venv

source .venv/bin/activate

python -m pip install pandas scikit-learn

python -m pip freeze > requirements.txt

```

On Windows, activation uses the corresponding script in `.venv\Scripts`. The environment prevents one project from changing the packages used by another. However, a virtual environment directory is usually not committed to Git because it is large, platform-dependent, and reproducible from dependency specifications.

Dependency specifications should balance flexibility and stability. A completely unpinned requirement may allow a future release to introduce incompatible behaviour. Overly rigid pinning can make maintenance difficult. Professional projects often distinguish direct dependencies, which the project explicitly uses, from transitive dependencies, which those packages require. Lock files record a resolved dependency set, while a higher-level project file may describe the intended direct dependencies.

Reproducibility has several levels. Computational reproducibility means that the same code and inputs produce the same result under a sufficiently similar environment. Environmental reproducibility means that the software environment itself can be reconstructed. Scientific reproducibility additionally requires clear data provenance, documented methods, and appropriate interpretation. No package manager can compensate for undocumented data revisions or an unexplained manual correction.

IDEs and Notebooks: Complementary Modes of Work

An integrated development environment combines an editor with features such as syntax highlighting, code navigation, debugging, testing, environment selection, and version-control integration. Examples include Visual Studio Code, PyCharm, and RStudio. An IDE is particularly useful when a project contains multiple modules, automated tests, command-line interfaces, configuration files, and deployment code.

A notebook combines executable code, output, explanatory text, equations, and visualisations in a document. Jupyter notebooks are widely used for exploration, teaching, reporting, and communication. Their interactive cells encourage rapid experimentation: an analyst can load a sample, inspect a distribution, fit a model, and revise an assumption without repeatedly running an entire program.

The flexibility of notebooks also creates risks. Cells may be executed out of order, leaving a hidden state that is not represented by the visible sequence. Outputs can become stale, large files can make repositories unwieldy, and sensitive data may be embedded in cell output. A notebook that runs successfully only because a previous session modified variables is not a reliable analytical artefact.

A strong workflow uses notebooks for exploration and explanation while moving stable logic into modules or scripts. The notebook then imports tested functions, documents the analysis, and records selected results. Before sharing a notebook, the author should restart the kernel, execute all cells in order, inspect outputs, remove secrets, and clarify data and environment requirements. Tools such as notebook execution checks and conversion utilities can support this discipline.

The distinction between state and source code is central. A notebook records both, but not always transparently. IDE-based scripts make execution order explicit and integrate naturally with testing. Neither format is universally superior; the appropriate choice depends on whether the task emphasises exploration, communication, reusable software, or operational reliability.

Git as a Model of Change

Git is a distributed version-control system. It records changes to files as a sequence of commits in a repository. Each commit should represent a meaningful, coherent state and should include a message that explains the change. Because the repository is distributed, a local copy contains substantial history and can support many operations without a network connection.

The conceptual model includes a working tree, a staging area, and a repository history. The working tree contains current files. The staging area is a proposed snapshot for the next commit. The repository stores committed snapshots and their relationships. A typical sequence is:

```bash

git init

git status

git add src/ tests/ README.md

git commit -m "Add validated data loading functions"

```

`git status` is one of the most useful commands because it reveals modified, untracked, and staged files. A commit is not a backup of every computer state; it is a recorded project snapshot. Generated files, local environments, temporary data, credentials, and editor-specific files should normally be excluded through `.gitignore`.

Branches provide separate lines of development. A feature branch allows work to proceed without immediately changing the main branch. When the work is ready, it can be merged after review and testing. Conflicts occur when Git cannot automatically reconcile competing edits to the same region of a file. Resolving a conflict requires understanding both changes, editing the file deliberately, testing the result, and committing the resolution.

Remote repositories hosted on platforms such as GitHub, GitLab, or Bitbucket support collaboration, access control, issue tracking, code review, and continuous integration. `git clone` creates a local copy; `git pull` retrieves and integrates remote changes; `git push` publishes local commits. A pull request or merge request provides a formal proposal for incorporating changes into a shared branch.

Version control is more than a historical record. It enables accountability, review, rollback, experimentation, and comparison. In data science, however, Git does not automatically version large datasets, database states, external APIs, model artefacts, or the software environment. These must be addressed through data versioning, immutable storage, metadata, experiment tracking, or documented retrieval procedures where appropriate.

A Reproducible Analytical Workflow

A controlled workflow begins with a clear project boundary. A practical structure might contain `data/`, `src/`, `notebooks/`, `tests/`, `reports/`, and configuration or documentation files. Raw data should be distinguished from processed data, and code should not silently overwrite original inputs. A README should explain the purpose of the project, setup instructions, expected inputs, execution steps, and limitations.

The workflow can be organised as follows:

1. Create an isolated environment and declare dependencies.

2. Establish a repository and an appropriate ignore file.

3. Separate configuration from source code and keep secrets outside version control.

4. Preserve raw inputs where legally and technically possible, recording provenance and acquisition dates.

5. Develop reusable functions and test important transformations.

6. Use notebooks for exploration or presentation, while keeping durable logic in source modules.

7. Commit small, meaningful changes with informative messages.

8. Review changes through branches and pull requests.

9. Automate tests, style checks, and notebook or pipeline validation where practical.

10. Record the data, code version, environment, and parameters associated with important results.

This workflow does not imply that every project needs an elaborate platform. Reproducibility is proportional to risk and importance. A classroom exercise may need a README and a dependency file; a regulated financial or health application may require access controls, audit trails, approval gates, data-retention policies, and reproducible deployment artefacts.

Applied Case: Rebuilding a Customer-Churn Analysis

Background

A fictional telecommunications company, Northstar Mobile, asks an analytics team to identify customers at risk of cancellation. An analyst initially delivers a notebook containing data preparation, model training, and charts. The reported accuracy is promising, but another analyst cannot reproduce the result because the notebook depends on an unrecorded package version, a manually edited CSV file, and cells executed in an undocumented order.

Situation and Analysis

The team identifies four sources of instability. First, the notebook uses a global Python environment that has changed since the original analysis. Second, the input file contains corrections made manually without a transformation log. Third, the model uses a random split but does not record the random seed. Fourth, the notebook output does not show which features were available at the prediction date, creating a possible leakage risk.

The problem is therefore not simply a choice of algorithm. It is a workflow-control problem involving environment, provenance, execution order, and analytical validity.

Approach and Solution

The team creates a repository with `src/`, `notebooks/`, `data/raw/`, `data/processed/`, and `tests/`. The raw extract is stored in controlled storage rather than committed to Git, and its location, acquisition timestamp, checksum, and access requirements are documented. A transformation script converts the raw extract into a processed dataset without modifying the original.

A project-specific virtual environment is created, and direct dependencies are recorded in a project configuration file with a resolved lock file. The data preparation logic is moved from notebook cells into tested functions. The notebook imports these functions and contains only exploratory analysis, model comparison, and explanatory visualisations. A fixed seed is recorded as a parameter, and a feature-availability review removes variables that would not have been known at prediction time.

The team uses a feature branch and opens a merge request. Automated checks run unit tests, style checks, and a clean notebook execution. Reviewers inspect both code and analytical assumptions. Credentials are supplied through protected environment variables rather than committed files.

Outcome

The revised workflow produces a result that a second analyst can reproduce from the documented inputs and environment. The measured performance is lower than the original result because leakage has been removed, but the estimate is more credible. The company gains a maintainable pipeline rather than a single opaque notebook.

Learning Points

The case demonstrates that reproducibility is an organisational and technical property. Version control protects code history but does not itself guarantee valid data, correct execution, or ethical modelling. Environment isolation, data provenance, testing, review, and explicit assumptions must work together.

Common Risks, Limitations, and Professional Responsibilities

A frequent mistake is to treat `git add .` as a complete version-control strategy. It may stage secrets, large files, generated outputs, or personal configuration. Analysts should inspect the staging area before committing and use secret-scanning controls where available. If a secret is committed, deleting it in a later commit may not be sufficient; the credential should be revoked and repository history handled appropriately.

Another risk is dependency drift. Installing packages directly into a global environment can break unrelated projects. Conversely, copying an environment directory between operating systems is unreliable. Dependency declarations, automated setup instructions, and periodic rebuilding provide stronger evidence that an environment is reproducible.

Poor branching practices can also reduce quality. Long-lived branches accumulate conflicts and make review difficult. Tiny commits with vague messages such as “changes” provide little explanatory value. A useful commit is focused, tested, and understandable to another team member.

Data science repositories can contain personal, confidential, proprietary, or regulated information. Access should be authorised, data minimised, and retention requirements observed. Personally identifiable information should not be placed in notebooks or test fixtures without appropriate controls. Analysts must consider whether publishing code, sample data, model outputs, or error logs could reveal protected information.

Open-source software introduces licensing obligations. Teams should track licences and comply with conditions concerning attribution, redistribution, or source availability. Security updates should be monitored, and unmaintained or suspicious packages should not be adopted merely because they are convenient.

Current practice increasingly includes continuous integration, container images, infrastructure-as-code, data and model versioning, and machine-readable project metadata. Containers can improve portability by packaging software and system libraries, although they do not eliminate the need to manage data, secrets, hardware differences, or security updates. Cloud development environments and remote notebooks reduce local setup effort but require careful control of permissions, costs, data location, and provider dependence.

Best Practices for Controlled Analytical Work

The following principles provide a practical standard:

• Prefer project-relative paths and configuration-driven execution.
• Keep raw data immutable and document transformations.
• Use isolated environments and declare dependencies.
• Separate exploratory notebooks from reusable production logic.
• Restart and rerun notebooks before publication.
• Write tests for data transformations, boundary conditions, and critical calculations.
• Commit coherent changes and review them through a shared repository.
• Never commit passwords, tokens, private keys, or unnecessary personal data.
• Record code version, data version, parameters, and environment for important results.
• Automate repeatable checks rather than relying solely on personal discipline.
• Match workflow complexity to the risk, scale, and regulatory context of the project.

Chapter Summary

Development environments provide the technical conditions in which analytical work is created, executed, and maintained. Operating systems manage files, processes, permissions, and environment variables; shells provide a scriptable interface for navigation, execution, and automation. Package managers and virtual environments control dependencies and reduce conflicts. IDEs support structured software development, while notebooks support interactive exploration and communication but require discipline because of hidden state and stale outputs.

Git records coherent changes and enables branching, review, collaboration, and rollback. Remote repository platforms extend these capabilities through access control, issues, merge requests, and automated checks. A reproducible workflow connects environment management, data provenance, testing, documentation, version control, and security. The central professional lesson is that a credible analytical result is not only a number or visualisation; it is an inspectable outcome supported by a controlled chain of code, data, assumptions, and execution.

Key Terms

Operating system: Software that manages computer hardware, files, processes, users, and application execution.

Filesystem: The operating system structure used to organise and access files and directories.

Shell: A command interpreter that executes commands and connects programs to operating-system services.

Process: A running instance of a program with an execution context and allocated resources.

Environment variable: A named configuration value made available to programs through the operating environment.

Dependency: A software component required by a project to operate correctly.

Package manager: A tool that installs, updates, removes, and records software packages.

Virtual environment: An isolated software environment containing project-specific interpreter packages.

IDE: An integrated development environment combining code editing with tools such as debugging, testing, and navigation.

Notebook: An interactive document combining executable code, outputs, text, and visualisations.

Reproducibility: The ability to obtain consistent results by repeating documented computational procedures with specified inputs and conditions.

Repository: A storage location containing project files and version-control history.

Commit: A recorded snapshot of selected project changes in a version-control repository.

Branch: An independent line of development within a version-control history.

Merge: The process of incorporating changes from one branch into another.

Remote: A repository location accessed through a network for collaboration or backup.

Pull request: A formal request to review and merge proposed changes into a shared branch.

Data provenance: Information describing the origin, history, transformations, and ownership of data.

Continuous integration: Automated execution of checks when changes are proposed or incorporated.

Least privilege: The security principle of granting only the access required for a task.

Review Questions

Short Answer Questions

1. What is the difference between an absolute path and a relative path?

2. Why are virtual environments useful in data-science projects?

3. What roles do the working tree, staging area, and repository play in Git?

4. Identify two risks associated with interactive notebooks.

5. Why should secrets not be committed to a repository?

Long Answer Questions

1. Explain how operating-system concepts influence the execution and security of analytical workflows.

2. Compare the appropriate uses of an IDE, a shell, and a notebook in a data-science project.

3. Describe how package managers, dependency declarations, and lock files contribute to reproducibility.

4. Explain a Git-based collaborative workflow from creating a branch to merging reviewed changes.

5. Discuss why version control alone cannot guarantee reproducible data-science results.

Analytical / Application Questions

1. A notebook produces a different model score when run by a colleague. Diagnose possible causes and propose a controlled investigation.

2. Design a repository structure and workflow for a project involving confidential customer data and a regulated reporting requirement.

3. A team has committed large datasets, generated plots, and an API token to its repository. Recommend immediate corrective actions and longer-term preventive practices.

22. Multiple Choice Questions with Answers

Question 1

Which practice most directly improves the portability of a data-science project across computers?

A. Using absolute paths to all input files

B. Using project-relative paths and declaring dependencies

C. Saving only the final model accuracy

D. Running all code from a global environment

Correct Answer: B. Using project-relative paths and declaring dependencies

Question 2

What is the primary purpose of a virtual environment?

A. To replace the operating system

B. To encrypt all project data

C. To isolate project-specific software dependencies

D. To provide a remote Git repository

Correct Answer: C. To isolate project-specific software dependencies

Question 3

Why can a notebook produce misleading results even when all visible cells appear correct?

A. Notebooks cannot execute Python code

B. Cells may have been executed out of order, creating hidden state

C. Notebooks automatically delete source code

D. Notebook outputs are always generated from remote data

Correct Answer: B. Cells may have been executed out of order, creating hidden state

Question 4

In Git, what is the function of the staging area?

A. It stores the operating system kernel

B. It identifies the files proposed for the next commit

C. It permanently publishes changes to a remote

D. It resolves every merge conflict automatically

Correct Answer: B. It identifies the files proposed for the next commit

Question 5

A team wants reviewers to inspect a feature before it changes the main branch. Which Git practice is most appropriate?

A. Editing the main branch directly

B. Deleting the repository history

C. Using a feature branch and a pull request

D. Ignoring all untracked files

Correct Answer: C. Using a feature branch and a pull request

Question 6

Which item should generally be excluded from a Git repository?

A. A README containing setup instructions

B. A tested source module

C. A project dependency specification

D. An API token stored in a configuration file

Correct Answer: D. An API token stored in a configuration file

Question 7

A churn model performs unusually well because it uses a variable recorded after a customer cancelled. What problem does this illustrate?

A. Data leakage

B. Filesystem fragmentation

C. Branch divergence

D. Package isolation

Correct Answer: A. Data leakage

Question 8

Which statement best describes a useful commit?

A. It contains every file changed over several months

B. It represents a coherent, understandable project change

C. It includes all local environments and temporary outputs

D. It has a message unrelated to the files changed

Correct Answer: B. It represents a coherent, understandable project change

Question 9

What should a team record to support reproducibility of an important analytical result?

A. Only the final chart

B. Only the analyst's username

C. The code version, data version, parameters, and environment

D. Only the number of notebook cells

Correct Answer: C. The code version, data version, parameters, and environment

Question 10

Which principle recommends granting a program only the permissions required for its task?

A. Maximum availability

B. Least privilege

C. Branch inheritance

D. Transitive installation

Correct Answer: B. Least privilege