Programming and Computational Tools

Introduction

Data science depends on more than statistical knowledge. A data scientist must also be able to express ideas as executable instructions, inspect and transform data, use specialist libraries, document decisions, and reproduce results. Programming provides the language for these activities, while computational tools provide the working environment.

This chapter introduces Python-based workflows, although most principles also apply to R. The emphasis is not on becoming a software engineer, but on developing reliable programming habits for data work. Python is widely used because it combines general-purpose programming with extensive data-science libraries such as NumPy, pandas, Matplotlib and scikit-learn.

A typical data-science workflow is iterative rather than strictly linear:

1. Define the question and success criteria.
2. Obtain and inspect data.
3. Clean and transform the data.
4. Explore patterns and relationships.
5. Build and evaluate an analysis or model.
6. Communicate results.
7. Record the process so that it can be repeated and reviewed.

A notebook is especially useful during exploration because it combines executable code, output, visualisations and explanatory text. However, notebooks can also become disorganised. Professional practice therefore requires clear structure, meaningful names, controlled dependencies, documentation and version control.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the role of programming in data science;
• use Python variables, common data structures, operators and control flow;
• define and use functions;
• import and apply libraries appropriately;
• work effectively in computational notebooks;
• organise a small, reproducible data-analysis project;
• document assumptions, transformations and results;
• explain the purpose of version control and apply basic Git practices;
• identify common sources of error and limitations in computational analysis.

1. Programming as a Tool for Data Science

A program is a sequence of instructions that a computer executes. In data science, programs may calculate summary statistics, reshape tables, generate graphs, train models or automate repeated tasks. Programming is valuable because it is precise, scalable and repeatable.

A spreadsheet may be convenient for a small table, but code is generally preferable when an analysis must be repeated, audited, shared or applied to larger datasets. Code also makes transformations explicit. For example, the instruction sales["revenue"] = sales["units"] * sales["price"] records exactly how revenue was calculated.

Programming errors are not always syntax errors. A syntax error violates the rules of the language and usually prevents execution. A runtime error occurs while a program is running, such as attempting to divide by zero. A logical error produces an apparently valid result that is wrong because the instructions do not represent the intended reasoning. Logical errors are particularly dangerous in data science because they may remain unnoticed.

2. Computational Environments and Notebooks

A computational environment includes the programming language, interpreter, installed packages, files, operating system and configuration used to execute an analysis. Common environments include Jupyter Notebook, JupyterLab, Google Colab and integrated development environments such as VS Code.

A notebook is divided into cells. Code cells contain executable instructions; Markdown cells contain headings, explanations, equations and links. Outputs may include text, tables, error messages and charts. The combination of narrative and computation supports an exploratory workflow.

A notebook has a state: variables created in one cell remain available to later cells. This is convenient but can create hidden dependencies. For example, a later cell may work only because an earlier cell was executed, even though the notebook’s visible order suggests otherwise. To test a notebook, restart its kernel and run all cells from beginning to end. This checks whether the documented sequence is sufficient.

Good notebook structure commonly includes:

• title, author, date and purpose;
• data sources and licensing information;
• environment and package requirements;
• imports;
• data loading;
• validation and cleaning;
• exploratory analysis;
• results and interpretation;
• limitations and conclusions.

Code should be separated from commentary, and long notebooks should be divided into logical sections. Outputs that are no longer relevant should be cleared before publication. Sensitive data should never be embedded in a notebook that will be shared publicly.

3. Python Foundations

3.1 Variables and data types

A variable is a name referring to a value. Assignment uses the equals sign:

age = 21
course = "Data Science"
passed = True

Here, age is an integer, course is a string and passed is a Boolean value. Python also supports floating-point numbers, such as 72.5, and a special value called None, which represents the absence of a value.

Variable names should communicate meaning. average_monthly_revenue is preferable to x, unless x has a clearly defined mathematical role. Python is case-sensitive: Sales, sales and SALES are different names.

Common operators include:

The operator / performs ordinary division, while // performs floor division. The expression 7 // 2 returns 3, whereas 7 / 2 returns 3.5. The operator % returns a remainder, so 7 % 2 returns 1.

3.2 Data structures

Data structures organise values for efficient access and manipulation.

A list is ordered and mutable:

temperatures = [18.2, 19.0, 17.5, 21.1]
temperatures.append(20.4)

A tuple is ordered but normally treated as immutable:

location = (51.5, -0.1)

A dictionary stores key–value pairs:

student = {"name": "Amina", "mark": 78, "complete": True}

A set stores unique values and is useful for membership tests or removing duplicates:

departments = {"sales", "finance", "sales"}

Indexing begins at zero. If values = [10, 20, 30], then values[0] is 10. Slicing values[1:] returns [20, 30]. Incorrect assumptions about indexing are a common source of errors when moving between tools that use different conventions.

3.3 Control flow

Conditional statements allow a program to choose between alternatives:

if mark >= 70:
    result = "distinction"
elif mark >= 40:
    result = "pass"
else:
    result = "fail"

The indentation is part of Python syntax. A for loop repeats an operation over a sequence:

squares = []
for number in [1, 2, 3, 4]:
    squares.append(number ** 2)

A while loop continues while a condition is true. It should be used carefully because an incorrectly designed condition can create an infinite loop.

List comprehensions provide a compact form for simple transformations:

squares = [number ** 2 for number in [1, 2, 3, 4]]

Readability is more important than compactness. A conventional loop is preferable when the logic is complex.

4. Functions and Modular Thinking

A function is a named, reusable unit of computation. It accepts inputs, performs operations and may return an output.

def percentage_change(old_value, new_value):
    if old_value == 0:
        raise ValueError("old_value must not be zero")
    return (new_value - old_value) / old_value * 100

change = percentage_change(80, 100)

The function returns 25.0, because the change is 20 relative to an original value of 80. The percentage-change formula is:

The formula assumes that the old value is non-zero and that both values use the same units. If the old value is negative or represents a quantity for which percentage change is not meaningful, interpretation requires care.

Functions improve consistency, testing and maintenance. A good function generally has one clear purpose, meaningful parameter names, a predictable return value and documentation. A docstring records the function’s purpose and assumptions:

def mean(values):
    """Return the arithmetic mean of a non-empty sequence."""
    if len(values) == 0:
        raise ValueError("values must not be empty")
    return sum(values) / len(values)

The arithmetic mean of values x₁, x₂, …, xₙ is:

Here, x̄ is the sample mean, n is the number of observations and xᵢ is observation i. The observations must be numeric, and the result has the same units as the observations. The mean can be strongly affected by extreme values.

5. Libraries for Data Science

A library is reusable code written by others. Importing a library provides tested functionality without requiring the user to implement every operation from first principles.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

NumPy provides arrays and numerical operations. pandas provides labelled one-dimensional Series and two-dimensional DataFrame objects. Matplotlib supports visualisation. scikit-learn provides many machine-learning algorithms and evaluation tools.

A DataFrame can be created directly:

import pandas as pd

records = {
    "product": ["A", "B", "C"],
    "units": [10, 7, 12],
    "price": [5.0, 8.0, 4.5]
}
sales = pd.DataFrame(records)
sales["revenue"] = sales["units"] * sales["price"]

The resulting revenue values are 50.0, 56.0 and 54.0. Each row represents a product record, while each column represents a variable. Before analysis, inspect structure and quality:

sales.head()
sales.info()
sales.describe()
sales.isna().sum()

These commands help identify data types, missing values, unusual ranges and possible errors. A summary is not a substitute for understanding the data dictionary. A column labelled age may contain years, age bands or an encoded category; its meaning must be established from documentation.

Filtering and grouping are central operations:

high_value = sales[sales["revenue"] > 52]
summary = sales.groupby("product", as_index=False)["revenue"].sum()

Libraries have versions and dependencies. A program that works on one computer may fail on another if package versions differ. Record the Python version and package versions, for example in a requirements.txt file or environment configuration. Do not install packages indiscriminately; use trusted sources and review licences and security implications.

6. Reproducible Workflows

Reproducibility means that another person can follow the recorded process and obtain the same result, given the same data, code, environment and relevant random seed. Reproducibility does not mean that every future dataset will produce the same findings.

A reproducible project separates inputs, processing and outputs. A practical directory structure is:

project/
├── README.md
├── data/
│   ├── raw/
│   └── processed/
├── notebooks/
├── src/
├── reports/
└── requirements.txt

Raw data should be preserved rather than overwritten. Derived data should be generated by documented code. File paths should be portable; hard-coded paths such as C:\Users\Name\Desktop\file.csv make collaboration difficult.

A reproducible workflow should record:

• the research or business question;
• the source, date and licence of each dataset;
• definitions of variables and units;
• cleaning rules and excluded records;
• package and software versions;
• random seeds where stochastic methods are used;
• output files and evaluation criteria;
• limitations and unresolved uncertainties.

Randomised algorithms may produce different outputs on different runs. Setting a seed makes a particular computational run repeatable:

import numpy as np
rng = np.random.default_rng(42)
sample = rng.choice([1, 2, 3, 4, 5], size=3, replace=False)

A seed does not make a method scientifically valid, nor does it guarantee identical results across all software versions and hardware. It simply controls a pseudo-random number generator under specified conditions.

7. Documentation and Code Quality

Documentation serves both present and future readers. Comments should explain why a non-obvious decision was made, not restate obvious syntax. For example, # remove cancelled transactions before calculating revenue is more useful than # subtract values.

Important documentation practices include:

• use consistent naming conventions;
• keep functions short and focused;
• validate assumptions early;
• avoid unexplained magic numbers;
• state units and time periods;
• distinguish missing, zero and not-applicable values;
• include citations for external methods or datasets;
• report limitations honestly.

Testing can be informal or systematic. A useful test checks expected behaviour for normal, boundary and invalid inputs. For the percentage_change function, tests should include a normal calculation, an old value of zero and values with decimal precision. Assertions can expose errors early:

assert percentage_change(80, 100) == 25.0

Exact equality may be unsuitable for floating-point results because computers represent many decimal fractions approximately. For numerical comparisons, a tolerance-based approach is often more appropriate, such as checking whether the absolute difference is below a small threshold.

8. Basic Version Control with Git

Version control records changes to files over time. Git allows a student or team to identify who changed a file, compare versions, restore earlier work and collaborate through branches and repositories.

A basic workflow is:

git init
git status
git add notebook.ipynb
git commit -m "Add initial data exploration"

git init creates a repository, git status reports changes, git add stages selected files and git commit records a snapshot. A remote platform such as GitHub or GitLab can support collaboration, but a remote repository is not automatically private or secure.

Commit messages should describe the change, for example Fix missing-value filter, rather than Update. Commits should be reasonably small and logically coherent. A .gitignore file should exclude temporary files, local environments, large generated outputs and secrets. API keys, passwords and personal data must not be committed.

Notebooks present a special challenge because their JSON structure can make line-by-line comparisons difficult. Teams may use clear notebook practices, pair notebooks with scripts, or convert notebooks into executable reports. The essential principle is that version control should preserve both the analysis and the ability to understand how it changed.

9. Applied Case: Reproducible Analysis of Retail Transactions

The following is an illustrative case. A small retailer wants to identify which products generated the most revenue during one month. The transaction file contains date, product, units, unit_price and status.

Background and problem

Management has received a total-revenue figure but cannot reproduce it. Some records are cancelled, several prices are missing, and the analyst’s calculations exist only in a personal spreadsheet. The problem is to create a transparent workflow that calculates revenue and produces a product summary.

Analysis and approach

First, the analyst records the data source, definitions and assumptions. Cancelled transactions are excluded because they do not represent completed sales. Rows with missing units or prices are not silently converted to zero; they are inspected and reported. Revenue is defined as:

Both quantities must refer to the same transaction, and the price must use a consistent currency. A simplified implementation is:

import pandas as pd

raw = pd.read_csv("data/raw/transactions.csv")

required = {"date", "product", "units", "unit_price", "status"}
missing_columns = required - set(raw.columns)
if missing_columns:
    raise ValueError(f"Missing columns: {missing_columns}")

valid = raw.loc[
    (raw["status"] == "completed") &
    raw["units"].notna() &
    raw["unit_price"].notna()
].copy()

valid["revenue"] = valid["units"] * valid["unit_price"]
product_summary = (
    valid.groupby("product", as_index=False)["revenue"]
         .sum()
         .sort_values("revenue", ascending=False)
)

The analyst then compares the number of raw records with the number retained, examines excluded records and checks whether units and prices fall within plausible ranges. The summary is saved as a derived output, while the raw file remains unchanged.

Outcome

Management receives a ranked product table and a short report showing the exclusion rules, total retained transactions and known limitations. A README explains how to recreate the result. A Git repository records the notebook, code, documentation and dependency information.

Learning points

The important achievement is not merely calculating revenue. It is establishing a defensible chain from source data to conclusion. Explicit assumptions prevent hidden decisions, validation reduces the risk of silent errors, and version control provides an audit trail. If the retailer later changes its definition of completed sales, the code and documentation can be updated transparently.

10. Challenges, Ethics and Governance

Computational tools can create an appearance of objectivity even when inputs or assumptions are flawed. Common challenges include poor data quality, missing documentation, incompatible software versions, accidental data leakage and overreliance on automated output.

Privacy is especially important. Data should be collected and used for legitimate purposes, access should be restricted, and personal information should be minimised where possible. Depending on jurisdiction and context, laws such as the General Data Protection Regulation may impose requirements concerning lawful processing, transparency, retention and individual rights. Students should consult applicable institutional and national policies rather than assuming that publicly accessible data is unrestricted for every use.

Security practices include protecting credentials, using approved storage, updating dependencies, checking package provenance and avoiding execution of untrusted code. Reproducibility must not be pursued by exposing confidential data. A secure synthetic or anonymised dataset may be preferable for teaching and sharing.

Artificial intelligence coding assistants can help explain syntax, generate draft functions or suggest tests. Their output must be reviewed because generated code can contain logical errors, insecure practices, licensing concerns or fabricated explanations. The analyst remains responsible for the final program and its consequences. AI assistance should be disclosed when institutional policy requires it, and confidential data should not be entered into an unapproved service.

11. Best-Practice Checklist

Before submitting a computational analysis, ask:

• Can another person identify the data source and reproduce the main result?
• Are raw data and processed data clearly distinguished?
• Are variable names, units and assumptions documented?
• Does the notebook run successfully from a fresh environment?
• Have missing values, duplicates and implausible observations been examined?
• Are code, outputs and narrative consistent?
• Are package versions and random seeds recorded where relevant?
• Have sensitive information and credentials been excluded?
• Are limitations and ethical considerations stated?
• Does version control show meaningful, understandable changes?

Chapter Summary

Programming enables data scientists to express analyses precisely, automate repeated work and create transparent transformations. Python variables hold values, data structures organise information, control flow directs execution and functions package reusable logic. Libraries such as NumPy, pandas, Matplotlib and scikit-learn extend the language for numerical analysis, tabular data, visualisation and modelling.

Notebooks combine code, output and explanation, but their stateful nature requires disciplined execution and organisation. Reproducible workflows preserve raw data, document sources and assumptions, control environments and record transformations. Documentation, validation and testing help distinguish reliable analysis from plausible-looking error. Git provides basic version control through repositories, staging and commits, while secure data handling and ethical judgement remain essential. Computational tools support professional data science only when they are used critically, transparently and responsibly.

Key Terms

Variable: A named reference to a value in a program.

Data type: The category of a value, such as integer, string, Boolean or floating point.

List: An ordered, mutable Python collection.

Dictionary: A collection of key–value pairs.

Function: A reusable block of code that performs a defined task.

Library: Reusable software providing functions, classes or tools.

Package: A distributable collection of related software modules.

Notebook: An interactive document containing code, outputs and explanatory text.

Kernel: The computational process that executes notebook code.

DataFrame: A labelled, two-dimensional tabular data structure in pandas.

Reproducibility: The ability to recreate an analysis under the same documented conditions.

Dependency: External software required by a program.

Data validation: Checking whether data conforms to expected structure, type and range.

Docstring: A documentation string associated with a Python function, class or module.

Version control: A system for recording and managing changes to files.

Repository: A directory containing files and their version history.

Commit: A recorded snapshot of staged changes in Git.

Raw data: Data preserved in its original or source form.

Data leakage: Inappropriate use of information that should not be available for an analysis or model.

Application programming interface (API): A defined way for software systems to communicate.

Review Questions

Short Answer Questions

1. What is the difference between a syntax error and a logical error?
2. Why should a notebook be restarted and run from beginning to end before sharing?
3. Name three Python data structures and give one use for each.
4. What information should a reproducible data-science project record?
5. What is the purpose of a Git commit?

Long Answer Questions

1. Explain how variables, data structures, control flow and functions work together in a Python data-analysis program.
2. Describe a well-organised notebook workflow from data loading to reporting, including quality checks.
3. Explain why package versions, file structure and random seeds may affect reproducibility.
4. Compare exploratory notebook work with a production-quality analytical script or application.
5. Discuss the ethical and security risks involved in sharing computational analyses containing personal or confidential data.

Analytical / Application Questions

1. A transaction dataset contains completed and cancelled orders, missing prices and duplicate transaction identifiers. Design a documented processing workflow, explaining which checks and decisions should occur before revenue is calculated.
2. A notebook produces a correct result only when cells are run in an unusual order. Diagnose the likely causes and propose changes that would make the notebook reproducible.
3. A team member commits an API key and a file containing customer names to a public repository. Analyse the risks and describe immediate and long-term corrective actions.

Multiple Choice Questions with Answers

Question 1

Which Python data structure stores key–value pairs?

A. List

B. Dictionary

C. Tuple

D. Set

Correct Answer: B. Dictionary

Question 2

What is the main purpose of a function in a data-analysis program?

A. To make every value global

B. To remove the need for documentation

C. To package reusable logic for a defined task

D. To prevent all runtime errors

Correct Answer: C. To package reusable logic for a defined task

Question 3

Which pandas expression selects rows where revenue is greater than 100?

A. sales[sales["revenue"] > 100]

B. sales.select("revenue" > 100)

C. sales["revenue" = 100]

D. sales.where(revenue greater 100)

Correct Answer: A. sales[sales["revenue"] > 100]

Question 4

Why should raw data normally be preserved unchanged?

A. It makes files smaller

B. It provides an original reference for validation and reproduction

C. It automatically removes missing values

D. It prevents all privacy risks

Correct Answer: B. It provides an original reference for validation and reproduction

Question 5

What does git add do in the basic Git workflow?

A. Deletes previous commits

B. Stages selected changes for a commit

C. Publishes files to every remote repository

D. Runs Python code

Correct Answer: B. Stages selected changes for a commit

Question 6

If units are 8 and unit price is 12.50, what is revenue using revenue = units × unit price?

A. 20.50

B. 96.00

C. 100.00

D. 156.25

Correct Answer: C. 100.00

Question 7

Which action best tests whether a notebook has hidden execution dependencies?

A. Change all variable names

B. Restart the kernel and run all cells in order

C. Delete all Markdown cells

D. Run only the final cell

Correct Answer: B. Restart the kernel and run all cells in order

Question 8

Why is setting a random seed useful?

A. It guarantees a method is unbiased

B. It controls pseudo-random generation for repeatable runs

C. It removes the need for training data

D. It improves internet connectivity

Correct Answer: B. It controls pseudo-random generation for repeatable runs

Question 9

Which practice is most appropriate for an API key?

A. Commit it to a public repository for convenience

B. Place it in notebook output

C. Store it securely outside version-controlled source files

D. Add it to a dataset README

Correct Answer: C. Store it securely outside version-controlled source files

Question 10

What does the arithmetic mean calculation require?

A. Multiplying all observations by the sample size

B. Adding observations and dividing by their count

C. Sorting observations and selecting the middle value

D. Removing every extreme observation

Correct Answer: B. Adding observations and dividing by their count