Python for Statistics, Machine Learning and AI

Introduction and context

Python is widely used in data science because one language can support data preparation, statistical analysis, machine learning, visualisation and interaction with artificial-intelligence systems. A professional workflow is not simply a sequence of library calls. It is a process of defining a question, understanding the data-generating context, preparing data without contamination, selecting an appropriate model, validating its performance and communicating uncertainty and limitations.

This chapter introduces practical statistical analysis with Python and the foundations of scikit-learn workflows. It also examines overfitting, data leakage, algorithmic bias and the responsible use of generative AI coding assistants. The emphasis is on reproducible and defensible work suitable for a diploma-level data scientist.

Learning objectives

By the end of this chapter, you should be able to:

• calculate and interpret introductory descriptive statistics;
• distinguish population quantities from sample estimates;
• construct a reproducible scikit-learn preprocessing and modelling pipeline;
• explain training, validation, testing and cross-validation;
• select suitable evaluation metrics for regression and classification;
• identify overfitting, leakage and important sources of bias;
• interpret model results in the context of a real decision;
• use generative AI coding tools critically, securely and transparently.

1. Statistical foundations in Python

1.1 Variables, observations and populations

A dataset contains observations, usually represented by rows, and variables or features, usually represented by columns. A target variable is the quantity to be predicted or explained. For example, in a house-price dataset, floor area and location may be features, while sale price is the target.

A population is the complete set of units of interest. A sample is the subset actually observed. A population mean is commonly denoted by μ, whereas a sample mean is denoted by x̄. For observations x₁, x₂, …, xₙ, the sample mean is:

Here, n is the sample size and xᵢ is observation i. The formula applies to quantitative observations for which an arithmetic average is meaningful. It has the same units as xᵢ.

The sample variance is:

The denominator n − 1 is used when s² estimates population variance from a sample under the usual independent-sampling assumptions. The sample standard deviation is s = √s², and therefore has the same units as the original variable. A variance has squared units.

Python provides several levels of support:

import numpy as np
import pandas as pd

scores = pd.Series([62, 70, 70, 81, 92])
print(scores.mean())       # 75.0
print(scores.median())     # 70.0
print(scores.std(ddof=1))  # sample standard deviation
print(scores.describe())

Pandas uses sample standard deviation by default for a Series because its default is ddof=1. NumPy's np.std defaults to ddof=0, which calculates the population-style divisor. The analyst must choose deliberately rather than treating these conventions as interchangeable.

1.2 Centre, spread and shape

The mean is sensitive to extreme values. The median is the middle ordered value and is more resistant to outliers. The mode is the most frequent value and is particularly useful for categorical variables. Measures of spread include the range, interquartile range (IQR), variance and standard deviation.

The IQR is:

where Q₁ and Q₃ are the first and third quartiles. A common exploratory rule flags values below Q₁ − 1.5IQR or above Q₃ + 1.5IQR as potential outliers. This is a screening convention, not proof that an observation is erroneous. An unusual value may represent a valid and important case.

For two quantitative variables, covariance indicates whether they tend to move together, while correlation standardises covariance:

Here, cov(X,Y) is sample covariance and sₓ and sᵧ are sample standard deviations. Pearson's r lies between −1 and 1 under its definition, but correlation does not establish causation. It can also be misleading for nonlinear relationships, mixtures of groups and influential observations.

numeric = df.select_dtypes(include="number")
print(numeric.describe().T)
print(numeric.corr(numeric_only=True))

1.3 Probability and uncertainty

A probability describes uncertainty about an event. A conditional probability is the probability of A given B:

In applied data science, uncertainty arises from sampling, measurement, model assumptions and changing environments. A point estimate alone is therefore incomplete. Confidence intervals, prediction intervals, resampling and sensitivity analysis can communicate uncertainty, although each depends on assumptions.

A 95% confidence interval is not the statement that there is a 95% probability that a fixed parameter lies in the calculated interval. Under repeated sampling and the method's assumptions, approximately 95% of intervals constructed by that procedure would contain the parameter. This distinction is important when reporting results to decision-makers.

2. From data question to scikit-learn workflow

A robust workflow separates the target from the predictors, splits data before learning transformations, fits preprocessing only on training data and evaluates on untouched data. Scikit-learn represents most supervised estimators with fit and predict methods.

from sklearn.model_selection import train_test_split

X = df.drop(columns="target")
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42
)

random_state supports reproducibility, but it does not guarantee that a model is generally valid. A single split can be unusually easy or difficult. If observations are time ordered, a random split may be inappropriate because it can allow future information to influence training. Time-based evaluation is usually more realistic in forecasting.

2.1 Preprocessing with pipelines

Common preprocessing operations include missing-value imputation, scaling numerical variables and one-hot encoding categorical variables. Standardisation transforms a value x into:

where μ_train and σ_train are calculated from the training data only. If σ_train is zero, the variable has no training variation and requires special handling. Standardisation is important for distance-based and regularised models, but tree-based models generally do not require it.

A ColumnTransformer applies different operations to different columns, and a Pipeline ensures that transformations are fitted as part of model training.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression

numeric_features = ["age", "income"]
categorical_features = ["region", "plan"]

numeric_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

preprocess = ColumnTransformer([
    ("num", numeric_pipe, numeric_features),
    ("cat", categorical_pipe, categorical_features)
])

model = Pipeline([
    ("preprocess", preprocess),
    ("classifier", LogisticRegression(max_iter=1000))
])

The handle_unknown="ignore" option prevents an error when a category appears in test data but was absent during training. It does not solve all problems of changing data distributions; it only provides a safe encoding behaviour.

3. Supervised learning models

3.1 Regression

Regression predicts a numerical outcome such as cost, demand or temperature. Ordinary least squares linear regression represents a conditional mean using:

where ŷ is the predicted response, β₀ is the intercept, βⱼ is the coefficient for feature xⱼ and p is the number of features. The interpretation of βⱼ as a change in the expected response per unit change in xⱼ assumes the other included features are held constant and the model form is appropriate. It is not automatically a causal effect.

Mean absolute error (MAE) is:

It is expressed in the target's units. Mean squared error (MSE) is:

MSE gives greater weight to large errors and has squared target units. Root mean squared error is RMSE = √MSE and returns to the target's units. R², the coefficient of determination, is:

The denominator compares errors with a baseline that always predicts the training or evaluation mean ȳ, depending on the context. R² can be negative on test data when the model performs worse than that baseline; it is not inherently a percentage accuracy measure.

3.2 Classification

Classification predicts categories such as fraud/not fraud or approved/rejected. A binary confusion matrix contains true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN).

Accuracy is:

Precision and recall are:

Precision answers, “Of predicted positives, how many were positive?” Recall answers, “Of actual positives, how many were found?” When false positives and false negatives have different consequences, accuracy alone is inadequate. F1 is the harmonic mean of precision and recall:

These ratios require non-zero denominators. If a classifier produces no predicted positives, precision needs an explicit zero-division policy.

The classification threshold changes the trade-off between precision and recall. A probability estimate of 0.51 is not intrinsically a positive decision; the threshold should reflect costs, capacity and risk.

from sklearn.metrics import classification_report, confusion_matrix

model.fit(X_train, y_train)
predictions = model.predict(X_test)
print(confusion_matrix(y_test, predictions))
print(classification_report(y_test, predictions, zero_division=0))

4. Validation, cross-validation and model selection

The training set is used to estimate model parameters. A validation set supports model choice and hyperparameter tuning. A test set is reserved for a final, relatively unbiased estimate after decisions are complete. Reusing the test set repeatedly turns it into an informal validation set and makes performance appear more certain than it is.

In k-fold cross-validation, the training data are partitioned into k folds. Each fold is used once for validation while the remaining k − 1 folds are used for fitting. The reported score is commonly the mean of the k validation scores:

The variation among fold scores is also informative. Cross-validation is not automatically suitable for clustered, repeated-measures or time-series data. Grouped or time-series splitters may be required.

from sklearn.model_selection import cross_validate, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(model, X, y, cv=cv, scoring=["accuracy", "f1"])
print(results["test_accuracy"].mean())
print(results["test_f1"].mean())

For imbalanced classification, stratification attempts to preserve class proportions in folds. It does not correct imbalance; it only changes how observations are allocated.

5. Overfitting, leakage and bias

5.1 Overfitting

Overfitting occurs when a model captures noise or accidental patterns in training data and performs poorly on new data. A large gap between training and validation performance is a warning sign. Causes include excessive model complexity, too many features relative to observations, repeated tuning and noisy labels.

Possible responses include simpler models, regularisation, more representative data, feature selection, early stopping where applicable and stronger validation design. Underfitting occurs when a model is too simple to represent important structure. The objective is not the highest training score; it is reliable performance on relevant future cases.

5.2 Data leakage

Leakage occurs when information unavailable at prediction time enters model fitting or evaluation. Examples include:

• scaling the entire dataset before the train-test split;
• imputing missing values using test observations;
• using a post-outcome variable as a predictor;
• allowing records from the same person to appear in both training and test sets;
• selecting features using the full dataset before cross-validation.

Pipelines reduce preprocessing leakage by fitting transformations inside each training fold. They cannot identify a business-process leak, such as a variable recorded only after a loan has been approved. Leakage is a question about information availability and time, not merely Python syntax.

5.3 Bias and fairness

Bias can enter through problem definition, sampling, measurement, labels, feature choices, deployment and feedback loops. A model may achieve good average performance while producing substantially different error rates across groups.

Fairness analysis should consider the decision's context, protected characteristics, legal requirements and potential harms. Useful diagnostics may include group-specific selection rates, false-positive rates, false-negative rates and calibration. These measures can conflict, particularly when base rates differ, so there is no universal fairness metric that resolves every ethical decision.

Sensitive attributes should not automatically be removed. They may be needed to audit disparities, and proxy variables can preserve discriminatory patterns even after removal. Access must nevertheless be controlled and justified.

6. A complete applied case: predicting customer attrition

Background and problem

This illustrative case concerns a subscription service that wants to identify customers at risk of cancelling within the next 30 days. The purpose is to offer support, not to deny service. The target is binary: cancellation within the defined future window.

Analysis and approach

The team first defines the prediction time: the end of each customer's current billing week. Only information available at that time may be used. It removes a field called cancellation_reason, because it is recorded after cancellation and would leak the outcome. Customers with multiple records are assigned entirely to either training or testing using a group-based split.

The team inspects class balance, missingness, duplicate records and whether some regions are under-represented. Numeric fields are median-imputed and standardised; categorical fields are imputed and one-hot encoded. A logistic regression pipeline provides an interpretable baseline, while a tree-based model provides a nonlinear comparison.

from sklearn.model_selection import GroupShuffleSplit
from sklearn.metrics import roc_auc_score, average_precision_score

X = data.drop(columns="cancelled")
y = data["cancelled"]
groups = data["customer_id"]

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=groups))

X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

model.fit(X_train, y_train)
probability = model.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, probability))
print(average_precision_score(y_test, probability))

ROC AUC measures ranking ability across thresholds, while average precision is often more informative when the positive class is rare. Neither metric alone determines whether a retention campaign is worthwhile. The team selects a threshold using the available support capacity and the relative cost of contacting a low-risk customer versus missing a high-risk customer.

Outcome and governance

The model performs better than a simple baseline, but performance is weaker for a small regional group because the training sample is limited. The team reports group-specific recall, conducts a qualitative review of influential variables and monitors performance after deployment. Customers are not automatically penalised. Staff receive a risk signal with an explanation of the principal factors and a route for human review.

Learning points

The most important achievement is not selecting the most complex algorithm. It is aligning the target with the decision, preventing temporal leakage, validating by customer, choosing metrics that reflect the operational objective and monitoring unequal performance.

7. Generative AI coding tools

Generative AI assistants can propose Python code, explain errors, generate tests and help document a pipeline. They can also invent library arguments, use obsolete APIs, produce insecure code, overlook leakage or confidently misinterpret a metric. Generated code is an unverified suggestion, not evidence.

A responsible workflow is:

1. state the task, data assumptions and required output precisely;
2. ask for a small, explainable solution rather than an opaque complete system;
3. inspect every import, transformation, metric and default;
4. run the code on a small known example and test edge cases;
5. consult official documentation for API behaviour and version compatibility;
6. check leakage, privacy, fairness, licensing and security implications;
7. record material AI assistance where institutional or professional policy requires it.

Never paste confidential personal, health, financial or proprietary data into an external tool without authorisation. Prompts and generated outputs may become part of an information trail. AI-generated comments should not be accepted as proof of correctness, and generated citations must be independently verified.

For assessment and employment, the learner must be able to explain the code, reproduce its results and defend its methodological choices. A useful principle is: automate typing, not responsibility.

8. Professional best practices and limitations

Use version-controlled notebooks or scripts, fixed seeds where appropriate, environment files, clear data dictionaries and reproducible preprocessing. Record dataset versions, exclusion rules, model versions and evaluation dates. Compare against simple baselines. Report uncertainty and confidence intervals where feasible rather than presenting a single impressive score.

Recognise limitations: a model trained on historical data may reproduce historical inequities; cross-validation estimates performance under a sampling design, not universal truth; feature importance is not necessarily causality; and deployment data may drift. Monitor input distributions, missingness, outcome rates and subgroup performance. Establish a process for retraining, rollback and human escalation.

Modern AI systems increasingly combine statistical models, neural networks and large language models. These systems can generate text or code but may hallucinate facts and can be sensitive to prompts. The same principles remain essential: define the task, evaluate against appropriate data, protect information, monitor harm and retain human accountability.

Chapter Summary

Python supports an integrated data-science workflow, but reliable results depend on statistical reasoning as much as programming. Descriptive statistics summarise centre, spread and association, while uncertainty reminds analysts that samples and measurements are imperfect. Scikit-learn pipelines combine preprocessing and estimators safely, particularly when transformations are fitted only on training folds. Regression and classification require different metrics, and metric selection should reflect the decision and its costs.

Training, validation and testing have different purposes. Cross-validation improves model comparison but must respect groups, time and dependence. Overfitting reduces generalisation; leakage creates unrealistically strong results; and bias can arise throughout the data and decision lifecycle. A professional analyst combines technical evaluation with fairness assessment, privacy protection, documentation and monitoring. Generative AI can accelerate coding, but all generated code and claims require human verification.

Key Terms

Population: The complete set of units about which conclusions are intended.

Sample: An observed subset of a population.

Feature: An input variable used by a model.

Target: The outcome a supervised-learning model predicts.

Mean: The arithmetic average of quantitative observations.

Variance: A measure of squared deviation from the mean.

Standard deviation: The square root of variance.

Interquartile range: The difference between the third and first quartiles.

Correlation: A standardised measure of linear association.

Pipeline: A sequence that combines transformations and an estimator.

Imputation: Replacing missing values using a defined rule or model.

Encoding: Representing categorical values numerically for modelling.

Cross-validation: Repeated fitting and validation across data folds.

Overfitting: Learning noise or accidental patterns that do not generalise.

Data leakage: Use of information unavailable at prediction time.

Confusion matrix: A table of correct and incorrect classification outcomes.

Precision: The proportion of predicted positives that are actual positives.

Recall: The proportion of actual positives that are detected.

Algorithmic bias: Systematic unequal error or treatment associated with data or design.

Data drift: Change in the distribution or relationship of data over time.

Review Questions

Short Answer Questions

1. Distinguish a population mean from a sample mean.
2. Why should a scaler normally be fitted only on training data?
3. Define overfitting and give one possible remedy.
4. What is the difference between precision and recall?
5. Why is a generative AI coding assistant not a substitute for testing?

Long Answer Questions

1. Explain how a scikit-learn ColumnTransformer and Pipeline support safe preprocessing.
2. Compare MAE, RMSE and R², including a situation in which each may be useful.
3. Describe the purposes of training, validation, cross-validation and test data.
4. Explain how temporal leakage can occur in a customer-attrition project.
5. Discuss why removing a sensitive attribute does not necessarily remove algorithmic bias.

Analytical / Application Questions

1. A fraud classifier has high accuracy but misses most fraudulent transactions. Which metrics and evaluation steps would you investigate, and why?
2. Design a validation strategy for data containing repeated observations from the same patient. Explain how you would prevent patient-level leakage.
3. A generative AI tool produces a preprocessing script that scales all rows before splitting the data. Identify the methodological problem and rewrite the workflow conceptually so that the evaluation is trustworthy.

Multiple Choice Questions with Answers

Question 1

Which operation should be fitted using training data only?

A. Calculating the test-set labels

B. Standardisation parameters

C. Displaying a confusion matrix

D. Counting the number of columns

Correct Answer: B. Standardisation parameters

Question 2

What does recall measure in binary classification?

A. The proportion of predicted positives that are correct

B. The proportion of all predictions that are correct

C. The proportion of actual positives that are detected

D. The proportion of actual negatives that are detected

Correct Answer: C. The proportion of actual positives that are detected

Question 3

Which situation is an example of data leakage?

A. Using a pipeline for imputation

B. Using a variable recorded after the outcome as a predictor

C. Holding out a test set

D. Using a fixed random state

Correct Answer: B. Using a variable recorded after the outcome as a predictor

Question 4

What is the principal purpose of a test set?

A. To fit preprocessing parameters repeatedly

B. To select every hyperparameter

C. To provide a final estimate after modelling decisions

D. To increase the number of training observations

Correct Answer: C. To provide a final estimate after modelling decisions

Question 5

Which metric is expressed in the same units as the regression target?

A. MSE

B. RMSE

C. R²

D. Correlation coefficient

Correct Answer: B. RMSE

Question 6

Why might accuracy be misleading for a highly imbalanced classification problem?

A. It cannot be calculated from a confusion matrix

B. It ignores the number of observations

C. A majority-class predictor can obtain high accuracy while missing positives

D. It is always smaller than recall

Correct Answer: C. A majority-class predictor can obtain high accuracy while missing positives

Question 7

What does handle_unknown="ignore" in one-hot encoding primarily do?

A. Removes all missing values

B. Prevents an error for unseen categories during transformation

C. Balances the target classes

D. Selects the most important features

Correct Answer: B. Prevents an error for unseen categories during transformation

Question 8

Which split is generally more appropriate when observations are time ordered?

A. A random split that mixes future and past

B. A time-aware split preserving temporal order

C. A split based only on column names

D. A split that uses the test target during training

Correct Answer: B. A time-aware split preserving temporal order

Question 9

Which statement about correlation is correct?

A. It proves that one variable causes another

B. It is unaffected by outliers

C. It measures linear association and does not by itself establish causation

D. It can take any value greater than 1

Correct Answer: C. It measures linear association and does not by itself establish causation

Question 10

What is the most responsible response to AI-generated Python code?

A. Run it in production without inspection

B. Assume comments prove correctness

C. Verify it with tests, documentation and methodological review

D. Use it only when it produces the highest training score

Correct Answer: C. Verify it with tests, documentation and methodological review