Machine Learning Foundations with Python

Introduction and Context

Machine learning (ML) is a branch of artificial intelligence in which computer systems learn patterns from data to make predictions, classifications, recommendations, or decisions. Unlike traditional rule-based programming, where a developer explicitly specifies the logic, machine learning estimates a model from examples.

A business problem becomes a machine-learning problem only when three conditions are present: a measurable outcome or structure of interest, relevant data, and a decision that can benefit from improved prediction or discovery. For example, a retailer may predict weekly sales, classify transactions as fraudulent or legitimate, group customers by purchasing behaviour, or reduce hundreds of sensor measurements to a smaller set of informative dimensions.

Python is widely used for machine learning because it combines readable syntax with a mature ecosystem. In this chapter, scikit-learn is used for classical supervised and unsupervised learning. The emphasis is not merely on calling an algorithm, but on framing the problem correctly, evaluating generalisation, controlling overfitting, and interpreting results responsibly.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish supervised, unsupervised, and dimensionality-reduction tasks;
• translate a business question into features, a target, and an evaluation metric;
• build regression, classification, clustering, and principal component analysis models with scikit-learn;
• explain training error, test error, bias, variance, overfitting, and underfitting;
• apply preprocessing, cross-validation, and regularisation appropriately;
• interpret model results while recognising limitations, fairness risks, and data leakage.

1. From Business Question to Learning Task

A useful first step is to express the data in matrix form. Let X ∈ ℝ^n×p be the feature matrix, where n is the number of observations and p is the number of features. The element x_ij is the value of feature j for observation i. In supervised learning, each observation also has a known target value y_i. The learning algorithm estimates a function that maps features to an outcome:

Here, X denotes input features, f is the fitted model, and ŷ is the prediction. In practice, the model is estimated from a training set and then evaluated on previously unseen data.

Supervised learning

Supervised learning uses labelled examples. The target determines the task:

In regression, a prediction such as 125.4 units is meaningful. In classification, the model may output a class probability, such as the probability that a loan applicant defaults.

Unsupervised learning

Unsupervised learning has no supplied target. The aim is to discover structure, such as customer segments or unusual observations. Clustering assigns observations to groups, whereas dimensionality reduction represents many variables using fewer derived variables.

Framing questions

Before choosing an algorithm, specify:

1. Unit of analysis: customer, transaction, product, machine, or time period.
2. Prediction moment: what information would be available when a decision is made?
3. Target definition: precisely what constitutes a positive class or numerical outcome?
4. Success metric: what error is costly to the organisation?
5. Action: what decision follows from a prediction?
6. Constraints: latency, interpretability, privacy, fairness, and regulatory requirements.

A model that predicts accurately but cannot be used operationally, or that relies on information unavailable at prediction time, is not a successful solution.

2. The Machine-Learning Workflow

A professional workflow generally consists of:

1. define the business objective;
2. collect, inspect, and document data;
3. split data into training and test sets;
4. preprocess features using only training information;
5. establish a simple baseline;
6. train candidate models;
7. evaluate with cross-validation and an untouched test set;
8. inspect errors, subgroup performance, and interpretability;
9. deploy, monitor, and periodically retrain.

A common split is 80% training and 20% testing, although the appropriate choice depends on sample size. For time-dependent data, a random split may be invalid because it allows future patterns to influence past predictions. A chronological split or time-series cross-validation is more appropriate.

Data leakage

Data leakage occurs when information unavailable at prediction time enters the features or evaluation process. Examples include using a final diagnosis to predict hospital admission, scaling the complete dataset before splitting, or selecting variables using the test set. Leakage produces deceptively strong results and is one of the most serious practical errors in ML.

Scikit-learn pipelines reduce leakage by fitting transformations within each training fold:

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000))
])
model.fit(X_train, y_train)

The stratify argument approximately preserves class proportions in a classification split. It should not be used blindly when classes are extremely rare or observations are grouped; in such cases, grouped or specialised sampling may be needed.

3. Regression

Regression estimates a continuous outcome. The simplest model is linear regression:

Here, β₀ is the intercept, βⱼ is the expected change in the target associated with a one-unit change in feature j while other features are held constant, and εᵢ is the error term. Ordinary least squares estimates coefficients by minimising the residual sum of squares:

The model is most interpretable when the relationship is approximately linear, observations are appropriately independent, features are measured consistently, and extreme outliers do not dominate the fit. For classical inferential interpretation, assumptions concerning error variance, independence, and distribution may also matter; predictive use does not require every assumption to hold perfectly, but violations can reduce reliability.

Regression metrics

Mean absolute error is:

It expresses the average absolute error in the target's units. Root mean squared error is:

RMSE penalises large errors more strongly than MAE. The coefficient of determination is:

where ȳ is the mean observed target. R² measures improvement over predicting the mean on the evaluated data; it is not a percentage accuracy measure and may be negative on test data.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

reg = Pipeline([
    ("scale", StandardScaler()),
    ("model", LinearRegression())
])
reg.fit(X_train, y_train)
pred = reg.predict(X_test)

mae = mean_absolute_error(y_test, pred)
rmse = np.sqrt(mean_squared_error(y_test, pred))
r2 = r2_score(y_test, pred)

Regularised regression

When features are numerous or correlated, ordinary least squares can have unstable coefficients. Ridge regression adds an L2 penalty:

Lasso regression uses an L1 penalty:

λ ≥ 0 controls the penalty. The intercept is usually not penalised. Larger λ generally reduces coefficient magnitude and model variance. Lasso can set some coefficients exactly to zero, providing a form of feature selection; with strongly correlated variables, its selected feature can be unstable. Standardisation is important because penalties otherwise depend on measurement scale.

4. Classification

Classification predicts a discrete class. Logistic regression models the probability of a binary outcome:

The output lies between 0 and 1. A threshold, often 0.5 but not necessarily, converts probability into a class. Threshold selection should reflect the costs of false positives and false negatives.

A confusion matrix contains true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Important metrics are:

Precision answers, “Among predicted positives, how many are positive?” Recall answers, “Among actual positives, how many were found?” Accuracy can be misleading when classes are imbalanced. ROC-AUC summarises ranking performance across classification thresholds, while precision–recall curves are often more informative for rare positive classes.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score

clf = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(class_weight="balanced", max_iter=1000))
])
clf.fit(X_train, y_train)
class_pred = clf.predict(X_test)
prob = clf.predict_proba(X_test)[:, 1]

print(classification_report(y_test, class_pred))
print(roc_auc_score(y_test, prob))

class_weight="balanced" changes the fitting emphasis; it does not automatically solve sampling bias or guarantee fairness. Tree-based classifiers, support vector machines, and nearest-neighbour methods are additional choices. Trees can represent nonlinear interactions and require little scaling, while nearest-neighbour and support-vector methods are sensitive to feature scale.

5. Clustering and Dimensionality Reduction

K-means clustering

K-means partitions observations into k clusters by minimising within-cluster squared Euclidean distance:

Cⱼ is cluster j and μⱼ is its centroid. The algorithm alternates between assigning each observation to its nearest centroid and recalculating centroids. It requires k in advance and works best when clusters are reasonably compact, separated, and comparable in scale. It is sensitive to outliers and initialisation.

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)
kmeans = KMeans(n_clusters=3, n_init=20, random_state=42)
labels = kmeans.fit_predict(X_scaled)

The elbow method compares within-cluster sum of squares for different k values, but an elbow may be ambiguous. The silhouette coefficient for an observation compares average within-cluster distance with the nearest other-cluster distance:

where a is mean distance to points in the same cluster and b is the smallest mean distance to another cluster. Values near 1 indicate better separation, values near 0 indicate overlap, and negative values may indicate poor assignment. Clustering quality must also be judged by domain meaning and stability, not by one score alone.

Principal component analysis

Principal component analysis (PCA) transforms correlated variables into orthogonal components ordered by explained variance. After centring features, PCA finds directions of greatest variance. If X is a centred data matrix, its covariance matrix is:

The principal directions are eigenvectors of S, and their associated eigenvalues indicate variance along those directions. In practice, scikit-learn commonly uses singular value decomposition. PCA is sensitive to scale, so standardisation is often necessary when variables have different units.

from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline

pca_model = make_pipeline(StandardScaler(), PCA(n_components=2))
X_2d = pca_model.fit_transform(X)

PCA is useful for visualisation, compression, noise reduction, and preparing features for some models. It is not inherently a predictive model, and components may be difficult to interpret because each is a weighted combination of original variables. PCA directions learned from training data must be applied unchanged to future data.

6. Generalisation, Bias, Variance, and Validation

A model should perform well on new observations, not merely reproduce its training data. Expected prediction error can be conceptually decomposed into bias, variance, and irreducible noise:

Bias is systematic error caused by an overly restrictive model. Variance is sensitivity to the particular training sample. A very simple linear model may underfit and have high bias; a deep tree may overfit and have high variance. Increasing model complexity often decreases training error but can eventually increase test error.

Cross-validation estimates performance across multiple training-validation splits. In k-fold cross-validation, the data are divided into k folds; each fold is used once for validation while the others are used for training. The reported score is commonly the mean across folds, and its variation indicates uncertainty due to splitting.

from sklearn.model_selection import cross_validate, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(clf, X, y, cv=cv, scoring=["accuracy", "roc_auc"])

Hyperparameter tuning must occur inside cross-validation, preferably with GridSearchCV or RandomizedSearchCV. The final test set should be used once, after model selection, to provide an approximately unbiased final estimate. Repeatedly checking the test set turns it into an informal training resource.

7. Interpretability and Responsible Use

Interpretability has several meanings. A linear model may be globally understandable through coefficients, whereas a complex model may require local explanations for individual predictions. A coefficient's meaning depends on preprocessing, coding, confounding, and the presence of correlated variables; it should not automatically be interpreted as causal influence.

Useful tools include:

• permutation importance, which measures performance decrease after shuffling a feature;
• partial dependence, which displays average predicted response as a feature changes;
• SHAP-style additive explanations, which allocate a prediction among features under specified assumptions;
• coefficients and decision paths for inherently simpler models.

Interpretation methods are themselves models or approximations and can be misleading with correlated features, extrapolation, or distribution shift. Explanations should be checked against domain knowledge and subgroup behaviour.

Responsible deployment requires attention to privacy, security, consent, fairness, and accountability. Personal data should be minimised, protected, and processed under applicable law and organisational policy. Sensitive attributes may require restricted access, but excluding them does not guarantee fairness because proxy variables can remain. Evaluate error rates across relevant groups, document data provenance, preserve audit trails, and provide human review for high-impact decisions. A prediction is not a justification for denying a person a service without due process.

8. Illustrative Case: Predicting Customer Churn

Background

A subscription company wants to reduce avoidable customer cancellations. It records tenure, monthly fee, service usage, support contacts, contract type, and a historical churn indicator.

Problem

The business asks, “Which customers are likely to churn in the next 30 days?” This is a binary classification problem. The prediction moment is the first day of each month, so features must be restricted to information available by that date. A future cancellation request must not appear as a feature.

Analysis and approach

The data team first defines churn precisely and checks duplicate accounts, missing values, class imbalance, and time ordering. It creates a chronological training period and later test period. Numerical variables are imputed and standardised; categorical variables are one-hot encoded within a pipeline. Logistic regression provides an interpretable baseline, while a tree-based model is considered for nonlinear relationships.

Because contacting customers is costly, precision matters; however, failing to identify a likely churner also has a cost. The team therefore examines precision, recall, the precision–recall curve, calibration, and expected financial value at several probability thresholds. Cross-validation is performed only within the historical training period. The final test period is reserved until model selection is complete.

Outcome

Suppose the selected model improves recall at an affordable precision compared with the baseline. The company does not automatically send identical offers to all flagged customers. Instead, it conducts a controlled retention experiment, monitors performance by contract type and region, and reviews whether interventions benefit customers rather than merely increasing unwanted contact.

Learning points

The important achievement is not choosing the most complex algorithm. It is aligning the target with the decision, preventing temporal leakage, selecting a threshold based on business costs, and monitoring outcomes after deployment. Churn predictions indicate risk; they do not establish why a customer will leave.

9. Best Practices and Limitations

• Begin with a transparent baseline before adopting complex models.
• Keep a data dictionary describing definitions, units, missingness, and collection dates.
• Fit imputation, scaling, encoding, feature selection, and dimensionality reduction inside pipelines.
• Use stratified, grouped, or time-aware validation when ordinary random splitting is inappropriate.
• Report uncertainty, subgroup metrics, calibration, and the practical cost of errors.
• Avoid interpreting correlation as causation.
• Monitor drift in features, target prevalence, calibration, and business outcomes.
• Version data, code, configurations, and model artifacts.

No algorithm is universally best. Performance depends on data quality, sample size, feature relevance, measurement processes, and deployment conditions. Unsupervised outputs are especially dependent on distance definitions, scaling, and the chosen number of groups. Models can degrade when the population, policy, or environment changes.

Chapter Summary

Machine learning transforms data into predictive or descriptive models. Supervised learning uses labelled targets for regression and classification; unsupervised learning discovers structure through methods such as clustering and PCA. A sound workflow begins with business and prediction-moment definitions, then uses leakage-safe preprocessing, appropriate data splitting, baselines, cross-validation, and a final untouched test set.

Regression metrics include MAE, RMSE, and R². Classification requires attention to confusion-matrix measures, class imbalance, thresholds, and the cost of errors. K-means seeks compact clusters, while PCA creates orthogonal directions that explain variance. Bias–variance analysis explains the tension between underfitting and overfitting, and regularisation controls complexity. Finally, responsible practice requires interpretability, privacy, fairness evaluation, documentation, monitoring, and human accountability.

Key Terms

Supervised learning: Learning from examples with known target values.

Unsupervised learning: Learning patterns without a supplied target.

Feature: An input variable used by a model.

Target: The outcome a supervised model learns to predict.

Regression: Prediction of a continuous numerical outcome.

Classification: Prediction of a discrete class or category.

Clustering: Grouping observations according to similarity.

Dimensionality reduction: Representing data with fewer variables or components.

Data leakage: Use of information that would not be available at prediction time.

Overfitting: Learning training-specific patterns that fail to generalise.

Underfitting: Using a model too simple to capture relevant structure.

Bias: Systematic prediction error associated with restrictive assumptions.

Variance: Sensitivity of a fitted model to the training sample.

Regularisation: Penalising model complexity to improve generalisation.

Cross-validation: Repeated train-validation splitting for performance estimation.

Precision: Proportion of predicted positives that are truly positive.

Recall: Proportion of actual positives that are correctly identified.

Pipeline: A sequence of preprocessing and modelling steps fitted systematically.

Interpretability: The extent to which model behaviour can be understood.

Concept drift: Change over time in relationships between inputs and outcomes.

Review Questions

Short Answer Questions

1. Distinguish regression from classification and give one business example of each.
2. What is data leakage, and why does it produce unreliable evaluation results?
3. Define precision and recall using confusion-matrix terms.
4. Why is feature scaling important for K-means and many regularised models?
5. What is the purpose of a final untouched test set?

Long Answer Questions

1. Explain a complete machine-learning workflow from business framing through deployment monitoring.
2. Compare MAE, RMSE, and R², including situations in which each is informative.
3. Explain bias, variance, underfitting, overfitting, and the role of regularisation.
4. Describe K-means and PCA, including their assumptions, uses, and limitations.
5. Discuss interpretability, privacy, fairness, and accountability in a high-impact classification system.

Analytical / Application Questions

1. A fraud dataset contains 1% fraudulent transactions. Which evaluation metrics would you prioritise, and how would you select an operating threshold? Justify your answer.
2. A model is trained on customer records from 2022 and deployed in 2025 after a major pricing-policy change. Design a validation and monitoring strategy that addresses temporal shift.
3. A clustering analysis produces four statistically separated groups, but marketing staff cannot assign meaningful business interpretations to them. Analyse possible causes and propose a responsible next step.

Multiple Choice Questions with Answers

Question 1

Which task is supervised learning?

A. Grouping customers without labels

B. Predicting house prices from labelled historical sales

C. Reducing variables with PCA

D. Detecting structure without a target

Correct Answer: B. Predicting house prices from labelled historical sales

Question 2

What is the primary purpose of a test set?

A. To fit preprocessing parameters

B. To select every hyperparameter repeatedly

C. To estimate performance on unseen data after model selection

D. To increase the training sample

Correct Answer: C. To estimate performance on unseen data after model selection

Question 3

Which metric is expressed in the same units as a continuous target?

A. MAE

B. ROC-AUC

C. Accuracy

D. Silhouette coefficient

Correct Answer: A. MAE

Question 4

If false negatives are especially costly, which measure should generally receive strong attention?

A. Recall

B. Precision only

C. R²

D. Within-cluster sum of squares

Correct Answer: A. Recall

Question 5

What does L1 regularisation characteristically encourage?

A. Larger coefficients

B. Exactly zero coefficients for some features

C. Removal of the intercept in every model

D. Higher training error without any generalisation benefit

Correct Answer: B. Exactly zero coefficients for some features

Question 6

Which situation is an example of data leakage?

A. Using a chronological test period

B. Fitting a scaler only on training folds

C. Including information recorded after the prediction event

D. Comparing MAE and RMSE

Correct Answer: C. Including information recorded after the prediction event

Question 7

K-means primarily minimises which quantity?

A. Within-cluster squared distances to centroids

B. Classification recall

C. The number of original features

D. The logistic loss only

Correct Answer: A. Within-cluster squared distances to centroids

Question 8

What is a principal component?

A. A class label supplied by a supervisor

B. A direction formed from variables that captures variance

C. A missing-value replacement

D. A confusion-matrix cell

Correct Answer: B. A direction formed from variables that captures variance

Question 9

A model with very low training error and much higher test error most clearly shows:

A. Underfitting

B. Overfitting

C. Perfect calibration

D. Unsupervised learning

Correct Answer: B. Overfitting

Question 10

Why should a classification threshold not automatically be fixed at 0.5?

A. Probabilities cannot be calculated

B. The best threshold depends on class proportions and error costs

C. Thresholds apply only to regression

D. A threshold changes the training features

Correct Answer: B. The best threshold depends on class proportions and error costs