Machine Learning with Python

Introduction

Machine learning (ML) enables a computer system to learn patterns from data and use those patterns to make predictions, classifications, or groupings. In data science, Python and scikit-learn provide a practical environment for building reproducible machine-learning workflows. The central professional challenge is not merely selecting an algorithm; it is designing a valid experiment in which data preparation, model fitting, evaluation, and interpretation are properly separated.

This chapter focuses on supervised learning—regression and classification—and unsupervised learning, particularly clustering. It also develops preprocessing, train-test design, cross-validation, baseline comparison, leakage prevention, and practical implementation using scikit-learn.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish supervised from unsupervised learning;
• formulate regression and classification problems;
• prepare numerical and categorical data using scikit-learn pipelines;
• divide data into training and test sets appropriately;
• train and evaluate regression, classification, and clustering models;
• use cross-validation and compare models with meaningful baselines;
• identify overfitting, underfitting, data leakage, and class imbalance;
• interpret results and communicate limitations responsibly.

1. Machine-Learning Problem Formulation

A dataset usually contains observations and variables. Let X denote the feature matrix and y the target variable. In supervised learning, the algorithm receives examples containing both features and known targets. It estimates a function f such that:

Here, ŷ is the predicted target. The learned function is an approximation: it should perform well not only on the observations used for training, but also on new observations drawn from a comparable population.

In regression, the target is numerical, such as revenue, temperature, or house price. In classification, the target is categorical, such as fraud/not fraud or churn/no churn. In unsupervised learning, target labels are absent; the objective is to discover structure, such as customer segments or groups of similar documents.

A useful distinction is between parameters and hyperparameters. Parameters are learned from training data, such as coefficients in linear regression. Hyperparameters are selected before or during model selection, such as the number of clusters in k-means or the regularization strength in logistic regression.

Common workflow

Business question → Data audit → Train/test design → Preprocessing
        ↓                                  ↓
Baseline ← Model fitting ← Cross-validation ← Candidate models
        ↓
Final test evaluation → Interpretation → Deployment and monitoring

The test set should be treated as an estimate of future performance, not as a resource for repeated model decisions.

2. Data Splitting and Experimental Design

A basic holdout design divides observations into a training set and a test set. The model and preprocessing steps are fitted using the training data only. The test set is held back until the final evaluation.

For a dataset with n observations, a training fraction r gives approximately rn training observations and (1-r)n test observations. The exact count can vary because of rounding and implementation details. A common starting point is 80% training and 20% testing, although the appropriate ratio depends on dataset size and the cost of an unreliable estimate.

For classification, stratified splitting attempts to preserve class proportions in both partitions. For time-dependent data, random splitting is often inappropriate because it can allow future information to influence the past. A chronological split is preferable.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.20,
    random_state=42,
    stratify=y  # use for classification; omit for ordinary regression
)

A fixed random_state makes an experiment reproducible. Reproducibility does not mean that one random split is universally correct; it means that the same experiment can be inspected and repeated.

Data leakage

Data leakage occurs when information unavailable at prediction time influences model training or evaluation. Examples include scaling the entire dataset before splitting, using a post-outcome variable as a feature, or selecting features using the test labels. Leakage usually produces performance estimates that are too optimistic.

The safest solution is to place transformations and the estimator inside a Pipeline. During cross-validation, each transformation is fitted separately on the relevant training fold.

3. Preprocessing with scikit-learn

Preprocessing converts raw variables into a form suitable for an algorithm. It may include missing-value imputation, scaling, categorical encoding, feature selection, and transformations of skewed variables.

Numerical scaling

Standardization converts a numerical feature to approximately zero mean and unit variance:

where x_i is an observed value, μ is the training-sample mean, and σ is the training-sample standard deviation. The result is dimensionless. Standardization is especially useful for distance-based methods, support vector machines, neural networks, and regularized linear models. Tree-based methods generally do not require it.

Min-max scaling transforms values to a selected interval, commonly [0, 1]:

This formula assumes x_max ≠ x_min. Extreme outliers can strongly affect both scaling approaches.

Categorical variables and missing values

Most estimators require numerical inputs. One-hot encoding represents a categorical variable with indicator columns. For a variable with categories Red, Blue, and Green, a row may be encoded as (1, 0, 0), (0, 1, 0), or (0, 0, 1). With an intercept in a linear model, one category may be dropped to avoid exact linear dependence; this is the dummy-variable trap.

Missing numerical values can be replaced with a statistic such as the training median. Missing categorical values can be replaced with the most frequent category or an explicit label such as “Missing”. Imputation is an assumption and should be documented.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_features = ['age', 'income']
categorical_features = ['region', 'plan']

numeric_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(handle_unknown='ignore'))
])

preprocessor = ColumnTransformer([
    ('numeric', numeric_pipeline, numeric_features),
    ('categorical', categorical_pipeline, categorical_features)
])

handle_unknown='ignore' prevents a prediction failure when a category appears in new data but was absent during training. It does not make the new category semantically meaningful; it merely encodes it safely.

4. Supervised Learning: Regression

Regression estimates a numerical outcome. Ordinary least-squares linear regression represents the conditional mean as a weighted sum of features:

Here, ŷ_i is the prediction for observation i, β₀ is the intercept, β_j is the coefficient of feature j, and p is the number of features. The coefficients are commonly estimated by minimizing the residual sum of squares:

Least squares assumes, for inferential interpretation, an appropriate linear conditional mean, independent errors, and often constant error variance. Prediction can still be useful when some inferential assumptions are imperfect, but extrapolation and strong nonlinearity remain risks.

Regression metrics

Mean absolute error (MAE) is:

It has the same units as the target and treats errors linearly. Mean squared error (MSE) is:

MSE gives greater influence to large errors and has squared target units. Root mean squared error is RMSE = √MSE, restoring the target units. The coefficient of determination is:

where ȳ is the mean target in the evaluated sample. Relative to a mean-prediction baseline, R² can be negative on test data if the model performs worse than that baseline. It is not automatically a percentage of “accuracy”.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

regression_model = Pipeline([
    ('preprocess', preprocessor),
    ('model', LinearRegression())
])
regression_model.fit(X_train, y_train)
predictions = regression_model.predict(X_test)

mae = mean_absolute_error(y_test, predictions)
rmse = np.sqrt(mean_squared_error(y_test, predictions))
r2 = r2_score(y_test, predictions)

Regularized models can control coefficient magnitude. Ridge regression minimizes squared error plus λ∑β_j², while lasso adds λ∑|β_j|, where λ ≥ 0 controls regularization and the intercept is usually excluded from the penalty. Regularization requires careful scaling and selection of λ using training data and cross-validation.

5. Supervised Learning: Classification

Classification predicts a discrete class. Logistic regression models the probability of a binary outcome:

The output is a probability between 0 and 1. A classification threshold, often 0.5 but not necessarily optimal, converts probability into a class label.

A confusion matrix records true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Accuracy is:

Precision and recall are:

Precision answers, “Among predicted positives, how many were positive?” Recall answers, “Among actual positives, how many were found?” The F1 score is the harmonic mean:

These metrics require nonzero denominators. Accuracy can be misleading under class imbalance. For example, a detector that labels every case as “normal” may achieve high accuracy when positive events are rare but have zero recall for the important class.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score

classifier = Pipeline([
    ('preprocess', preprocessor),
    ('model', LogisticRegression(max_iter=1000, class_weight='balanced'))
])
classifier.fit(X_train, y_train)
labels = classifier.predict(X_test)
probabilities = classifier.predict_proba(X_test)[:, 1]

print(confusion_matrix(y_test, labels))
print(classification_report(y_test, labels))
print(roc_auc_score(y_test, probabilities))

ROC-AUC measures ranking performance across classification thresholds. It can appear strong even when the positive class is very rare; precision-recall analysis may then be more informative.

Other useful classifiers include decision trees, random forests, gradient boosting, support vector machines, and k-nearest neighbors. Trees partition feature space and can capture nonlinear interactions. Ensembles often improve predictive performance but may reduce transparency and increase computational cost. Nearest-neighbor methods depend strongly on scaling and the definition of distance.

6. Cross-Validation and Baselines

A single train-test split can produce a noisy estimate. In k-fold cross-validation, the training data are divided into k folds. Each fold is used once for validation while the other k−1 folds are used for fitting. The cross-validation score is commonly the arithmetic mean:

where score_j is the score from fold j. The variation among fold scores indicates uncertainty due to partitioning, although it is not a complete confidence interval.

from sklearn.model_selection import StratifiedKFold, cross_validate

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(
    classifier, X_train, y_train, cv=cv,
    scoring=['accuracy', 'precision', 'recall', 'f1']
)
print(results['test_f1'].mean())

For regression, use KFold; for grouped observations, use GroupKFold; for temporal data, use TimeSeriesSplit. The splitting method must reflect how predictions will be made in practice.

A baseline is a simple reference model. A regression baseline may always predict the training mean. A classification baseline may always predict the most frequent class, although a domain rule may be a stronger reference. A complex model has practical value only if it improves on a credible baseline using an appropriate metric and acceptable cost.

Hyperparameter tuning can be performed with GridSearchCV or RandomizedSearchCV. The test set must remain untouched while the search occurs. When many alternatives are tried, the final test score can still be optimistic; nested cross-validation is a stronger design for rigorous model comparison.

7. Unsupervised Learning: Clustering

Clustering groups observations according to a similarity measure without known target labels. K-means seeks k centroids and assigns each observation to the nearest centroid. Its objective is to minimize within-cluster squared Euclidean distance:

Here, C_j is cluster j, μ_j is its centroid, and ||x_i−μ_j||² is squared Euclidean distance. K-means assumes that numerical features and Euclidean distance are meaningful, and it is sensitive to scale, outliers, initialization, and the chosen value of k.

from sklearn.cluster import KMeans
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

cluster_model = make_pipeline(
    StandardScaler(),
    KMeans(n_clusters=3, n_init=20, random_state=42)
)
cluster_labels = cluster_model.fit_predict(customer_features)

The silhouette coefficient for an observation compares average within-cluster distance a with the lowest average distance to another cluster b:

It ranges from −1 to 1 when defined: higher values suggest better separation, but the result is not proof that clusters represent meaningful business segments. Domain interpretation and stability are essential. Hierarchical clustering, DBSCAN, and Gaussian mixture models provide alternatives when clusters are non-spherical, density-based, or probabilistic.

8. Worked Application: Predicting Customer Churn

The following is an illustrative case. A subscription company wants to identify customers likely to cancel within the next 30 days. Available variables include tenure, monthly usage, support contacts, plan type, region, and whether a payment failed during the previous month.

Problem

The target is binary: churn or no churn. Missing values occur in usage and region. The company wants to prioritize recall because failing to identify a customer at risk is more costly than contacting some customers who would not have churned.

Analysis and approach

First, the team defines the prediction time. Only variables known at that time are included. A random split is acceptable only if records represent independent customers and the future relationship is stable. If repeated monthly records exist for the same customer, a customer-level group split is needed to prevent the same customer appearing in both training and test data. If the goal is future-month prediction, a temporal split is preferable.

The team constructs a pipeline with median imputation and scaling for numerical features, and most-frequent imputation plus one-hot encoding for categorical features. A majority-class classifier provides a baseline. Logistic regression and random forest are compared using five-fold stratified cross-validation, with recall and precision reported together.

Suppose the test results are as follows:

The logistic model identifies more churners, whereas the forest produces fewer false alarms. The correct choice depends on the relative costs of missed churners and unnecessary retention offers. The team may also select a probability threshold below 0.5 to increase recall, then evaluate the resulting precision, contact volume, and financial return on a temporally separated test period.

Outcome and learning points

The model should not be deployed solely because it has the highest F1 score. The company should test whether retention interventions actually reduce churn, monitor performance by region and plan type, document the threshold, and review whether payment variables create unfair or unintended treatment. A predictive model identifies risk; it does not establish that a customer will churn or that a particular intervention will work.

9. Professional Practice, Ethics, and Governance

Machine-learning systems can affect access to credit, employment, healthcare, insurance, and public services. Important governance practices include:

• document the data source, target definition, features, preprocessing, and intended use;
• distinguish correlation from causation;
• assess performance across relevant demographic and operational groups;
• minimize collection and retention of personal data;
• protect data through access control and secure storage;
• provide human review where decisions have substantial consequences;
• monitor drift in feature distributions, class proportions, and error rates;
• preserve model versioning and reproducible training records.

Fairness cannot be inferred from one aggregate metric. Different fairness criteria may conflict, particularly when base rates differ between groups. Evaluation should therefore involve domain experts, affected stakeholders, legal requirements, and documented risk controls. Explainability tools can help inspect model behavior, but an explanation is not a guarantee that the model is valid or fair.

10. Best Practices and Limitations

A strong scikit-learn workflow follows these principles:

1. Define the prediction target and decision point precisely.
2. Audit data types, missingness, duplicates, outliers, and leakage risks.
3. Split data according to the deployment situation.
4. Fit preprocessing only within the training workflow.
5. Establish a simple, meaningful baseline.
6. Select metrics according to business and social costs.
7. Use cross-validation for model and hyperparameter comparison.
8. Reserve the test set for final evaluation.
9. Inspect errors, subgroup performance, calibration, and stability.
10. Monitor the deployed system and retrain only under controlled procedures.

No algorithm is universally best. Results depend on data quality, sampling, target definition, measurement error, distribution shift, and the action taken after prediction. A high test score may fail in production if users change behavior, the population changes, or the training data encode historical inequities.

Chapter Summary

Machine learning with Python involves more than calling a fitting method. Supervised learning uses labelled examples for numerical regression or categorical classification; unsupervised learning discovers structure without target labels. Valid experimentation requires a suitable train-test design, leakage-resistant preprocessing, appropriate metrics, and comparison with a baseline. Pipelines combine transformations and estimators so that cross-validation remains valid. Regression can be evaluated with MAE, RMSE, and R²; classification requires careful use of confusion-matrix metrics, threshold selection, and class-imbalance analysis. K-means clustering minimizes within-cluster squared distance but depends on scaling, initialization, and the choice of cluster number. Finally, responsible deployment requires documentation, fairness assessment, privacy protection, interpretability, and monitoring for drift.

Key Terms

Supervised learning: Learning from observations with known target values.

Unsupervised learning: Learning patterns from data without labelled targets.

Feature: An input variable used for prediction or grouping.

Target: The outcome a supervised model predicts.

Regression: Prediction of a numerical outcome.

Classification: Prediction of a categorical outcome.

Pipeline: An ordered scikit-learn sequence of preprocessing and modelling steps.

Data leakage: Use of information unavailable at prediction time.

Imputation: Replacement of missing values using a defined rule.

One-hot encoding: Representation of categories as binary indicator variables.

Standardization: Rescaling using the training mean and standard deviation.

Baseline: A simple reference method used for comparison.

Cross-validation: Repeated training and validation across data folds.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positives that are identified.

Regularization: A penalty used to control model complexity.

Clustering: Grouping observations by similarity without target labels.

Centroid: The representative mean point of a k-means cluster.

Concept drift: Change in the relationship between inputs and outcomes over time.

Model governance: Processes for documenting, evaluating, deploying, and monitoring models.

Review Questions

Short Answer Questions

1. What is the difference between supervised and unsupervised learning?
2. Why should preprocessing be fitted only on training data?
3. Define precision and recall.
4. What purpose does a baseline model serve?
5. Why is feature scaling important for k-means clustering?

Long Answer Questions

1. Explain how a train-test split, pipeline, and cross-validation work together to reduce misleading model evaluation.
2. Compare MAE, RMSE, and R² and describe a situation in which each may be useful.
3. Explain the assumptions and limitations of ordinary least-squares linear regression.
4. Describe how class imbalance can distort classification evaluation and propose suitable responses.
5. Explain the k-means objective, its assumptions, and two alternatives for data with non-spherical or density-based structure.

Analytical / Application Questions

1. A fraud model has 99% accuracy but detects only 10% of fraudulent transactions. Diagnose the evaluation problem and recommend metrics and a decision strategy.
2. A dataset contains monthly records for the same customers from 2021–2024. Design a train-validation-test strategy for predicting churn in 2024 and justify your choice.
3. A clustering analysis produces a high silhouette score, but marketing managers cannot identify meaningful differences between clusters. Explain possible reasons and recommend further analysis.

Multiple Choice Questions with Answers

Question 1

Which practice most directly prevents preprocessing leakage during cross-validation?

A. Fitting transformations inside a scikit-learn Pipeline

B. Increasing the number of test observations

C. Removing the target variable from the dataset

D. Using a more complex estimator

Correct Answer: A. Fitting transformations inside a scikit-learn Pipeline

Question 2

Which metric has the same units as the regression target?

A. Mean squared error

B. Root mean squared error

C. R-squared

D. Accuracy

Correct Answer: B. Root mean squared error

Question 3

Recall answers which question?

A. Among predicted positives, how many are correct?

B. Among all observations, how many are correct?

C. Among actual positives, how many were identified?

D. How well are numerical errors calibrated?

Correct Answer: C. Among actual positives, how many were identified?

Question 4

Why can accuracy be misleading for a rare-event classifier?

A. It cannot be calculated from a confusion matrix

B. A majority prediction can achieve high accuracy while missing positives

C. It always gives a lower value than recall

D. It requires continuous targets

Correct Answer: B. A majority prediction can achieve high accuracy while missing positives

Question 5

What does standardization calculate for a value x?

A. x + μ

B. x / σ

C. (x − μ) / σ

D. (x − σ) / μ

Correct Answer: C. (x − μ) / σ

Question 6

What does k-means minimize?

A. Within-cluster squared distances to centroids

B. The number of features in the dataset

C. Classification false negatives

D. The test-set error only

Correct Answer: A. Within-cluster squared distances to centroids

Question 7

Which split is generally most appropriate when predicting future observations from time-ordered data?

A. Random split with labels shuffled

B. Chronological split

C. Split based only on target value

D. Split after fitting the model

Correct Answer: B. Chronological split

Question 8

What is the primary purpose of a baseline model?

A. To guarantee fairness

B. To replace the test set

C. To provide a simple reference for judging improvement

D. To maximize model complexity

Correct Answer: C. To provide a simple reference for judging improvement

Question 9

In binary logistic regression, the model output before thresholding is commonly interpreted as:

A. A cluster identifier

B. A probability of the positive class

C. A standardized residual

D. A missing-value indicator

Correct Answer: B. A probability of the positive class

Question 10

Which statement about a held-out test set is correct?

A. It should be repeatedly used to tune hyperparameters

B. It should be used to fit the scaler

C. It is reserved for final evaluation after model decisions

D. It must contain no target values

Correct Answer: C. It is reserved for final evaluation after model decisions