Machine-Learning Lifecycle and Problem Framing

Introduction

Machine learning projects rarely fail because an algorithm cannot be trained. They more often fail because the team solves the wrong problem, predicts an unusable target, measures success incorrectly, or cannot place the model into an operational process. Problem framing is therefore the bridge between a business need and a technically valid machine-learning task.

This chapter presents a structured lifecycle for developing and managing machine-learning solutions. It explains how to translate business needs into analytical questions, define observations and targets, assess feasibility, establish baselines, select success metrics, and move from development to deployment and review. The CRISP-DM framework—Business Understanding, Data Understanding, Data Preparation, Modelling, Evaluation, and Deployment—is used as a practical organising model. In professional work, these stages are iterative rather than strictly linear.

Learning Objectives

By the end of this chapter, you should be able to:

1. Translate a business objective into a precise analytical problem.
2. Distinguish observations, prediction horizons, targets, features, and decisions.
3. Select suitable technical, business, and operational success metrics.
4. Assess data, organisational, technical, and ethical feasibility.
5. Establish simple and meaningful baselines.
6. Apply the CRISP-DM lifecycle to a machine-learning project.
7. Identify risks such as leakage, bias, drift, and deployment failure.
8. Communicate assumptions and recommendations to technical and non-technical stakeholders.

1. From Business Need to Analytical Question

A business statement is usually broader than a machine-learning task. For example, “reduce customer churn” expresses a desired outcome, not yet a prediction problem. A data-science team must clarify:

• What decision will be made?
• Who or what is the unit of analysis?
• When must the decision be made?
• What outcome is to be predicted or estimated?
• What action follows the prediction?
• What costs and benefits are associated with errors?

A useful problem statement contains five elements:

The resulting analytical question might be: “Using information available on a customer’s review date, estimate the probability that the customer will cancel within the next 30 days, so that the retention team can prioritise permitted interventions.” This is more precise than “predict churn.”

Descriptive, Diagnostic, Predictive, and Prescriptive Questions

Data-science questions can be classified according to the type of knowledge required:

• Descriptive: What happened? For example, how many orders were late last month?
• Diagnostic: Why did it happen? For example, which operational factors are associated with late delivery?
• Predictive: What is likely to happen? For example, which orders have a high probability of being late?
• Prescriptive: What should be done? For example, which delivery intervention is expected to reduce lateness at acceptable cost?

Machine learning is most directly associated with predictive tasks, but a predictive model does not automatically provide a causal explanation or an optimal action. A variable associated with churn may not be a suitable intervention target. Prediction, explanation, causation, and optimisation must not be treated as identical.

Classification, Regression, Ranking, and Forecasting

The target determines the primary task type.

The choice should follow the decision, not the popularity of an algorithm. If a manager needs a ranked list for limited follow-up capacity, ranking or calibrated probabilities may be more useful than a forced yes/no label.

2. The Formal Structure of a Prediction Problem

A supervised-learning problem can be represented by observations indexed by i. Let x_i denote the features available for observation i, and let y_i denote its target. A model produces a prediction:

Here, ŷ_i is the predicted value, f is the chosen model family, and θ represents learned parameters. The notation is applicable when a target is defined and the input information is available at prediction time.

For binary classification, a model may estimate:

where p_i is the estimated probability of the positive class. A classification threshold t converts the probability into a decision: predict class 1 when p_i ≥ t. The threshold should reflect consequences and capacity, not merely default to 0.5.

Observation Unit and Target Definition

The observation unit is the entity represented by one modelling record: a customer, transaction, patient visit, machine, claim, or day. Confusing units creates invalid analyses. A customer-level target should not be joined carelessly to transaction-level features, because frequent customers would contribute disproportionately unless the data is deliberately aggregated.

A target definition must specify:

1. the event;
2. the observation date;
3. the prediction horizon;
4. inclusion and exclusion rules;
5. how missing or ambiguous outcomes are treated.

For example, “late delivery” may mean arrival after the promised timestamp, after the original estimated timestamp, or after a service-level threshold. These alternatives produce different labels and different business meanings.

Prediction Time and Leakage

At prediction time, only information available before or at the decision timestamp may be used. Target leakage occurs when a feature contains information created after the target event or information that would not be available when the decision is made. A “refund issued” field may predict whether an order will be classified as a complaint, but it is not legitimate if refunds occur after complaints.

Leakage can produce excellent validation scores and poor real-world performance. To prevent it, define an explicit time boundary, construct features using historical data only, and review every feature with domain experts.

Label Quality

A model learns the target supplied to it, not the business concept in someone’s mind. Labels may be incomplete, delayed, subjective, or influenced by past decisions. If only investigated fraud cases receive a fraud label, uninvestigated cases are not necessarily legitimate. Label audits, sampling, adjudication rules, and documentation are essential.

3. Success Metrics and Baselines

A project requires at least three kinds of success criteria:

1. Business metrics: value created or harm avoided, such as reduced losses or improved service.
2. Model metrics: predictive quality, such as recall, precision, or mean absolute error.
3. Operational metrics: whether the system works in practice, such as latency, coverage, adoption, and alert volume.

A model with high accuracy may still be useless if its predictions arrive too late or if staff cannot act on them.

Classification Metrics

For binary classification, outcomes are commonly summarised as:

Important measures include:

Accuracy is the proportion of all predictions that are correct. Precision is the proportion of predicted positives that are actually positive. Recall, also called sensitivity, is the proportion of actual positives detected. These formulas assume a clearly defined binary outcome and nonzero denominators.

The F1 score is the harmonic mean of precision and recall:

It is useful when both precision and recall matter, but it does not represent financial value and can hide the effect of class prevalence. For highly imbalanced problems, precision-recall curves, expected cost, and performance at an operational threshold may be more informative than accuracy.

A probability model should also be assessed for calibration: among cases assigned probability 0.7, approximately 70% should experience the event over a sufficiently large and representative group. Discrimination and calibration are different properties.

Regression Metrics

For numeric targets, common metrics include mean absolute error (MAE) and root mean squared error (RMSE):

Here, n is the number of evaluated observations, y_i is the actual value, and ŷ_i is the prediction. Both metrics have the same units as the target. RMSE penalises large errors more strongly than MAE because errors are squared before averaging. MAE is often easier to explain as average absolute error.

Cost-Sensitive Evaluation

Suppose a false positive costs C_FP and a false negative costs C_FN. A simple expected error cost is:

This expression is useful when costs are known or reasonably estimated. It excludes true-positive benefits and operational costs unless they are added explicitly. In practice, teams should compare candidate thresholds using a documented cost-benefit model rather than selecting a threshold from a metric alone.

Establishing Baselines

A baseline is a reference against which a model is judged. Examples include:

• majority-class prediction for classification;
• historical average for numeric prediction;
• last-period value for a time series;
• existing rules or human review;
• a random or capacity-based selection strategy.

A sophisticated model that does not outperform a transparent baseline has not demonstrated value. Baselines also expose data problems: an unexpectedly strong baseline may indicate that the problem is easy, the metric is unsuitable, or leakage exists.

4. Feasibility Assessment

Before modelling, conduct a feasibility assessment across six dimensions.

A project can be statistically feasible but operationally infeasible. For example, a model may predict equipment failure accurately but provide no benefit if replacement parts take six weeks to obtain and no maintenance capacity is available.

Data feasibility includes volume, quality, granularity, historical coverage, label availability, missingness, and data access permissions. More data is not automatically better: biased or duplicated records may increase confidence in a wrong conclusion.

Economic feasibility can be assessed with a simple value model. If a proposed intervention produces benefit B per successful case, costs C_A per attempted intervention, and the model produces TP true positives and FP false positives, an illustrative net value is:

where C_M represents modelling and operational costs. This is a simplified model: it assumes constant benefit and cost and does not include uncertainty, indirect effects, or time value of money. It is nevertheless useful for early feasibility discussions.

5. The CRISP-DM Lifecycle

CRISP-DM is a widely used process model with six phases:

1. Business Understanding
2. Data Understanding
3. Data Preparation
4. Modelling
5. Evaluation
6. Deployment

The phases are iterative. Findings in data understanding may change the business question; evaluation may reveal that the target is unsuitable; deployment may expose data or workflow constraints.

Figure: CRISP-DM phases form an iterative lifecycle rather than a one-way sequence.

Business Understanding

Define the business objective, stakeholders, constraints, decision process, risks, and success criteria. Produce a problem statement, project scope, assumptions, and a plan. A useful deliverable is a decision log showing what the model will and will not support.

Data Understanding

Identify data sources, examine distributions and missingness, inspect label quality, and assess representativeness. Record data lineage and access restrictions. Exploratory analysis should be guided by the problem rather than used as an unlimited search for correlations.

Data Preparation

Clean, join, transform, and select data. Build reproducible pipelines and document transformations. Split data according to the deployment setting. Random splits may be inappropriate for temporal or grouped data; future records must not influence evaluation of past decisions.

Modelling

Choose models appropriate to the target, data, interpretability needs, computational resources, and decision context. Begin with a baseline, then compare candidate models. Use cross-validation only in a way that respects time, groups, and leakage constraints. Hyperparameter tuning must be separated from final testing.

Evaluation

Evaluate on data representing future use. Examine overall performance, subgroup performance, calibration, robustness, error types, and operational consequences. Ask whether the model solves the original business problem, not merely whether a score is high.

Deployment

Deployment may mean a real-time API, a batch score, a dashboard, a report, or a rule embedded in a workflow. Specify ownership, access controls, latency, retraining triggers, monitoring, rollback procedures, and user training. Deployment is the beginning of operational learning, not the end of the project.

6. Worked Example: Customer-Retention Prioritisation

The following is an illustrative case.

Background and Problem

A subscription service wants to reduce avoidable cancellations. The retention team can contact only 1,000 customers per week. The proposed system will rank active customers by estimated probability of cancellation within 30 days.

Framing

The observation unit is an active customer on the weekly scoring date. The target is cancellation within the following 30 days. Features include historical usage, tenure, support contacts, and payment history, provided they are available on the scoring date. The team must exclude post-cancellation records and must not use sensitive attributes for unauthorised targeting.

The business success criterion is not simply maximum recall. The team wants more retained customers per 1,000 contacts than the existing selection rule, without unfairly excluding groups or creating unacceptable customer pressure.

Baseline and Evaluation

The baseline selects customers using the existing rule. A second baseline selects the 1,000 customers with the highest historical cancellation rate by a simple segment. Candidate models are evaluated using a time-based holdout: older data is used for training and later data for testing. Precision among the top 1,000 customers is an operationally meaningful metric. Calibration is also examined because intervention resources are limited.

Suppose the test period contains 10,000 scored customers, of whom 800 cancel. A model contacts 1,000 customers and correctly identifies 240 future cancellations. It therefore has precision:

Its recall is:

These figures do not by themselves prove value. The retention offer may be expensive, some customers may cancel regardless of intervention, and contacting customers may have negative effects. A controlled trial is needed to estimate incremental retention: compare eligible contacted customers randomly assigned to an intervention with a suitable control group. Prediction identifies whom to consider; experimentation estimates whether the action works.

Outcome and Learning Points

The project should proceed only if the model improves the decision relative to the baseline and the organisation can deliver a permissible, effective intervention. Monitoring should track data drift, top-list precision, subgroup outcomes, offer costs, customer complaints, and incremental retention. If cancellation behaviour changes because of pricing or market conditions, the model may require recalibration or retraining.

The central lesson is that a churn model is not the product. The product is a governed decision process that uses predictions responsibly to allocate retention effort.

7. Deployment, Monitoring, and Operational Review

A production lifecycle requires monitoring at several levels:

• Data quality: missing fields, unexpected ranges, schema changes, duplicate records.
• Data drift: changes in feature distributions.
• Concept drift: changes in the relationship between features and target.
• Performance: precision, recall, calibration, MAE, or other agreed metrics when labels become available.
• Service health: latency, uptime, failures, and scoring completeness.
• Business impact: financial value, service outcomes, adoption, and unintended effects.
• Fairness: performance and error differences across relevant groups.

Monitoring metrics require thresholds and actions. “Monitor accuracy” is incomplete unless the team specifies how often it is measured, what minimum level is acceptable, and who responds when it falls below that level.

Operational review should ask whether the model remains necessary, whether users follow it appropriately, whether the intervention still works, and whether data collection remains justified. Models should be retired when their assumptions no longer hold or when a simpler process produces equal value.

8. Governance, Ethics, and Responsible Practice

Machine-learning systems can reproduce historical discrimination, obscure accountability, or expose personal information. Responsible practice includes:

• data minimisation and purpose limitation;
• lawful processing and appropriate consent or authority;
• access control, retention rules, and secure storage;
• documentation of data sources, assumptions, and model versions;
• human review for high-impact decisions;
• testing for disparate error rates and harmful proxy variables;
• explanations appropriate to the users and affected people;
• appeal, correction, and incident-response processes.

Fairness is not a single universal metric. Different definitions can conflict, especially when base rates differ between groups. Teams should identify the relevant harm, consult affected stakeholders, report subgroup performance, and justify the selected approach. Removing a protected attribute does not guarantee fairness because other variables may act as proxies.

Generative AI and automated coding assistants can accelerate data preparation and documentation, but generated code and interpretations require review. Confidential data should not be placed into unapproved tools. Reproducibility, testing, provenance, and human accountability remain necessary.

9. Best Practices for Diploma-Level Practice

1. Begin with the decision and action, not the algorithm.
2. Write the target, observation date, horizon, and exclusions explicitly.
3. Create a data dictionary and record lineage.
4. Establish a simple baseline before tuning complex models.
5. Use validation procedures that reflect deployment time and grouping.
6. Inspect errors and subgroup performance, not only one headline score.
7. Separate model performance from causal or business impact.
8. Document assumptions, limitations, owners, and escalation procedures.
9. Design monitoring before deployment.
10. Treat deployment and review as part of the lifecycle.

Chapter Summary

Machine-learning success depends on framing a useful and feasible decision problem. A business objective must be translated into an analytical question with a defined observation unit, target, prediction horizon, available features, action, and constraints. Classification, regression, ranking, and forecasting tasks are distinguished by the target and decision context.

Success should be evaluated at business, model, and operational levels. Accuracy, precision, recall, F1, calibration, MAE, RMSE, and cost-based measures each answer different questions. Baselines provide essential references and can reveal leakage or weak problem definitions.

CRISP-DM structures work through Business Understanding, Data Understanding, Data Preparation, Modelling, Evaluation, and Deployment. These phases are iterative. Feasibility includes business, data, technical, economic, organisational, legal, and ethical considerations. Production systems require monitoring for data quality, drift, performance, service health, business impact, and fairness. The responsible data scientist recognises that a model is only one component of a governed decision process.

Key Terms

Analytical question: A precise question that connects data to a decision or measurable outcome.

Baseline: A simple or existing method used as a reference for judging a model.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Concept drift: Change over time in the relationship between inputs and the target.

CRISP-DM: An iterative framework containing six phases for data-mining and machine-learning projects.

Data leakage: Use of information unavailable at prediction time or derived from the target or future.

Deployment: Placing a model or its outputs into an operational environment.

F1 score: The harmonic mean of precision and recall.

Feature: An input variable used by a model.

Label: The observed target value used for supervised learning.

Observation unit: The entity represented by one modelling record.

Prediction horizon: The future period over which an outcome is predicted.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives detected by a model.

Regression: Prediction of a numeric target.

Target: The outcome a supervised-learning model is trained to predict.

Target leakage: A form of leakage in which target-related information improperly enters the features.

Threshold: A rule for converting a score or probability into a decision.

Concept drift: Change in the input-target relationship after deployment.

Operational metric: A measure of whether a deployed system functions effectively in practice.

Review Questions

Short Answer Questions

1. What five elements should be included in a well-framed analytical problem?
2. Define target leakage and give one example.
3. Distinguish precision from recall.
4. Why is a baseline necessary in a machine-learning project?
5. Name the six CRISP-DM phases.

Long Answer Questions

1. Explain how a broad business objective can be converted into a supervised-learning problem.
2. Compare business, model, and operational success metrics, using an example.
3. Discuss why random train-test splitting may be inappropriate for temporal data.
4. Explain the main dimensions of a machine-learning feasibility assessment.
5. Describe the responsibilities involved in monitoring a model after deployment.

Analytical / Application Questions

1. A hospital wants to predict missed appointments. Define an appropriate observation unit, target, prediction horizon, action, baseline, and evaluation metric. Identify two possible sources of leakage.
2. A fraud model has high accuracy but detects very few fraudulent transactions. Explain why this can occur and recommend a better evaluation approach.
3. A company wants to use a churn model to send discounts to customers. Design a small evaluation plan that distinguishes predictive performance from the incremental effect of the discounts.

Multiple Choice Questions with Answers

Question 1

Which statement best defines an observation unit?

A. The algorithm used to train a model

B. The entity represented by one modelling record

C. The percentage of correct predictions

D. The time required to deploy a model

Correct Answer: B. The entity represented by one modelling record

Question 2

Which situation is an example of target leakage?

A. Using customer tenure measured before scoring

B. Using a transaction amount recorded at purchase

C. Using a refund issued after a complaint as a complaint predictor

D. Using a customer’s historical purchase count

Correct Answer: C. Using a refund issued after a complaint as a complaint predictor

Question 3

A model identifies 80 true positives among 200 predicted positives. What is its precision?

A. 20%

B. 40%

C. 60%

D. 80%

Correct Answer: B. 40%

Question 4

Which metric is especially useful for measuring the average absolute size of regression errors in the target’s units?

A. MAE

B. Recall

C. Precision

D. F1 score

Correct Answer: A. MAE

Question 5

Which is the most appropriate baseline for a last-period demand forecast?

A. A randomly selected forecast

B. The previous period’s demand

C. A model with maximum complexity

D. The number of available features

Correct Answer: B. The previous period’s demand

Question 6

Which CRISP-DM phase primarily defines the business objective and project constraints?

A. Business Understanding

B. Data Preparation

C. Modelling

D. Deployment

Correct Answer: A. Business Understanding

Question 7

Why can accuracy be misleading for a rare-event classification problem?

A. Accuracy cannot be calculated for binary outcomes

B. A model can appear accurate by mostly predicting the common class

C. Accuracy always measures calibration

D. Accuracy has the same meaning as recall

Correct Answer: B. A model can appear accurate by mostly predicting the common class

Question 8

What does calibration assess?

A. Whether features have identical units

B. Whether predicted probabilities correspond to observed frequencies

C. Whether a model has the shortest runtime

D. Whether all groups have equal population sizes

Correct Answer: B. Whether predicted probabilities correspond to observed frequencies

Question 9

Which activity best estimates whether a retention intervention causes additional customer retention?

A. Increasing model complexity

B. Removing the target variable

C. Comparing randomly assigned intervention and control groups

D. Selecting the threshold of 0.5

Correct Answer: C. Comparing randomly assigned intervention and control groups

Question 10

Which is an operational monitoring concern?

A. Whether the model’s source code uses a loop

B. Whether prediction latency and scoring completeness remain acceptable

C. Whether the training dataset has a column name

D. Whether the model has exactly two features

Correct Answer: B. Whether prediction latency and scoring completeness remain acceptable