The AI and ML Project Lifecycle

Introduction

An artificial intelligence or machine learning (ML) project is not simply the act of selecting an algorithm and training it on data. Successful projects connect a real organisational need to a measurable outcome, reliable data, an appropriate technical method, responsible deployment, and continuing monitoring. Many technically accurate models fail because the original problem was poorly defined, the data did not represent the intended users, or no plan existed for maintaining the model after deployment.

This chapter presents the ML project as a lifecycle rather than a one-time programming task. It introduces the Cross-Industry Standard Process for Data Mining (CRISP-DM), explains how business problems are translated into ML tasks, examines stakeholders and success measures, and discusses feasibility, experimentation, documentation, deployment, governance, and model maintenance.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the stages of an AI/ML project lifecycle;
• use CRISP-DM to structure a project;
• translate a business objective into a suitable ML task;
• identify stakeholders, constraints, risks, and success criteria;
• distinguish business, model, and operational metrics;
• design a valid experimental workflow and avoid data leakage;
• document datasets, experiments, models, and decisions;
• explain deployment, monitoring, retraining, and retirement;
• recognise ethical, legal, security, and governance responsibilities.

1. From Business Need to ML Problem

A business problem describes a desired change in the world. An ML problem describes how data and a computational model may support that change. These are related but not identical.

For example, “reduce customer churn” is a business objective. A possible ML task is to estimate the probability that each customer will cancel within the next 30 days. The organisation may then use those estimates to offer assistance to selected customers. The model is therefore one component of a decision process, not the entire solution.

A useful problem statement identifies:

1. Decision: What decision or action will be supported?
2. Unit of analysis: Is the prediction about a customer, transaction, image, machine, or document?
3. Prediction time: When must the prediction be available?
4. Target: What outcome is being predicted or generated?
5. Horizon: Over what future period is the outcome measured?
6. Users: Who will interpret or act on the output?
7. Constraints: What limits exist concerning cost, speed, privacy, safety, or regulation?
8. Value: How will success create benefit or reduce harm?

A precise formulation prevents a common error: optimising a model for a target that is easy to measure but irrelevant to the actual decision.

Selecting the ML task

The task should follow the decision requirement and the available evidence.

Not every problem needs ML. A rule-based system, database query, statistical report, process redesign, or additional human staffing may be cheaper, safer, and more effective. ML is appropriate when a repeatable relationship can be learned from relevant data and when the expected benefit justifies the risks and operational costs.

2. CRISP-DM as a Project Framework

CRISP-DM is a widely used framework containing six iterative phases:

1. business understanding;
2. data understanding;
3. data preparation;
4. modelling;
5. evaluation;
6. deployment.

The phases are shown as a cycle because project teams frequently return to earlier stages. A modelling result may reveal that the target is poorly defined. Evaluation may show that data preparation needs revision. Deployment may expose latency, fairness, or maintenance problems that require a new project iteration.

Business understanding

The team clarifies the organisational objective, decision process, expected value, constraints, and project scope. A project charter should record the problem statement, assumptions, stakeholders, deliverables, timeline, risks, and acceptance criteria.

A business success measure should be stated independently of model performance. For instance, a fraud project might seek to reduce financial loss while maintaining an acceptable rate of legitimate transactions incorrectly blocked. A high classification score is not sufficient if the system does not improve the operational outcome.

Data understanding

Data understanding involves identifying available sources, examining structure and quality, and checking whether the data represents the intended population. Activities include data profiling, visualisation, label inspection, missing-value analysis, duplicate detection, and investigation of sampling and measurement processes.

Important questions include:

• Who generated the data and for what original purpose?
• Does each record correspond to the correct unit of analysis?
• Are labels reliable, timely, and consistently defined?
• Are there features that would not be available at prediction time?
• Which groups or situations are under-represented?
• Are personal, confidential, or regulated data involved?

Data quality is not a single number. Completeness, accuracy, consistency, timeliness, validity, uniqueness, and representativeness all matter.

Data preparation

Data preparation may include joining sources, correcting errors, handling missing values, encoding categories, scaling numerical variables, creating features, and dividing data into training, validation, and test sets. Each transformation must be designed so that information from the future or from the test set does not leak into training.

For time-dependent data, a chronological split is often more realistic than a random split. A model predicting next month’s demand should not be trained using records from after the date at which predictions would have been made.

A reproducible data pipeline is preferable to undocumented manual spreadsheet operations. It should record source versions, transformation logic, validation checks, and the resulting dataset version.

Modelling

The team selects baseline methods and candidate algorithms. A baseline may be a simple rule, the majority class, a previous-period value, or linear/logistic regression. Baselines establish whether added complexity produces meaningful improvement.

Model choice depends on the task, data size, interpretability requirement, computational resources, latency, robustness, and regulatory context. A more complex model is not automatically a better model. In high-stakes settings, explainability and auditability may be more important than a small improvement in predictive performance.

Evaluation

Evaluation compares model results with both technical and business criteria. It should use data that was not used to fit the final model and should reflect the real operating environment. Evaluation also examines subgroup performance, calibration, robustness, error types, resource use, and potential harms.

A model should not be declared successful merely because it has a high average score. For example, an overall accuracy of 95% may be poor for a rare-event problem if nearly all positive cases are missed.

Deployment

Deployment places the model into a process where predictions are generated and used. Possible forms include batch scoring, an API, an embedded application, or a decision-support dashboard. Deployment includes data pipelines, access control, logging, alerting, rollback procedures, user training, and documentation.

Deployment is not the end of CRISP-DM. Real-world data and behaviour change, so a deployed model must be monitored and periodically reviewed.

3. Stakeholders and Success Metrics

An ML project normally involves more people than data scientists. Typical stakeholders include:

• Sponsor: provides funding and resolves strategic conflicts;
• domain expert: explains the process and evaluates practical meaning;
• product or project manager: controls scope, schedule, and priorities;
• data engineer: builds reliable data access and pipelines;
• data scientist or ML engineer: develops and evaluates models;
• IT or platform team: supports infrastructure and deployment;
• security and privacy specialists: assess threats and data handling;
• legal, compliance, or ethics officers: interpret obligations;
• frontline users: act on outputs and identify usability problems;
• affected individuals or communities: experience the consequences of decisions.

Stakeholder mapping should identify influence, interests, responsibilities, and communication needs. A responsible team also considers people who are not represented in project meetings but may be affected by the system.

Layers of success

Success should be defined at several levels.

Business metrics measure organisational outcomes, such as cost reduction, revenue, processing time, retention, or service quality.

Model metrics measure predictive or generative behaviour. For binary classification:

Here, TP is the number of true positives, TN true negatives, FP false positives, and FN false negatives. Accuracy is the proportion of all predictions that are correct. Precision is the proportion of predicted positives that are actually positive. Recall is the proportion of actual positives that are detected. These formulas apply to a defined classification dataset and require clear positive and negative labels. They do not by themselves indicate financial value or fairness.

The F1 score is the harmonic mean of precision and recall:

It is defined when precision and recall are meaningful and their sum is non-zero. F1 gives equal importance to precision and recall, which may not match the cost structure of a particular application.

For regression, a common metric is mean absolute error:

Here, n is the number of observations, yᵢ is the observed value, and ŷᵢ is the prediction. MAE has the same units as the target and treats errors linearly. It is useful when an error of twice the size should incur approximately twice the penalty.

Operational metrics measure whether the system works in production, including response time, availability, failure rate, data freshness, cost per prediction, and alert volume.

Responsible-use metrics may include subgroup error rates, privacy incidents, complaints, override rates, and the percentage of decisions receiving human review.

A metric specification should include its definition, measurement population, calculation period, data source, target value, owner, and known limitations.

4. Feasibility, Risk, and Project Planning

Feasibility is multidimensional. A project may be technically feasible but legally unacceptable, or economically attractive but impossible because required labels do not exist.

A feasibility assessment considers:

• data feasibility: availability, quality, representativeness, access, and labelling effort;
• technical feasibility: suitable algorithms, infrastructure, integration, latency, and scale;
• economic feasibility: development, operation, maintenance, and opportunity costs;
• organisational feasibility: skills, leadership support, workflow readiness, and change management;
• legal and ethical feasibility: privacy, discrimination, safety, intellectual property, and sector rules.

Risk analysis should identify the risk, its cause, likelihood, impact, affected parties, controls, owner, and residual risk. Examples include data leakage, cyberattack, biased labels, model drift, misuse, incorrect automation, vendor dependency, and loss of service.

A useful project plan separates discovery from commitment. A short proof-of-concept can test whether the signal exists, but it should not be mistaken for production readiness. A staged plan may include discovery, data audit, baseline experiment, pilot, controlled release, and scale-up. At each stage, a go/no-go decision should be based on predefined evidence.

5. Experimentation and Technical Discipline

Experiments should answer specific questions. “Can a model predict churn?” is less useful than “Does adding service-history features improve recall at the operational precision threshold without increasing subgroup disparity?”

A sound experiment includes a hypothesis, dataset version, feature definition, split strategy, baseline, algorithm configuration, evaluation metrics, random seed where relevant, results, and interpretation. Changing several elements at once makes it difficult to identify the cause of improvement.

Training, validation, and test data

The training set is used to fit model parameters. The validation set supports model selection and tuning. The test set provides a final estimate after choices have been made. Repeatedly checking the test set turns it into a de facto validation set and can produce optimistic estimates.

Cross-validation can provide a more stable estimate when data is limited. In k-fold cross-validation, the data is divided into k folds; the model is trained on k−1 folds and evaluated on the remaining fold, repeating until each fold has served as validation. The mean score summarises performance, while variation across folds indicates uncertainty. Cross-validation must still respect groups, time order, or other dependencies when random folding would create leakage.

Class imbalance and thresholds

For rare events, accuracy can be misleading. Teams should inspect precision, recall, confusion matrices, precision–recall curves, and the consequences of false positives and false negatives. A probability threshold converts a score into a class decision. Changing the threshold usually changes precision and recall; the choice should reflect operational costs and capacity rather than a default value alone.

Reproducibility

Reproducibility requires versioned code, data references, environment specifications, configuration files, experiment logs, and clear randomisation procedures. Reproducibility does not guarantee validity: a perfectly repeatable experiment may still use biased data or an inappropriate target.

6. Documentation and Communication

Documentation is an engineering and governance control, not clerical overhead. Important artefacts include:

• project charter and decision log;
• data inventory and data dictionary;
• dataset statement describing source, population, collection, limitations, and permitted uses;
• feature definitions and transformation pipeline;
• experiment records and evaluation report;
• model card describing intended use, performance, limitations, and risks;
• deployment specification and operating procedure;
• monitoring plan, incident process, and retirement criteria.

Documentation should be understandable to technical and non-technical audiences. A model card, for example, should state what the model does, what it must not be used for, the evaluation population, subgroup results, known failure cases, and human oversight requirements.

Good communication distinguishes fact, assumption, estimate, and unresolved question. It also reports uncertainty rather than presenting a single score as certainty.

7. Deployment and the Operational Lifecycle

A model becomes an operational system when its output influences a real process. Deployment patterns include:

• batch inference: predictions generated periodically for a stored population;
• online inference: predictions generated in response to a request;
• edge inference: prediction performed near the data source, such as a device;
• human-in-the-loop decision support: the model recommends and a person reviews;
• automated decision-making: the system acts without routine human approval.

The appropriate pattern depends on latency, connectivity, privacy, cost, and risk. High-impact decisions commonly require stronger human review, appeal mechanisms, and audit trails.

Production monitoring should cover:

1. data quality: missingness, ranges, schema changes, and unusual values;
2. data drift: changes in the distribution of input variables;
3. concept drift: changes in the relationship between inputs and outcomes;
4. model performance: accuracy or task-specific measures when labels become available;
5. service health: latency, uptime, errors, throughput, and cost;
6. impact: user complaints, subgroup outcomes, overrides, and unintended effects.

A model may remain technically available while becoming unsuitable because customer behaviour, regulations, products, or economic conditions have changed. Retraining should therefore be triggered by evidence and governed by testing, not performed automatically without controls. A rollback version should be available, and retirement should be planned when a model is unsafe, obsolete, uneconomic, or no longer needed.

8. Ethical, Legal, and Security Considerations

Responsible AI applies throughout the lifecycle. Key principles include fairness, transparency, accountability, privacy, safety, security, and human agency. These principles can conflict. For example, collecting more data may improve prediction but increase privacy risk; a highly interpretable model may be less accurate in a particular setting. Such trade-offs should be made explicitly and documented.

Privacy practices include purpose limitation, data minimisation, access control, retention limits, encryption, de-identification where appropriate, and secure deletion. De-identification reduces risk but does not guarantee anonymity, especially when datasets can be linked.

Fairness assessment should examine whether performance and impacts differ across relevant groups. There is no single fairness measure that resolves every context. Teams must define the harm of concern, consult affected stakeholders, and assess legal requirements. Removing a sensitive attribute does not necessarily remove unfairness because correlated variables may act as proxies.

Security threats include data poisoning, adversarial inputs, model extraction, prompt injection in generative systems, credential compromise, and supply-chain vulnerabilities. Controls include least-privilege access, input validation, secure software practices, monitoring, incident response, and human review of high-risk outputs.

Generative AI adds risks such as fabricated content, confidential-data disclosure, copyright concerns, and persuasive misuse. Evaluation must include factuality, refusal behaviour, provenance, and misuse scenarios rather than relying only on fluency.

9. Illustrative Case: Predicting Missed Appointments

The following is a fictional illustrative case.

Background and problem

A community clinic experiences missed appointments, reducing capacity and delaying care. The clinic proposes an ML system to identify appointments at higher risk of non-attendance so that staff can offer reminders or alternative scheduling. The goal is not to deny appointments or penalise patients.

Analysis

Stakeholders include patients, clinicians, reception staff, the clinic manager, an IT team, a privacy officer, and a community representative. The unit of analysis is a scheduled appointment. The prediction must be generated 48 hours before the appointment, using only information available at that time.

The team defines the target as whether an appointment is missed without cancellation. It discovers that historical records contain inconsistent cancellation codes and that some patients have very few digital contact details. These findings create data-quality and fairness risks.

Approach

The team first establishes a baseline based on the overall historical missed-appointment rate. It creates a time-based training and test split so that earlier appointments predict later appointments. Features include appointment lead time, prior attendance history, clinic type, and reminder history. Sensitive attributes are used for evaluation of potential disparities only where lawful and appropriately governed, not automatically as prediction features.

The team selects a threshold using the clinic’s reminder capacity and the relative costs of unnecessary reminders and missed appointments. It evaluates recall, precision, calibration, subgroup performance, staff workload, and patient feedback. The workflow includes a human review option and does not automatically cancel or restrict appointments.

Outcome

In the pilot, the model identifies a useful group for additional reminders, but subgroup analysis shows lower calibration for patients with limited historical records. The clinic therefore introduces a “new or insufficient history” pathway rather than treating low information as low risk. It also improves data-entry standards and continues monitoring.

Learning points

The valuable outcome is not merely a model score. The project succeeds because it defines a safe decision, respects prediction-time information, considers affected people, measures operational consequences, and changes the process when evaluation reveals unequal performance.

Challenges and Limitations

ML lifecycle frameworks provide structure, not guarantees. CRISP-DM does not specify every modern requirement for privacy, continuous delivery, foundation models, or algorithmic impact assessment. Project phases may also appear more orderly than actual work, which is often iterative and political.

Historical data can encode past discrimination or reflect decisions made under old policies. Labels may measure administrative behaviour rather than the phenomenon of interest. A model can perform well in testing but fail after deployment because of drift, feedback loops, or changes in user behaviour. Metrics can be gamed if teams optimise a target without considering the wider system.

Finally, some problems should not be automated. If the consequences are severe, evidence is weak, or affected people cannot reasonably challenge a decision, a non-ML process may be more appropriate.

Best-Practice Checklist

Before deployment, the team should be able to answer “yes” to the following questions:

• Is the business decision and intended use explicit?
• Is the target available and valid at the prediction time?
• Are stakeholders, affected groups, and responsibilities identified?
• Has a non-ML baseline been tested?
• Are data provenance, quality, permissions, and limitations documented?
• Are train, validation, and test procedures free from leakage?
• Are business, model, operational, and responsible-use metrics defined?
• Have subgroup performance and foreseeable harms been assessed?
• Are security, privacy, human oversight, appeal, and incident procedures ready?
• Can the system be monitored, rolled back, retrained, and retired?

Chapter Summary

The AI and ML project lifecycle connects a business need with a responsibly operated technical system. CRISP-DM structures work through business understanding, data understanding, data preparation, modelling, evaluation, and deployment, while allowing iteration between phases. A business objective must be translated into a precise decision, unit of analysis, target, time horizon, and constraint set. Stakeholders include technical teams, decision-makers, domain experts, governance specialists, users, and affected communities.

Feasibility involves data, technology, economics, organisation, and law. Evaluation must combine business, model, operational, and responsible-use measures. Careful experimentation requires suitable data splitting, baseline comparisons, leakage prevention, documentation, and reproducibility. Deployment introduces monitoring for data quality, drift, performance, service health, and impact. Models require controlled updates, rollback, and eventual retirement. Ethical, privacy, fairness, security, and regulatory considerations are continuous lifecycle responsibilities rather than final checks.

Key Terms

Business understanding: Defining the organisational objective, decision, value, scope, and constraints.

CRISP-DM: An iterative framework for data-mining and ML projects.

Data leakage: Unintended use of information unavailable at prediction time or from evaluation data during training.

Data drift: Change in the distribution of input data over time.

Concept drift: Change in the relationship between inputs and the target outcome.

Deployment: Placing a model into an operational process.

Evaluation metric: A defined measure used to assess model or system performance.

Feature: An input variable supplied to a model.

F1 score: The harmonic mean of precision and recall.

False negative: A positive case incorrectly predicted as negative.

False positive: A negative case incorrectly predicted as positive.

Model card: Documentation describing a model’s purpose, performance, limitations, and risks.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are correctly detected.

Stakeholder: A person or group that influences or is affected by the project.

Test set: Data reserved for final performance estimation after model choices are made.

Training set: Data used to fit model parameters.

Validation set: Data used for model selection and tuning.

Threshold: A rule for converting a score or probability into a decision.

Review Questions

Short Answer Questions

1. Why is a business objective not identical to an ML task?
2. Name the six phases of CRISP-DM.
3. What is the difference between precision and recall?
4. Why is a chronological data split useful for many forecasting problems?
5. Give two examples of operational metrics for a deployed model.

Long Answer Questions

1. Explain how a team should translate “reduce customer churn” into a precise ML project specification.
2. Discuss the responsibilities of technical and non-technical stakeholders across the ML lifecycle.
3. Explain why a high average model score may be insufficient for approving deployment.
4. Describe the purpose of training, validation, and test data, including the risk of repeatedly using the test set.
5. Discuss how privacy, fairness, security, and human oversight should influence model design and deployment.

Analytical / Application Questions

1. A hospital wants to predict missed appointments. Design a CRISP-DM-based plan that identifies the target, prediction time, stakeholders, metrics, risks, and monitoring requirements.
2. A fraud model has high accuracy but misses most fraudulent transactions because fraud is rare. Analyse why accuracy is inadequate and recommend additional metrics and threshold considerations.
3. After deployment, a demand model’s input distributions change and its error increases. Determine whether this suggests data drift, concept drift, or both, and propose a controlled response.

Multiple Choice Questions with Answers

Question 1

Which statement best describes the relationship between a business problem and an ML task?

A. Every business problem should be solved with deep learning.

B. An ML task specifies how data and a model may support a business decision.

C. A business problem is always a classification problem.

D. The ML task replaces the business decision process.

Correct Answer: B. An ML task specifies how data and a model may support a business decision.

Question 2

Which CRISP-DM phase primarily defines organisational objectives, scope, and constraints?

A. Business understanding

B. Data preparation

C. Modelling

D. Deployment

Correct Answer: A. Business understanding

Question 3

A model identifies 80 of 100 actual positive cases and produces 20 false-positive predictions. What is its precision?

A. 0.20

B. 0.50

C. 0.80

D. 0.833

Correct Answer: D. 0.833

Question 4

What is the main purpose of a validation set?

A. To fit the model parameters exclusively

B. To support model selection and tuning

C. To replace all production monitoring

D. To store personal data not used by the project

Correct Answer: B. To support model selection and tuning

Question 5

Which practice most directly helps prevent future information from leaking into model training?

A. Using a chronological split for time-dependent prediction

B. Increasing the number of model layers

C. Removing the project documentation

D. Evaluating only on the training data

Correct Answer: A. Using a chronological split for time-dependent prediction

Question 6

Which metric is measured in the same units as a continuous target?

A. Accuracy

B. Precision

C. Mean absolute error

D. Recall

Correct Answer: C. Mean absolute error

Question 7

What does concept drift mean?

A. The input schema has gained a new column

B. The relationship between inputs and the target has changed

C. The model file has been compressed

D. The training set contains duplicate rows

Correct Answer: B. The relationship between inputs and the target has changed

Question 8

Which item is primarily an operational metric?

A. Response latency

B. F1 score on the test set

C. Number of training features

D. Model parameter count

Correct Answer: A. Response latency

Question 9

Why is a baseline model valuable?

A. It guarantees regulatory approval.

B. It provides a reference against which added complexity can be judged.

C. It eliminates the need for a test set.

D. It always has better performance than advanced models.

Correct Answer: B. It provides a reference against which added complexity can be judged.

Question 10

Which action is most appropriate when monitoring reveals that a deployed model has become unsafe?

A. Ignore the result until the next annual review.

B. Remove all monitoring records.

C. Use a controlled rollback or suspend use while investigating.

D. Automatically increase the decision threshold without documentation.

Correct Answer: C. Use a controlled rollback or suspend use while investigating.